2019-08-08 DSpace 7 Working Group Meeting

Date

08 Aug 2019 from 14:00-15:00 UTC

Location: https://lyrasis.zoom.us/my/dspace (Meeting ID: 502 527 3040).

• More connection options available at DSpace Meeting Room

Agenda

• (15 mins) Developer Stand Up - Developers give brief updates on their effort (or their team's effort).

  • Update/see "Current Work" section below based on your status. Please feel free to update prior to meeting.
  • Please highlight any new work (needing reviews/testing), any blockers (for you), and any discussion topics you may have.
• (30 mins) General Discussion Topics
  
  1. Ben Bosman Support for bundles in REST (DS-4317) for:
     1. reorder bitstreams (bundle2bitstream.bitstream_order, not to be confused with bitstream sequence)
     2. retrieving item's bitstreams limited to one or a few bundles, or even to one thumbnail
  2. (Topic #2)
  3. Improved Estimation Strategy for Beta release. (Estimates should include time for discussion / code reviews, also should be done by multiple developers)
     1. Development Planning Spreadsheet needs to be updated, so that we can determine what (outstanding) tasks need estimation.
     2. Will be using Wideband Delphi estimation technique
• (15 mins) Planning for next week
  
  • Assigning PRs for Review
  • Next tasks from Development Planning Spreadsheet

Attendees

• Art Lowel (Atmire) 

• Andrea Bollini (4Science)  - On holiday, will return on Aug 8 meeting.

• Tim Donohue 

• Heather Greer Klein

• Lieven Droogmans

• Mark H. Wood
• Giuseppe Digilio (4Science) - Out on Aug 8 and 15
• Ben Bosman
• Pascal-Nicolas Becker
• Chris Wilper
• Terrence W Brady
• Paulo Graça 
• Dimitris Pierrakos - Out of the office until Aug 19
• Julius Gruber
• Laura Henze

Current Work

Tickets / PRs In Progress

1. (Angular) Adding Accessibility via Travis CI  https://github.com/DSpace/dspace-angular/pull/356 (work in progress) (Lower priority)
2.  (Angular Bug) https://github.com/DSpace/dspace-angular/issues/368 ( Art Lowel (Atmire) )
3. (REST Contract) Edit Homepage news: https://github.com/DSpace/Rest7Contract/pull/45 (Ben Bosman  - has outstanding questions/comments) (Lower priority)
4. (REST) DS-4043: Revisit the security layer of the submission  (work in progress) Andrea Bollini (4Science)
5.  (REST Contract) Group and eperson management: https://github.com/DSpace/Rest7Contract/pull/41 (Waiting on updates fromBen Bosman )

PRs Needing Review

1. (REST Contract) Collecting statistics - https://github.com/DSpace/Rest7Contract/pull/63 (Dimitris Pierrakos, Andrea Bollini (4Science)  Feedback provided, Ben Bosman , Tim Donohue )
2. (REST Contract) Update bitstream properties and format https://github.com/DSpace/Rest7Contract/pull/68 (Tim Donohue- minor improvements noted, Andrea Bollini (4Science) Feedback provided)
3. (REST) Pagination bug with withdrawn items: https://github.com/DSpace/DSpace/pull/2406 (Dimitris Pierrakos , Ben Bosman - Feedback provided)
4. (REST) implement upload bitstream to archived item https://github.com/DSpace/DSpace/pull/2473 (Tim Donohue -  Feedback provided,  Andrea Bollini (4Science)-  Feedback provided)
5. (Angular) (Entities) Deleting relationships: https://github.com/DSpace/dspace-angular/pull/402 (Paulo Graça - will test again, Tim Donohue )
6. (Angular) Move Item Component: https://github.com/DSpace/dspace-angular/pull/335 (Giuseppe Digilio (4Science) reviewed and provided feedback, Tim Donohue )
7. (Angular) Item-Collection Mapper:  https://github.com/DSpace/dspace-angular/pull/348 (UNBLOCKED as PR#2282 was merged. Tim Donohue NEEDS SECOND REVIEWER)
8. (Angular) Shibboleth integration support: https://github.com/DSpace/dspace-angular/pull/429  (Julius running into an error with 'yarn start' only) (Giuseppe Digilio (4Science) feedback provided, Fernando FCT/FCCN, Paulo Graça )
9. (Angular) Submission Miscellaneous fixes: https://github.com/DSpace/dspace-angular/pull/432 (Art Lowel (Atmire) will re-review, Julius Gruber )
10. (Angular) Edit current item query https://github.com/DSpace/dspace-angular/pull/440 (Ben Bosman, Tim Donohue)
11. (Angular) UI Language Cookie https://github.com/DSpace/dspace-angular/pull/443 ( Tim Donohue, Paulo Graça)
12. (Updated) (Angular) Convert i18n files to JSON5 format https://github.com/DSpace/dspace-angular/pull/439 (Tim Donohue, NEEDS SECOND REVIEWER)
13. (New) (Angular) flatten i18n files https://github.com/DSpace/dspace-angular/pull/457 ( Tim Donohue, Paulo Graça )
14. (New) (Angular) bitstream format registries https://github.com/DSpace/dspace-angular/pull/455 (Tim Donohue, Art Lowel (Atmire))
15. (New) (Angular) Disable e2e tests in travis https://github.com/DSpace/dspace-angular/pull/454 (CAN BE MERGED)
16. Indirect entity refs during CSV import https://github.com/DSpace/DSpace/pull/2471 (Ben Bosman, Tim Donohue )

PRs Merged this week!

1.  (Angular) Fix for failing travis tests https://github.com/DSpace/dspace-angular/pull/444
2.  (Angular) Replace model factories https://github.com/DSpace/dspace-angular/pull/434
3.  (REST Contract) OAI Harvesting configuration - https://github.com/DSpace/Rest7Contract/pull/66
4.  (REST) Item Mapper functionality: https://github.com/DSpace/DSpace/pull/2282
   1.  Minor update to Contract per this PR: https://github.com/DSpace/Rest7Contract/pull/70
5.  (Backend) Travis CI - Continue to use Ubuntu Trusty 14.04: https://github.com/DSpace/DSpace/pull/2477

Blocked

1. (Blocked PRs go here)

Delayed / Needs Discussion

1. (REST) Scripts & Processes endpoint: https://github.com/DSpace/Rest7Contract/pull/17
   1. How do we move this forward? In our Jan 24 meeting, Tim noted a risk of "scope creep" with this feature suggestion. This idea has been tabled since then.
   2. In July 25 meeting, Atmire said they'd come back with notes on the proposed backend implementation.
2. Managing Authorization info in Angular UI: How to pass Authorization rights (i.e. logged in user's access rights) from REST API to Angular?  See for example: https://github.com/DSpace/dspace-angular/issues/393
   1. Can this be achieved via passed HAL "_links" (e.g. the existence of an "edit" link in REST response means you must have Edit rights)?
   2. In July 25 meeting, we noted this probably cannot be resolved with just one simple solution. May need to look at different options for different scenarios
      1. Also likely to need to store/cache a user's Groups in UI layer, as some areas (e.g. Administrative) require knowledge of user group membership
3. Initial Performance Testing from Chris.
   1. https://cwilper.github.io/dspace-perftest/
4. (REST Contract) Edit Homepage News: https://github.com/DSpace/Rest7Contract/pull/45
   
   1. Delayed until after Preview release. General agreement (in meeting on March 21, 2019) that storing HTML in metadata fields is not really ideal behavior.  Metadata (from a librarian standpoint) tends to be free of format-related markup (as that allows for easier sharing, understanding of metadata.  Currently Community & Collection homepage information is HTML-based and is stored in metadata that is appropriate for a minor subset of information (like the title) but it is better to move large/rich text to bitstreams.  
   2. Proposal here is to consider storing HTML-based markup (for Site, Community & Collection homepages) in Bitstream(s) associated with the object in question.  May allow for more CMS-lite behavior in the future
   3. Timeline for this is uncertain.  Possibly in 7 or 8. May depend on how/whether it can be scoped.
5. Concurrency in DSpace 7 (or 8).  What do we want to do when multiple editors are editing the same object?  Needs further analysis regarding implementation details
   1. We've decided (in meeting on March 7, 2019) to use ETags to implement concurrency. REST Contract notes on ETags: https://github.com/DSpace/Rest7Contract#etags--conditional-headers
   2. ETags only update of the two fields match. If someone edits first, your edit would fail and you would get a fail response (422?)
   3. ETags seems to have broader support in other REST APIs.  Recommended also by both Art and Andrea.

6. Not discussed much, but could there be opportunities to use Travis CI + Docker Compose for testing of Angular??

   • Docs: https://docs.travis-ci.com/user/docker/#using-docker-compose
   • Blogpost on how to do it: http://elliot.land/post/using-docker-compose-on-travis-ci
   • An example travis.yml file: https://github.com/Ortus-Solutions/docker-buildfiles/blob/master/.travis.example.yml

Notes

• Support for bundles in REST (DS-4317)
  • General agreement that we need to support Bundles in REST API
  • This also means that the "/item/{uuid}/bitstreams" endpoint should go away: https://github.com/DSpace/Rest7Contract/blob/master/items.md#bitstreams
  • Should be replaced by a "/item/{uuid}/bundles" endpoint.  May also want to (eventually) think about how to use Projections to limit the Bundles/Bitstreams embedded
  • Keep in mind, we should also implement the "primary" bitstream option (supported in DSpace v6).  This allows an easier way to select the main Bitstream (and Thumbnail) for display on Item Page (or in Item lists).
  • We recommend a staged approach here.  The "/item/{uuid}/bitstreams" endpoint may exist in parallel to the new bundles endpoint until everything (including Angular) moves over to Bundles...then it should be removed in a followup PR
• End to end (e2e) tests in Angular UI
  • Should we disable immediately per this PR? https://github.com/DSpace/dspace-angular/pull/454  They are causing issues in Travis CI.  Travis servers are taking a very long time to run these tests against the demo REST API and often timeout
  • The tests themselves are very minimal right now.  Some are not useful. The only useful ones are the Search e2e tests.
  • We all agree that e2e tests are necessary, and we'd like to have more of them.  They will find bugs that are not evident in normal tests...if Angular only tests itself and REST only tests itself, then bugs can still exist if their tests/assumptions are not completely aligned.  We need e2e tests to test Angular against the current REST responses.
  • Next Steps
    • Disable e2e tests for now (all agree)
    • Leave this e2e ticket open: https://github.com/DSpace/dspace-angular/issues/453
    • Reach out to the Docker experts (Terrence W Brady and/or Pascal-Nicolas Becker) to see if there's any interest/ideas for potentially spinning up Docker instances in Travis to run the e2e tests.
    • If we can find a better strategy for running e2e tests without reliance on the demo REST site then we can create more e2e tests to more fully test the two applications working together.
• Security discussion on newly proposed statistics endpoints.  See this PR https://github.com/DSpace/Rest7Contract/pull/63
  • Tim noted some possible concerns about whether this endpoint should "know" whether a request came from the UI or elsewhere: https://github.com/DSpace/Rest7Contract/pull/63#issuecomment-519182491
  • Great discussion here of what is really doable. 
  • In the end, we've realized this is really really complex.  There's no straightforward way to identify a UI request versus a different REST client.  So, honestly, there's nothing we can do in the REST Contract here.  In the implementation if we notice issues here we could potentially check the Referer header as a resource to understand how the request was generated. But, even headers can be "faked" by a bot.  So, a perfect filtering mechanism doesn't exist
  • Next steps: Move forward with this REST contract & implementation as described.  This is at least as trustworthy as any web based statistics generated in DSpace v6.  If we find/discover ways to improve it, we can do that later (possible even post DSpace v7)
• Update on Estimation process
  • Tim did a thorough review of  Development Planning Spreadsheet 
  • First column now identifies tasks that are well defined and can be immediately estimated ("Estimation") versus tasks that are not well defined yet ("Needs More Info") versus tasks that have been implemented already ("Completed"). 
    • Please note that "Completed" doesn't necessary mean the feature is 100% perfect, just that the feature exists.  If further improvements are known to be necessary, they should be added as a separate task which can then be estimated (or discussed as needed).
  • In this review, three main "big topics" were pulled out that effect a lot of other estimates and/or scoping of tasks
    • REST API Projections - affects performance of REST API, and a few outstanding features need projections to ensure they can function without slowing down the entire system
    • Authorization in Angular - currently Angular UI has hardcoded some authorizations, as we never determined exactly how to pass authorization information from REST API to Angular
    • Scripts & Processes REST endpoint - an early idea which was tabled for some time. Many administrative tools could potentially use a generic "scripts & processes" endpoint if it exists. More discussion & scoping of this idea would better define whether it should exist & based on that how administrative tools (export/import/curation) should be scoped/estimated.
  • Tim will be bringing these topics forward in upcoming meetings for revisiting.
  • Tim will also be working with Heather to start estimation process for those features which are well defined...we can begin estimations with those, and do a second round of estimations for other features (once defined/scoped)
• NEXT MEETING
  • There is a holiday in much of Europe on Aug 15. 
  • WE WILL NEXT MEET IN TWO WEEKS, on Aug 22.  However, we will update status next week (via Slack/Wiki) so that development can proceed as normal