Developers Meeting on Weds, July

...

10, 2019

Agenda

Quick Reminders

Friendly reminders of upcoming meetings, discussions etc

• DSpace 7 Working Group: Next meeting is Thurs, July 11 at 14:00 UTC. 

• DSpace 7 Entities Working Group: Next meeting is Tues, July 9 16 at 15:00 UTC. Agenda: 2019-07-02 DSpace 7 Entities WG Meeting

• DSpace Developer Show and Tell Meetings:  On hold until interesting topics arise.
• 2019 DSpace North American User Group Meeting (Sept 23-24) Call for proposals out!

...

1. (Ongoing Topic) Quick Updates from other meetings

   1. DSpace 7 Status Updates for this week (from DSpace 7 Working Group or DSpace 7 Entities Working Group)

   2. DSpace 6.x Status Updates for this week

      1. 6.4 will surely happen at some point, but no definitive plan or schedule at this time.  Please continue to help move forward / merge PRs into the dspace-6.x branch, and we can continue to monitor when a 6.4 release makes sense.
2. Ongoing Work
   1. Upgrading Solr Server for DSpace (Mark H. Wood )
      
      1. Auto-reindexing in Solr: 

         Jira
         server DuraSpace JIRA serverId c815ca92-fd23-34c2-8fe3-956808caf8c5 key DS-3658

         
         
         1. Should this only happen for major releases?  Should it be configurable?  Can we find a more precise trigger?  When do we need to reindex?
      2. Dump/restore tool for the authority core.  

         Jira
         server DuraSpace JIRA columns key,summary,type,created,updated,due,assignee,reporter,priority,status,resolution serverId c815ca92-fd23-34c2-8fe3-956808caf8c5 key DS-4187

           Or should we use solr-export-statistics?
   2. DSpace Docker and Cloud Deployment Goals (Terrence W Brady )
      

      1. Update sequences on initialization

         1. https://github.com/DSpace/DSpace/pull/2362 - update sequences port

         2. https://github.com/DSpace/DSpace/pull/2361  - update sequences port

      2. DSpace Launcher Dashboard - Deploy a PR on AWS for Testing
         1. There is a 2 minute video that illustrates this proposal.
3. For Discussion: Brainstorming how to improve DSpace database usage (now that we use Hibernate)

   1. Jira
      server DuraSpace JIRA serverId c815ca92-fd23-34c2-8fe3-956808caf8c5 key DS-4276

   2. See also Session management discussion under "Tabled Topics" section below.  In DSpace 5, each "Context" object had a separate DB connection.  In DSpace 6+, each thread has a separate DB Connection (but Context objects may share a DB connection if they share the same thread).
4. Tickets, Pull Requests or Email threads/discussions requiring more attention? (Please feel free to add any you wish to discuss under this topic)
   1. Quick Win PRs: https://github.com/DSpace/DSpace/pulls?q=is%3Aopen+review%3Aapproved+label%3A%22quick+win%22

Tabled Topics

These topics are ones we've touched on in the past and likely need to revisit (with other interested parties). If a topic below is of interest to you, say something and we'll promote it to an agenda topic!

1. Brainstorms / ideas
   
   1. (On Hold, pending Steering/Leadership approval) Follow-up on "DSpace Top GitHub Contributors" site (Tim Donohue ): https://tdonohue.github.io/top-contributors/
   2. Bulk Operations Support Enhancements (from Mark H. Wood)
   3. Curation System Needs (from Terrence W Brady  )
2. Management of database connections for DSpace going forward (7.0 and beyond). What behavior is ideal? Also see notes at DSpace Database Access
   1. In DSpace 5, each "Context" established a new DB connection. Context then committed or aborted the connection after it was done (based on results of that request).  Context could also be shared between methods if a single transaction needed to perform actions across multiple methods.
   2. In DSpace 6, Hibernate manages the DB connection pool.  Each thread grabs a Connection from the pool. This means two Context objects could use the same Connection (if they are in the same thread). In other words, code can no longer assume each new Context() is treated as a new database transaction.
      
      1. Should we be making use of SessionFactory.openSession() for READ-ONLY Contexts (or any change of Context state) to ensure we are creating a new Connection (and not simply modifying the state of an existing one)?  Currently we always use SessionFactory.getCurrentSession() in HibernateDBConnection, which doesn't guarantee a new connection: https://github.com/DSpace/DSpace/blob/dspace-6_x/dspace-api/src/main/java/org/dspace/core/HibernateDBConnection.java
   3. Bulk operations, such as loading batches of items or doing mass updates, have another issue:  transaction size and lifetime.  Operating on 1 000 000 items in a single transaction can cause enormous cache bloat, or even exhaust the heap.
      1. Bulk loading should be broken down by committing a modestly-sized batch and opening a new transaction at frequent intervals.  (A consequence of this design is that the operation must leave enough information to restart it without re-adding work already committed, should the operation fail or be prematurely terminated by the user.  The SAF importer is a good example.)
      2. Mass updates need two different transaction lifetimes:  a query which generates the list of objects on which to operate, which lasts throughout the update; and the update queries, which should be committed frequently as above.  This requires two transactions, so that the updates can be committed without ending the long-running query that tells us what to update.

Ticket Summaries

1. Help us test / code review! These are tickets needing code review/testing and flagged for a future release (ordered by release & priority)

   Expand
   Jira server DuraSpace JIRA columns key,summary,type,created,updated,assignee,reporter,priority,status,fixversions maximumIssues 20 jqlQuery filter=13905 ORDER BY fixVersion DESC, priority DESC serverId c815ca92-fd23-34c2-8fe3-956808caf8c5

   
   

2. Newly created tickets this week:

   Expand
   Jira server DuraSpace JIRA columns key,summary,type,created,assignee,reporter,priority,status maximumIssues 20 jqlQuery filter=13902 serverId c815ca92-fd23-34c2-8fe3-956808caf8c5

   
   

3. Old, unresolved tickets with activity this week:

   Expand
   Jira server DuraSpace JIRA columns key,summary,type,created,updated,assignee,reporter,priority,status maximumIssues 20 jqlQuery filter=13906 serverId c815ca92-fd23-34c2-8fe3-956808caf8c5

   
   

4. Tickets resolved this week:

   Expand
   Jira server DuraSpace JIRA columns key,summary,type,created,assignee,reporter,priority,status,resolution maximumIssues 20 jqlQuery filter=13903 serverId c815ca92-fd23-34c2-8fe3-956808caf8c5

   
   

5. Tickets requiring review. This is the JIRA Backlog of "Received" tickets: 

   Expand
   Jira server DuraSpace JIRA columns key,summary,type,created,updated,assignee,reporter,priority maximumIssues 20 jqlQuery filter=10152 serverId c815ca92-fd23-34c2-8fe3-956808caf8c5

   
   

Meeting Notes

Meeting Transcript 

...