Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Extend database Tabled Topic to discuss large bulk operations

...

  1. Management of database connections for DSpace going forward (7.0 and beyond). What behavior is ideal? Also see notes at DSpace Database Access
    1. In DSpace 5, each "Context" established a new DB connection. Context then committed or aborted the connection after it was done (based on results of that request).  Context could also be shared between methods if a single transaction needed to perform actions across multiple methods.
    2. In DSpace 6, Hibernate manages the DB connection pool.  Each thread grabs a Connection from the pool. This means two Context objects could use the same Connection (if they are in the same thread). In other words, code can no longer assume each new Context() is treated as a new database transaction.
      1. Should we be making use of SessionFactory.openSession() for READ-ONLY Contexts (or any change of Context state) to ensure we are creating a new Connection (and not simply modifying the state of an existing one)?  Currently we always use SessionFactory.getCurrentSession() in HibernateDBConnection, which doesn't guarantee a new connection: https://github.com/DSpace/DSpace/blob/dspace-6_x/dspace-api/src/main/java/org/dspace/core/HibernateDBConnection.java
    3. Bulk operations, such as loading batches of items or doing mass updates, have another issue:  transaction size and lifetime.  Operating on 1 000 000 items in a single transaction can cause enormous cache bloat, or even exhaust the heap.
      1. Bulk loading should be broken down by committing a modestly-sized batch and opening a new transaction at frequent intervals.  (A consequence of this design is that the operation must leave enough information to restart it without re-adding work already committed, should the operation fail or be prematurely terminated by the user.  The SAF importer is a good example.)
      2. Mass updates need two different transaction lifetimes:  a query which generates the list of objects on which to operate, which lasts throughout the update; and the update queries, which should be committed frequently as above.  This requires two transactions, so that the updates can be committed without ending the long-running query that tells us what to update.


Ticket Summaries

  1. Help us test / code review! These are tickets needing code review/testing and flagged for a future release (ordered by release & priority)

    Expand

    Jira
    serverDuraSpace JIRA
    columnskey,summary,type,created,updated,assignee,reporter,priority,status,fixversions
    maximumIssues20
    jqlQueryfilter=13905 ORDER BY fixVersion DESC, priority DESC
    serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5


  2. Newly created tickets this week:

    Expand
    Jira
    serverDuraSpace JIRA
    columnskey,summary,type,created,assignee,reporter,priority,status
    maximumIssues20
    jqlQueryfilter=13902
    serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5


  3. Old, unresolved tickets with activity this week:

    Expand

    Jira
    serverDuraSpace JIRA
    columnskey,summary,type,created,updated,assignee,reporter,priority,status
    maximumIssues20
    jqlQueryfilter=13906
    serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5


  4. Tickets resolved this week:

    Expand

    Jira
    serverDuraSpace JIRA
    columnskey,summary,type,created,assignee,reporter,priority,status,resolution
    maximumIssues20
    jqlQueryfilter=13903
    serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5


  5. Tickets requiring review. This is the JIRA Backlog of "Received" tickets: 

    Expand

    Jira
    serverDuraSpace JIRA
    columnskey,summary,type,created,updated,assignee,reporter,priority
    maximumIssues20
    jqlQueryfilter=10152
    serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5


...