Time/Place

This meeting is a hybrid teleconference and slack chat. Anyone is welcome to join...here's the info:

Attendees 


**Each week a meeting chair will be assigned based on a rotating schedule.**

(star) - denotes note taker

Agenda

  1. Announcements:
    1. Fedora Showcase - Sept 12-14, 2023
      1. CFP is available until July 31, 2023 - Apply now
    2. OR Debrief
  2. Pop-up/Other Topics:
    1. Tombstone decision from Governance
      1. Would like to see more clear definitions outlined of PURGE vs DELETE
      2. Approve Jared's recommendations via this doc so long as the definitions of purge and delete are clear
    2. Thomas Bernhart Problems with DB connection reset when trying to commit long-running transaction
      Error message: HikariPool-1 - Failed to validate connection org.postgresql.jdbc.PgConnection@66a670f4 (This connection has been closed.). Possibly consider using a shorter maxLifetime value.
      Make more configuration options for Hikari? Can Hikari be configured directly instead of fcrepo-properties?
      Frequently used Hikary properties: https://github.com/brettwooldridge/HikariCP#frequently-used

  3. Updates on:

    1. Open Tickets (but assigned in some cases): 
      1. FCREPO-3898 - Getting issue details... STATUS
      2. FCREPO-3897 - Getting issue details... STATUS
      3. FCREPO-3896 - Getting issue details... STATUS
      4. FCREPO-3895 - Getting issue details... STATUS
      5. FCREPO-3894 - Getting issue details... STATUS
      6. FCREPO-3884 - Getting issue details... STATUS
      7. FCREPO-3883 - Getting issue details... STATUS / FCREPO-3882 - Getting issue details... STATUS / FCREPO-3881 - Getting issue details... STATUS
      8. FCREPO-3866 - Getting issue details... STATUS
      9. FCREPO-3865 - Getting issue details... STATUS
      10. FCREPO-3864 - Getting issue details... STATUS

    2.  In Progress and older but still relevant open tickets:

      1. OAI-PMH Stuff
      2. FCREPO-3875 - Getting issue details... STATUS
      3. FCREPO-3871 - Getting issue details... STATUS

    3. In Review:

      1. FCREPO-3893 - Getting issue details... STATUS
      2. FCREPO-3891 - Getting issue details... STATUS
      3. FCREPO-3850 - Getting issue details... STATUS
      4. FCREPO-3849 - Getting issue details... STATUS

  4. New tickets:

  5. Backlog Tickets to consider working: NA at present
  6. Next Meeting Chair:
    1. Chair: James Alexander
    2. Note Taker: Ben Pennell
    3. See Rotating Schedule here 

Notes

  1. Announcements
    1.  Fedora Showcase
      1. Will be two to three half-days, depending on response level
      2. Please consider submitting something!
        1. Suggestions
          1. Reports on interesting uses of Fedora
          2. Reports from sister communities (Samvera, Islandora, etc.)
          3. "How to report an issue or engage with the community"
        2. Arran has been reaching out to old Google Group admins to see if there's interest in getting user groups back together again. There will be space made for this during the event.
    2. Open Repositories Debrief
      1. Fedora was not particularly well-represented; focus of conference has swung more toward academic institutions and repositories for research data (more emphasis on grant-related data preservation with limited lifespans, rather than long-term preservation). Discussion of full-stack applications for research data discovery and storage, etc. There was an OCFL catch-up meeting which had some interesting conversations. Different focus of conference may have been related to location of conference in South Africa, with many representatives from institutions on the continent – these research-related interests may be of greater priority to that regional community.
      2. Is there an additional/better venue for these types of conversations?
      3. Discussion: Fedora certainly could be used for research data management.
  2. Pop-up/Other Topics
    1. Tombstone decision
      1. Jared's draft document was approved as long as definitions of PURGE and DELETE are clear in the documentation. (PURGE = history is erased, DELETE = not readily visible, but still exists as historical record).
      2. The documentation work is not a blocker for the technical work – we just need to be sure that the documentation is clear and accurate when the work is done.
      3. Jared proposed creating a wiki page to discuss options and differences; he will create a JIRA ticket to track this part of the project.
    2. DB connection reset problem
      1. Thomas' team was having problems because a centrally managed PostgreSQL server is in use and its settings can't be changed. Thus, Fedora needs to be adapted to work with the existing PG environment.
      2. Question: is there a way to more directly set Hikari properties in Fedora to make testing/troubleshooting easier? Jared suggested trying to set the Hikari properties as system properties on the JVM. Thomas hasn't tried that yet but will attempt it. Jared suggests that some settings may work this way, but others may not if they're being explicitly set in Fedora code based on other Fedora properties (though Mike suspects that it will not work for any settings). Whatever the result is, we should probably document this behavior (e.g. list properties that must be defined through Fedora and show how things are mapped). It might also be possible, if necessary, to add Fedora properties with a Hikari-specific prefix or config file and write some simple code to pass appropriate values along to Hikari.
      3. Problem seems to be that database connection is timing out and Hikari is not reopening the connection automatically, leading to failures on commits. Thomas wants to try changing some settings to see if this behavior can be addressed. No ticket has been opened yet because it's not yet clear exactly what needs to go in the ticket.
      4. Thomas will do more testing and will report back.
    3. Doron reported that migration work is ongoing – latest test run should yield results tomorrow (seems okay so far).
  3. Updates on:
    1. Open tickets

      1. FCREPO-3898 (document logging configuration)

        1. Simple documentation ticket to add a note about a setting.

      2. FCREPO-3897 (incorrect error when nesting archival groups in the UI)

        1. Dan discovered this problem while trying something. Error message inappropriately mentions a binary (wrong header on pop-up box). Very minor problem.
        2. Led to general discussion about updating the overall HTML web UI:
          1. Can we develop a better tool to fit a role more like the old F3 Flash UI?
          2. Is building a new admin tool worth the maintenance costs?
          3. Should such a tool be a separate project?
          4. Due to limitations of what can be done in the browser (e.g. creating headers required for transactions), a feature-rich client might work better as a separate Node.js application (for example) that runs on a server and talks to the Fedora API.
          5. Moving administration to a separate project could also potentially allow the removal of the HTML API from the core project, which could simplify the core while adding richer functionality elsewhere.
          6. Clearly defining scope up front is important (e.g. this is not intended to replace Islandora, just to allow administration of data).
          7. This proposal could lead to a "Fedora 7" which has no breaking API changes but removes some previously built-in functionality to push it elsewhere.
          8. Dan pointed out that the challenges of building this overlap with some of the challenges of building the OAI-PMH provider (e.g. accessing Fedora functionality exclusively via the API as an external consumer).
          9. Likely first step: begin writing a proposal document.

      3. FCREPO-3896 (ingests > 2GB prone to errors on upload)

        1. There was a session expiry problem that was impacting some very large uploads (now works up to 100GB).
          1. This did not require a bug fix, just the fcrepo.session_timeout setting, which defaults to 3 minutes but can be increased for uploads that take longer than this.
          2. When this problem occurs, it will present as a timeout followed by a rollback
          3. Jared raised the point that if we're actively receiving data in an upload, we should not be timing out because we're actively doing something! Can this be fixed?
        2. Dan is following up to see if this is related to the same problem, or something else.

      4. FCREPO-3895 (expose effective ACL functionality via API)

        1. Jared talked to Paul about this; he wanted to be able to find out which ACL controls an object. Jared does not believe this is possible right now because WebAC is a filter rather than part of the core layer. (This is useful if we want to change the ACL library in future, but makes it difficult/impossible to pass information because there is no dependency relationship between the components). A solution would require significant refactoring to make logic available to the core.

      5. FCREPO-3894 (UI broken for non fedoraAdmin users)

      6. FCREPO-3884 (OAI-PMH)

        1. Dan is currently running into challenges with listIdentifiers because a call to a root container with a large number of children can take a very long time with serialization (e.g. multiple hours for multiple millions). Jared suggested that the search API might allow paginated retrieval of data in a more efficient way... or maybe this is a use case where we have to leverage a tool like Solr (though Dan is trying to avoid adding new dependencies). Dan will investigate the search API solution next.

      7. FCREPO-3883/FCREPO-3882/FCREPO-3881/FCREPO-3866 (all related to tombstone behavior/access to past versions of deleted objects/etc.)

        1. Related to discussion above – we now have an approach, so work just needs to begin on the tickets.

      8. FCREPO-3865 (add the ability to run a reindex from the CLI or web interface on demand)

        1. Still needs to be done (by someone who can figure out the SQL complexities involved – e.g. how to rebuild the index without breaking the running system; e.g., use temporary tables and then swap in via transaction?). Challenge/trade-off: you'll have a period of read-only state while index is going into place. Challenge: what happens if more objects are added while the index is being generated? How do we keep track of those?

      9. FCREPO-3864

        1. We ran out of time before further tickets could be discussed.
    2. In progress and older but still relevant

      1. FCREPO-3875

      2. FCREPO-3871

    3. In review

      1. FCREPO-3893

      2. FCREPO-3891

      3. FCREPO-3850

      4. FCREPO-3849

  4. New tickets:
  5. Backlog Tickets to consider working
  • No labels

All content on the LYRASIS Wiki is licensed under the CC BY (Attribution) license, unless otherwise noted.