Part 4: Integrating The Packager

As the software which interacts with the Packager, the Intake services need to be updated to apply the new file layout to any incoming data in order to support new functionality.

Tokenization

As one of the processes which creates ACE Tokens for our collections, any changes to how we deal with when we create tokens and over what files would occur in intake. This is a good time to think about important questions regarding our current practices: what files do we need to create tokens for and is it wasteful to create tokens for packaging metadata? 

In general the payload files are what we care the most about being synchronized between files, as the metadata for the file layout is used for verification on transfer, not for long term preservation. Therefore we might want to consider a workflow where we only create ACE Tokens for the payload files of a package, and let each replication service validate the metadata.

OCFL Packaging

New Packages

The workflow for creating a new OCFL Object should be very similar to the current flow, with the exception that we are now applying a different packaging format. There are additional steps which may be taken depending on what we decide is included.

Workflow

1. Query OTM Bridge service for a preservation package waiting to be ingested
2. Query Ingest to check for existence of the preservation package
   1. For this workflow, assume we see that it does not exist
3. Use OCFL Packager to create a new OCFL Storage Root and OCFL Object
   1. Handles creation of the namaste, inventory.json, version directory, etc
      
   2. Handles moving files into the content directory
   3. Checks fixity for files during movement of data
4. Create optional packaging files
   1. Generate ACE Tokens and put them in a Token Store
   2. Put any application logging into the package
5. Finalize the package
   1. Packager generates inventory.json.sha512
6. Register the OCFL Object with the Ingest Server
   
   1. Load Files
   2. Load Fixity
      1. Is it necessary to track fixity of payload files in the ingest server?
   3. Load ACE Tokens
      1. If these are embedded in the package, is it still needed?

Applying Versions

Depending on how much work we want to do, the workflow for applying a new payload to a package has additional steps which must be taken. In addition, this process requires more communication as data will need to be re-staged so that it can be modified.

Workflow

1. Query Bridge service for a preservation package waiting to be ingested
2. Query Ingest to check for existence of the preservation package
1. For this workflow, we assume that the Ingest tells us that a package does exists
3. Request staging of the preservation package metadata
1. Ideally should go through the same Chronopolis node that Ingested the package to begin with
2. Could be a workflow of its own, perhaps needs a staging service or something similar
3. We might need additional data to be staged for calculating deltas, and should aim to stage the minimum amount of data needed
4. Modify the preservation package
   1. Validate fixity
   1. Add new version timestamp, number, files
   2. Ensure any additional metadata is added as required
1. Optional: Create deltas for newest versions of a file
2. Move the payload files in to place
3. Update package metadata for versioning
5. Create optional packaging files
   1. Files which are not modified do not need new tokens
1. Generate ACE Tokens
2. Generate logging information
6. Finalize the package
1. Regenerate the tagmanifest-sha256.txt or inventory.json.sha512
7. Apply content addressing if wanted
8. Update the package in the Ingest Server
1. Register new versions of files
2. Register new fixity for files
3. Register new tokens for files
4. Files which do not change are not re-distributed

Deleting Versions

todo