High-level storage interface

Overview

This page is for discussion and description of a proposed "high-level" storage interface for fedora. A draft proposal can be found below. In a nutshell, this proposal aims to remove certain hard-coded storage assumptions in Fedora, and present a storage layer/interface that would allow a place for extensions to Fedora that implement multiplexing, non-blob storage, lock-free updates, etc.

Documents

• Draft proposal (pdf) - Initial proposal for HighlevelStorage layer. Discusses motivation, separation of concerns, and possible use cases.
• Aaron's Presentation - Presented at Feb. 2010 ondon meeting. Summarizes proposal, and introduces a possible configuration of internal modules connected by chaining.
• Asger's Presentation - Presented at Feb. 2010 London meeting. Introduces Writable/Readable store interfaces used for plugging in indexing, caching, etc.

Issues for Discussion

• Interface to DOManager layer
  • Do we present a single HighLevelStorage interface to the DOManager for reads and writes, or
    do we present Readable and Writable?
• Datastream versioning.
  • There is a proposal to get rid of datastream versions in the model, and present versioning as a storage layer concern (TODO: get proposal and link to it)
  • If versioning becomes a concern of the storage layer, how does this affect the proposed interface?
    • Add get(PID pid, String version)? Is this sufficient?
• DigitalObject representation
  • Is the existing DigitalObject interface sufficient? Can it be improved? Should it be replaced?
  • If versioning moves from the object model (datastream versions) to the storage layer, then presumably DigitalObject would have to be updated so that it no longer exposes versions
  • Is there anything that would obviously present a problem to performance or scalability with corner-case objects (lots of datastreams or versions)?
• Return values
  • Operation or transaction ID?
  • A generic map that any module in the storage layer can populate?
• Chaining of modules
  • Should HighLevelStorage modules be assembled into a chain, a tree, both?
  • Tree configuration implies a list Writable modules, all seeing the same input.
  • Chain configuration implies modules that implement HighLevelStorage, but also may have a delegate HighLevelStorage class configured beneath
    • May need to pay attention to InputStreams passed up/down chain

Relevant tracker items

Prerequisites and Cleanup

These are issues that represent necessary (or desirable) cleanup of existing code. These issues are general improvements that will make implementing a HighLevelStorage layer easier.

Issues to revisit

These are issues that may need need to be re-stated in the context of HighLevelStorage, re-implemented using HighLevelStorage concepts, or may be affected in some other way.