View Source

About This Page

The purpose of this page is to inform the design and development of Akubra by providing an analysis of existing Blob Storage APIs and implementations.

Here we define Pluggable Blob Storage API as a programming interface with the following characteristics:

A Pluggable Blob Storage API MUST:

Be primarily concerned with providing direct read/write access to named files.
Be designed with back-end pluggability in mind.

A Pluggable Blob Storage API MAY:

Provide read/write access to metadata about stored files.
Provide a way to list or query stored files.
Provide transactional capabilities.

A Pluggable Blob Storage API MUST NOT:

Be concerned with container (directory/folder/collection) semantics.
Be concerned with versioning semantics (e.g. revision history).

APIs and Implementations

API	Language	Target Application	Known Implementations
AKU-API: Akubra BlobStore	Java	General-Purpose	AKU-FS: FSBlobStore S3 (Planned) Transactional Impl (Planned)
ALF-API: Alfresco ContentStore	Java	Alfresco CMS	ALF-FS: FileContentStore ALF-HTT: HttpAlfrescoStore ALF-REP: ReplicatingContentStore ALF-ROU: TenantRoutingFileContentStore
BIT-API: DSpace BitStore	Java	DSpace 2 (Unreleased)	BIT-FED: FedoraBitStore BIT-FS: DSBitStore BIT-HON: HoneycombBitStore BIT-S3: S3BitStore BIT-SRB: SRBBitStore
DJA-API: Django Storage	Python	Any Using Django	DJA-FS: DefaultStorage DJA-S3: S3
DSY-API: Daisy BlobStore	Java	Daisy CMS	DSY-FS: FSBlobStore
EBS-API: Windows Sharepoint EBS	COM	MS SQL Server	EBS-FS: Custom Implementation
JKB-API: JackRabbit BLOBStore (deprecated)	Java	Apache JackRabbit	JKB-DB: DbBlobStore JKB-FS: FileSystemBLOBStore
JKD-API: JackRabbit DataStore	Java	Apache JackRabbit	JKD-DB: DbDataStore JKD-FS: FileDataStore
LLS-API: Fedora ILowLevelStorage	Java	Fedora Repository (2.x to 3.1)	LLS-FS: GenericFileSystem LLS-HON: Honeycomb LLS-IRD: iRODS LLS-S3: S3 LLS-SRB1: Fedora22Ext LLS-SRB2: DARTFedoraDB
OTM-API: Topaz OTM BlobStore	Java	Topaz OTM	OTM-FED: FedoraBlobStore OTM-FS: SimpleBlobStore
XAM-API: SNIA XAM API	Java / C	General-Purpose	XAM-CEN: EMC Centera XAM-HON: Honeycomb XAM-IAP: HP IAP

Capability Matrix

API/Impl	ID Type	ID Gen	Write?	Rewrite?	Read-Only MD	Writable MD	Listing Options	Transactions
AKU-API	URI	Internal or External	Yes	Yes	Size	?	All By Prefix	Optional, JTA
AKU-FS	file:/// URI	Internal	Yes	Yes	Size	None	All By Prefix	No
API/Impl	ID Type	ID Gen	Write?	Rewrite?	Read-Only MD	Writable MD	Listing Options	Transactions
ALF-API	URL	Internal or External	Yes	Yes	Size Last-Modified	None	All Last-Modified Range	Optional, JTA
ALF-FS	store:// URL	Internal or External	Yes	Yes	Size Last-Modified	None	All Last-Modified Range	Optional, JTA
ALF-HTT	http:// URL	N/A	No	No	Size Last-Modified	None	None	Optional, JTA
ALF-REP	URL	Internal or External	Yes	Yes	Size Last-Modified	None	All Last-Modified Range	Optional, JTA
ALF-ROU	URL	Internal or External	Yes	Yes	Size Last-Modified	None	All Last-Modified Range	Optional, JTA
API/Impl	ID Type	ID Gen	Write?	Rewrite?	Read-Only MD	Writable MD	Listing Options	Transactions
BIT-API	Unicode String	Internal	Yes	Yes	Any (Map)	None	None	No
BIT-FED	(NCNameChar)*	Internal	Yes	No	Size Checksum (Multiple types) Last-Modified	None	None	No
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="7c09ef1a-9596-466f-83ff-11bb9846f362"><ac:plain-text-body><![CDATA[	BIT-FS	([DEV:0-9]){38,40}	Internal	Yes	Yes	Size ]]></ac:plain-text-body></ac:structured-macro> MD5 Checksum Last-Modified	None	None	No
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="236e6a16-fe87-4604-a707-6ac44724458d"><ac:plain-text-body><![CDATA[	BIT-HON	([DEV:0-9]){38,40}	Internal	Yes	Yes	?	None	None	No	]]></ac:plain-text-body></ac:structured-macro>
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="45d72a16-86b6-4562-8ee5-0fe2996dee44"><ac:plain-text-body><![CDATA[	BIT-S3	([DEV:0-9]){38,40}	Internal	Yes	Yes	Size ]]></ac:plain-text-body></ac:structured-macro> MD5 Checksum Last-Modified	None	None	No
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="41a0ef9f-9289-4e4f-b67d-12cdf32e9ded"><ac:plain-text-body><![CDATA[	BIT-SRB	([DEV:0-9]){38,40}	Internal	Yes	Yes	Size ]]></ac:plain-text-body></ac:structured-macro> MD5 Checksum	None	None	No
API/Impl	ID Type	ID Gen	Write?	Rewrite?	Read-Only MD	Writable MD	Listing Options	Transactions
DJA-API	Unicode String	External	Yes	Yes	Size URL for Reading	None	None	No
DJA-FS	Unicode String	External	Yes	Yes	Size URL for Reading	None	None	No
DJA-S3	Unicode String	External	Yes	Yes	Size URL for Reading	None	None	No
API/Impl	ID Type	ID Gen	Write?	Rewrite?	Read-Only MD	Writable MD	Listing Options	Transactions
DSY-API	Unicode String	Internal	Yes	No	None	None	None	No
DSY-FS	Unicode String	Internal	Yes	No	None	None	None	No
API/Impl	ID Type	ID Gen	Write?	Rewrite?	Read-Only MD	Writable MD	Listing Options	Transactions
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="fd683b37-21ca-4d9d-b5c1-5a2005d5cdd3"><ac:plain-text-body><![CDATA[	EBS-API	byte[]	Internal	Yes	No	Size ]]></ac:plain-text-body></ac:structured-macro> Last-Modified Created Accessed	None	None	No
EBS-FS	GUID or String	Internal	Yes	No	Size Last-Modified Created Accessed	None	None	No
API/Impl	ID Type	ID Gen	Write?	Rewrite?	Read-Only MD	Writable MD	Listing Options	Transactions
JKB-API	Unicode String	Internal	Yes	Yes	None	None	None	No
JKB-DB	propertyId + index	Internal	Yes	Yes	None	None	None	No
JKB-FS	Filesystem Path	Internal	Yes	Yes	None	None	None	No
API/Impl	ID Type	ID Gen	Write?	Rewrite?	Read-Only MD	Writable MD	Listing Options	Transactions
JKD-API	Hex String	Internal	Yes	No	Size Last-Modified	None	All	No
JKD-DB	SHA-1 Digest of Content	Internal	Yes	No	Size Last-Modified	None	All	No
JKD-FS	SHA-1 Digest of Content	Internal	Yes	No	Size Last-Modified	None	All	No
API/Impl	ID Type	ID Gen	Write?	Rewrite?	Read-Only MD	Writable MD	Listing Options	Transactions
LLS-API	pid or pid + dsId	External	Yes	Yes	None	None	None	No
LLS-FS	pid or pid + dsId	External	Yes	Yes	None	None	None	No
LLS-HON	pid or pid + dsId	External	Yes	Yes	None	None	None	No
LLS-IRD	pid or pid + dsId	External	Yes	Yes	None	None	None	No
LLS-S3	pid or pid + dsId	External	Yes	Yes	None	None	None	No
LLS-SRB1	pid or pid + dsId	External	Yes	Yes	None	None	None	No
LLS-SRB2	pid or pid + dsId	External	Yes	Yes	None	None	None	No
API/Impl	ID Type	ID Gen	Write?	Rewrite?	Read-Only MD	Writable MD	Listing Options	Transactions
OTM-API	Unicode String	External	Yes	No	None	None	None	Optional, JTA
OTM-FED	Unicode String	External	Yes	No	None	None	None	Best-effort, JTA
OTM-FS	Unicode String	External	Yes	No	None	None	None	Yes, JTA
API/Impl	ID Type	ID Gen	Write?	Rewrite?	Read-Only MD	Writable MD	Listing Options	Transactions
XAM-API	XUID	Internal	Yes	No ¹	Last-Modified Created Accessed	Several Predefined Custom ²	All SQL-Like Queries	No
XAM-CEN	XUID	Internal	Yes	No	Last-Modified Created Accessed	Several Predefined Custom	All SQL-Like Queries	No
XAM-HON	XUID	Internal	Yes	No	Last-Modified Created Accessed	Several Predefined Custom	All SQL-Like Queries	No
XAM-IAP	XUID	Internal	Yes	No	Last-Modified Created Accessed	Several Predefined Custom	All SQL-Like Queries	No

¹ XAM is designed with a fixed-content assumption - data cannot be modified without changing its XUID.
² In XAM, all metadata fields are either binding or non-binding. Binding metadata cannot change while keeping the same XUID for an XSet, while non-binding metadata can.

Other APIs/Protocols/Impls of Interest

Apache Sling - REST interface on JCR, need to look at this more...
AtomPub - Collection semantics
Azure Blob Storage API - Collection semantics, REST-based
CMIS - Draft; has versioning and container semantics.
DSpace 1 BitstreamStorageManager - Single impl, supports either local filesystem or SRB
Eclipse - BlobStore- Single impl, maps UUIDs to files on disk
Google Datastore API - Google's "cloud" underneath, REST-ish, container semantics, transactions!...but quite limited today (1MB limit on content, 1000 limit on local files...etc)
JAF DataSource - Not a Store API; just models a file with a MIME type
JSR170/283 - JCR; has versioning and container semantics.
Jakarta Slide ContentStore - Defunct; has versioning semantics
Jargon API for SRB and iRODS - Models files and directories, with metadata
JSPWiki FileSystemProvider - Fedora Repository Impl available; API has versioning semantics.
MediaWiki FileStore - PHP source
MogileFS - Obtaining
NFS - Directory semantics.
OKI Repository OSID - Has container semantics (Assets may contain other assets)
Plone Large File Management - Several alternatives to ZODB. Appear to be explicitly configured as part of a "content type". For example, see FileSystemStorage
Sakai Content Hosting Service - Collection semantics. JCR impl
Skwish - Transactions and reads/writes over a collection of "segments" on a filesystem. See Motivation
SMB/CIFS - Directory semantics.
Storage Resource Manager - Used by LHC; does not provide direct data access, only references.