About This Page
The purpose of this page is to inform the design and development of Akubra by providing an analysis of existing Blob Storage APIs and implementations.
Here we define Pluggable Blob Storage API as a programming interface with the following characteristics:
A Pluggable Blob Storage API MUST:
- Be primarily concerned with providing direct read/write access to named files.
- Be designed with back-end pluggability in mind.
A Pluggable Blob Storage API MAY:
- Provide read/write access to metadata about stored files.
- Provide a way to list or query stored files.
- Provide transactional capabilities.
A Pluggable Blob Storage API MUST NOT:
- Be concerned with container (directory/folder/collection) semantics.
- Be concerned with versioning semantics (e.g. revision history).
APIs and Implementations
API |
Language |
Target Application |
Known Implementations |
|---|---|---|---|
Java |
General-Purpose |
AKU-FS: FSBlobStore |
|
Java |
Alfresco CMS |
ALF-FS: FileContentStore |
|
Java |
DSpace 2 (Unreleased) |
BIT-FED: FedoraBitStore |
|
Python |
Any Using Django |
||
Java |
Daisy CMS |
||
COM |
MS SQL Server |
||
Java |
Apache JackRabbit |
||
Java |
Apache JackRabbit |
||
Java |
Fedora Repository (2.x to 3.1) |
LLS-FS: GenericFileSystem |
|
Java |
Topaz OTM |
||
Java / C |
General-Purpose |
Capability Matrix
API/Impl |
ID Type |
ID Gen |
Write? |
Rewrite? |
Read-Only MD |
Writable MD |
Listing Options |
Transactions |
|---|---|---|---|---|---|---|---|---|
AKU-API |
URI |
Internal or External |
Yes |
Yes |
Size |
? |
All |
Optional, JTA |
AKU-FS |
file:/// URI |
Internal |
Yes |
Yes |
Size |
None |
All |
No |
API/Impl |
ID Type |
ID Gen |
Write? |
Rewrite? |
Read-Only MD |
Writable MD |
Listing Options |
Transactions |
ALF-API |
URL |
Internal or External |
Yes |
Yes |
Size |
None |
All |
Optional, JTA |
ALF-FS |
store:// URL |
Internal or External |
Yes |
Yes |
Size |
None |
All |
Optional, JTA |
ALF-HTT |
http:// URL |
N/A |
No |
No |
Size |
None |
None |
Optional, JTA |
ALF-REP |
URL |
Internal or External |
Yes |
Yes |
Size |
None |
All |
Optional, JTA |
ALF-ROU |
URL |
Internal or External |
Yes |
Yes |
Size |
None |
All |
Optional, JTA |
API/Impl |
ID Type |
ID Gen |
Write? |
Rewrite? |
Read-Only MD |
Writable MD |
Listing Options |
Transactions |
BIT-API |
Unicode String |
Internal |
Yes |
Yes |
Any (Map) |
None |
None |
No |
BIT-FED |
(NCNameChar)* |
Internal |
Yes |
No |
Size |
None |
None |
No |
BIT-FS |
([DEV:0-9]){38,40} |
Internal |
Yes |
Yes |
Size |
None |
None |
No |
BIT-HON |
([DEV:0-9]){38,40} |
Internal |
Yes |
Yes |
? |
None |
None |
No |
BIT-S3 |
([DEV:0-9]){38,40} |
Internal |
Yes |
Yes |
Size |
None |
None |
No |
BIT-SRB |
([DEV:0-9]){38,40} |
Internal |
Yes |
Yes |
Size |
None |
None |
No |
API/Impl |
ID Type |
ID Gen |
Write? |
Rewrite? |
Read-Only MD |
Writable MD |
Listing Options |
Transactions |
DJA-API |
Unicode String |
External |
Yes |
Yes |
Size |
None |
None |
No |
DJA-FS |
Unicode String |
External |
Yes |
Yes |
Size |
None |
None |
No |
DJA-S3 |
Unicode String |
External |
Yes |
Yes |
Size |
None |
None |
No |
API/Impl |
ID Type |
ID Gen |
Write? |
Rewrite? |
Read-Only MD |
Writable MD |
Listing Options |
Transactions |
DSY-API |
Unicode String |
Internal |
Yes |
No |
None |
None |
None |
No |
DSY-FS |
Unicode String |
Internal |
Yes |
No |
None |
None |
None |
No |
API/Impl |
ID Type |
ID Gen |
Write? |
Rewrite? |
Read-Only MD |
Writable MD |
Listing Options |
Transactions |
EBS-API |
byte[] |
Internal |
Yes |
No |
Size |
None |
None |
No |
EBS-FS |
GUID or String |
Internal |
Yes |
No |
Size |
None |
None |
No |
API/Impl |
ID Type |
ID Gen |
Write? |
Rewrite? |
Read-Only MD |
Writable MD |
Listing Options |
Transactions |
JKB-API |
Unicode String |
Internal |
Yes |
Yes |
None |
None |
None |
No |
JKB-DB |
propertyId + index |
Internal |
Yes |
Yes |
None |
None |
None |
No |
JKB-FS |
Filesystem Path |
Internal |
Yes |
Yes |
None |
None |
None |
No |
API/Impl |
ID Type |
ID Gen |
Write? |
Rewrite? |
Read-Only MD |
Writable MD |
Listing Options |
Transactions |
JKD-API |
Hex String |
Internal |
Yes |
No |
Size |
None |
All |
No |
JKD-DB |
SHA-1 Digest of Content |
Internal |
Yes |
No |
Size |
None |
All |
No |
JKD-FS |
SHA-1 Digest of Content |
Internal |
Yes |
No |
Size |
None |
All |
No |
API/Impl |
ID Type |
ID Gen |
Write? |
Rewrite? |
Read-Only MD |
Writable MD |
Listing Options |
Transactions |
LLS-API |
pid or pid + dsId |
External |
Yes |
Yes |
None |
None |
None |
No |
LLS-FS |
pid or pid + dsId |
External |
Yes |
Yes |
None |
None |
None |
No |
LLS-HON |
pid or pid + dsId |
External |
Yes |
Yes |
None |
None |
None |
No |
LLS-IRD |
pid or pid + dsId |
External |
Yes |
Yes |
None |
None |
None |
No |
LLS-S3 |
pid or pid + dsId |
External |
Yes |
Yes |
None |
None |
None |
No |
LLS-SRB1 |
pid or pid + dsId |
External |
Yes |
Yes |
None |
None |
None |
No |
LLS-SRB2 |
pid or pid + dsId |
External |
Yes |
Yes |
None |
None |
None |
No |
API/Impl |
ID Type |
ID Gen |
Write? |
Rewrite? |
Read-Only MD |
Writable MD |
Listing Options |
Transactions |
OTM-API |
Unicode String |
External |
Yes |
No |
None |
None |
None |
Optional, JTA |
OTM-FED |
Unicode String |
External |
Yes |
No |
None |
None |
None |
Best-effort, JTA |
OTM-FS |
Unicode String |
External |
Yes |
No |
None |
None |
None |
Yes, JTA |
API/Impl |
ID Type |
ID Gen |
Write? |
Rewrite? |
Read-Only MD |
Writable MD |
Listing Options |
Transactions |
XAM-API |
XUID |
Internal |
Yes |
No 1 |
Last-Modified |
Several Predefined |
All |
No |
XAM-CEN |
XUID |
Internal |
Yes |
No |
Last-Modified |
Several Predefined |
All |
No |
XAM-HON |
XUID |
Internal |
Yes |
No |
Last-Modified |
Several Predefined |
All |
No |
XAM-IAP |
XUID |
Internal |
Yes |
No |
Last-Modified |
Several Predefined |
All |
No |
1 XAM is designed with a fixed-content assumption - data cannot be modified without changing its XUID.
2 In XAM, all metadata fields are either binding or non-binding. Binding metadata cannot change while keeping the same XUID for an XSet, while non-binding metadata can.
Other APIs/Protocols/Impls of Interest
- Apache Sling - REST interface on JCR, need to look at this more...
- AtomPub - Collection semantics
- Azure Blob Storage API - Collection semantics, REST-based
- CMIS - Draft; has versioning and container semantics.
- DSpace 1 BitstreamStorageManager - Single impl, supports either local filesystem or SRB
- Eclipse - BlobStore- Single impl, maps UUIDs to files on disk
- Google Datastore API - Google's "cloud" underneath, REST-ish, container semantics, transactions!...but quite limited today (1MB limit on content, 1000 limit on local files...etc)
- JAF DataSource - Not a Store API; just models a file with a MIME type
- JSR170/283 - JCR; has versioning and container semantics.
- Jakarta Slide ContentStore - Defunct; has versioning semantics
- Jargon API for SRB and iRODS - Models files and directories, with metadata
- JSPWiki FileSystemProvider - Fedora Repository Impl available; API has versioning semantics.
- MediaWiki FileStore - PHP source
- MogileFS - Obtaining
- GlusterFS- a clustered file-system capable of scaling to several peta-bytes
- NFS - Directory semantics.
- OKI Repository OSID - Has container semantics (Assets may contain other assets)
- Plone Large File Management - Several alternatives to ZODB. Appear to be explicitly configured as part of a "content type". For example, see FileSystemStorage
- Sakai Content Hosting Service - Collection semantics. JCR impl
- Skwish - Transactions and reads/writes over a collection of "segments" on a filesystem. See Motivation
- SMB/CIFS - Directory semantics.
- Storage Resource Manager - Used by LHC; does not provide direct data access, only references.
