Large File Ingest and Retrieval

Java command-line options and System properties can be used

• -Xmx2048m – maximum memory Java can use
• -Dfcrepo.home=/path/to/data – set the directory for permanent data
• -Djava.io.tmpdir=/path/to/tmpdir – set the directory for temp files
  
  

Ingesting Large Files via the REST API

Based on the tests below, we believe arbitrarily-large files can be ingested and downloaded via the REST API (tested up to 1TB).  The only apparent limitations are disk space available to store the files, and a sufficiently large Java heap size.

Note: To enable fast access to large files, it is necessary to set "contentBasedSha1" : "false".  Otherwise the repository will run a SHA1 on the content for identification that could take on the order or hours when reaching into the range of > 50Gb.  For  more on this benchmarking see: https://wiki.duraspace.org/display/FF/Design+-+Large+Files.

REST API Upload/Download Roundtrip

• Platform: Linux 3.12.1-1-ARCH #1 SMP PREEMPT x86_64 GNU/Linux 16GB RAM
• Repository Profile: Single-File
• Workflow Profile: Upload/Download Roundtrip

REST API Upload/Download Roundtrip

• Platform: lib-devsandbox1.ucsd.edu (all data on NAS to handle large files)
• Repository Profile: Minimal
• Workflow Profile: Upload/Download Roundtrip

Serving Large Files via Filesystem Federation

Based on the tests below, we believe arbitrarily-large files can be projected into the repository via filesystem federation and downloaded via the REST API (tested up to 1TB).  The only apparent limitations are disk space available to store the files, and a sufficiently large Java heap size.

Filesystem Federation Download Tests

• Platform: Linux 3.12.1-1-ARCH #1 SMP PREEMPT x86_64 GNU/Linux  16GB RAM

• Repository Profile: Single-File with an additional external Resource:

  "externalSources" : {
"home-directory" : {
    "classname" : "org.modeshape.connector.filesystem.FileSystemConnector",
    "directoryPath" : "/tmp/projection",
    "projections" : [ "default:/projection => /" ],
    "readOnly" : true,
    "addMimeTypeMixin" : true
    }
}
  
  

Filesystem Federation Download Tests

• Platform: lib-devsandbox1.ucsd.edu (all data on NAS to handle large files)
• Repository Profile: Minimal, with filesystem federation:

"externalSources" : {
    "filesystem" : {
        "classname" : "org.modeshape.connector.filesystem.FileSystemConnect
        "directoryPath" : "/mnt/isilon/fedora-dev/federated",
        "projections" : [ "default:/projection => /" ],
        "readOnly" : true,
        "addMimeTypeMixin" : true,
        "contentBasedSha1" : "false"
    }
}

Related articles