How to handle large files

Large Files on a Single Node Fedora 4 Installation

Because of https://issues.jboss.org/browse/MODE-2103 for large file ingests only the single-file configuration can be used.

Setting the Java Property fcrepo.modeshape.configuration to classpath:/config/single-file/repository.json and allowing the heap to grow up to 2gb is required.

Example

Running Fedora 4 for large file ingests in Tomcat7 

Large File Upload/Download Roundtrip via REST API Tests

• Platform: lib-devsandbox1.ucsd.edu (all data on NAS to handle large files)
• Repository Profile: Minimal
• Workflow Profile: Upload/Download Roundtrip

• Conclusion: Arbitrarily-large files can be ingested via the REST API.  The only apparent limitations are disk space available to store the file, and sufficiently large Java heap size (tested with -Xmx2048m).

Large File Upload/Download Roundtrip Tests

• Platform: Linux 3.12.1-1-ARCH #1 SMP PREEMPT x86_64 GNU/Linux 16GB RAM
• Repository Profile: Single-File
• Workflow Profile: Upload/Download Roundtrip

Federated Content Large File Download Roundtrip Tests

• Platform: Linux 3.12.1-1-ARCH #1 SMP PREEMPT x86_64 GNU/Linux  16GB RAM

• Repository Profile: Single-File with an additional external Resource:

  "externalSources" : {
"home-directory" : {
    "classname" : "org.modeshape.connector.filesystem.FileSystemConnector",
    "directoryPath" : "/tmp/projection",
    "projections" : [ "default:/projection => /" ],
    "readOnly" : true,
    "addMimeTypeMixin" : true
    }
}
  
  

• Platform: lib-devsandbox1.ucsd.edu (all data on NAS to handle large files)
• Repository Profile: Minimal, with filesystem federation:

"externalSources" : {
    "filesystem" : {
        "classname" : "org.modeshape.connector.filesystem.FileSystemConnect
        "directoryPath" : "/mnt/isilon/fedora-dev/federated",
        "projections" : [ "default:/projection => /" ],
        "readOnly" : true,
        "addMimeTypeMixin" : true,
        "contentBasedSha1" : "false"
    }
}

Related articles