...
Note: To enable fast access to large files, it is necessary to set "contentBasedSha1" : "false". Otherwise the repository will run a SHA1 on the content for identification that could take on the order or hours when reaching into the range of > 50Gb. For more on this benchmarking see: Design - LargeFiles.
- Platform: lib-devsandbox1.ucsd.edu (all data on NAS to handle large files)
- Repository Profile: Minimal
- Workflow Profile: Upload/Download Roundtrip
| File Size | Upload | Download |
|---|---|---|
| 256GB | 3h51m34s (18.87MB/sec) | 43m09s (101.25MB/sec) |
| 512GB | 7h49m43s (18.60MB/sec) | 1h29m15s (97.90MB/sec) |
| 1TB | 15h41m21s (18.57MB/sec) | 3h21m44s (86.63MB/sec) |
...
Based on the tests below, we believe arbitrarily-large files can be projected into the repository via filesystem federation and downloaded via the REST API (tested up to 1TB). The only apparent limitations are disk space available to store the files, and a sufficiently large Java heap size.
- Platform: lib-devsandbox1.ucsd.edu (all data on NAS to handle large files)
- Repository Profile: Minimal, with filesystem federation:
| Code Block |
|---|
"externalSources" : {
"filesystem" : {
"classname" : "org.modeshapefcrepo.connector.filesystemfile.FileSystemConnectorFedoraFileSystemConnector",
"directoryPath" : "/mnt/isilon/fedora-dev/federated",
"projections" : [ "default:/projection => /" ],
"readOnlyreadonly" : true,
"addMimeTypeMixin" : true,
"contentBasedSha1" : "false"
}
} |
...
Based on the tests below, we believe arbitrarily-large files can be uploaded and downloaded via the REST API, using either repository storage or a federated filesystem (tested up to 1TB). The only apparent limitations are disk space available to store the files, temp directory capacity, and a sufficiently large Java heap size.
- Platform: lib-devsandbox1.ucsd.edu (all data on NAS to handle large files)
- Repository Profile: Federation
- Workflow Profile: Repository/Federation/NFS/SCP Comparison
Comparison of Upload and Download Times for Different Transfer Methods
...