...
| Excerpt |
|---|
Large files can be uploaded via the REST API, or projected into the repository using filesystem federation. Transfer times for uploading to the repository via the REST API are about the same as copying using NFS, and moderately faster than using SCP. Uploading via the REST API to a federated filesystem is significantly slower and requires a large temp directory capacity. |
Java command-line options and System properties can be used
-Xmx2048m– maximum memory Java can use-Dfcrepo.home=/path/to/data– set the directory for permanent data-Djava.io.tmpdir=/path/to/tmpdir– set the directory for temp files. Data uploaded via the REST API is written to a temp file in this directory before being moved to the federated filesystem, so this directory should have enough free space for the largest files you will upload. (
if running under tomcat, tomcat replaces your java.io.tmpdir with the value of CATALINA_TMPDIR, so you should set that environment variable instead)
Ingesting Large Files via the REST API
Based on the tests below, we believe arbitrarily-large files can be ingested and downloaded via the REST API (tested up to 1TB). The only apparent limitations are disk space available to store the files, and a sufficiently large Java heap size.
...
.
- Platform: lib-devsandbox1.ucsd.edu (all data on NAS to handle large files)
- Repository Profile: Minimal
- Workflow Profile: Upload/Download Roundtrip
| File Size | Upload | Download |
|---|---|---|
| 256GB | 3h51m34s (18.87MB/sec) | 43m09s (101.25MB/sec) |
| 512GB | 7h49m43s (18.60MB/sec) | 1h29m15s (97.90MB/sec) |
| 1TB | 15h41m21s (18.57MB/sec) | 3h21m44s (86.63MB/sec) |
Serving Large Files via Filesystem Federation
Based on the tests below, we believe arbitrarily-large files can be projected into the repository via filesystem federation and downloaded via the REST API (tested up to 1TB). The only apparent limitations are disk space available to store the files, and a sufficiently large Java heap size.
- Platform: lib-devsandbox1.ucsd.edu (all data on NAS to handle large files)
- Repository Profile: Minimal, with filesystem federation:
...
| / |
...
| File Size | Download |
|---|---|
| 256 GB | 1h09m26s (62.92MB/sec) |
| 512 GB | 2h00m15s (72.67MB/sec) |
| 1 TB | 3h57m25s (73.61MB/sec) |
...
Direct Comparison of Different Transfer Methods
...