...
- Platform: Linux 3.12.1-1-ARCH #1 SMP PREEMPT x86_64 GNU/Linux 16GB RAM
- Repository Profile: Single-File
- Workflow Profile: Upload/Download Roundtrip
| File Size | Upload | Download |
|---|---|---|
| 256GB | 15,488,156ms (16.9MB/sec) | 3,306,756ms (79.3MB/sec) |
REST API Upload/Download Roundtrip
- Platform: lib-devsandbox1.ucsd.edu (all data on NAS to handle large files)
- Repository Profile: Minimal
- Workflow Profile: Upload/Download Roundtrip
| File Size | Upload | Download |
|---|---|---|
| 256GB | 15,488,156ms (16.9MB/sec) | 3,306,756ms (79.3MB/sec) |
| 512GB | 31,262,610ms (16.77MB/sec) | 5,386,542ms (97.33MB/sec) |
| 1TB | 59,631,142ms (17.58MB/sec) | 15,120,135ms (69.35MB/sec) |
...
Filesystem Federation Download Tests
- Platform: lib-devsandbox1.ucsd.edu (all data on NAS to handle large files)
- Repository Profile: Minimal, with filesystem federation:
...
Based on the tests below, we believe arbitrarily-large files can be uploaded and downloaded via the REST API, using either repository storage or a federated filesystem (tested up to 1TB). The only apparent limitations are disk space available to store the files, temp directory capacity, and a sufficiently large Java heap size.
- Platform: lib-devsandbox1.ucsd.edu (all data on NAS to handle large files)
- Repository Profile: Federation
- Workflow Profile: Repository/Federation/NFS/SCP Comparison
Comparison of Upload and Download Times for Different Transfer Methods
...
| Source | Destination | File Size | Copy Time |
|---|---|---|---|
| Repository storage | Federated filesystem | 1TB | 402 min (153 GB/sec) |
| Federated filesystem | Repository storage | 1TB | 345 min (178 GB/sec) |
Range Retrieval
Retrieving a byte range is supported and has been tested with 1TB files for both repository storage and federated filesystem. There is an integration test in the standard test suite for verifying that range retrieval works. By default, this test uses a small datastream size to avoid slowing down the test suite, but the size is configurable so it is easy for a developer to test files as large as local disk space allows.