Page History
...
The Replication Task Suite is a DSpace Add-On which provides a set of curation system tasks to assist in performing replication (backup/restore/audit) of DSpace contents to other locations. The DSpace content is packaged in containers known as AIPs (OAIS speak: 'archival information packages'). By default, AIPs are generated in the default DSpace AIP Format (the same format used by the AIP Backup and Restore tool). If desired, there is an option to generate BagIt-based AIPs instead of using the default DSpace AIP format.
...
Info | ||
---|---|---|
| ||
Based on the version of DSpace you are running, here are the compatible latest releases of the Replication Task Suite:
|
...
Replication Task Suite Version | Supported DSpace Version(s) | Supported Java Version | Supported Interfaces | Notes |
---|---|---|---|---|
3.2 | DSpace version 3.x and , 4.x or 5.x | Java 7 or above | XMLUI and/or commandline | The 3.2 stable version of the Replication Task Suite is nearly identical to the 1.x stable version. It just includes minor bug fixes to ensure the Replication Task Suite is compatible with the new newer DSpace 3.x / 4.x API.APIs. |
1.3 | DSpace version 1.8.x | Java 6 or above | XMLUI and/or commandline | Highly recommended to use either DSpace 1.8.1 or above. DSpace 1.8.0 has a known bug where running a Replication Task will always return a NullPointerException - see DS-1077 |
Installation instructions for each version are included below:
...
As the Replication Suite is just a suite of Curation System tasks, it may be called (like any Curation Tasks) from the following locations:
...
For more information see the Curation System details on Task Invocation.
Installation on DSpace 3.x, 4.x or
...
5.x
- In your DSpace Source directory (
[dspace-src]
), you will need to modify the following POM file:[dspace-src]/dspace/modules/additions/pom.xml
(This POM will ensure that the "dspace-replicate" dependency is made available to commandline and ALL DSpace interfaces)
For this pom.xml file, add the following
<dependency>
section at the end of the existing<dependencies>
section (just before the closing</dependencies>
tag).Code Block <dependencies> ... <!-- Adding this dependency will install the Replication Task Suite Addon --> <dependency> <groupId>org.dspace</groupId> <artifactId>dspace-replicate</artifactId> <version>3.2</version> </dependency> </dependencies>
Once you've finished modifying both pom.xml files, rebuild DSpace by running the following from your
[dspace-src]/dspace/
folder:Code Block mvn clean package
- Follow the instructions in the Configuration section below in order to enable & configure the Replication Task Suite Add-On.
- You may wish to ensure these configurations exist in your
[dspace-src]/dspace/config/
directory. That way they will be auto-installed/copied whenever you run "ant update" (see next step).
- You may wish to ensure these configurations exist in your
You will need to update your existing DSpace 3.x installation, by running the following from your
[dspace-src]/dspace/target/dspace-[version]-build/
directoryCode Block ant update
Note Alternatively, if you don't want to do a full DSpace update, you can just update your existing binaries & webapps by running the following two commands:
ant update_code
(Updates the existing[dspace]/lib/
directory)ant update_webapps
(Updates the existing[dspace]/webapp/
directory)
...
...
- DSpace AIP Format (METS-based) (default) - This is the same AIP format utilized by the DSpace AIP Backup and Restore feature, so it is 100% compatible with that DSpace feature. In fact when using this format, the Replication Task Suite just "wraps" calls to the AIP Backup and Restore feature itself.
- BagIt AIP Format (beta) - This is a new AIP format provided by the Replication Task Suite. It generates AIPs in the BagIt File Packaging Format. Institutions which already are familiar with BagIt or use it elsewhere may find this format preferable. (Please note that this AIP format does not yet support all DSpace objects. See the below table for more information.)
...
| DSpace AIP Format (METS-based AIPs) | BagIt AIP Format |
Supported Backup/Restore Types |
|
|
---|---|---|
Can Backup & Restore all DSpace Content easily | Yes | Yes |
Can Backup & Restore a Single Community/Collection/Item easily | Yes | Yes |
Backups can be used to move one or more Community/Collection/Items to another DSpace system easily. | Yes (Using the Replication Task Suite or using the command line AIP Backup and Restore tools) | Yes (though the Replication Task Suite add-on must be installed on both systems) |
Can Backup & Restore Item Versions (added in DSpace 3.x) | No (Item Versioning not yet compatible with AIP format. Only the most recent version of an Item is described in the AIP.) | No (Item Versioning not yet compatible with AIP format. Only the most recent version of an Item is described in the AIP.) |
Supported DSpace Object Types |
|
|
Supports backup/restore of all Communities/Collections/Items (including metadata, files, logos, etc.) | Yes | Yes |
Supports backup/restore of all People/Groups/Permissions | Yes | No (Not yet supported) |
Supports backup/restore of all Collection-specific Item Templates | Yes | No (Not yet supported) |
Supports backup/restore of all Collection Harvesting settings (only for Collections which pull in all Items via OAI-PMH or OAI-ORE) | No (The harvest settings are not preserved, but previously harvested items are preserved in their own AIPs) | No (The harvest settings are not preserved, but previously harvested items are preserved in their own AIPs) |
Supports backup/restore of all Withdrawn (but not deleted) Items | Yes | Yes |
Supports backup/restore of Item Mappings between Collections | Yes | Yes |
Supports backup/restore of all in-process, uncompleted Submissions (or those currently in an approval workflow) | No (AIPs are only generated for objects which are completed and considered "in archive") | No (AIPs are only generated for objects which are completed and considered "in archive") |
Supports backup/restore of Items using custom Metadata Schemas & Fields | Yes | Yes |
Supports backup/restore of all local DSpace Configurations and Customizations | No (You are expected to backup your DSpace configurations and customizations separately. AIPs only backup content held within DSpace.) | No (You are expected to backup your DSpace configurations and customizations separately. AIPs only backup content held within DSpace.) |
...
This section goes through the steps of configuring the Replication Suite to use the default DSpace AIP format, which utilizes METS packaging. This is the default & recommended setting.
- General Curation Configuration: First, in your
[dspace]/config/modules/curate.cfg
you will want to enable & configure the METS-based replication tasks. (NOTE: there is a samplecurate.cfg
file provided in https://github.com/DSpace/dspace-replicate/tree/master/config/modules which is pre-configured to use METS-based AIPs).Enable the Replication Tasks: In the list of "Task Class implementations" (
plugin.named.org.dspace.curate.CurationTask
), add the following.
REMEMBER to add a comma and backslash (", \") after each line (except the final line).Code Block plugin.named.org.dspace.curate.CurationTask = \ ... (YOUR EXISTING TASKS) ... , \ org.dspace.ctask.replicate.EstimateAIPSize = estaipsize, \ org.dspace.ctask.replicate.ReadOdometer = readodometer, \ org.dspace.ctask.replicate.TransmitAIP = transmitaip, \ org.dspace.ctask.replicate.TransmitSingleAIP = transmitsingleaip, \ org.dspace.ctask.replicate.VerifyAIP = verifyaip, \ org.dspace.ctask.replicate.FetchAIP = fetchaip, \ org.dspace.ctask.replicate.CompareWithAIP = auditaip, \ org.dspace.ctask.replicate.RemoveAIP = removeaip, \ org.dspace.ctask.replicate.METSRestoreFromAIP = restorefromaip, \ org.dspace.ctask.replicate.METSRestoreFromAIP = replacewithaip, \ org.dspace.ctask.replicate.METSRestoreFromAIP = restorekeepexisting, \ org.dspace.ctask.replicate.METSRestoreFromAIP = restoresinglefromaip, \ org.dspace.ctask.replicate.METSRestoreFromAIP = replacesinglewithaip
Give Each Task a Human-Friendly Task Name: Under the
ui.tasknames
setting, give each of the above Tasks a human-friendy name. Here are some recommended values, but you are welcome to tweak them.
REMEMBER to add a comma and backslash (", \") after each line (except the final line).Code Block ui.tasknames = \ ... (YOUR EXISTING TASK NAMES) ... , \ estaipsize = Estimate Storage Space for AIP(s), \ readodometer = Read Odometer, \ transmitaip = Transmit AIP(s) to Storage, \ verifyaip = Verify AIP(s) exist in Storage, \ fetchaip = Fetch AIP(s) from Storage, \ auditaip = Audit against AIP(s), \ removeaip = Remove AIP(s) from Storage, \ restorefromaip = Restore Missing Object(s) from AIP(s), \ replacewithaip = Replace Existing Object(s) with AIP(s), \ restorekeepexisting = Restore Missing Object(s) but Keep Existing Objects,\ restoresinglefromaip = Restore Single Object from AIP, \ replacesinglewithaip = Replace Single Object with AIP
Optionally Create a Task Group: Finally, if you'd like to create a Task Group for these tasks, you can create a group named "replicate" and add them all to it. The below is just an example for how you may wish to set the
ui.taskgroups
andui.taskgroup.*
settings. It creates two Task Groups: (1) a "General Purpose Tasks" group for a few default DSpace Curation Tasks, and (2) a "Replication Suite Tasks" group for all these new Replication tasks.Code Block # Tasks may be organized into named groups which display together in UI drop-downs ui.taskgroups = \ general = General Purpose Tasks, \ replicate = Replication Suite Tasks # Group membership is defined using comma-separated lists of task names, one property per group ui.taskgroup.general = profileformats, requiredmetadata, checklinks ui.taskgroup.replicate = estaipsize, readodometer, transmitaip, verifyaip, fetchaip, auditaip, removeaip, restorefromaip, replacewithaip, restorekeepexisting, restoresinglefromaip, replacesinglewithaip
Replication Suite Configuration: Next, in your
[dspace]/config/modules/replicate.cfg
you will want to ensure it is setup to properly use METS-based AIPs. Under the "AIP Packaging Settings" you'll want the following settings enabled:Code Block # Package type. Permitted values: 'mets', 'bagit' # mets = Generate default DSpace AIPs as described in: https://wiki.duraspace.org/display/DSDOC18/AIP+Backup+and+Restore # bagit = Generate AIPs based on the BagIt packaging format: https://wiki.ucop.edu/display/Curation/BagIt packer.pkgtype = mets # Format of package compression. Permitted values: 'zip' or 'tgz' # for 'mets' packages, only 'zip' is supported packer.archfmt = zip # Whether or not the name packages with a DSpace type prefix. # When 'true', package files are named [type]@[handle].[format] (e.g. ITEM@123456789-1.zip) # When 'false', package files are named [handle].[format] (e.g. 123456789-1.zip) # Defaults to 'true'. For 'mets' packages, this must be 'true'. packer.typeprefix = true
- Optionally tweak the AIP Restore/Replace settings: Optionally, you can decide to tweak the way AIPs are restored or replaced (using AIP Backup and Restore options). These settings normally should not need to be tweaked, but are available in the
[dspace]/config/modules/replicate-mets.cfg
configuration file. See that configuration file for more details.
...
Once you've setup your Consumer & restarted DSpace, you'll start to see a (plain text file) queue of tasks (in the location specified by the consumer.queue
setting in "replicate.cfg") that need to be performed in order to synchronize your AIP backup with what is in your DSpace instance. This replication queue is just a normal DSpace Curation System queue, and it can be processed via command line or a cron job (recommended).
...
In DSpace, by default, duplicate tasks in a Curation System queue will each be processed individually. So, that means if an Item is updated 10 times, it will appear in the queue 10 times, and its AIP will be (re-)generated and (re-)transmitted to storage 10 times when that queue is processed. (DuraCloud Note: Some storage platforms, e.g. DuraCloud, provide a way to determine whether a newly generated AIP actually differs from the one in replica storage. So, in the case of DuraCloud storage, the AIP will be re-generated 10 times, but it will only be transmitted to DuraCloud ONCE. The other 9 times, the DuraCloud storage plugin will determine that the checksum of the new AIP is identical to the one in DuraCloud and skip the transmission step. See How DuraCloud storage works section above for more info.)
...
This section goes through the steps of configuring the usage of Checkm manifest tasks. These tasks provide a capability to store DSpace content checksums external from DSpace in the Checkm Manifest format. Some institutions may find this to be a useful replacement for the default DSpace Checksum Checker/Validator, which only stores/validates checksums internal to the DSpace system.
...
The default values will create a METS-based AIP in the default DSpace AIP Format, compressed into a 'zip' archive. The other alternative supported by the replication task suite is Library of Congress 'Bagit' packaging, which may compressed either into a 'zip' file or a 'tgz' ('gzipped tar'), a compression standard more common in Unix systems.
...
- Restore Single Object from AIP (
restoresinglefromaip
)- This task acts the same as the default "restorefromaip" task, but it does NOT restore any child objects. So, if it is run on a collection, just the collection itself will be restored (items in that collection will not be restored).
- Restore Missing Object(s) but Keep Existing Objects (
restorekeepexisting
)- This task acts similar to the default "restorefromaip" task, but it attempts to skip over any objects which already exist in the repository. In other words, an error is not thrown if an object already exists – rather that entire object (and all its child objects) are skipped over during processing and left unchanged. This mode is identical to the "Keep Existing" mode of the DSpace AIP Backup and Restore tool.
Replacing Object(s)
Replication Tasks Used: | Replace Existing Object(s) with AIP(s) | Task ID: |
---|---|---|
Replace Single Object with AIP (*METS-AIP Only) | Task ID: |
...