Page History
| Excerpt |
|---|
The University of Michigan and @mire are initiating work to create a more robust and configurable embargo feature than the one that currently exists in DSpace. We would like to invite others to tweak the feature set if there's additional features of interest. |
Business Requirements
Without going into detail about the technical implementation, the basic feature set we are looking at, which will build on the current implementation includes the ability to provide
- Full embargo: no access to either the metadata or bitstream of an item until embargo period ends. Items Even if a match existsItems do not appear in: search or results, browse results, even if a match exists; though , RSS feeds, OAI-PMH feeds, etc. Though the metadata and bitstreams are stored, it's as if the item and its bitstreams had never been deposited, or indexed.
- Partial embargo: metadata visible, some/all bitstreams inaccessible. In this case, items appear in browse or search indexes so searchers and browsers know an item and its bitstreams exist. However, they can't download/view bitstreams. (This is the behavior of the current implementation.)
- Group access to embargoed items: embargoed items (full or partial) are viewable and all contents are completely accessible to members of select groups. This will address needs often expressed by archivists, who need a dark archive of materials in DSpace
...
To summarise, in the LCoNZ model, items can either be open or restricted. Restrictions can be partial or complete; they can also be permanent ("until further notice") or temporary (with a fixed expiry date). Temporary restrictions still require manual action to be lifted. Any of the permanent item states can apply after an embargo is lifted, but not all combinations of partial and temporary restrictions are used in practice.
Technical Requirements
Associated JIRA Tasks
| Info |
|---|
LCoNZ adds "Email based Embargo Termination Notification" and "Require Manual Embargo Release" business requirements to those already defined in general requirements. Another LCoNZ addition is the distinction of permanent state vs temporary restrictions (where the expiry of a temporary restriction changes the item permissions to the underlying permanent state). |
Technical Requirements
Associated JIRA Tasks
DS-908 Embargo Overhaul: Utilize ResourcePolicy Start and DS-908 Embargo Overhaul: Utilize ResourcePolicy Start and Stop datestamps for enforcing embargo in DSpace
...
Restricting Discovery of Embargoed Items in Search and Browse
Tapir/IDEALS Solution:
Item will Item will need to be restricted from search and browse, the Tapir solution for restricting Item from viewing in Search and Browse is to set Resource Policies inArchive=false on the Item that limit access to it, during the embargo period, an an and add it to a "restricted" table for access in the "Embargoed Item" Administrators View. An "EmbargoLifter" command is responsible for removing these embargo settings fromt he from the Item to lift the embargo
Restricting viewing of Embargoed Items
Resource Policies are currently set into place on DSpace Bitstreams during the desired period defined in the (usually dc.date.embargountil
Problem: Lifting the embargo removes any record of the original embargo being in effect/
Solution: Use of the provided "Start" and "End dates on the Resource Policies would allow the creation of an Embargo Period that could be preserved with the Item and would not need physical alteration to release the Item from Embargo.
DSpace Items Private
The AuthorizationManager already supports the enforcement of timeframes in ResourcePolicies. I would like to propose that we expand ResourcePolicy in the following manner:
Migration from existing Embargo solutions
Metadata Concerning the setting of an embargo term and lift date may still be recoreded in the metadata. ALterations may include:
- Not relying on date described in metadata to manage access rights
- Exposure of the embargo details for any item in OAI/METS, XMLUI/METS or JSPUI Item views by evaluation of the ResourcePolices that are in force.
Related Projects and Solutions
IDEALS Embargo Functionality
The current //DSpace 1.5.2 //version has been customized with an embargo feature that allows three different group restrictions during the submission process:
# Limit access to a group:
this sends an email to an administrator, so the administrator can set this up manually
# University Only access:
either IP based or login
# Complete embargo:
the item is fully withdrawn until the embargo is released.
The embargo lift date is currently not defined as a date, but stored as a number of months. It is stored as a date in the database after selecting the amount of months. The submitter also needs to supply a reason for the embargo. In the item metadata it should be exposed that the item is under embargo, and also mention the embargo lift date.
There is also a custom mechanism for batch uploads, and a script for mapping the correct embargo settings on these uploaded items, and the import mapfile. Embargo will also effect OAI-PMH exposure of content through withdrawn/archived item state. Also the submitter should be able to access the item. The client requests these functionalities to be retained.
At the moment, only entire items can be embargoed, but the client is optionally interested in being able to set an embargo on the bitstream level as well. Optionally the client is requesting an implementation to support for current user to contact submitter for "request to access" the embargoed content.
Description
The current implementation of the embargo functionality can be migrated to DSpace 1.6.2, but requires some optimizations to limit changes to the DSpace core classes.
@mire recommends the client to invest in re-working the embargo implementation to decrease any further upgrade issues to future versions of DSpace, which can be achieved using two different approaches. The approach @mire advises, is detailed here, and the second approach is described in Alternative 1.
The advised solution does not use the embargo framework because there are limitations to the extensibility of this framework. We will create a custom solution which limits changes to existing DSpace core classes as much as possible, and maintains the same storage solution for the embargo settings from the current repository. Both the interface during the submission and the edit interface (for admin & submitter) for archived items will be ported.
Custom Embargo Per Bitstream
Embargo per bitstream instead of per item. This solution will be an extension to the solution (and alternative) mentioned above. The main difference is that all embargo functionality will need to be entered and stored per uploaded bitstream (file). Both the interface during the submission and the edit interface for archived items will be adjusted to ensure the settings can be configured for each bitstream individually.
Embargo per Item - Technical description
The current solution provides a form in which the submitter can indicate if the item is private or pubic. If private is chosen at least one of the following private options has to be selected:
- Visible to University of Illinois users ONLY
- Visible to a Smaller Group of users ONLY
- Make publicly visible after a period of time (field to indicate number of months)
In case the user selects one of the first two options ("//Visible to University of Illinois users ONLY//" or "//Visible to a Smaller Group of users ONLY//") the following operations are performed:
- a record is added into *restrict_item* table
- a metadata field is added dc.provenance.description (Item marked as restricted to the 'UIUC Users.....)
The Item results to be:
- SEARCHABLE
- WITH FILES RESTRICTED
In case the user select only the third option ("//Make publicly visible after a period of time//") the following operations are performed:
- a record is added into *restrict_item* table
- a record is added into *bi_embargoed* table
- a metadata field is added dc.provenance.description (Item marked as restricted to the 'UIUC Users.....)
The Item results to be:
- NOT SEARCHABLE
Summarizing the previous description, we have:
...
Option
...
=record in //restrict_item//
...
=record in //bi_embargoed//
...
=Searchable?
...
= Bitstream(s) Visible?
...
Visible to University of Illinois users ONLY
...
Yes
...
No
...
Yes
...
No
General Solution:
A possibility in the general approach will be not to add the item to a restricted table,, but continue to encode the embargo period and conditions in the ResourcePolicies and create a view over the Items based on this criteria (ideally by using Discovery to indicate that either the Item or one of its Bitstreams is embargoed)
Restricting viewing of Embargoed Items
Resource Policies are currently set into place on DSpace Bitstreams during the desired period defined in the (usually dc.date.embargountil
Problem: Lifting the embargo removes any record of the original embargo being in effect/
Solution: Use of the provided "Start" and "End dates on the Resource Policies would allow the creation of an Embargo Period that could be preserved with the Item and would not need physical alteration to release the Item from Embargo.
DSpace Items Private
The AuthorizationManager already supports the enforcement of timeframes in ResourcePolicies. I would like to propose that we expand ResourcePolicy in the following manner:
Encoding of Embargo Rights in AIP
It should be possible to export and restore Embargo Rights on Bitstreams and Items based on PREMIS RIghts sections in the AIP
Migration from existing Embargo solutions
Metadata Concerning the setting of an embargo term and lift date may still be recoreded in the metadata. ALterations may include:
- Not relying on date described in metadata to manage access rights
- Exposure of the embargo details for any item in OAI/METS, XMLUI/METS or JSPUI Item views by evaluation of the ResourcePolices that are in force.
Related Projects and Solutions
IDEALS Embargo Functionality
The current //DSpace 1.5.2 //version has been customized with an embargo feature that allows three different group restrictions during the submission process:
# Limit access to a group:
this sends an email to an administrator, so the administrator can set this up manually
# University Only access:
either IP based or login
# Complete embargo:
the item is fully withdrawn until the embargo is released.
The embargo lift date is currently not defined as a date, but stored as a number of months. It is stored as a date in the database after selecting the amount of months. The submitter also needs to supply a reason for the embargo. In the item metadata it should be exposed that the item is under embargo, and also mention the embargo lift date.
There is also a custom mechanism for batch uploads, and a script for mapping the correct embargo settings on these uploaded items, and the import mapfile. Embargo will also effect OAI-PMH exposure of content through withdrawn/archived item state. Also the submitter should be able to access the item. The client requests these functionalities to be retained.
At the moment, only entire items can be embargoed, but the client is optionally interested in being able to set an embargo on the bitstream level as well. Optionally the client is requesting an implementation to support for current user to contact submitter for “request to access” the embargoed content.
Description
The current implementation of the embargo functionality can be migrated to DSpace 1.6.2, but requires some optimizations to limit changes to the DSpace core classes.
@mire recommends the client to invest in re-working the embargo implementation to decrease any further upgrade issues to future versions of DSpace, which can be achieved using two different approaches. The approach @mire advises, is detailed here, and the second approach is described in Alternative 1.
The advised solution does not use the embargo framework because there are limitations to the extensibility of this framework. We will create a custom solution which limits changes to existing DSpace core classes as much as possible, and maintains the same storage solution for the embargo settings from the current repository. Both the interface during the submission and the edit interface (for admin & submitter) for archived items will be ported.
Custom Embargo Per Bitstream
Embargo per bitstream instead of per item. This solution will be an extension to the solution (and alternative) mentioned above. The main difference is that all embargo functionality will need to be entered and stored per uploaded bitstream (file). Both the interface during the submission and the edit interface for archived items will be adjusted to ensure the settings can be configured for each bitstream individually.
Embargo per Item - Technical description
The current solution provides a form in which the submitter can indicate if the item is private or pubic. If private is chosen at least one of the following private options has to be selected:
- Visible to University of Illinois users ONLY
- Visible to a Smaller Group of users ONLY
...
Visible to a Smaller Group of users ONLY
...
Yes
...
No
...
Yes
...
- Make publicly visible after a period of time (field to indicate number of months)
...
Yes
...
Yes
...
No
...
-
In case the user selects one of the first two options ("//Visible to University of Illinois users ONLY//" or "//Visible to a Smaller Group of users ONLY//") the following operations are performed:
- a record is added into *restrict_item* table
- a metadata field is added dc.provenance.description (Item marked as restricted to the 'UIUC Users.....)
The Item results to be:
- SEARCHABLE
- WITH FILES RESTRICTED
In case the user select only the third option ("//Make publicly visible after a period of time//") the following operations are performed:
- a record is added into *restrict_item* table
- a record is added into *bi_embargoed* table
- a metadata field is added dc.provenance.description (Item marked as restricted to the 'UIUC Users.....)
The Item results to be:
- NOT SEARCHABLE
Summarizing the previous description, we have:
Option | =record in //restrict_item// | =record in //bi_embargoed// | =Searchable? | = Bitstream(s) Visible? |
Visible to University of Illinois users ONLY | Yes | No | Yes | No |
Visible to a Smaller Group of users ONLY | Yes | No | Yes | No |
Make publicly visible after a period of time (field to indicate number of months) | Yes | Yes | No | - |
The tables involved in the current solution have to following structure:
*RESTRICT_ITEM*
=Field | =Description |
id | unique identifier |
item_id | unique item identifier |
release_date | filled only if "//Make publicly visible after a period of time//" is chosen |
eperson_group_id | unique group identifier |
reason | reason why the access at the item is restricted |
*BI_EMBARGOED*
=Field | =Description |
id | unique table identifier |
item_id | unique item identifier |
sort_1 | item title |
sort_2 | item date issued |
sort_3 | item data available |
Embargo per Bitstream - Technical description
The new solution provides the users with two ways to define restricted access at bitstream level:
- Restrict Access
- Upload/Edit File
These functions can be activated in:
- UI Submission
- Edit Item
- Workflow
Restrict Access
This function is the same contemplated in the previous version, but the form has been modified.
At this level the user can define general access settings options. This means that every new bitstream uploaded by default will have the same access settings defined by "Restrict Access" function.
*Note:*
1. *If the user will change the general access settings during the workflow or after the item is installed the new settings will be applied to all the bitstream(s) attached to the item.*
1. *If the general settings are "//Private/Closed Access"// won't be possible define custom settings at bitstream level.*
Upload/Edit File
In the modules //UploadStep// and //EditItem// the user can define the custom access settings per bitstream, overriding the general defined previously.
Administrative Item Access
To override the general settings the user must to check "//Enable Custom Settings//".
Back-End
The settings are defined in the table //resourcepolicy, //here a summary about what happens when the settings are defined:
= SETTING | = ACTION in RESOURCEPOLICY |
PUBLIC | A record with DEFALT_READ Anonymous policy is added |
PRIVATE_NO_EMBARGO | A record with DEFALT_READ Group policy is added |
PRIVATE_EMBARGO | A record with DEFALT_READ Group is added |
Note: The class //InstallItem// has been overridden because when the Item came installed the policies of all the bitstreams came overridden.
*Comments:*
Current Workflow Help Documentation
https://services.ideals.illinois.edu/wiki/bin/view/IDEALS/IDEALSHelpPage#Submit_Choose_Access_and_Privacy
Feature supports placing item into withdrawn.
Embargo by Item only applies to Items that are Open Access, Visible to a smaller group or university of illimois users. This is because in those cases the item is in an archived state in the repository.
Item restrictions in the LCoNZ IRRs
See Business requirements for item restrictions in the LCoNZ repositories above. The description in this section is for DSpace 1.7.2 and XMLUI.
User interface description
The restriction type as well as the embargo type + expiry date can be set in the DSpace deposit form as well as the SWORD thesis deposit forms, with the exception of dark items. Items deposited into a certain community in one of the repositories (which happens via a third-party application) are automatically turned dark.
Repository staff can review and change an item's restriction status in the workflow or via "edit item" once the item is archived. The first screenshot shows a repository that allows both temporary and permanent restrictions; all temporary restrictions in this repository are full embargoes. The second screenshot shows a repository that allows only Open as the underlying item state, but both full and partial embargoes.
Temporary and permanent restrictions are currently configured on separate screens, though this is mainly for historic reasons. These screens could, and most likely should, be consolidated into one screen; however, it may be desirable to let certain repository staff groups change only temporary or only permanent restrictions.
The first screenshot below shows the embargo screen where files can be fully embargoed only. The second screenshot shows a repository in which full/partial/no embargo can apply. The third screenshot shows permanent restriction settings.
Repository staff have access to a list of all embargoed items via an entry in the "Administrative" menu in the sidebar.
Technical description
The LCoNZ IRR restriction model uses item metadata and resource policies. Separate metadata fields hold the permanent state ("access level"), the embargo type (full vs partial) and the embargo expiry date. Customisations to core DSpace classes ensure that the appropriate resource policies are set up for each item and bitstream whenever the permanent or temporary state changes. Likewise, customisations to core DSpace classes ensure that these resource policies are actually honoured, building on the dark item modifications described by the DSpace@Cambridge folks.
The tables involved in the current solution have to following structure:
*RESTRICT_ITEM*
=Field | =Description |
id | unique identifier |
item_id | unique item identifier |
release_date | filled only if "//Make publicly visible after a period of time//" is chosen |
eperson_group_id | unique group identifier |
reason | reason why the access at the item is restricted |
*BI_EMBARGOED*
=Field | =Description |
id | unique table identifier |
item_id | unique item identifier |
sort_1 | item title |
sort_2 | item date issued |
sort_3 | item data available |
Embargo per Bitstream - Technical description
The new solution provides the users with two ways to define restricted access at bitstream level:
- Restrict Access
- Upload/Edit File
These functions can be activated in:
- UI Submission
- Edit Item
- Workflow
Restrict Access
This function is the same contemplated in the previous version, but the form has been modified.
At this level the user can define general access settings options. This means that every new bitstream uploaded by default will have the same access settings defined by "Restrict Access" function.
*Note:*
1. *If the user will change the general access settings during the workflow or after the item is installed the new settings will be applied to all the bitstream(s) attached to the item.*
1. *If the general settings are "//Private/Closed Access"// won't be possible define custom settings at bitstream level.*
Upload/Edit File
In the modules //UploadStep// and //EditItem// the user can define the custom access settings per bitstream, overriding the general defined previously.
To override the general settings the user must to check "//Enable Custom Settings//".
Back-End
The settings are defined in the table //resourcepolicy, //here a summary about what happens when the settings are defined:
= SETTING | = ACTION in RESOURCEPOLICY |
PUBLIC | A record with DEFALT_READ Anonymous policy is added |
PRIVATE_NO_EMBARGO | A record with DEFALT_READ Group policy is added |
PRIVATE_EMBARGO | A record with DEFALT_READ Group is added |
Note: The class //InstallItem// has been overridden because when the Item came installed the policies of all the bitstreams came overridden.
*Comments:*
Current Workflow Help Documentation
https://services.ideals.illinois.edu/wiki/bin/view/IDEALS/IDEALSHelpPage#Submit_Choose_Access_and_Privacy
Feature supports placing item into withdrawn.
Embargo by Item only applies to Items that are Open Access, Visible to a smaller group or university of illimois users. This is because in those cases the item is in an archived state in the repository.
Overview
Content Tools