Versions Compared

Old Version 37

changes.mady.by.user Maureen Walsh

Saved on

compared with

New Version Current

changes.mady.by.user Sarah Potvin

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

 

Table of Contents

Introduction

Recommendations towards “Updating the Qualified Dublin Core registry in DSpace to the latest standards of the DCMI,” a priority identified in the October 2011 community survey on improving metadata support. It also seeks to comply with the proposal to Standardize the Default Namespace. Discussion of this issue thus far is stored as a DCAT Discussion Forum topic, . 

 

Note: In addition to the child page of mappings linked below, see the grandchild pages, "Samples and decision points for mappings." and " Proposed phased schemas"

Glossary of Terms

Main goal of these recommendations

  • The ultimate goal of these recommendations is to implement DCTERMS as the default metadata schema, thus ensuring compliance with current standards endorsed by DCMI.

Possible Phases of Update

Ultimate goal = 'dcterms' schema as metadata registry default schema

For proposed phased schema changes see: "Proposed phased schemas"

Phase One: Add schema "dcterms" to the DSpace 4.0 default registry as an added schema (not the default schema)

  • Schema "dcterms" is added to DSpace 4.0 for testing.
  • Provide documentation.
  • Curation task for metadata migration added to 4.0.

Phase Two: Change default schema from "dc" to "dcterms". The "dc" schema is updated. "local" and "dspace" schemas are added to default registry

  • Develop and implement flat "dcterms" schema as the default DSpace schema.
  • Develop and implement "local" schema.
  • Develop and implement a DSpace admin/internal metadata schema - "dspace"
  • Ship DSpace with "dc," "dcterms," "dspace," and "local" schemas in metadata registry.

  • Create new Application Profile for DSpace.

  • Address all areas of code affected (search/browse, import/export, crosswalks, hard-coded 'dc' elements/qualifiers, etc.). Resolve issues with features that rely on metadata solutions (i.e., Creative Commons, RequestCopy, embargo, etc.). Consider plugins, add-ons, interfaces, etc. as well.
  • Lock down the "dcterms" schema from the UI.
  • Provide tools for current DSpace repositories to migrate to these schemas (i.e. edit their metadata registry and data), if desirable (i.e. provide tools for migrating elements not compliant with dcterms to "local" registry).
  • DSpace add-on for remote query to profile the metadata usage of DSpace repositories added.
  • Provide documentation.

 

Outstanding issues for committers and community

  • Is it possible to ultimately implement DCTERMS with full functionality (vocabularies, etc.)? What changes to the data model will be necessary?
  • How will this proposal integrate with other suggested changes to DSpace metadata, including Proposal for Metadata Enhancement? How might it affect integration with Fedora? How might it affect other desired changes to metadata in DSpace, including implementing functional structured metadata such as MODS, METS, and PREMIS?
  • What challenges will this proposal present—or solve—for harvesting?
  • To enable repositories to migrate existing metadata to the DCTERMS schema, we will need to develop robust tools for repositories to deploy. (Note: A curation task has been added to 4.0.)
  • Should DSpace admin/internal metadata (not including DIM) have its own schema ("dspace"), or use 'local' schema?

Recommendation background

The original DCAT Discussion forum topic that lead to this proposal can be found at "Updating the Qualified Dublin Core registry in DSpace."

Recommendation (tentative, pending further community input):

  • Update current default 'dc' schema in DSpace metadata registry to current qualified Dublin Core (QDC)
    standards

    • Background:provide brief definition and reference (link?) for "DCMI Terms" and "QDC"
      • The default DSpace metadata registry ships with
      one schema -
      • the 'dc' schema (which is the default DSpace metadata schema). It was designed to comply with the Dublin Core Libraries Working Group Application Profile, modeled on
      the
      • flat, extensible
      Qualified Dublin Core standard. Soon after, the Dublin Core Metadata Initiative (DCMI) updated its
      • Qualified Dublin Core
      (QDC) standard
      • . 

        • [The default DSpace metadata registry schema is "namespace" =
         
        •   http://dublincore.org/documents/dcmi-terms/ and "name" = dc. This use of "dcmi-terms" should not be confused with DCMI Metadata Terms. It is a qualified 'dc' schema for DSpace based on an application profile (Dublin Core Libraries Working Group Application Profile (LAP)). DSpace used the LAP as the starting point for its application of Dublin Core, borrowing most of the qualifiers from it and adapting others to fit. Some qualifiers were also added to suit DSpace needs.The 'namespace' it is declaring is not a DCMI namespace. The default DSpace schema is not dc: namespace http://purl.org/dc/elements/1.1/ (the collection of legacy properties that make up the Dublin Core Metadata Element Set, Version 1.1 [DCMES]) or dcterms: namespace http://purl.org/dc/terms/ (the collection of all DCMI properties, classes and encoding schemes (other than the properties in the Dublin Core Metadata Element Set, Version 1.1 [DCMES], the classes in the DCMI Type Vocabulary [DCMI-TYPE] and the terms used in the DCMI Abstract Model) http://purl.org/dc/terms/)]
      provide reasoning - why is compliance with QDC (ultimately DCTERMS) important/desirable

    • Desirability of and demand for DCTERMS compliance: 
      .
    • provide a few specific examples of what fields would change from/to, perhaps identify some of what the basic differences would be from current DCMI
    • Updates range from the very basic (change dc.identifier.citation to dc.identifier.bibliographicCitation; dc.description.provenance to dc.provenance; dc.date.copyright to dc.date.copyrighted) to the addition of QDC elements like dc.accrualMethod and dc.audience. If we are continuing to allow qualifiers to be added to QDC elements in the 'dc' schema, we may also continue to support the current qualified elements that are not specifically compliant with QDC (e.g., dc.contributor.editor). A preliminary mapping can be found here: https://wiki.duraspace  
      • .
      org/download/attachments/32478705/DCAT+QDC+preliminary+%28posted+to+DCAT+8.15.2012%29.xlsx


  • Add DCTERMS as new, parallel schema in the default metadata registry

    • Background:provide brief definition and reference (link?) for "DCTERMS"
      • DCMI has not updated its Qualified Dublin Core standard since 2005. The community standard has shifted towards DCMI Metadata Terms, which, unlike QDC, is not a flat schema based on the schema.element.qualifier format. DCTERMS include range and domain values. A particular term may link to another term that it refines or is refined by (for example: the dcterm "hasPart" refines "relation"; "created" refines "date").

    • Rationale:provide reasoning - why is compliance DCTERMS important/desirable
      The ultimate goal, as described below, is to implement full compliance with DCTERMS, which would involve supporting the standard's range and domain values. This goal, however, is not possible with the current DSpace data model. For now, DCTERMS could be provided as a flat schema. Unlike our proposal for the updated 'dc' (QDC) schema, the DCTERMS schema will not be an update of what currently ships with DSpace but a whole new set of properties. Some of these terms, however, are easily mapped between the existing 'dc' (QDC) schema. For example, dc.date.created maps to dcterms:created. dc.format maps to dcterms:format. dc.date.updated maps to dcterms:modified
      • DCTERMS is the currently maintained DCMI standard.
        • As Sarah Shreeves recently commented:
          "I want to strongly urge the group to look at conforming with DCMI terms (http://dublincore.org/documents/dcmi-terms/) - even if we can't conform to the vocabulary, etc, this is the most up to date and current form of the namespace. If we use the dc qualifiers document we will be perpetuating the same problem, IMO. I think we can, as Tim suggests, have a graceful path forward. I will admit that a real part of my fear of just moving to DC Qualified is that DSpace--in terms of metadata--will continue to be seen as out of touch with where much of the metadata world is headed."

        • Also, from http://dublincore.org/documents/dces/:
          "Since 1998, when these fifteen elements [dc: namespace] entered into a standardization track, notions of best practice in the Semantic Web have evolved to include the assignment of formal domains and ranges in addition to definitions in natural language. Domains and ranges specify what kind of described resources and value resources are associated with a given property. Domains and ranges express the meanings implicit in natural-language definitions in an explicit form that is usable for the automatic processing of logical inferences. When a given property is encountered, an inferencing application may use information about the domains and ranges assigned to a property in order to make inferences about the resources described thereby.
        • Since January 2008, therefore, DCMI includes formal domains and ranges in the definitions of its properties. So as not to affect the conformance of existing implementations of "simple Dublin Core" in RDF, domains and ranges have not been specified for the fifteen properties of the dc: namespace (http://purl.org/dc/elements/1.1/). Rather, fifteen new properties with "names" identical to those of the Dublin Core Metadata Element Set Version 1.1 have been created in the dcterms: namespace (http://purl.org/dc/terms/). These fifteen new properties have been defined as subproperties of the corresponding properties of DCMES Version 1.1 and assigned domains and ranges as specified in the more comprehensive document "DCMI Metadata Terms" [DCTERMS].
        • Implementers may freely choose to use these fifteen properties either in their legacy dc: variant (e.g., http://purl.org/dc/elements/1.1/creator) or in the dcterms: variant (e.g., http://purl.org/dc/terms/creator) depending on application requirements. The RDF schemas of the DCMI namespaces describe the subproperty relation of dcterms:creator to dc:creator for use by Semantic Web-aware applications. Over time, however, implementers are encouraged to use the semantically more precise dcterms: properties, as they more fully follow emerging notions of best practice for machine-processable metadata
        ."
    • provide a few specific examples of what fields would change from/to, perhaps identify some of what the basic differences would be from current DCMI
        • .
       Some of these mappings remain to be decided and finalized. For example, DCTERMS provides a controlled list of syntax and vocabulary encoding schemes. QDC and
        • "
      DCMI Terms" have often designated vocabulary and syntax encoding specifications as qualifiers (e.g., dc.subject.mesh, dc.identifier.uri). If we flatted DCTERMS, do we similarly extend with qualifiers (e.g., dcterms:subject.mesh)?  
      A preliminary mapping can be found here: https://wiki.duraspace.org/download/attachments/32478705/DCAT+QDC+preliminary+%28posted+to+DCAT+8.15.2012%29.xlsx

  • Lockdown schemas offering migratory tools to pull out local customizations and push into new local schema. Make it possible but not easy to delete or edit elements in 'dc' (QDC) and DCTERMS schemasDCTERMS schema. Continue to enable the addition of qualifiers in the 'dc' (QDC) schemaClarify end result - how many metadata schemas in the metadata registry will DSpace ship with – 3?: 
  • 1) 'dc' (QDC) - which will be an update of the current default 'dc' schema, and will be set as the default metadata schema 
    • .

  • For staging purposes, we recommend that DSpace ship with 4 registries in Phase 2
  • 2) 'dcterms' (DCTERMS) - which will be an optional metadata schema, ultimate goal of replacing 'dc' (QDC) at some point in the future
  • 3) Local schema - which would ship empty, for the purpose of local customizations
    • Yes, the idea is for DSpace to ship with three schemas in the default metadata registry, to support ultimate migration to DCTERMS , allow for the continuing use of QDC, and and to standardize namespaces by pushing local customizations not compliant with QDC DC or DCTERMS into a local schema.

Main goal of these recommendations:

  • Take steps toward ultimate goal of full compliance with DCTERMS, thus ensuring compliance with current standards endorsed by DCMI and linked data capabilities. By bringing the existing 'dc' (QDC) schema  into QDC compliance, we also provide an intermediate migration step, enabling repositories to meet compliance with QDC upon upgrade, which will ease the transition to DCTERMS.
  • By locking down schemas, ensure compliance with QDC and DCTERMS standards but provide tools to allow customizations not compliant with QDC/DCTERMS to persist in local schema. 

Outstanding issues for committers and community:

  • Is it possible to ultimately implement DCTERMS with full functionality? What changes to the data model will be necessary?
  • If both QDC and DCTERMS are included, which will be considered the default? Is this projected to change?
  • How will this proposal integrate with other suggested changes to DSpace metadata, including Proposal for Metadata Enhancement? How might it affect integration with Fedora? How might it affect other desired changes to metadata in DSpace, including implementing functional structured metadata such as MODS, METS, and PREMIS?
  • What challenges will this proposal present—or solve—for harvesting?
  • To enable repositories to migrate existing metadata to QDC and DCTERMS schemas, we will need to develop robust tools for repositories to deploy. One outstanding issue is the design and development of these tools.  
  • Need help on how best to map between schemas - QDC vs. DCTERMS schemas and how they will work together  

Possible Phases of Update:

1) Move to DCTERMS

Update current default 'dc' schema in metadata registry to latest qualified Dublin Core, add 'dcterms' and 'local' as parallel schemas in registry, goal = 'dcterms' schema as metadata registry default schema

  • :
    • 1) 'dcterms' (DCTERMS) - which will be the default metadata schema
    • 2) 'dc' schema
    • 3) 'dspace' schema for system/admin metadata
    • 4) 'local' schema - which would ship with some elements migrated out of 'dc' because not compliant with QDC, and enabled for the purpose of local customizations

Relevant JIRA tickets

...

(please add any JIRA tickets that could be affected by this proposal!)

It would be useful if we could make a guess at which JIRA issues would be RESOLVED (issue will be specifically addressed by the project) vs. which ones will be AFFECTED (a decision will have to be made that may impact, but not necessarily resolve the issue) vs. which ones are RELATED (issues that relate in some way to metadata schema options in DSpace, but we either aren't sure it will be resolved or it will likely not be resolved) to the DC update project.

RESOLVED

DS-433: Update DublinCore Registry to Implement latest DC Standards

DS-805: QDC schema registry needs to be brought into conformity with the current DCMI standards

 

RELATED/AFFECTED

DS-125: Date type can't be repeatable in the submission

Would be RESOLVED

Jira
serverDuraSpace JIRA
serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
keyDS-433

Jira
serverDuraSpace JIRA
serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
keyDS-805

Jira
serverDuraSpace JIRA
serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
keyDS-716

RELATED/Would be AFFECTED

Jira
serverDuraSpace JIRA
serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
keyDS-125

Jira
serverDuraSpace JIRA
serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
keyDS-202
DS-202: Metadata Generator Plugin

  • Not sure whether this is related. But I assume if DCMI requires some properties to be unique (for example identifiers), I guess you would need a generator to ensure unique identifiers get generated.  

Jira
serverDuraSpace JIRA
serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
keyDS-651

Jira
serverDuraSpace JIRA
serverId

DS-716: Add an administrative metadata schema to DSpace

DS-800: Manage visibility of metadata fields as field attribute rather than in dspace.cfg 

DS-815: DCDate throws NullPointerException with mangled dates

DS-1134: Multilingual metadata for communities/collections

c815ca92-fd23-34c2-8fe3-956808caf8c5
keyDS-716

Jira
serverDuraSpace JIRA
serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
keyDS-800

Jira
serverDuraSpace JIRA
serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
keyDS-815

Jira
serverDuraSpace JIRA
serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
keyDS-1134

Jira
serverDuraSpace JIRA
serverIdc815ca92-fd23-34c2-8fe3-956808caf8c5
keyDS-1420
DS-1420: Exception handling for deleting a metadata field

Areas/processes that will be affected by registry update

...

What areas and processes will be affected by these shifts? Is there any documentation of what features in DSpace are making use of certain fields? Where will the code be affected? Where are metadata elements hardcoded?

...

  • Any process that displays metadata in the web used user interface:
    • item pages
    • search, browse, DSpace discovery

...