Version 5.0

Table of Contents:

New features in DSpace 5.0

DSpace 5.0 ships with a number of new features. Certain features are automatically enabled by default while others require deliberate activation.  The following non-exhaustive list contains the major new features in 5.0
	Easier Upgrading to 5.x from ANY previous DSpace version (1.x.x, 3.x or 4.x). Your underlying DSpace database now upgrades itself automatically when you first run a newer version of DSpace (see DS-2167, by Tim Donohue with support/feedback from 5.0 Release Team).  Solr/Lucene indexes now upgrade automatically during the "ant update" step of the Upgrade process (see DS-2297 by Tim Donohue, with support/feedback from Ivan Masár and 5.0 Release Team).
	Perform Batch Imports from the User Interface (in both XMLUI and JSPUI) XMLUI version (XMLUI Documentation, also see DS-1641) by Peter Dietz with the support of Ohio State University Libraries and Longsight JSPUI version (JSPUI Documentation, also see DS-2177) by the Greek National Documentation Centre/EKT
	XMLUI new features Mirage 2 Responsive Theme, based on Bootstrap (disabled by default, see DS-2052 for screenshots) by @mire ORCID Integration (disabled by default, see DS-2049) by @mire Report Google Analytics statistics from Admin UI (disabled by default, see DS-2108) by Robin Taylor Track file downloads in Google Analytics statistics (see DS-2008) by Robin Taylor Autogenerate PDF citation "cover pages" for all PDFs (disabled by default, see DS-2175) by Peter Dietz with the support of Ohio State University Libraries and Longsight  Sherpa/Romeo lookup during item submission process, see DS-2053 by Kevin Van de Velde and Bert Vanderhallen with the support of @mire (previously available only in JSPUI) Rendering MathML code in abstracts using MathJax, see DS-635 by Peter Dietz with the support of Ohio State University Libraries and Longsight Ensure "page not found" error pages use configured theme, see DS-1596 by Tim Donohue with the support of DuraSpace Performance improvements for "Select Collection" dropdown in submission process, see DS-682 by Peter Dietz with the support of Ohio State University Libraries and Longsight
	JSPUI new features Drag and drop file upload (using HTML5), see DS-1994 by Pascal-Nicolas Becker with the support of TU Berlin Item Visual Indicators in Browse/Search results, see DS-2162 by Greek National Documentation Centre/EKT Track file downloads in Google Analytics statistics (see DS-2008) by Robin Taylor
	REST API new features DSpace REST API now includes CRUD (Create/Read/Update/Delete) endpoints, see DS-2168 by the Czech Technical University in Prague
	RDF Interface to support Linked (Open) Data (NEW) DSpace can now provide its content as Linked (Open) Data via a new RDF interface (provided as an "rdf" webapp), see DS-2061 by Pascal-Nicolas Becker
	OAI-PMH interface enhancements / bug fixes OpenAIRE v3 compliance (operators over filters) OAI respects item READ rights /oai displays the list of available contexts; contexts have descriptions See DS-1649 by João Melo
	Enhanced Thumbnail Quality (disabled by default) Enhanced image thumbnails can now be generated using ImageMagick Enhanced PDF Thumbnails can now be generated using ImageMagick and Ghostscript See DS-2105 by Terry Brady with the support of Georgetown University
	Bug fixes / improvements to Biblio-Transformation-Engine (BTE) BTE: batch import from various bibliographic formats was upgrade to the latest version (see DS-2183) Kindly contributed by the Greek National Documentation Centre/EKT
	Enhancements to DOI Support (disabled by default) Enhanced EZID IdentifierProvider Metadata Mapping via XSLT, see DS-2119 by Mohamed Mohideen Abdul Rasheed
	Apache Solr libraries were upgraded for all interfaces (JSPUI, XMLUI, and OAI) See DS-2253 by Roeland Dillen with the support of @mire
	Add a place for third-party JARs / plugins to be "found" by DSpace (disabled by default) DSpace will now look for JARs / plugins in the locations specified by "plugin.classpath" value specified in dspace.cfg. See DS-2107 by Mark H. Wood with the support of IUPUI University Library
	All objects now have metadata support All DSpace objects (Communities, Collections, Items, EPeople, Groups) now have metadata, and most now use the default "dc" (Dublin Core) metadata schema. NOTE: The only exception is EPeople metadata, which is stored in a new "eperson" metadata schema. The User Interfaces don't yet take advantage of this enhancement in DSpace 5.0. Instead, this is an internal restructuring of data within DSpace. In the future, this provides the potential to create more enhanced metadata (or even more configurable metadata) on all objects See DS-1582 by Mark H. Wood with the support of IUPUI University Library and Kevin Van de Velde with the support of @mire

Release TODO list

• Check documentation for all new features, add links to them the list of new features above. Incomplete list of missing docs:
  • Unable to locate Jira server for this macro. It may be due to Application Link configuration.
  • Unable to locate Jira server for this macro. It may be due to Application Link configuration.
  • Unable to locate Jira server for this macro. It may be due to Application Link configuration.
  • Unable to locate Jira server for this macro. It may be due to Application Link configuration.
• Current list of release blockers: 

  key	summary	type	created	updated	due	assignee	reporter	priority	status	resolution
  Unable to locate Jira server for this macro. It may be due to Application Link configuration.

• Unable to locate Jira server for this macro. It may be due to Application Link configuration.  Seems like we have 2 options:
  
  1. put Tomcat-specific code to JSPUI's web.xml
  2. just document this Tomcat 7-specific warning
     On DevMtg 2014-11-05 we decided to document for RC1 and decide before RC2
     
     
• Update the list of contributors on the Release Notes in Documentation (see "TBA" in text)
• Draft up announcement for mailing lists
• Update DSpace.org

Automatic Solr upgrade

Findings

Solr is distributed on Maven Central as a .war file, which includes the Lucene .jars. No Lucene version relevant to DSpace is available from Maven Central.

Solr is a wrapper for Lucene providing concurrency. Lucene can work with index files directly if we don't need concurrent access (i.e. before we start up Solr).

DSpace 1.6 was the first version to use Solr for statistics. In order to upgrade to DSpace 5, we need to upgrade the Solr index format in two steps. The second step is easy, upgrading from DSpace 3 (Solr 3.5) to DSpace 5 means just running the Solr 4.10.2 optimize command, which can be run even concurrently when the UIs are running. However, Solr 4.4 doesn't understand the legacy index formats used in Solr before 3.5. Therefore, in the first step, we need to use Solr 3.5 or Lucene 3.5 to upgrade from the older index versions to 3.5.

Suggested upgrade

We need to avoid the conflict of having both Solr 3.5 and 4.10.2 in the DSpace classpath. Therefore I suggest to run the first step before the Solr 4.10.2 webapp is started as part of DSpace 5.

To do that, we need to:

1. Detect the oldest segment version using SegmentInfo.getVersion() of Lucene 4.10
   1. General logic is in this area of the CheckIndex script: https://github.com/apache/lucene-solr/blob/lucene_solr_4_10/lucene/core/src/java/org/apache/lucene/index/CheckIndex.java#L426
2. If that oldest segment version is < 3.5 go to 3. Otherwise go to 5.
3. get lucene-core-3.5.0.jar from Maven Central
4. run the IndexUpgrader class of Lucene 3.5 (or optimize())
5. start up DSpace in order to start up Solr 4
6. run the IndexUpgrader class of Lucene 4.10.2

The manual steps would be:

wget "http://search.maven.org/remotecontent?filepath=org/apache/lucene/lucene-core/3.5.0/lucene-core-3.5.0.jar" -O lucene-core-3.5.0.jar
# check index version, see table below:
java -cp lucene-core-3.5.0.jar org.apache.lucene.index.CheckIndex /dspace/solr/statistics/data/index/
java -cp lucene-core-3.5.0.jar org.apache.lucene.index.CheckIndex  /dspace/solr/search/data/index/
# upgrade index version:
java -cp lucene-core-3.5.0.jar org.apache.lucene.index.IndexUpgrader /dspace/solr/statistics/data/index/
java -cp lucene-core-3.5.0.jar org.apache.lucene.index.IndexUpgrader /dspace/solr/search/data/index/
# check index version again, should be "version=3.5 format=FORMAT_3_1 [Lucene 3.1+]"

DSpace/Solr/Lucene versions and compatibility

Luke

https://code.google.com/p/luke/wiki/Compatibility

Java version

Minimum version of JVM required to run Luke 1.0.1 and earlier is 1.5.

Minimum version of JVM required to run Luke 3.x and 4.x is 1.6.

Lucene Java indexes

Luke 1.0.1 should be able to open indexes built using the following versions of Lucene-Java:

• 3.0.x
• 2.x (any version)

Starting with version 4.x Luke cannot open indexes created with Lucene 3.0.0 or earlier.

Starting with version 3.1.0 Luke releases will use the same numbering as Lucene releases, to avoid confusion.

Changes

The following changes have already been made to the codebase and will be released in 5.0. Please note that the below listing is dynamically generated, so it will be changing as we continue to add new features, improvements and bug fixes to the 5.0 release.

New Features in 5.0

General Improvements in 5.0

Bug Fixes in 5.0

Organizational Details

Release Coordination

Instead of a single "Release Coordinator", the DSpace 5.0 release will be managed by a "Release Team".

Release Team Leader

• Peter Dietz (Longsight)

Release Team Members

• Peter Dietz (Longsight)
• Hardy Pottinger (U of Missouri)
• Ivan Masár
• Mark H. Wood (Indiana University)
• Robin Taylor (University of Edinburgh)
• Pascal-Nicolas Becker (Technische Universität Berlin)

All Release Team membership information is as of November 4, 2014. Please volunteer (by emailing Tim Donohue), if you are interested in joining the team!

Timeline and Processing

Your contributions are welcome now! Code and documentation need not be finished, so long as it is working and we can all see what it is for. Time is set aside for fixing, polishing, and integration. We have some general Code Contribution Guidelines available, but you are also welcome to ask questions on the dspace-devel mailing list.

Release Process needs to proceed according to the following Maven release process: Release Procedure