Solr

Solr in DSpace

What is Solr: http://lucene.apache.org/solr/features.html

DSpace uses Solr as a part of Discovery as index to speed up access to content metadata and data about access to DSpace (for statistics). It also provides faceting and search results filtering. If Discovery is enabled, the DSpace search field accepts Solr search syntax.
Discovery is an optional part of DSpace since 1.7 (with big improvements and configuration format changes in 1.8). When enabled, Discovery replaces DSpace Search and Browse and provides Solr-based statistics.

Connecting to Solr

By default, the DSpace Solr server is configured to listen only on localhost port 8080 (unless you specified another port in Tomcat configuration and the [dspace]/config/modules/discovery.cfg config file). That means that you cannot connect from another machine to the dspace server port 8080 and request a Solr URL - you'll get a HTTP 403 error. This configuration was done for security considerations - Solr index contains some data that is not accessible via public DSpace interfaces and some of the data might be sensitive.

While you could make Solr publicly accessible by changing this default configuration (if you want to do so, search for LocalHostRestrictionFilter), this is not recommended. Instead, use one of following simple means to bypass this restriction temporarily. All of them will make Solr accessible only to the machine you're connecting from for as long as the connection is open.

1. OpenSSH client - port forwarding
   connect to DSpace server and forward its port 8080 to localhost (machine we're connecting from) port 1234

   ssh -L 1234:127.0.0.1:8080 mydspace.edu

   makes mydspace.edu:8080 accessible via localhost:1234 (type http://localhost:1234 in browser address bar); also opens ssh shell
   exit ssh to terminate port forwarding
   Alternatively:

   ssh -N -f -L 1234:127.0.0.1:8080 mydspace.edu

   run with -N and -f flags if you want ssh to go to background
   kill the ssh process to terminate port forwarding
2. Putty client - port forwarding
   The same with Putty:

   Connection - SSH - Tunnels
   Source port: 8080
   Destination: localhost:1234
   Local
   Auto
   Add
   

3. OpenSSH client - SOCKS proxy
   connect to DSpace server and run a SOCKS proxy server on localhost port 1234; configure browser to use localhost:1234 as SOCKS proxy and remove "localhost" and "127.0.0.1" from addresses that bypass this proxy
   all browser requests now originate from dspace server (source IP is dspace server's IP) - dspace is the proxy server
   type http://localhost:8080 in browser address bar - localhost here is the dspace server

   ssh -D 1234 mydspace.edu

Note about Putty as SOCKS proxy - while it can be configured, it raises a security exception when Solr is accessed. If you figure this out, please add this method here.

Accessing Solr

Solr cores

DSpace contains a so-called multicore installation of Solr. That means that there are multiple Solr indexes and configurations sharing one Solr codebase. If you're familiar with Apache HTTPD, it is analogous to multiple virtual hosts running on one Apache server (separate configuration and webpages), except that individual Solr cores are accessible via different URL (as opposed to virtualhost IP:port).

The two Solr instances in DSpace Discovery are called "search" and "statistics". search contains data about communities, collections, items and bitstreams. statistics contains data about searches, accessing users, IPs etc. The two instances are accessible at following URLs (relative to the dspace server):

http://localhost:8080/solr/search/
http://localhost:8080/solr/statistics/

Solr admin interface

Both Solr cores have separate administration interfaces which let you view their respective schemas, configurations, set up logging and submit queries. The schema browser here is very useful to list fields (and their types) included in each index and even see an overview of most common values of individual fields with their frequency.

http://localhost:8080/solr/search/admin/
http://localhost:8080/solr/statistics/admin/

Solr queries

The base URL of the default Solr search handler is as follows:

http://localhost:8080/solr/search/search
http://localhost:8080/solr/statistics/search

Using the knowledge of particular fields from Solr Admin and Solr syntax (SolrQuerySyntax, CommonQueryParameters) you can make your own search requests.
You can also look at the Tomcat log file to see queries generated by XMLUI in real time

tail -f /var/log/tomcat6/catalina.out

(depending on your OS, Tomcat installation method and logging settings, the path may be different)

Solr responses

By default, Solr responses are returned in XML format. However, Solr can provide several other output formats including JSON and CSV. Discovery uses the javabin format. The Solr request parameter is wt (e.g. &wt=json). For more information, see Response Writers, QueryResponseWriters.
An interesting option is to specify an XSLT stylesheet that can transform the XML response (server-side) to any format you choose, typically HTML. Append &wt=xslt&tr=example.xsl to the Solr request URL. The .xsl files must be provided in the [dspace]/solr/search/conf/xslt/ directory.
For more information, see XsltResponseWriter.

Examples

Date of last deposited item

To get all items (search.resourcetype:2) sorted by date accessioned (dc.date.accessioned_dt) in order from newest to oldest (desc; %20 is just an url-encoded space character):

http://localhost:8080/solr/search/select?q=search.resourcetype:2&sort=dc.date.accessioned_dt%20desc

Note:

To get only the first (newest) item (rows=1) with all but the date accessioned field filtered out (fl=dc.date.accessioned) and without the Solr response header (omitHeader=true):

http://localhost:8080/solr/search/select?q=search.resourcetype:2&sort=dc.date.accessioned_dt%20desc&rows=1&fl=dc.date.accessioned&omitHeader=true

Top downloaded items by a specific user

http://localhost:8080/solr/statistics/select?indent=on&version=2.2&start=0&rows=10&fl=*%2Cscore&qt=standard&wt=standard&explainOther=&hl.fl=&facet=true&facet.field=epersonid&q=type:0

Note:

Querying Solr from XMLUI

Since Solr returns its responses in XML, it's possible and easy to call custom Solr queries from XMLUI, process the XML response with XSLT and display the results in human-readable form on the HTML page.
There are two ways how to do that - synchronously in Cocoon or asynchronously using AJAX (JavaScript) after the page is loaded. Solr queries are usually very fast, so only synchronous calls will be shown here.

You can include another XML document to be processed by XSLT using the document() function. The parameter to this function is a string with the path to the XML document to process. This can be either a static .xml file stored on the server filesystem or a URL, which will be fetched at time of processing. For Solr, the latter is what we need. Furthermore, we need to distinguish templates for processing this external XML document as opposed to the input XML document. We'll do this using the mode attribute and define a different processing mode for each query.

<xsl:apply-templates select="document('http://localhost:8080/select?q=search.resourcetype:2&amp;sort=dc.date.accessioned_dt%20desc&amp;rows=1&amp;fl=dc.date.accessioned_dt&amp;omitHeader=true'))"
mode="solr-response"/>

Now we need to define a template with the same mode that matches elements contained in the Solr response XML:

<xsl:template match="/response/result/doc/date" mode="solr-response">
    Last item was imported: <xsl:value-of select="text()"/>
</xsl:template>

Furthermore, we don't want to hardcode the http://localhost:8080 Solr URL, because this can be changed in config file and that would break the template. So we'll call a Java function from XSLT to retrieve the configured Solr URL. See the complete example in the next section.

Examples

Date of last deposited item

For description of the query parameters, see above.

1. Add confman namespace and "confman" to exclude-result-prefixes.

   <xsl:stylesheet
   ...
       xmlns:confman="org.dspace.core.ConfigurationManager"
       exclude-result-prefixes="... confman">
   

2. Add this simple template to process Solr result. More complex date formatting can be done easily in XSLT 2.0 (see XSLT 2.0 spec), however Cocoon still uses XSLT 1.0 (see DS-995). It is currently also possible to call Java functions to do date formatting.

   <xsl:template match="/response/result/doc/date" mode="lastItem">
       Last item was imported: <xsl:value-of select="substring(text(), 1, 10)"/>
   </xsl:template>
   

3. Add the following code to the place where you want the resulting text to appear:

   <xsl:variable name="solr-search-url" select="confman:getProperty('discovery', 'search.server')"/>
   <xsl:apply-templates select="document(concat($solr-search-url, '/select?q=search.resourcetype:2&amp;sort=dc.date.accessioned_dt%20desc&amp;rows=1&amp;fl=dc.date.accessioned_dt&amp;omitHeader=true'))"
   mode="lastItem"/>
   

   For example, to add it after the list of Recent items in Mirage, override its template like this:

   <xsl:template match="dri:referenceSet[@type = 'summaryList' and @n='site-last-submitted']" priority="2">
       <xsl:apply-templates select="dri:head"/>
       <!-- Here we decide whether we have a hierarchical list or a flat one -->
       <xsl:choose>
           <xsl:when test="descendant-or-self::dri:referenceSet/@rend='hierarchy' or ancestor::dri:referenceSet/@rend='hierarchy'">
               <ul>
                   <xsl:apply-templates select="*[not(name()='head')]" mode="summaryList"/>
               </ul>
           </xsl:when>
           <xsl:otherwise>
               <ul class="ds-artifact-list">
                   <xsl:apply-templates select="*[not(name()='head')]" mode="summaryList"/>
               </ul>
           </xsl:otherwise>
       </xsl:choose>
       <xsl:variable name="solr-search-url" select="confman:getProperty('discovery', 'search.server')"/>
       <xsl:apply-templates select="document(concat($solr-search-url, '/select?q=search.resourcetype:2&amp;sort=dc.date.accessioned_dt%20desc&amp;rows=1&amp;fl=dc.date.accessioned_dt&amp;omitHeader=true'))"
   mode="lastItem"/>
   </xsl:template>
   

Guidepost

Other pages on this wiki describing Solr and Discovery.

• Discovery Official DSpace 1.8 documentation
• DSpace Discovery Discovery proposal & purpose, intro video, Discovery 1.8 changes & configuration
• Discovery Configuration Configuration of Discovery in DSpace 1.7
• DSpace Discovery HowTo Discovery screenshots (before Discovery was included in DSpace), most content obsolete (pre-1.7.0)

See also:

• Solr Tutorial
• ajax-solr, a JavaScript library for creating user interfaces to Solr.