By Robert Tansley, Google engineer and architect of DSpace 1.0

Anyone who has analyzed traffic to their DSpace site (e.g. using Google Analytics) will notice that a significant (and in many cases a majority) of visitors arrive via a search engine such as Google or Yahoo. Hence, to help maximize the impact of content and thus encourage further deposits, it's important to ensure that your DSpace instance is indexed effectively. Here is how.

Very briefly:

DSpace 1.5 and 1.5.1 ship with a bad robots.txt. Remove the line that reads Disallow: /browse. If you do not, your site will not be correctly indexed.

Contents

Upgrade to the latest possible DSpace.

Ensure your content can be indexed

First ensure your DSpace instance is visible, e.g. with: https://www.google.com/webmasters/tools/sitestatus

If your site is not indexed at all, all search engines have a way to add your URL, e.g.:

Add HTML Sitemap support.

This is as simple as running \[dspace\]/bin/generate-sitemaps once a day.

Just set up a cron job (or scheduled task in Windows), e.g. (cron):

{{0 6 * * * \[dspace\]/bin/generate-sitemaps}}

Also, if you've customized your site's look and feel (as most have), ensure that there is a link to \[dspace-url\]/htmlmap in your DSpace's front or home page. (This is present in the footer of the default template.) E.g.:

<a href="/htmlmap"></a>

Search engines will now look at /htmlmap, which (after generate-sitemaps has been run) serves one or more pre-generated (and thus served with minimal impact on your hardware) HTML files linking directly to items, collections and communities in your DSpace instance. Crawlers will not have to work their way through any browse screens, which are intended more for human consumption, and more expensive for the server.

Create a good robots.txt

DSpace 1.5 and 1.5.1 ship with a bad robots.txt file. Delete it, or specifically the line that says Disallow: /browse. If you do not, your site will not be correctly indexed.

The trick here is to minimize load on your server, but without actually blocking anything vital for indexing. Search engines need to be able to index item, collection and community pages, and all bitstreams within items – full-text access is critically important for effective indexing, e.g. for citation analysis as well as the usual keyword searching.

If you have restricted content on your site, search engines will not be able to access it; they access all pages as an anonymous user.

Ensure that your robots.txt file is at the top level of your site: i.e. at http://repo.foo.edu/robots.txt, and NOT e.g. http://repo.foo.edu/dspace/robots.txt. If your DSpace instance is served from e.g. http://repo.foo.edu/dspace/, you'll need to add /dspace to all the paths in the examples below (e.g. /dspace/browse-subject).

DO NOT BLOCK THESE

Some URLs can be disallowed without negative impact, but be ABSOLUTELY SURE the following URLs can be reached by crawlers, i.e. DO NOT put these on Disallow: lines, or your DSpace instance might not be indexed properly.

Example good robots.txt

If you do not provide sitemaps, the only way for crawlers to reach content in this case is via the browse pages. You can use this to prevent crawlers hitting all but the date browse pages (the most reliable for indexing) in DSpace 1.4 or earlier. (Selective blocking is not possible in 1.5, but if you're on 1.5 you can use sitemaps.)

 User-agent: *
 Disallow: /browse-subject
 Disallow: /browse-author
 Disallow: /*/browse-subject
 Disallow: /*/browse-author

Other content to consider blocking

If you have configured Solr statistics to be publicly available then you probably do not want this content to be indexed.

Metadata in the HTML HEAD

It's possible to greatly customize the look and feel of your DSpace, which makes it harder for search engines, and other tools and services such as Zotero, Connotea and SIMILE Piggy Bank, to correctly pick out item metadata fields. To address this, DSpace 1.5 includes item metadata in the HEAD element of each item's HTML display page.

If you're using DSpace 1.5 and the JSP UI, or 1.5.2 with either UI, you should already see this if you VIEW SOURCE in your browser on an item display page, e.g.:

 <meta name="DC.type" content="Article" />
 <meta name="DCTERMS.contributor" content="Tansley, Robert" />

If you don't see anything like this, ensure that the following is in your site's layout/header-default.jsp, within the <head> element:

 <% if (extraHeadData != null)
    { %>
    <%= extraHeadData %>
 <% } %>


If you have heavily customized your metadata fields away from Dublin Core, you can modify the crosswalk that generates these elements by modifying {{\[dspace\]/config/crosswalks/xhtml-head-item.properties}}.

In general, OAI-PMH isn't useful to Search Engines

Feel free to support OAI-PMH, but be aware that in general it is not useful for search engines:

Other methods

See: http://wiki.lib.sun.ac.za/index.php/SUNScholar/Web_Analytics