...

• /bitstream
• /browse UNLESS USING SITEMAPS
• /*/browse UNLESS USING SITEMAPS
• /browse-date UNLESS USING SITEMAPS
• /*/browse-date UNLESS USING SITEMAPS
• /community-list UNLESS USING SITEMAPS
• /handle
• /html
• /htmlmap

...

 User-agent: *
 Disallow: /browse-subject
 Disallow: /browse-author
 Disallow: /*/browse-subject
 Disallow: /*/browse-author

Other content to consider blocking

If you have configured Solr statistics to be publicly available then you probably do not want this content to be indexed.

• /displaystats

Metadata in the HTML HEAD

...

• NOTE: Back in 2008, Google officially announced they were retiring support for OAI-PMH based Sitemaps. So, OAI-PMH will no longer help you get better indexing through Google. Instead, you should be using the DSpace 'generate-sitemaps' feature described above.

Other methods

See: http://wiki.lib.sun.ac.za/index.php/SUNScholar/Web_Analytics^{Image Removed}