All Versions
- DSpace 7.x (Current Release)
- DSpace 8.x (Unreleased)
- DSpace 6.x (EOL)
- DSpace 5.x (EOL)
- More Versions...
Page History
| Table of Contents | ||||||
|---|---|---|---|---|---|---|
|
DSpace Log Converter
With the release of DSpace 1.6, new statistics software component was added. The use of Solr for statistics in DSpace makes it possible to have a database of statistics. With this in mind, there is the issue of the older log files and how a site can use them. The following command process is able to convert the existing log files and then import them for Solr use. The user will need to perform this conversion only once.
...
Although the DSpace Log Convertor applies basic spider filtering (googlebot, yahoo slurp, msnbot), it is far from complete. Please refer to Managing Usage Statistics Filtering and Pruning Spiders for spider removal operations, after converting your old logs.
Filtering and Pruning Spiders
Command used: |
|
Java class: | org.dspace.statistics.util.StatisticsClient |
Arguments (short and long forms): | Description |
| Reindex the bitstreams to ensure we have the bundle name |
-r or --remove-deleted-bitstreams | While indexing the bundle names remove the statistics about deleted bitstreams |
| Update Spider IP Files from internet into |
| Delete Spiders in Solr By isBot Flag. Will prune out all records that have |
| Delete Spiders in Solr By IP Address, DNS name, or Agent name. Will prune out all records that match spider identification patterns. |
| Update isBot Flag in Solr. Marks any records currently stored in statistics that have IP addresses matched in spiders files |
| Calls up this brief help table at command line. |
...
Spiders may also be excluded by DNS name or Agent header value. Place one or more files of patterns in the directories [dspace]/config/spiders/domains and/or [dspace]/config/spiders/agents. Each line in a pattern file should be either empty, a comment starting with "#" in column 1, or a regular expression which matches some names to be recognized as spiders.
Routine Solr Index Maintenance
Command used: |
|
Java class: | org.dspace.statistics.util.StatisticsClient |
Arguments (short and long forms): | Description |
| Run maintenance on the SOLR index. Recommended to run daily, to prevent your servlet container from running out of memory |
...
The usage of this this option is strongly recommended, you should run this script daily (from crontab or your system's scheduler), to prevent your servlet container from running out of memory.
Solr Sharding By Year
Command used: |
|
Java class: | org.dspace.statistics.util.StatisticsClient |
Arguments (short and long forms): | Description |
| Splits the data in the main core up into a separate solr core for each year, this will upgrade the performance of the solr. |
...
| Code Block |
|---|
# At 12:00AM on January 1, "shard" the DSpace Statistics Solr index. Ensures each year has its own Solr index - this improves performance. 0 0 1 1 * [dspace]/bin/dspace stats-util -s |
Technical implementation details
After sharding, the SOLR data cores are located in the [dspace.dir]/solr directory. There is no need to define the location of each individual core in solr.xml because they are automatically retrieved at runtime. This retrieval happens in the static method located in theorg.dspace.statistics.SolrLogger class. These cores are stored in the statisticYearCores list each time a query is made to the solr these cores are added as shards by the addAdditionalSolrYearCores method. The cores share a common configuration copied from your original statistics core. Therefore, no issues should be resulting from subsequent ant updates.
The actual sharding of the of the original solr core into individual cores by year is done in the shardSolrIndex method in the org.dspace.statistics.SolrLogger class. The sharding is done by first running a facet on the time to get the facets split by year. Once we have our years from our logs we query the main solr data server for all information on each year & download these as csv's. When we have all data for one year we upload it to the newly created core of that year by using the update csvhandler. One all data of one year has been uploaded that data is removed from the main solr (by doing it this way if our solr crashes we do not need to start from scratch).
Overview
Content Tools