Versions Compared

Old Version 8

changes.mady.by.user Bram Luyten (Atmire)

Saved on

compared with

New Version 9

changes.mady.by.user Ivan Masár

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

With the release of DSpace 1.6, new statistics software component was added. DSpace's The use of SOLR Solr for statistics in DSpace makes it possible to have a database of statistics. This With this in mind, there is the issue of the older log files and how a site can use them. The following command process is able to convert the existing log files and then import them for SOLR Solr use. The user will need to perform this conversion only once.

The Log Converter program converts log files from dspace.log into an intermediate format that can be inserted into SOLRSolr.

Command used:

[dspace]/bin/dspace stats-log-converter

Java class:

org.dspace.statistics.util.ClassicDSpaceLogConverter

Arguments short and long forms):

Description

-i or --in

Input file

-o or --out

Output file

-m or --multiple

Adds a wildcard at the end of input and output, so it would mean dspace.log* would be converted. (For example, the following files would be included because of this argument: dspace.log, dspace.log.1, dspace.log.2, dspace.log.3, etc.)

-n or --newformat

If the log files have been created with DSpace 1.6

-v or --verbose

Display verbose output (helpful for debugging)

-h or --help

Help

The command loads the intermediate log files that have been created by the aforementioned script into SOLR.

Command used:

[dspace]/bin/dspace stats-log-importer

Java class:

org.dspace.statistics.util.StatisticsImporter

Arguments (short and long forms):

Description

-i or --in

input file

-m or --multiple

Adds a wildcard at the end of the input, so it would mean dspace.log* would be imported

-s or --skipdns

To skip the reverse DNS lookups that work out where a user is from. (The DNS lookup finds the information about the host from its IP address, such as geographical location, etc. This can be slow, and wouldn't work on a server not connected to the internet.)

-v or --verbose

Display verbose ouput (helpful for debugging)

-l or --local

For developers: allows you to import a log file from another system, so because the handles won't exist, it looks up random items in your local system to add hits to instead.

-h or --help

Help

Although the DSpace Log Convertor applies basic spider filtering (googlebot, yahoo slurp, msnbot), it is far from complete. Please refer to Filtering and Pruning Spiders for spider removal operations, after converting your old logs.

...

Command used:

[dspace]/bin/dspace stats-util

Java class:

org.dspace.statistics.util.StatisticsClient

Arguments (short and long forms):

Description

-b or --reindex-bitstreams

Reindex the bitstreams to ensure we have the bundle name
-r or --remove-deleted-bitstreams

While indexing the bundle names remove the statistics about deleted bitstreams

-u or --update-spider-files

Update Spider IP Files from internet into /dspace/config/spiders. Downloads Spider files identified in dspace.cfg under property solr.spiderips.urls. See DSpace SOLR Configuration settings for Statistics Configuration

-f or --delete-spiders-by-flag

Delete Spiders in Solr By isBot Flag. Will prune out all records that have isBot:true

-i or --delete-spiders-by-ip

Delete Spiders in Solr By IP Address. Will prune out all records that have IP's that match spider IPs.

-m or --mark-spiders

Update isBog Flag in Solr. Marks any records currently stored in statistics that have IP addresses matched in spiders files

-h or --help

Calls up this brief help table at command line.

...

Command used:

[dspace]/bin/dspace stats-util

Java class:

org.dspace.statistics.util.StatisticsClient

Arguments (short and long forms):

Description

-o or --optimize

Run maintenance on the SOLR index. Recommended to run daily, to prevent your servlet container from running out of memory

...

Command used:

[dspace]/bin/dspace stats-util

Java class:

org.dspace.statistics.util.StatisticsClient

Arguments (short and long forms):

Description

-s or --shard-solr-index

Splits the data in the main core up into a separate solr core for each year, this will upgrade the performance of the solr.

...