...

In some cases, poor VIVO performance has been traced to search engine robots that either ignore or misread directives in VIVO's robots.txt file, or which issue requests for large pages at a rate that greatly exceeds the demand otherwise encountered in typical production use.  If the search engine in question is not critical to VIVO's visibility, it may be advisable to restrict access to the associated robots.  In some situations, institutional search appliances are responsible for the excessive server load.  Here, discussions with local IT staff may be warranted.

Additional discussion

Work in progress at https://docs.google.com/a/symplectic.co.uk/document/d/1ylp9HEzJiBsBP6vx1vd-Irf8o3Ff-5vDhytOVTI5_Ho/edit#heading=h.vdtjwwvnjdn7