Many Members Performance Testing

Introduction

The Fedora Community has raised significant concerns about retrieval times for resources with many members.   By "members" we are talking about at least three scenarios

1. Many children
2. Many hasMember outlinks
3. Many memberOf inlinks 

Using a set of tests originally devised by Esmé Cowles I modified them slightly and added a couple of new tests.  The latest versions can be found here : test scripts.   The test results, in most cases, reflect that fastest retrieval speeds I was able to get. Initial retrieval times tended to be slower in most cases due to the fact that modeshape needs to warm up the cache.  The scripts now make an initial GET call to warm the cache before timing a second call to the same resource.

Optimization Strategies

I pursued three main strategies for improving the performance of the "many members" use cases.   I first tried configuring custom modeshape indexes in conjunction with attempts to pull the member ids using targeted jcr-sql2 queries that queried specific property rather than iterating through nodes.  The hope here was that by using jcr-sql2 we could by pass the potentially costly operation of loading the node before accessing a property on that node.   No matter how I went about it, it seemed that this approach surprisingly did not move the needle at all.  In some cases it look like it made the problem slightly worse.  Eventually I abandoned this line of thinking.   The next thing I tried was to "turn on"  parallel processing of streams.  This required a minor tweak to the code and promised significant improvement on multiprocessor machines.  Finally I tried setting the cacheSize param in repository.json to a large number in the hopes that this might improve the performance.

Conclusions

Looking at the hasMember and memberOf use cases it would seem that the most significant improvements could be seen by combining the parallelization changes with the increased cacheSize using a local postgresql database.   hasMember improved by 5x while memberOf performance improved by a 8x.

Setup:

In order to set up the machines for the tests,  I performed the following steps.  

1.  Install postgresql 
2. Install mysql-server 
3. Build  the fcrepo4 project from the specified commit. 
4. Run fcrepo-webapp using mvn clean jetty:run and the appropriate MAVEN_OPTS such as fcrepo.modeshape.configuration, fcrepo.mysql.username, fcrepo.mysql.password,  fcrepo.postgres.username, fcrepo.postgres.password.
5. git clone https://github.com/fcrepo4-labs/fcrepo-performance-test-scripts.git
6. For each test, I ran the scripts with 1000 and 10000 items. 

Instance Definition

1. AWS / Ubuntu 16 / Oracle Java 8 / m3.medium  (3.7 GiB memory  / Intel Xeon E5-2670 (Sandy Bridge) Processor @  2.6 GHz x 1)
2. Lenovo / Ubuntu 16.10 / Java HotSpot 1.8.0_111 / 11.6 GiB memory / Intel i7-4600U CPU @ 2.10GHz x 4
3. AWS / Ubuntu 16 / Oracle Java 8 / m3.xlarge (14 GiB memory  / Intel Xeon E5-2670 (Sandy Bridge) Processor @  2.6 GHz x 4)
4. AWS / Ubuntu 16 / Oracle Java 8 / c4.xlarge (7.5 GiB memory  / Intel Xeon E5-2666 v3 (Haswell)  @  2.9 GHz x 4)
5. AWS / Ubuntu 16 / Oracle Java 8 / c4.2xlarge (15 GiB memory  /Intel Xeon E5-2666 v3 (Haswell) @  2.9 GHz x 8)
6. AWS / Ubuntu 16 / Oracle Java 8 / c4.4xlarge (30 GiB memory  /Intel Xeon E5-2666 v3 (Haswell) @  2.9 GHz x 16)

n-children.sh

n-members.sh

n-memberof.sh

Number of relations: 1000

n-uris.sh

n-properties.sh

n-binaries.sh  (time for loading binary files)

Conclusions