Page tree
Skip to end of metadata
Go to start of metadata

Introduction

The Fedora Community has raised significant concerns about retrieval times for resources with many members.   By "members" we are talking about at least three scenarios

  1. Many children
  2. Many hasMember outlinks
  3. Many memberOf inlinks 

Using a set of tests originally devised by Esmé Cowles I modified them slightly and added a couple of new tests.  The latest versions can be found here : test scripts.   The test results, in most cases, reflect that fastest retrieval speeds I was able to get. Initial retrieval times tended to be slower in most cases due to the fact that modeshape needs to warm up the cache.  The scripts now make an initial GET call to warm the cache before timing a second call to the same resource.

Optimization Strategies

I pursued three main strategies for improving the performance of the "many members" use cases.   I first tried configuring custom modeshape indexes in conjunction with attempts to pull the member ids using targeted jcr-sql2 queries that queried specific property rather than iterating through nodes.  The hope here was that by using jcr-sql2 we could by pass the potentially costly operation of loading the node before accessing a property on that node.   No matter how I went about it, it seemed that this approach surprisingly did not move the needle at all.  In some cases it look like it made the problem slightly worse.  Eventually I abandoned this line of thinking.   The next thing I tried was to "turn on"  parallel processing of streams.  This required a minor tweak to the code and promised significant improvement on multiprocessor machines.  Finally I tried setting the cacheSize param in repository.json to a large number in the hopes that this might improve the performance.

Conclusions

Looking at the hasMember and memberOf use cases it would seem that the most significant improvements could be seen by combining the parallelization changes with the increased cacheSize using a local postgresql database.   hasMember improved by 5x while memberOf performance improved by a 8x.

Setup:

In order to set up the machines for the tests,  I performed the following steps.  

  1.  Install postgresql 
  2. Install mysql-server 
  3. Build  the fcrepo4 project from the specified commit. 
  4. Run fcrepo-webapp using mvn clean jetty:run and the appropriate MAVEN_OPTS such as fcrepo.modeshape.configuration, fcrepo.mysql.username, fcrepo.mysql.password,  fcrepo.postgres.username, fcrepo.postgres.password.
  5. git clone https://github.com/dbernstein/fcrepo-performance-tests.git
  6. For each test, I ran the scripts with 1000 and 10000 items. 

Instance Definition

  1. AWS / Ubuntu 16 / Oracle Java 8 / m3.medium  (3.7 GiB memory  / Intel Xeon E5-2670 (Sandy Bridge) Processor @  2.6 GHz x 1)
  2. Lenovo / Ubuntu 16.10 / Java HotSpot 1.8.0_111 / 11.6 GiB memory / Intel i7-4600U CPU @ 2.10GHz x 4
  3. AWS / Ubuntu 16 / Oracle Java 8 / m3.xlarge (14 GiB memory  / Intel Xeon E5-2670 (Sandy Bridge) Processor @  2.6 GHz x 4)
  4. AWS / Ubuntu 16 / Oracle Java 8 / c4.xlarge (7.5 GiB memory  / Intel Xeon E5-2666 v3 (Haswell)  @  2.9 GHz x 4)
  5. AWS / Ubuntu 16 / Oracle Java 8 / c4.2xlarge (15 GiB memory  /Intel Xeon E5-2666 v3 (Haswell) @  2.9 GHz x 8)
  6. AWS / Ubuntu 16 / Oracle Java 8 / c4.4xlarge (30 GiB memory  /Intel Xeon E5-2666 v3 (Haswell) @  2.9 GHz x 16)


n-children.sh

FCREPO Version
RepoBranch
Commit
Environment
Modeshape Config

# of relations

Test Duration

(seconds)

Tester

4.8.0-SNAPSHOTdbernstein
b60d4e5file-simple100005.222
4.8.0-SNAPSHOTdbernstein
daa11f35file-simple100005.567
4.8.0-SNAPSHOTdbernstein
b60d4e6file-simple100004.859
4.8.0-SNAPSHOTdbernstein
daa11f36file-simple100004.841

n-members.sh

FCREPO VersionRepoBranchCommitmodeshapeEnvironment

# of relations

Test Duration

(seconds)

Tester
4.8.0-SNAPSHOTfcrepo4master2df32file-simple110001.408
4.7.1fcrepo44.7.1546f5a5file-simple210001.45
4.8.0-SNAPSHOTfcrepo4master2df32file-simple110,00028.583
4.7.1fcrepo44.7.1546f5a5file-simple210,00024.79
4.8.0-SNAPSHOTfcrepo4masterb60d4efile-simple310,0009.58
4.8.0-SNAPSHOTdbernstein
fcrepo-2105-v4-parallelization
daa11f3file-simple310,00012.16Danny Bernsteinparallel streams enabled.
4.8.0-SNAPSHOTdbernsteinfcrepo-2105-v4-parallelizationdaa11f3file-simple410,0009.71Danny Bernsteinparallel streams enabled.
4.8.0-SNAPSHOTdbernsteinfcrepo-2105-v4-parallelizationdaa11f3file-simple510,0008.51Danny Bernsteinparallel streams enabled.
4.8.0-SNAPSHOT

b60d4efile-simple510,0008.815
4.8.0-SNAPSHOTdbernsteinfcrepo-2105-v4-parallelizationdaa11f3file-simple610,0008.29Danny Bernsteinparallel streams enabled.
4.8.0-SNAPSHOT

b60d4efile-simple610,0008.661
4.8.0-SNAPSHOTbbrananfcrepo-2402f0a51ejdbc-postgresql110001.543
4.8.0-SNAPSHOTbbrananfcrepo-2402f0a51ejdbc-postgresql110,00061.381perhaps postgres needs caching configured?
4.8.0-SNAPSHOTdbernsteinfcrepo-2105-v4-parallelizationdaa11f3jdbc-postgresql310000.592
4.8.0-SNAPSHOTdbernsteinfcrepo-2105-v4-parallelizationdaa11f3jdbc-postgresql310,00039.671
4.8.0-SNAPSHOTbbrananfcrepo-2402f0a51ejdbc-postgresql310004.435
4.8.0-SNAPSHOTbbrananfcrepo-2402f0a51ejdbc-postgresql310,00039.486
4.8.0-SNAPSHOTdbernsteinfcrepo-2105-large-cachef453ajdbc-postgresql310000.5611 million item cache
4.8.0-SNAPSHOTdbernsteinfcrepo-2105-large-cachef453ajdbc-postgresql310,0005.3341 million item cache
4.8.0-SNAPSHOTdbernsteinfcrepo-2105-large-cache5138b4jdbc-postgresql310000.6331 million item cache + parallelized
4.8.0-SNAPSHOTdbernsteinfcrepo-2105-large-cache5138b4jdbc-postgresql310,0005.3981 million item cache + parallelized
4.8.0-SNAPSHOTdbernsteinfcrepo-2105-large-cache5138b4mysql-postgresql310000.8201 million item cache + parallelized
4.8.0-SNAPSHOTdbernsteinfcrepo-2105-large-cache5138b4mysql-postgresql310,0007.7261 million item cache + parallelized
4.8.0-SNAPSHOTdbernsteinfcrepo-240275dd1jdbc-postgresql-s3310000.7011 million item cache + parallelized
4.8.0-SNAPSHOTdbernsteinfcrepo-240275dd1jdbc-postgresql-s3310,0005.4851 million item cache + parallelized
4.8.0-SNAPSHOTdbernsteinfcrepo-240275dd1jdbc-mysql-s3310000.8641 million item cache + parallelized
4.8.0-SNAPSHOTdbernsteinfcrepo-240275dd1jdbc-mysql-s3310,0007.3951 million item cache + parallelized
4.7.1fcrepo44.7.1546f5a5file-simple210,00013.07cacheSize = 50,000
4.7.1fcrepo44.7.1546f5a5file-simple210,00010.30cacheSize = 1,000,000


n-memberof.sh

Number of relations: 1000

FCREPO VersionRepobranchCommitmodeshape configEnvironment# of relations

Test Duration

(seconds)

TesterNotes
4.8.0-SNAPSHOTfcrepo4master2df32file-simple110006.570
4.7.1fcrepo44.7.1546f5a5file-simple210002.80
4.8.0-SNAPSHOTfcrepo4master2df32file-simple110,00086.000
4.7.1fcrepo44.7.1546f5a5file-simple210,00057.35
4.8.0-SNAPSHOTfcrepo4masterb60d4efile-simple310,00030.02Unlike the n-member example, results begin streaming right away - so the response begins streaming within 2 seconds.
4.8.0-SNAPSHOTdbernsteinfcrepo-2105-v4-parallelizationdaa11f3file-simple310,00029.975Danny Bernsteinparallel streams enabled.
4.8.0-SNAPSHOTdbernsteinfcrepo-2105-v4-parallelizationdaa11f3file-simple410,00024.790Danny Bernsteinparallel streams enabled.
4.8.0-SNAPSHOTdbernsteinfcrepo-2105-v4-parallelizationdaa11f3file-simple510,00020.992Danny Bernsteinparallel streams enabled.
4.8.0-SNAPSHOTfcrepo4masterb60d4efile-simple510,00026.357
4.8.0-SNAPSHOTdbernsteinfcrepo-2105-v4-parallelizationdaa11f3file-simple610,00020.337Danny Bernsteinparallel streams enabled.
4.8.0-SNAPSHOTfcrepo4masterb60d4efile-simple610,00025.782
4.8.0-SNAPSHOTbbrananfcrepo-2402f0a51ejdbc-postgresql1100011.414
4.8.0-SNAPSHOTbbrananfcrepo-2402f0a51ejdbc-postgresql110,000194
4.8.0-SNAPSHOTdbernsteincrepo-2105-v4-parallelizationdaa11f3jdbc-postgresql310003.961parallel streams enabled.
4.8.0-SNAPSHOTdbernsteincrepo-2105-v4-parallelizationdaa11f3jdbc-postgresql310,00053.452parallel streams enabled.
4.8.0-SNAPSHOTbbrananfcrepo-2402f0a51ejdbc-postgresql3100010.833
4.8.0-SNAPSHOTbbrananfcrepo-2402f0a51ejdbc-postgresql310,000109.530
4.8.0-SNAPSHOTdbernsteinfcrepo-2105-large-cachef453ajdbc-postgresql31000022.9631 million item cache.
4.8.0-SNAPSHOTdbernsteinfcrepo-2105-v4-parallelization5138b4jdbc-postgresql310,00011.5591 million item cache + parallelized
4.8.0-SNAPSHOTdbernsteinfcrepo-2105-v4-parallelization5138b4mysql-postgresql310002.3371 million item cache + parallelized
4.8.0-SNAPSHOTdbernsteinfcrepo-2105-v4-parallelization5138b4mysql-postgresql310,00014.9981 million item cache + parallelized
4.8.0-SNAPSHOTdbernsteinfcrepo-240275dd1jdbc-postgresql-s3310001.8831 million item cache + parallelized
4.8.0-SNAPSHOTdbernsteinfcrepo-240275dd1jdbc-postgresql-s3310,000
1 million item cache + parallelized
4.8.0-SNAPSHOTdbernsteinfcrepo-240275dd1jdbc-mysql-s3310002.3741 million item cache + parallelized
4.8.0-SNAPSHOTdbernsteinfcrepo-240275dd1jdbc-mysql-s3310,00014.9561 million item cache + parallelized
4.7.1fcrepo44.7.1546f5a5file-simple210,00031.91cacheSize = 50,000
4.7.1fcrepo44.7.1546f5a5file-simple210,00023.07cacheSize = 1,000,000

n-uris.sh

FCREPO VersionCommitEnvironment# of relations

Test Duration

(seconds)

 
4.8.0-SNAPSHOT2df32110000.039
4.7.1546f5a5210000.14
4.8.0-SNAPSHOT2df32110,0000.063
4.7.1546f5a5210,0000.17

n-properties.sh

FCREPO VersionCommitEnvironment# of relations

Test Duration

(seconds)

Tester
4.8.0-SNAPSHOT2df321 10000.060
4.7.1546f5a5210000.089
4.8.0-SNAPSHOT2df32110,0000.119
4.7.1546f5a5210,0000.281
      


n-binaries.sh  (time for loading binary files)

repobranchCommitEnvironmentModeshape Config# of binariessize in KB

Test Duration

(seconds)

Tester
https://github.com/bbranan/fcrepo4.gitfcrepo-2402f0a51e1file-s3 1000100000:06:02
https://github.com/bbranan/fcrepo4.git fcrepo-2402f0a51e1file-simple1000100000:02:16
https://github.com/bbranan/fcrepo4.gitfcrepo-2402f0a51e1jdbc-postgresql1000100000:02:13
https://github.com/bbranan/fcrepo4.gitfcrepo-2402f0a51e3jdbc-postgresql-s31000100000:06:36
https://github.com/bbranan/fcrepo4.gitfcrepo-2402f0a51e3jdbc-mysql-s31000100000:06:32



Conclusions

  • No labels

1 Comment

  1. I'm not sure if this is helpful, but I wanted to see the effect of altering the cacheSize variable when polling our top-level container, which has 461,818 'ldp:contains' relations.

    This was performed against fcrepo4.7.1, running in tomcat7 and using a postgres database. JVM has 8GB of memory.

    "Warm" means I ran the same CURL request following the "Cold" request.



    curl -i -X GET https://aicdamstest08.artic.edu/fcrepo/rest/prod

    Number of ldp:contains relations: 461818

    Modeshape cache: default
    Cold: 10m42.066s
    Warm: 11m23.620s

    Modeshape cache: 100000
    Cold: 10m29.661s
    Warm: 10m30.366s

    Modeshape cache: 500000
    Cold: 11m58.420s
    Warm: 11m56.306s

    Modeshape cache: 1000000
    Cold: 20m3.242s
    Warm: 21m22.747s

    Modeshape cache: 2000000
    Cold:  11m29.792s   [Resulted in: java.lang.OutOfMemoryError: Java heap space]
    Warm: 35m7.886s   [Resulted in: java.lang.OutOfMemoryError: Java heap space]

    Modeshape cache: 3000000
    Cold: 10m26.067s   [Resulted in: java.lang.OutOfMemoryError: Java heap space]
    Warm: 37m33.297s   [Resulted in: java.lang.OutOfMemoryError: Java heap space]