This document is the beginning of a new performance testing initiative for the Fedora Repository and components integrated into Fedora Commons Framework. This work builds upon the superb performance tests and analysis performed by Fiz Karlsruhe, Sun Microsystems (now Oracle) and the performance testing included in each release of the Fedora Repository. We hope this will provide a common performance testing framework and begin a community effort to characterize Fedora performance to document best practices for configuration and identify bottlenecks for subsequent software development. It is presumed that most performance testing will be black-box (through the APIs) but instrumentation may be added to help find bottlenecks.
Goals Include:
Performance testing will be accomplished using testbeds kindly provided by a number of community sources. Testing can also be accomplished using Cloud infrastructure such as Amazon's offerings. Individual user organization can stand up their own versions of the testing framework for their unique infrastructure. Details of the test configurations will be documented in this Wiki as the configurations are implemented.
In most cases a test configuration will always consists of at least four parts:
Other typical components include:
A number of frameworks are used by the community including several home-grown test platforms. Two well supported open-source frameworks are Apache JMeter and The Grinder. Either are well suited for this purpose and have their proponents. In many cases tests written for one framework can easily be re-purposed for another. Hopefully, experience with more than one framework and tests from all will be shared here. Just as important is publication of test results and analyses.
To be valid, a performance test must minimize or isolate external influences. This can be done with a dedicated platform (and network if a multi-host configuration is used). Unfortunately, cloud platforms are shared and the performance allocated to any given set of processes varies with time. To utilize a ahared platform, you must monitor it and factor its load into the performance calculations. This can be complicated since there are so many potential factors to consider but "rough order of magnitude" results are still quite useful. This is still useful and may be the most practical approach for interim performance test.
There can be any combination of the above but it can start simply, built incrementally.
What is the minimal useful starting point?
What is the first reference platform?
What to we consider the first key tests?
Who will write tests?
When (what events) are tests run?
Can we test a branch?
What is the overlap with integration/functional tests?
Configuration 1 -
Configuration 2 -