Last updated: July 6, 2005
What sort of dimension of performance is measured?
CPU time is measured by the Eclipse performance suite.
What setup (OS, Eclipse, JRE) was used to run the EMF performance suite?
EMF:
The base build is EMF 2.0.1 Release build:
http://archive.eclipse.org/modeling/emf/emf/downloads/drops/2.0.1/R200409171617/
The compared build is EMF 2.1.0 RC1:
http://archive.eclipse.org/modeling/emf/emf/downloads/drops/2.1.0/R200507070200/
Other than EMF, the other components are the same:
Eclipse:
The Eclipse build used is Eclipse 3.1RC2.
Early trials indicated that using the same Eclipse driver that was used to build the EMF driver (eg., 2.0.1 with 3.0.1 vs. 2.1 with 3.1) introduced extra platform-specific changes. In order to minimize these platform-based effects, tests were performed using the same Eclipse version in all tests, even if that build did not exist at the time the driver was published or a newer driver was available.
Java:
The JRE used was IBM j9 (J2RE 1.4.2 IBM build cxia32142sr1a-20050209), with a heap size of 512MB, and J9 enabled.
Ant:
Apache Ant version 1.6.2 (Sept 28 2004) was used in conjunction with bash shell scripts to run the tests. These in turn then launched headless Eclipse to calculate the performance of the tests.
Operating System:
A Debian-based OS was chosen to run the tests due to its simplicity in setup. The particular variant was SimplyMEPIS 3.3, based on Debian GNU/Linux Unstable/Testing, running kernel 2.6.10. Very little configuration was done above the default MEPIS install, in order to minimize the work required to reproduce the configuration.
Hardware:
The machine used to run the tests is an IBM ThinkCentre PC with an Intel(R) Pentium(R) 4 3.00GHz CPU and 1.5 GB memory.
What is the EMF performance suite? Where do results come from?
The major characteristic of the EMF performance suite is that most of the APIs tested have CPU times in the order of the micro-second or less. It should be seen as a micro-performance suite.
The EMF performance suite uses an extension of the Eclipse performance suite (org.eclipse.test.performance), where a method is run several times (repetitions). The main difference is that the EMF performance suite uses the minimum instead of the average of all repetitions as the result of a test. To get the CPU time of a given method, that method is run for a number of repetitions, and then the result is the minimum of all the CPU times from all the repetitions. Therefore, the published results should be seen as the best possible time it can take to perform the operation. We wanted to have stable and reproducible results and we realized using the minimum provided the best measure of stability, as it excluded duration spikes and CPU swapping issues which contributed to both less stable data and wider ranging results.
Because the APIs tested were faster than a millisecond, iterations were introduced (between when the measurements start and stop) to get results generally in the order of a few thousand milliseconds. We realized that increasing iterations was generally helping stability as well. The result of a single API call is computed from the result of the performance test divided by the number of iterations.
The EMF performance tests suite is packaged as the org.eclipse.emf.test.performance plugin. You can get the EMF extension of the Eclipse performance suite from CVS here:
For each build (e.g. EMF release 2.0.1), the EMF performance suite was run 10 times, to produce 10 minimum CPU times for each of the test methods. Minimum values were found based on a set of repetitions of a given test, discarding all but the smallest value.
At most one of the 10 data points may have been deleted if it was much greater than the other 9 points. Then, average and standard deviation were computed on the reduced set of CPU times.
How were improvements found?
A test method is considered to have improved if, over 10 EMF performance suite runs, its average 2.1.0 CPU time is at least 5% less than its average 2.0.1 CPU time, and (2.1.0 average + 2.1.0 standard deviation) is less than (2.0.1 average - 2.0.1 standard deviation). Graphically, this means that the 2.0.1 and 2.1.0 average times and their error bars do not overlap on a scatter plot of the values.
What is each method doing? Where is the source code?
You can browse the javadoc of the org.eclipse.emf.test.performance plugin here:
http://www.eclipse.org/emf/docs/performance/javadoc/
You can browse the source of the org.eclipse.emf.test.performance plugin here:
You can also extract the plugin from CVS. For details, see:
http://wiki.eclipse.org/EMF/Getting_Source
Generally, a test method is run twice: once with a static EMF or SDO model instance, and once with a dynamic EMF or SDO model instance. This is implemented by having a static class extending the dynamic class with the same name, and having the static class override model initialization to use static instead of dynamic model. For this reason, the source of a (static or dynamic) method generally resides in the dynamic class.