EMF Performance: EMF 2.0.1 vs. EMF 2.1.0 RC1

  EMF Load and Save

Significant1 Changes ## Test 28.4% 01 DynamicIpo#testSer 52.4% 02 DynamicIpo#testSerCaching 2 27.5% 03 StaticIpo#testSer 50.8% 04 StaticIpo#testSerCaching 2 05 DynamicIpoLoad#testLoad 10.6% 06 DynamicIpoLoad#testLoadParserCache 7.4% 07 DynamicIpoLoad#testLoadParserAndFeatureMapCache 08 StaticIpoLoad#testLoad 8.8% 09 StaticIpoLoad#testLoadParserCache 5.8% 10 StaticIpoLoad#testLoadParserAndFeatureMapCache

  SDO Load and Save

Significant1 Changes ## Test 22.2% 11 DynamicIPOSDOSaveTest#saveWithNoOptions 46.8% 12 DynamicIPOSDOSaveTest#saveWithCaching 2 20.6% 13 StaticIPOSDOSaveTest#saveWithNoOptions 45.9% 14 StaticIPOSDOSaveTest#saveWithCaching 2 15 DynamicIPOSDOLoadTest#loadWithNoOptions 16 DynamicIPOSDOLoadTest#loadWithParserCache 17 DynamicIPOSDOLoadTest#loadWithParserAndFeatureMapCache 18 StaticIPOSDOLoadTest#loadWithNoOptions 19 StaticIPOSDOLoadTest#loadWithParserCache 20 StaticIPOSDOLoadTest#loadWithParserAndFeatureMapCache

  Save

Significant1 Changes ## Test 28.4% 01 DynamicIpo#testSer 52.4% 02 DynamicIpo#testSerCaching 2 27.5% 03 StaticIpo#testSer 50.8% 04 StaticIpo#testSerCaching 2 22.2% 11 DynamicIPOSDOSaveTest#saveWithNoOptions 46.8% 12 DynamicIPOSDOSaveTest#saveWithCaching 2 20.6% 13 StaticIPOSDOSaveTest#saveWithNoOptions 45.9% 14 StaticIPOSDOSaveTest#saveWithCaching 2

  Load

Significant1 Changes ## Test 05 DynamicIpoLoad#testLoad 10.6% 06 DynamicIpoLoad#testLoadParserCache 7.4% 07 DynamicIpoLoad#testLoadParserAndFeatureMapCache 08 StaticIpoLoad#testLoad 8.8% 09 StaticIpoLoad#testLoadParserCache 5.8% 10 StaticIpoLoad#testLoadParserAndFeatureMapCache 15 DynamicIPOSDOLoadTest#loadWithNoOptions 16 DynamicIPOSDOLoadTest#loadWithParserCache 17 DynamicIPOSDOLoadTest#loadWithParserAndFeatureMapCache 18 StaticIPOSDOLoadTest#loadWithNoOptions 19 StaticIPOSDOLoadTest#loadWithParserCache 20 StaticIPOSDOLoadTest#loadWithParserAndFeatureMapCache

  Static IPO Access

Significant1 Changes ## Test 75 StaticIPOSDOAccessorTest#getByGenerated 76 StaticIPOSDOAccessorTest#setByGenerated 26.1% 77 StaticIPOSDOAccessorTest#getByProperty 78 StaticIPOSDOAccessorTest#setByProperty 26.5% 79 StaticIPOSDOAccessorTest#getByShortPath 80 StaticIPOSDOAccessorTest#setByShortPath

  Dynamic IPO Access

Significant1 Changes ## Test 25.9% 50 DynamicIPOSDOAccessorTest#getByProperty -6% 51 DynamicIPOSDOAccessorTest#setByProperty 20.2% 52 DynamicIPOSDOAccessorTest#getByShortPath 53 DynamicIPOSDOAccessorTest#setByShortPath

  Sequence/FeatureMap

Significant1 Changes ## Test 23 StaticSequenceTest#getSequenceByGenerated 20.8% 24 StaticSequenceTest#getDerivedByGenerated 25 StaticSequenceTest#getSequenceWithEGet 23.3% 26 StaticSequenceTest#getDerivedWithEGet 24.1% 27 StaticSequenceTest#getSequence 23.2% 28 StaticSequenceTest#getDerived

  Bidirectional Set

Significant1 Changes ## Test 10.7% 30 StaticBidirectionalTest#setAdaptedByGenerated 31 StaticBidirectionalTest#setByGenerated 32 StaticBidirectionalTest#setWithESet 57 StaticIPOSDOAccessorTest#setObjectWithESet

  Dynamic Access (1)

Significant1 Changes ## Test 14.8% 33 DynamicIPOSDOAccessorTest#getObjectWithEGet 15.6% 34 DynamicIPOSDOAccessorTest#setObjectWithESet 29.6% 35 DynamicIPOSDOAccessorTest#getObjectByProperty 12.9% 36 DynamicIPOSDOAccessorTest#setObjectByProperty 42.2% 37 DynamicIPOSDOAccessorTest#getObjectByIndex 20.6% 38 DynamicIPOSDOAccessorTest#setObjectByIndex 30.5% 39 DynamicIPOSDOAccessorTest#getBigIntegerByProperty 7.7% 40 DynamicIPOSDOAccessorTest#setBigIntegerByProperty 41.9% 41 DynamicIPOSDOAccessorTest#getBigIntegerByIndex 21.1% 42 DynamicIPOSDOAccessorTest#setBigIntegerByIndex 13% 43 DynamicIPOSDOAccessorTest#getBigIntegerByPath 30.2% 44 DynamicIPOSDOAccessorTest#getBigDecimalByProperty 34.6% 45 DynamicIPOSDOAccessorTest#getBigDecimalByIndex 11.3% 46 DynamicIPOSDOAccessorTest#getBigDecimalByPath 22.9% 47 DynamicIPOSDOAccessorTest#getStringByShortPath 22.8% 48 DynamicIPOSDOAccessorTest#getDataObjectByProperty 49 DynamicIPOSDOAccessorTest#setDataObjectByProperty

  Dynamic Access (2)

Significant1 Changes ## Test 14.8% 33 DynamicIPOSDOAccessorTest#getObjectWithEGet 15.6% 34 DynamicIPOSDOAccessorTest#setObjectWithESet 29.6% 35 DynamicIPOSDOAccessorTest#getObjectByProperty 12.9% 36 DynamicIPOSDOAccessorTest#setObjectByProperty 42.2% 37 DynamicIPOSDOAccessorTest#getObjectByIndex 20.6% 38 DynamicIPOSDOAccessorTest#setObjectByIndex 30.5% 39 DynamicIPOSDOAccessorTest#getBigIntegerByProperty 7.7% 40 DynamicIPOSDOAccessorTest#setBigIntegerByProperty 41.9% 41 DynamicIPOSDOAccessorTest#getBigIntegerByIndex 21.1% 42 DynamicIPOSDOAccessorTest#setBigIntegerByIndex 30.2% 44 DynamicIPOSDOAccessorTest#getBigDecimalByProperty 34.6% 45 DynamicIPOSDOAccessorTest#getBigDecimalByIndex 22.9% 47 DynamicIPOSDOAccessorTest#getStringByShortPath 22.8% 48 DynamicIPOSDOAccessorTest#getDataObjectByProperty

  Static Access (1)

Significant1 Changes ## Test 54 StaticIPOSDOAccessorTest#getFromMap 55 StaticIPOSDOAccessorTest#setInMap 56 StaticIPOSDOAccessorTest#getObjectWithEGet 57 StaticIPOSDOAccessorTest#setObjectWithESet 33.5% 58 StaticIPOSDOAccessorTest#getObjectByProperty 59 StaticIPOSDOAccessorTest#setObjectByProperty 40.4% 60 StaticIPOSDOAccessorTest#getObjectByIndex 28.1% 61 StaticIPOSDOAccessorTest#setObjectByIndex 62 StaticIPOSDOAccessorTest#getBigIntegerByGenerated 63 StaticIPOSDOAccessorTest#setBigIntegerByGenerated 34.1% 64 StaticIPOSDOAccessorTest#getBigIntegerByProperty 65 StaticIPOSDOAccessorTest#setBigIntegerByProperty 41.7% 66 StaticIPOSDOAccessorTest#getBigIntegerByIndex 24.8% 67 StaticIPOSDOAccessorTest#setBigIntegerByIndex 16.7% 68 StaticIPOSDOAccessorTest#getBigIntegerByPath 43.6% 69 StaticIPOSDOAccessorTest#getBigDecimalByProperty 32% 70 StaticIPOSDOAccessorTest#getBigDecimalByIndex 13.6% 71 StaticIPOSDOAccessorTest#getBigDecimalByPath 36.7% 72 StaticIPOSDOAccessorTest#getStringByShortPath 20.3% 73 StaticIPOSDOAccessorTest#getDataObjectByProperty 74 StaticIPOSDOAccessorTest#setDataObjectByProperty

  Static Access (2)

Significant1 Changes ## Test 54 StaticIPOSDOAccessorTest#getFromMap 55 StaticIPOSDOAccessorTest#setInMap 56 StaticIPOSDOAccessorTest#getObjectWithEGet 57 StaticIPOSDOAccessorTest#setObjectWithESet 33.5% 58 StaticIPOSDOAccessorTest#getObjectByProperty 59 StaticIPOSDOAccessorTest#setObjectByProperty 40.4% 60 StaticIPOSDOAccessorTest#getObjectByIndex 28.1% 61 StaticIPOSDOAccessorTest#setObjectByIndex 62 StaticIPOSDOAccessorTest#getBigIntegerByGenerated 63 StaticIPOSDOAccessorTest#setBigIntegerByGenerated 34.1% 64 StaticIPOSDOAccessorTest#getBigIntegerByProperty 65 StaticIPOSDOAccessorTest#setBigIntegerByProperty 41.7% 66 StaticIPOSDOAccessorTest#getBigIntegerByIndex 24.8% 67 StaticIPOSDOAccessorTest#setBigIntegerByIndex 43.6% 69 StaticIPOSDOAccessorTest#getBigDecimalByProperty 32% 70 StaticIPOSDOAccessorTest#getBigDecimalByIndex 20.3% 73 StaticIPOSDOAccessorTest#getDataObjectByProperty

  Getters

Significant1 Changes ## Test 14.8% 33 DynamicIPOSDOAccessorTest#getObjectWithEGet 29.6% 35 DynamicIPOSDOAccessorTest#getObjectByProperty 42.2% 37 DynamicIPOSDOAccessorTest#getObjectByIndex 56 StaticIPOSDOAccessorTest#getObjectWithEGet 33.5% 58 StaticIPOSDOAccessorTest#getObjectByProperty 40.4% 60 StaticIPOSDOAccessorTest#getObjectByIndex

  Setters

Significant1 Changes ## Test 15.6% 34 DynamicIPOSDOAccessorTest#setObjectWithESet 12.9% 36 DynamicIPOSDOAccessorTest#setObjectByProperty 20.6% 38 DynamicIPOSDOAccessorTest#setObjectByIndex 57 StaticIPOSDOAccessorTest#setObjectWithESet 59 StaticIPOSDOAccessorTest#setObjectByProperty 28.1% 61 StaticIPOSDOAccessorTest#setObjectByIndex

  Open Content

Significant1 Changes ## Test 72.8% 21 OpenContentTest#demandFeature 47.9% 22 OpenContentTest#createDemandFeatureDO

  All Test Methods

2.0.1 2.1.0 Significant1 Avg Stdv% Avg Stdv% Changes ## Test 248 2.9% 177.5 2% 28.4% 01 DynamicIpo#testSer 248 2.9% 118 2.2% 52.4% 02 DynamicIpo#testSerCaching 2 230.5 1.2% 167 1.5% 27.5% 03 StaticIpo#testSer 230.5 1.2% 113.5 2.1% 50.8% 04 StaticIpo#testSerCaching 2 1767 0.4% 1712 0.6% 05 DynamicIpoLoad#testLoad 708.5 0.7% 633.5 1.1% 10.6% 06 DynamicIpoLoad#testLoadParserCache 581.5 0.6% 538.5 1.5% 7.4% 07 DynamicIpoLoad#testLoadParserAndFeatureMapCache 1746 0.3% 1700 0.3% 08 StaticIpoLoad#testLoad 690.5 0.7% 630 1% 8.8% 09 StaticIpoLoad#testLoadParserCache 571 0.8% 538 1% 5.8% 10 StaticIpoLoad#testLoadParserAndFeatureMapCache 303.6 1.7% 236.2 2.2% 22.2% 11 DynamicIPOSDOSaveTest#saveWithNoOptions 303.6 1.7% 161.4 1.9% 46.8% 12 DynamicIPOSDOSaveTest#saveWithCaching 2 296.3 1.5% 235.2 2% 20.6% 13 StaticIPOSDOSaveTest#saveWithNoOptions 296.3 1.5% 160.2 2.3% 45.9% 14 StaticIPOSDOSaveTest#saveWithCaching 2 1788 0.5% 1799 0.5% 15 DynamicIPOSDOLoadTest#loadWithNoOptions 693.5 1.1% 697 0.9% 16 DynamicIPOSDOLoadTest#loadWithParserCache 582 0.8% 603 1.4% 17 DynamicIPOSDOLoadTest#loadWithParserAndFeatureMapCache 1768 0.6% 1786 0.5% 18 StaticIPOSDOLoadTest#loadWithNoOptions 673 1.2% 687 0.9% 19 StaticIPOSDOLoadTest#loadWithParserCache 567.5 0.7% 595.5 1.3% 20 StaticIPOSDOLoadTest#loadWithParserAndFeatureMapCache 6546 1.2% v 1779 2% 72.8% 21 OpenContentTest#demandFeature 2752 1.1% ^ 1434 1.2% 47.9% 22 OpenContentTest#createDemandFeatureDO 0.00709 2% 0.00708 1.7% 23 StaticSequenceTest#getSequenceByGenerated 0.2486 2.7% 0.197 3.7% 20.8% 24 StaticSequenceTest#getDerivedByGenerated 0.03563 0.7% 0.0363 0.2% 25 StaticSequenceTest#getSequenceWithEGet 0.2933 3.4% 0.225 2.8% 23.3% 26 StaticSequenceTest#getDerivedWithEGet 0.1593 2% ^ 0.121 2.1% ^ 24.1% 27 StaticSequenceTest#getSequence 0.2271 5.6% 0.1744 4.8% 23.2% 28 StaticSequenceTest#getDerived 1.477 1.5% 1.341 5.3% ^ 9.2% 29 StaticSequenceTest#contains 1.667 2.5% ^ 1.489 2.6% 10.7% 30 StaticBidirectionalTest#setAdaptedByGenerated 5.167 4.7% ^ 5.227 5.4% ^ 31 StaticBidirectionalTest#setByGenerated 5.219 7.3% ^ 5.294 7% ^ 32 StaticBidirectionalTest#setWithESet 0.1176 14.8% ^ 0.1002 2.6% ^ 14.8% 33 DynamicIPOSDOAccessorTest#getObjectWithEGet 0.1964 9.6% 0.1658 7.1% ^ 15.6% 34 DynamicIPOSDOAccessorTest#setObjectWithESet 0.2398 6.8% ^ 0.1688 4.4% 29.6% 35 DynamicIPOSDOAccessorTest#getObjectByProperty 0.1906 7.3% 0.166 4.6% 12.9% 36 DynamicIPOSDOAccessorTest#setObjectByProperty 0.2679 6% 0.1549 4.9% ^ 42.2% 37 DynamicIPOSDOAccessorTest#getObjectByIndex 0.2014 6.6% 0.16 6.2% ^ 20.6% 38 DynamicIPOSDOAccessorTest#setObjectByIndex 0.2542 7.3% ^ 0.1768 4.4% 30.5% 39 DynamicIPOSDOAccessorTest#getBigIntegerByProperty 0.4078 5.9% 0.3763 4.6% 7.7% 40 DynamicIPOSDOAccessorTest#setBigIntegerByProperty 0.2728 7.5% 0.1586 3.2% ^ 41.9% 41 DynamicIPOSDOAccessorTest#getBigIntegerByIndex 0.218 4.8% 0.172 4.3% 21.1% 42 DynamicIPOSDOAccessorTest#setBigIntegerByIndex 3.465 1.6% 3.016 1.4% 13% 43 DynamicIPOSDOAccessorTest#getBigIntegerByPath 0.4918 8.1% 0.3433 6.9% 30.2% 44 DynamicIPOSDOAccessorTest#getBigDecimalByProperty 0.5386 7.3% 0.3524 8.8% 34.6% 45 DynamicIPOSDOAccessorTest#getBigDecimalByIndex 4.245 1.9% ^ 3.767 1.9% 11.3% 46 DynamicIPOSDOAccessorTest#getBigDecimalByPath 0.3292 5.7% 0.254 7.8% ^ 22.9% 47 DynamicIPOSDOAccessorTest#getStringByShortPath 0.2791 6.4% 0.2155 3.6% ^ 22.8% 48 DynamicIPOSDOAccessorTest#getDataObjectByProperty 1.327 5.7% 1.318 4.1% ^ 49 DynamicIPOSDOAccessorTest#setDataObjectByProperty 6.828 5.8% ^ 5.061 3.6% 25.9% 50 DynamicIPOSDOAccessorTest#getByProperty 26.85 6.9% 28.34 9.9% ^ -6% 51 DynamicIPOSDOAccessorTest#setByProperty 13.08 3.9% 10.44 11.6% ^ 20.2% 52 DynamicIPOSDOAccessorTest#getByShortPath 27.7 2.3% 26.93 4.6% ^ 53 DynamicIPOSDOAccessorTest#setByShortPath 0.03323 7.3% ^ 0.03308 6.9% 54 StaticIPOSDOAccessorTest#getFromMap 0.03223 7.1% ^ 0.0316 3% ^ 55 StaticIPOSDOAccessorTest#setInMap 0.02703 0.8% 0.02783 0.7% 56 StaticIPOSDOAccessorTest#getObjectWithEGet 0.03319 6% ^ 0.03315 0.7% ^ 57 StaticIPOSDOAccessorTest#setObjectWithESet 0.1415 5.8% ^ 0.09411 8.2% ^ 33.5% 58 StaticIPOSDOAccessorTest#getObjectByProperty 0.05563 10.1% ^ 0.05597 3% ^ 59 StaticIPOSDOAccessorTest#setObjectByProperty 0.1651 6.7% ^ 0.09836 1.8% ^ 40.4% 60 StaticIPOSDOAccessorTest#getObjectByIndex 0.07062 4.7% ^ 0.05076 2.6% ^ 28.1% 61 StaticIPOSDOAccessorTest#setObjectByIndex 0.002023 0.1% 0.002023 0.1% 62 StaticIPOSDOAccessorTest#getBigIntegerByGenerated 0.02119 1.1% ^ 0.02142 1.6% ^ 63 StaticIPOSDOAccessorTest#setBigIntegerByGenerated 0.1375 8.9% ^ 0.09062 18.5% ^ 34.1% 64 StaticIPOSDOAccessorTest#getBigIntegerByProperty 0.05637 10.8% ^ 0.05585 2.8% ^ 65 StaticIPOSDOAccessorTest#setBigIntegerByProperty 0.1598 6.9% 0.09307 16.1% 41.7% 66 StaticIPOSDOAccessorTest#getBigIntegerByIndex 0.07488 4.6% 0.05634 3.4% ^ 24.8% 67 StaticIPOSDOAccessorTest#setBigIntegerByIndex 3.396 1.7% 2.83 1.5% ^ 16.7% 68 StaticIPOSDOAccessorTest#getBigIntegerByPath 0.1391 10.3% ^ 0.0785 22.2% ^ 43.6% 69 StaticIPOSDOAccessorTest#getBigDecimalByProperty 0.1526 5% ^ 0.1037 18.7% 32% 70 StaticIPOSDOAccessorTest#getBigDecimalByIndex 4.065 2.5% 3.512 1.3% ^ 13.6% 71 StaticIPOSDOAccessorTest#getBigDecimalByPath 0.2213 4.7% ^ 0.14 14.2% ^ 36.7% 72 StaticIPOSDOAccessorTest#getStringByShortPath 0.1033 5.3% ^ 0.08228 8.1% 20.3% 73 StaticIPOSDOAccessorTest#getDataObjectByProperty 1.163 6.8% 1.125 8.2% ^ 74 StaticIPOSDOAccessorTest#setDataObjectByProperty 0.04246 1.5% 0.04062 6.2% 75 StaticIPOSDOAccessorTest#getByGenerated 5.174 5.3% 5.073 7.4% 76 StaticIPOSDOAccessorTest#setByGenerated 1.872 5.8% ^ 1.383 1.8% ^ 26.1% 77 StaticIPOSDOAccessorTest#getByProperty 5.149 6.6% ^ 5.276 7.1% ^ 78 StaticIPOSDOAccessorTest#setByProperty 4.743 5.5% 3.487 14.4% ^ 26.5% 79 StaticIPOSDOAccessorTest#getByShortPath 13.68 2% 13.26 3.3% ^ 80 StaticIPOSDOAccessorTest#setByShortPath Average Of 49 Improved Methods: 27.7%

Build(s) used to generate this page:

  EMF 2.0.1/R200409171617 vs. EMF 2.1.0/S200506300200 (RC1)

Legend

1	A 'significant change' is defined as a delta where: |(new value - old value) / (old value)| x 100% >= 5%, that is, the absolute value of the percent difference between the new and old value is at least 5%; for an improvement, (old avg value - std dev) > (new avg value + std dev), that is, error bars do no overlap; and, for a regression, (old avg value + std dev) < (new avg value - std dev), that is, error bars do no overlap
2	Since this method only exists in the second (newer) test data, the preceding test was used as the baseline for comparison. For example, DynamicIpo#testSerCaching() was compared to DynamicIpo#testSer() in order to be able to compare the best available method option available in the old build compared to the new build.
^	In order to improve standard deviation of the data set, the highest value was removed and the standard deviation recomputed. If the new standard deviation (expressed as a percentage of the mean) was found to be better by at least 1%, the highest data point was omitted and the new standard deviation used.
v	In order to improve standard deviation of the data set, the lowest value was removed and the standard deviation recomputed. If the new standard deviation (expressed as a percentage of the mean) was found to be better by at least 20%, the lowest data point was omitted and the new standard deviation used.

EMF Performance Data

To review the data used to calculate the above plots, please see the EMF Performance Data as either tab separated values (TSV) or Excel (XLS). Note also that by clicking on the plots themselves, you can review the values used by Ploticus 2.3 to create the plots.

EMF Performance FAQ

For additional information, please see the EMF Perfomance FAQ.