Eclipse OpenJ9
Eclipse Incubation

Performance

Benchmark testing demonstrates significantly better performance for both OpenJDK 8 and OpenJDK 9 when using Eclipse OpenJ9, than with Hotspot.

Many different metrics exist to measure the performance of an application including startup time, ramp up time, footprint, response time, as well as throughput. At Eclipse OpenJ9, we keep a watchful eye on all of these metrics, making sensible tradeoffs and providing tuning options that allow the virtual machine (VM) to be optimized for different workloads.

Our original tests were carried out with OpenJDK version 9, but as most users are more interested in version 8, we have taken the opportunity to re-evaluate the performance of OpenJ9 when running OpenJDK version 8 using the same metrics as before, namely footprint after start up and during ramp up, start-up time, throughput, and ramp-up time.

The results for OpenJDK 8 with OpenJ9 are very much in line with our previous tests using OpenJDK 9, and demonstrate a significantly better performance than OpenJDK 8 with Hotspot.

Take a look at these results:

Note: For the evaluation of OpenJDK v8, we used the Daytrader 7 benchmark (download from GitHub). Although Daytrader 3, which we used previously, is an application conforming to the Java EE 6 specification, Daytrader 7 has been re-designed to use the latest Java EE 7 features, including the new WebSockets specification. Thus, we feel that the new benchmark is more representative of the types of Java applications and technologies in use today and therefore more relevant to existing Java users.

While the specific levels of performance shown in these tests might change from benchmark to benchmark or even from machine to machine, the same trends are expected to hold for a large class of server-type applications.

These results show that OpenJ9 achieves a good balance between (often conflicting) performance metrics: it offers excellent footprint savings and faster start-up time with the help of AOT technology, while also delivering throughput performance that is competitive with Hotspot.

Due to its low memory footprint, OpenJ9 is particularly well suited for cloud computing environments where memory savings translate into cost savings for cloud users and providers alike.

You can read the full details about how the data was generated at OpenJDK 8 performance with OpenJ9. (You can also see the results of previous tests for OpenJDK 9 with OpenJ9.)

Memory footprint after startup

OpenJ9 is highly optimized for cloud workloads. The memory footprint (Resident Set Size) is around a third the size of other VM implementations.

Fig 1. DayTrader 7 footprint size after start up with a maximum Java heap size set to 1 GB.

×

Fig 1. DayTrader 7 footprint size after start up with a maximum Java heap size set to 1 GB.
Comparison between OpenJDK 8 with OpenJ9 and OpenJDK 8 with Hotspot.

Immediately after startup OpenJDK 8 with OpenJ9 showed a footprint size about 66% less than OpenJDK 8 with Hotspot.

When OpenJ9 was tested with AOT enabled, and with AOT and -Xquickstart enabled, the footprint size remained at around 66% less than Hotspot.

Memory footprint during ramp up

Fig 2. DayTrader 7 footprint size during ramp up with a maximum Java heap size set to 1 GB.

×

Fig 2. DayTrader 7 footprint size during ramp up with a maximum Java heap size set to 1 GB.
Comparison between OpenJDK 8 with OpenJ9 and OpenJDK 8 with Hotspot.

In all three scenarios, memory footprint (Resident Set Size) increased abruptly when load was applied to the system (time = 0). But at steady state, OpenJDK 8 with OpenJ9 was found to use approximately 63% less physical memory than OpenJDK 8 with HotSpot. This difference is even more striking than in the experiments with OpenJDK 9.

Enabling AOT on the OpenJ9 VM made almost no difference to these results.

Startup time

Shared classes and Ahead-of-Time (AOT) technologies typically reduce startup time while improving the overall ramp-up time of applications. As a developer you can achieve even faster startup times by using -Xquickstart mode as well.

Fig 3. DayTrader 7 startup time with a maximum Java heap size set to 1 GB.

×

Fig 3. DayTrader 7 startup time with a maximum Java heap size set to 1 GB.
Comparison between OpenJDK 8 with OpenJ9 and OpenJDK 8 with Hotspot.

With AOT enabled, startup time for OpenJDK 8 with OpenJ9 is 36% lower than Hotspot. Moreover, with AOT enabled and using the -Xquickstart option, start-up time for OpenJ9 is almost 42% lower than Hotspot.

Throughput

Fig 4. DayTrader 7 throughput during ramp up with a maximum Java heap size set to 1 GB.

×

Fig 4. DayTrader 7 throughput during ramp up with a maximum Java heap size set to 1 GB.
Comparison between OpenJDK 8 with OpenJ9 and OpenJDK 8 with Hotspot.

Although both OpenJDK 8 with OpenJ9 and OpenJDK 8 with Hotspot reach a similar peak throughput, OpenJDK 8 with OpenJ9 reaches the peak about 1 minute faster. The small amount of AOT code (8 MB) does not make a difference in this case.

Ramp up time in CPU-constrained environments

Virtualization is heavily used in the cloud to split big computing machines into many smaller VM guests. These guests are typically provisioned with a small number of virtual CPUs in order to achieve a high application density and to control the resources dedicated to applications. One side-effect is that Java applications can take longer to ramp-up because JIT compilation threads compete with application threads over the more limited computing resources.

Fig 5. DayTrader 7 throughput during ramp up with a maximum Java heap size set to 1 GB and the VM pinned to 1 core.

×

Fig 5. DayTrader 7 throughput during ramp up with a maximum Java heap size set to 1 GB and the VM pinned to 1 core.
Comparison between OpenJDK 8 with OpenJ9 and OpenJDK 8 with Hotspot.

Without any additional configuration set, OpenJ9 already ramps up much better than HotSpot. Although we haven't shown it on the graph, HotSpot needs about 30 minutes to reach its peak throughput; in contrast, OpenJ9 needs only 8.5 minutes!

With our recommended configuration for the cloud, the ramp-up curve for OpenJ9 improves substantially compared to its default settings. While OpenJ9 with default settings obtains a higher throughput in the end, it takes a full 5 minutes to equal the peak throughput of our configured OpenJ9. The lower peak throughput seen in the chart results from the use of a lot of AOT-compiled code, which is less optimized, and the presence of -Xtune:virtualized, which prevents most recompilations.

The total work done by the VM after load is applied, is represented by the area under the curve. The results show that our cloud configuration performs more work than the default OpenJ9 configuration and much more work than Hotspot in the first 12 minutes. Therefore, configuring OpenJ9 to use AOT and -Xtune:virtualized is an excellent solution for short lived VMs running in a CPU-constrained environment.

You can read the full details about how the data was generated at OpenJDK 8 performance with OpenJ9.