Re: [openj9-dev] Performance questions for low-latency app

[akrus <akrus@xxxxxxxxxxx>]

Hi Peter,

Can't find this GC log, but here is the latest one:

http://nya.im/jvm_20180904.log

Example times when app logged 'pauses' (and GC logged activity):

2018-09-06 00:14:55.030 - 17 ms

2018-09-06 01:06:55.027 - 17 ms

2018-09-06 03:36:25.033 - 16 ms

2018-09-06 05:41:20.016 - 17 ms

Thanks!

On Wed, 2018-09-05 at 13:28 -0400, Peter Shipton wrote:

Hi akrus,

Just back from vacation and I see nobody responded to you. The gc log download is no longer available, can you please re-post it.



Regards,

Peter Shipton
Java 9 tech lead; J9 Team Architect and Release Manager, IBM Java SDK
IBM Runtime Technologies (Runtimes) previously called the Java Technology Center (JTC)
IBM Software Group (SWG)

Phone: 1-613-356-5382 E-mail: Peter_Shipton@xxxxxxxxxx Follow IBM Runtimes on:	3755 Riverside Drive Ottawa, ON K1G 4K9 Canada

akrus ---2018-08-06 05:39:06 AM---Hello again, I think I collected enough test results for now, so I'd like to share

From: akrus <akrus@xxxxxxxxxxx>
To: openj9 developer discussions <openj9-dev@xxxxxxxxxxx>
Date: 2018-08-06 05:39 AM
Subject: Re: [openj9-dev] Performance questions for low-latency app
Sent by: openj9-dev-bounces@xxxxxxxxxxx

Hello again,

I think I collected enough test results for now, so I'd like to share them here and ask for any ideas/suggestions.
- with almost default settings, I saw the app being 'paused' for ~20 msec on almost every GC
- with hugepages enabled it improved a bit, but still about 90% of 'pauses' were seen during GC
- with isolcpus/CPUAffinity configured on Java process, this improved dramatically, although still these 20 msec spikes appear from time to time (depending on the load) - with the difference that only some threads are affects, not the whole app

I tried changing regionSize to 262144 and enabling -Xtune:virtualized, but cannot see any changes actually.

I have GC logs on, here is when app logged pause:
09:22:45.027

And actual GC log:
https://pastebin.com/T9N5Ppnv

Not sure if I understand GC log correctly, but it looks okay, right?

On Thu, 2018-06-28 at 12:39 +0000, Peter Shipton wrote:

Hi akrus,

It seems weird to debug a problem via the mailing list, can you please open an issue at [1] about using large pages.

Please let us know which platform, what command line arguments you are using, and see [2] for large page configuration information (as the OpenJ9 User Guide isn't yet published).

[1] https://github.com/eclipse/openj9/issues

[2] https://www.ibm.com/support/knowledgecenter/SSYKE2_8.0.0/com.ibm.java.vm.80.doc/docs/j9_configure_large_page.html

Peter

----- Original message -----
From: akrus <akrus@xxxxxxxxxxx>
Sent by: openj9-dev-bounces@xxxxxxxxxxx
To: openj9 developer discussions <openj9-dev@xxxxxxxxxxx>
Cc:
Subject: Re: [openj9-dev] Performance questions for low-latency app
Date: Thu, Jun 28, 2018 5:12 AM

Hi Aleksandar,

I'm afraid I don't know if there is any benchmark available and there is nothing I can share.

Btw, I've tried rebooting the server with allocated hugepages (much more than app actually needs), but still get this message:
JVMJ9GC088I Requested objectheap page size 2M could not be satisfied; using 4K instead

Is it supposed to work?

On Wed, 2018-06-27 at 21:49 -0400, Aleksandar Micic wrote:
Hi akrus,

is there a low-latency benchmark (low-latency trading like) that you recommend us trying. Either publicly available, or if you can share some of your own.

We are willing to explore opportunities for improvement in Metronome (or other GC policies in OpenJ9/OMR).

Regards,
Aleks


akrus ---06/20/2018 04:10:02 AM---Hello everyone! I was able to switch one of our app instances to OpenJ9 finally and it

From: akrus <akrus@xxxxxxxxxxx>
To: openj9 developer discussions <openj9-dev@xxxxxxxxxxx>
Date: 06/20/2018 04:10 AM
Subject: [openj9-dev] Performance questions for low-latency app
Sent by: openj9-dev-bounces@xxxxxxxxxxx



Hello everyone!

I was able to switch one of our app instances to OpenJ9 finally and it
works good so far. It was running on Azul Zing before, so I can compare
what we have now and what we had before.

There are couple of questions:
1) I have the following flags set so far: -Xmx200g
-Xshareclasses:name=myapp -Xgcpolicy:metronome -Xgc:targetPauseTime=1
Are there any other flags to consider for low-latency app?

2) I've tried enabling hugepages support for Java heap, but it was
complaining that such amount of memory is unavailable (which is not
true). I suppose that this can happen because hugepages were enabled on
a working system so it's advised in docs somewhere to reboot the
machine to fix this. Is anyone using hugepages for heap? Is there any
performance comparison for OpenJ9 with hugepages and without?

3) there is CUDA support available. Although we have no nVidia cards on
the servers, is it worth adding one? Is there any performance
comparison for this as well?

4) I've tried switching JIT optLevel to scorching and this is consuming
too much CPU. Is it worth changing default JIT level to any other?

5) I can see that Java memory usage never exceeds 50% of Xmx value, is
it supposed to be like that? Is there any flag to change this?

6) OpenJ9 supports RDMA, but there is no -Dcom.ibm.net.rdma.conf
configuration example I found. As I can see this was tested with
Mellanox cards, did anyone test Mellanox VMA?

7) comparing with Zing again, I can see that some app threads are
sometimes 'hanging' for ~20-40 msec and this happens quite frequently.
While the server is quite loaded and this can be normal behaviour, with
Zing it's happening much less frequently, but pauses are more
noticeable.

8) are there support options available (IBM?)?

Sorry for such a bunch of questions, but this mailing list is the only
place to find out the answers :)

Regards,
akrus.
_______________________________________________
openj9-dev mailing list
openj9-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/openj9-dev




_______________________________________________
openj9-dev mailing list
openj9-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/openj9-dev

_______________________________________________
openj9-dev mailing list
openj9-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/openj9-dev


_______________________________________________
openj9-dev mailing list
openj9-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/openj9-dev
_______________________________________________
openj9-dev mailing list
openj9-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/openj9-dev

_______________________________________________

openj9-dev mailing list

openj9-dev@xxxxxxxxxxx

To change your delivery options, retrieve your password, or unsubscribe from this list, visit

https://dev.eclipse.org/mailman/listinfo/openj9-dev