Eclipse Community Forums: Remote Application Platform (RAP) » High usage of memory and final CPU collapse

Help

Home

Home » Eclipse Projects » Remote Application Platform (RAP) » High usage of memory and final CPU collapse

Show: Today's Messages :: Show Polls :: Message Navigator

High usage of memory and final CPU collapse [message #1759917]

Wed, 19 April 2017 22:11

Vincenzo Caselli

Messages: 208
Registered: January 2012

Senior Member

Hi,
we have a RAP (based on 3.x workbench, with Perspective, Views, StatusLine, Jobs and so on) Application deployed on Tomcat over a virtualized Linux system with 8 Gb RAM + 2 Gb Swap + 2 CPUs.
The application serves thousands of access per day and server is always on (no nightly restart).
Tomcat is launched with the following memory options:
-Xms256m -Xmx2048m -XX:PermSize=128m -XX:MaxPermSize=512m

After some (about 6-10) day of usage the memory of Tomcat process goes far beyond the Xmx limit; here are the values showed with Linux 'top' command:
VIRT=7300m RES=4.8g
Then, at some point, the CPU usage goes to 100%, both the application and Tomcat manager are unreachable and the only option is restarting Tomcat (not the whole system) and this restores to normal values and usage.
We used some profiling tools (JProfiler and MAT) and did some optimizations based on analyzing the Heap Dump, but still didn't find what is causing the real problem.

So we have some questions about RAP:
- are there some guidelines that we can follow in order to make sure that every user session is cleaned correctly?
- what tools or techniques can we use to analyze deeply the problem?
- why Heap dumps are so small (just about some hundreds of Mb in size) with respect to the total RAM (well beyond 2 Gb) usage by the Tomcat process? how can we inspect all the process memory?

Any suggestion is highly welcome!
Thank you very much in advance.
Vincenzo

Report message to a moderator

Re: High usage of memory and final CPU collapse [message #1759942 is a reply to message #1759917]

Thu, 20 April 2017 08:54

Ivan Furnadjiev

Messages: 2285
Registered: July 2009
Location: Sofia, Bulgaria

Senior Member

Hi Vincenzo,
to find a memory leak I'm using VisualVM. You can start with a single user session. Start the app, do something with it and than reload the browser. Check with VisualVM for Display, UISessionImpl and UIThread instances in the heap dump. If there are no instances of these classes the RAP user session is correctly terminated. You can use VisualVM for CPU usage profiling too.
HTH,
Ivan

Report message to a moderator

Re: High usage of memory and final CPU collapse [message #1759988 is a reply to message #1759942]

Thu, 20 April 2017 22:38

Chris Fairhall

Messages: 205
Registered: February 2011

Senior Member

How many threads are running when the server dies? Threads allocate non-heap memory. If the UIThread's are failing to terminate, they will build up pretty quickly.

Report message to a moderator

Re: High usage of memory and final CPU collapse [message #1760023 is a reply to message #1759942]

Fri, 21 April 2017 10:03

Vincenzo Caselli

Messages: 208
Registered: January 2012

Senior Member

Hi Ivan,
thank you for the suggestion.
I will investigate further in the proposed way.
Thank you
Vincenzo

Report message to a moderator

Re: High usage of memory and final CPU collapse [message #1760024 is a reply to message #1759988]

Fri, 21 April 2017 10:10

Vincenzo Caselli

Messages: 208
Registered: January 2012

Senior Member

Hi Chris,
indeed the application makes large use of Jobs and asynch threads to update the UI.

Are there guidelines on how to ensure that these threads are cleaned correctly when problems arise (browser closed improperly, connection problems, ... ) ?

How can non-heap memory leaks can be examined?

Thank you very much
Vincenzo

Report message to a moderator

Re: High usage of memory and final CPU collapse [message #1760149 is a reply to message #1760024]

Sun, 23 April 2017 20:57

Chris Fairhall

Messages: 205
Registered: February 2011

Senior Member

I still run in to the odd hard to reproduce scenario where the UIThread fails to terminate when the session dies.

If that happens, the servlet container's background thread that cleans up old HTTP sessions can get stuck. This stops all sessions from being cleaned up and you run in to a memory leak.

When memory gets low the JVM can spend all its time running garbage collection.

One of the symptoms of this is the applications runs fine for a while then gradually gets slower and slower as the GC takes more and more time. Eventually the applications becomes completely unresponsive.

Report message to a moderator

Re: High usage of memory and final CPU collapse [message #1760151 is a reply to message #1760149]

Sun, 23 April 2017 22:11

Vincenzo Caselli

Messages: 208
Registered: January 2012

Senior Member

Chris,
you described exactly what is happening, in fact, after days of correct working, in a matter of 20-30 minutes:
- log shows huge times for every call
- the GC is taking all the CPU time
- the application, along with Tomcat Manager, is unresponsive

What strategy may be set up in this case?
Keeping a per-session list of references to threads and Jobs and cancel or stop them in some disposing event may be of help?
What event may be used in order to catch also a badly closed/crashed client browser?
Any better strategies?
Thank you

Report message to a moderator

Re: High usage of memory and final CPU collapse [message #1760152 is a reply to message #1760151]

Sun, 23 April 2017 22:47

Chris Fairhall

Messages: 205
Registered: February 2011

Senior Member

Try and find what is causing the scenario and fixing it?

It can be one of many things...

You really need a thread dump to verify if this was the cause. There will be a container thread blocked calling HttpSession#removeAttribute or HttpSession#invalidate somewhere in it's stack, waiting on UIThread#terminateThread

When you see that, you need to take a heap dump figure out why the UIThread didn't terminate. You'll need to use the thread dump to find the right UIThread instance to look at.

To make it even harder, the JVM might not give you either of those dumps once it falls in to the continuous GC cycle.

Report message to a moderator

Re: High usage of memory and final CPU collapse [message #1766212 is a reply to message #1760152]

Sat, 17 June 2017 19:26

Vincenzo Caselli

Messages: 208
Registered: January 2012

Senior Member

Hi all,
we were able to analyze a heap dump file and discovered that a big amount of the space was occupied by many instances of FileUploadHandler objects used in the application main view.
Then we put an explicit call to dispose of this object when the view is disposed.
With this update the application seems to behave very well up to now.
Thank you
Vincenzo

Report message to a moderator

Previous Topic:	ApplicationConfiguration is not called anymore, after adding new dependencies and plugins
Next Topic:	Deploy problem with RAP on WebSphere 8.5 for z/OS

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Sat Aug 18 01:17:25 GMT 2018

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter