| High usage of memory and final CPU collapse [message #1759917] |
Wed, 19 April 2017 22:11  |
|
Hi,
we have a RAP (based on 3.x workbench, with Perspective, Views, StatusLine, Jobs and so on) Application deployed on Tomcat over a virtualized Linux system with 8 Gb RAM + 2 Gb Swap + 2 CPUs.
The application serves thousands of access per day and server is always on (no nightly restart).
Tomcat is launched with the following memory options:
-Xms256m -Xmx2048m -XX:PermSize=128m -XX:MaxPermSize=512m
After some (about 6-10) day of usage the memory of Tomcat process goes far beyond the Xmx limit; here are the values showed with Linux 'top' command:
VIRT=7300m RES=4.8g
Then, at some point, the CPU usage goes to 100%, both the application and Tomcat manager are unreachable and the only option is restarting Tomcat (not the whole system) and this restores to normal values and usage.
We used some profiling tools (JProfiler and MAT) and did some optimizations based on analyzing the Heap Dump, but still didn't find what is causing the real problem.
So we have some questions about RAP:
- are there some guidelines that we can follow in order to make sure that every user session is cleaned correctly?
- what tools or techniques can we use to analyze deeply the problem?
- why Heap dumps are so small (just about some hundreds of Mb in size) with respect to the total RAM (well beyond 2 Gb) usage by the Tomcat process? how can we inspect all the process memory?
Any suggestion is highly welcome!
Thank you very much in advance.
Vincenzo
|
|
|
|
|
|
| Re: High usage of memory and final CPU collapse [message #1760024 is a reply to message #1759988] |
Fri, 21 April 2017 10:10  |
|
Hi Chris,
indeed the application makes large use of Jobs and asynch threads to update the UI.
Are there guidelines on how to ensure that these threads are cleaned correctly when problems arise (browser closed improperly, connection problems, ... ) ?
How can non-heap memory leaks can be examined?
Thank you very much
Vincenzo
|
|
|
|
| Re: High usage of memory and final CPU collapse [message #1760151 is a reply to message #1760149] |
Sun, 23 April 2017 22:11 |
|
Chris,
you described exactly what is happening, in fact, after days of correct working, in a matter of 20-30 minutes:
- log shows huge times for every call
- the GC is taking all the CPU time
- the application, along with Tomcat Manager, is unresponsive
What strategy may be set up in this case?
Keeping a per-session list of references to threads and Jobs and cancel or stop them in some disposing event may be of help?
What event may be used in order to catch also a badly closed/crashed client browser?
Any better strategies?
Thank you
|
|
|
|
| Re: High usage of memory and final CPU collapse [message #1766212 is a reply to message #1760152] |
Sat, 17 June 2017 19:26 |
|
Hi all,
we were able to analyze a heap dump file and discovered that a big amount of the space was occupied by many instances of FileUploadHandler objects used in the application main view.
Then we put an explicit call to dispose of this object when the view is disposed.
With this update the application seems to behave very well up to now.
Thank you
Vincenzo
|
|
|
Powered by
FUDForum. Page generated in 0.02293 seconds
Search
Help
Register
Login
Home
