| birt kills tomcat randomly [message #934321] |
Fri, 05 October 2012 21:33  |
 Chris F
Messages: 4
Registered: October 2012 |
Junior Member |
|
|
i'm running birt 2.5.1 w/ tomcat-7 & sun-jdk-1.6 and the apache jk connector, on a gentoo linux box. also running the espreadsheet engine for xls exporting. randomly tomcat becomes unresponsive, the service wont stop and you have to use kill -9 to terminate. The catalina logs usually dont show any errors around the time it stopped, and I haven't had the best luck replicating it. It seems to be fairly random, sometimes it wont happen for weeks sometimes it happens a couple times a day. I have replicated it a handful of time when I run really large reports (but usually when I run large reports it's fine), reports that may take several minutes to generate. I've tried playing around CATALINA_OPTS and JAVA_OPTS in the tomcat config file, and so far it hasn't seemed to help. Sometimes there is a recent error in the catalina log right before it crashes, but most times there is not. Here is one that happened today :
SEVERE: Error processing request
java.nio.BufferOverflowException
at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:327)
at org.apache.coyote.ajp.AjpAprProcessor.output(Unknown Source)
at org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(Unknown Source)
at org.apache.coyote.Response.doWrite(Unknown Source)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(Unknown Source)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(Unknown Source)
at org.apache.catalina.connector.OutputBuffer.doFlush(Unknown Source)
at org.apache.catalina.connector.OutputBuffer.close(Unknown Source)
at org.apache.catalina.connector.Response.finishResponse(Unknown Source)
at org.apache.catalina.connector.CoyoteAdapter.service(Unknown Source)
at org.apache.coyote.ajp.AjpAprProcessor.process(Unknown Source)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Unknown Source)
at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
this is a prod system with at least a thousand users, so it's quite the problem when tomcat freezes and renders the site useless. I've been battling with this for almost a year, it has been stable for the past couple months up until today. No nothing changed at all, no updates, no system files, nothing besides a reboot a few days ago when I was fixing some mysql issues.
I have this in my current /etc/conf.d/tomcat-7
CATALINE_OPTS="-Djava.library.path=/usr/lib -server -Xms1024M -Xmx4096M -XX:MaxPermSize=1024m"
JAVA_OPTS="$JAVA_OPTS -Xms1024M -Xmx4096M -XX:MaxPermSize=1024m"
Sometimes I have gotten memory heap errors around the time tomcat has crashed, so ive probably tried a dozen different opt settings to see if that would help, but it hasn't. i have 32gb of ram on this server so memory is not really an issue.
I know some people might suggests upgrading birt - but of the reports i tested none worked on the newer birt web deployment, and it would take quite some time to port everything over with birt/eclipse and figure out what all is broken on the old reports. Just not a task I've had time to entertain, hoping there's another way to get it stable - for the time being.
I'm open to any suggestions, thoughts, advice, or questions. Please help!
Thanks!
[Updated on: Fri, 05 October 2012 21:42] Report message to a moderator |
|
|
|
| Re: birt kills tomcat random ly [message #935942 is a reply to message #934662] |
Sun, 07 October 2012 13:45  |
donino donino
Messages: 183
Registered: July 2011 |
Senior Member |
|
|
The most likely cause is a memory issue. The 32 GB RAM of your server does not matter, only your JVM heap/permgen memory is important.
I think you should enable JMX services on your JVM. Thus you will be able to use the Java console or Visual VM, it will help to see what's happening on your server. In particular i see you have 4Gb heap memory, you should inspect if the memory used is constant after a Garbage Collect, or keeps rising slowly.
Furthermore, you should set up your JVM so that it creates a heap dump when it is Out Of Memory. Use if necessary Eclipse MAT to analyze a heap dump, especially the "Memory dominator tree" and the "Leak suspect" reports.
Furthermore, i assume your spreadsheet engine is using Apache POI? This library has currently an awful memory management: may be you should sligthly modify your largest reports to limit the size of the result set: add more parameters, limit the dataset rows, etc.
[Updated on: Sun, 07 October 2012 13:48] Report message to a moderator |
|
|
Powered by
FUDForum. Page generated in 0.01393 seconds
] 
Search
Help
Register
Login
Home
