Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » BIRT » birt kills tomcat randomly
birt kills tomcat randomly [message #934321] Fri, 05 October 2012 21:33 Go to next message
Chris F is currently offline Chris FFriend
Messages: 4
Registered: October 2012
Junior Member
i'm running birt 2.5.1 w/ tomcat-7 & sun-jdk-1.6 and the apache jk connector, on a gentoo linux box. also running the espreadsheet engine for xls exporting. randomly tomcat becomes unresponsive, the service wont stop and you have to use kill -9 to terminate. The catalina logs usually dont show any errors around the time it stopped, and I haven't had the best luck replicating it. It seems to be fairly random, sometimes it wont happen for weeks sometimes it happens a couple times a day. I have replicated it a handful of time when I run really large reports (but usually when I run large reports it's fine), reports that may take several minutes to generate. I've tried playing around CATALINA_OPTS and JAVA_OPTS in the tomcat config file, and so far it hasn't seemed to help. Sometimes there is a recent error in the catalina log right before it crashes, but most times there is not. Here is one that happened today :

SEVERE: Error processing request
java.nio.BufferOverflowException
        at java.nio.DirectByteBuffer.put(DirectByteBuffer.java:327)
        at org.apache.coyote.ajp.AjpAprProcessor.output(Unknown Source)
        at org.apache.coyote.ajp.AbstractAjpProcessor$SocketOutputBuffer.doWrite(Unknown Source)
        at org.apache.coyote.Response.doWrite(Unknown Source)
        at org.apache.catalina.connector.OutputBuffer.realWriteBytes(Unknown Source)
        at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(Unknown Source)
        at org.apache.catalina.connector.OutputBuffer.doFlush(Unknown Source)
        at org.apache.catalina.connector.OutputBuffer.close(Unknown Source)
        at org.apache.catalina.connector.Response.finishResponse(Unknown Source)
        at org.apache.catalina.connector.CoyoteAdapter.service(Unknown Source)
        at org.apache.coyote.ajp.AjpAprProcessor.process(Unknown Source)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Unknown Source)
        at org.apache.tomcat.util.net.AprEndpoint$SocketWithOptionsProcessor.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)



this is a prod system with at least a thousand users, so it's quite the problem when tomcat freezes and renders the site useless. I've been battling with this for almost a year, it has been stable for the past couple months up until today. No nothing changed at all, no updates, no system files, nothing besides a reboot a few days ago when I was fixing some mysql issues.

I have this in my current /etc/conf.d/tomcat-7
CATALINE_OPTS="-Djava.library.path=/usr/lib -server -Xms1024M -Xmx4096M -XX:MaxPermSize=1024m"
JAVA_OPTS="$JAVA_OPTS -Xms1024M -Xmx4096M -XX:MaxPermSize=1024m"


Sometimes I have gotten memory heap errors around the time tomcat has crashed, so ive probably tried a dozen different opt settings to see if that would help, but it hasn't. i have 32gb of ram on this server so memory is not really an issue.

I know some people might suggests upgrading birt - but of the reports i tested none worked on the newer birt web deployment, and it would take quite some time to port everything over with birt/eclipse and figure out what all is broken on the old reports. Just not a task I've had time to entertain, hoping there's another way to get it stable - for the time being.

I'm open to any suggestions, thoughts, advice, or questions. Please help!

Thanks!

[Updated on: Fri, 05 October 2012 21:42]

Report message to a moderator

Re: birt kills tomcat random ly [message #934662 is a reply to message #934321] Sat, 06 October 2012 06:38 Go to previous messageGo to next message
Colin Sutton is currently offline Colin SuttonFriend
Messages: 72
Registered: July 2009
Member
Christian Frye <forums-noreply@xxxxxxxx> wrote:
[...]
>
> this is a prod system with at least a thousand users, so it's quite the
> problem when tomcat freezes and renders the site useless. I've been
> battling with this for almost a year, it has been stable for the past
> couple months up until today. No nothing changed at all, no updates, no
> system files, nothing besides a reboot a few days ago when I was fixing some mysql issues.
>
> I have this in my current /etc/conf.d/tomcat-7
>
> CATALINE_OPTS="-Djava.library.path=/usr/lib -server -Xms1024M -Xmx4096M
> -XX:MaxPermSize=1024m"
> JAVA_OPTS="$JAVA_OPTS -Xms1024M -Xmx4096M -XX:MaxPermSize=1024m"
>[...]
Misspelt CATALINA?


--
Colin Sutton
Re: birt kills tomcat random ly [message #935942 is a reply to message #934662] Sun, 07 October 2012 13:45 Go to previous message
donino donino is currently offline donino doninoFriend
Messages: 165
Registered: July 2011
Senior Member
The most likely cause is a memory issue. The 32 GB RAM of your server does not matter, only your JVM heap/permgen memory is important.

I think you should enable JMX services on your JVM. Thus you will be able to use the Java console or Visual VM, it will help to see what's happening on your server. In particular i see you have 4Gb heap memory, you should inspect if the memory used is constant after a Garbage Collect, or keeps rising slowly.

Furthermore, you should set up your JVM so that it creates a heap dump when it is Out Of Memory. Use if necessary Eclipse MAT to analyze a heap dump, especially the "Memory dominator tree" and the "Leak suspect" reports.

Furthermore, i assume your spreadsheet engine is using Apache POI? This library has currently an awful memory management: may be you should sligthly modify your largest reports to limit the size of the result set: add more parameters, limit the dataset rows, etc.

[Updated on: Sun, 07 October 2012 13:48]

Report message to a moderator

Previous Topic:Display Data in a row separated by a comma
Next Topic:[chart] with multiple categories (on x)
Goto Forum:
  


Current Time: Sun Dec 21 04:21:32 GMT 2014

Powered by FUDForum. Page generated in 0.01660 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software