Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » BIRT » OutOfMemory exceptions in birt engine
OutOfMemory exceptions in birt engine [message #729551] Mon, 26 September 2011 13:31 Go to next message
sam va is currently offline sam va
Messages: 121
Registered: July 2011
Senior Member
Hello All

We are doing load testing for our reporting engine. Here is our stack

OS - Linux
CPU's - 2 - 2GHz processors
RAM - 3GB
Weblogic - 10.0.2
Java - jrockit_150_17
BIRT - 3.7

We are running simultaneously 5 huge reports (30,000 records each. rptDocument, pdf, excel). Failing with DIFFERENT out of memory errors.

java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 353040, Num elements: 88256
at java.util.ArrayList.ensureCapacity(ArrayList.java:169)
-----------------------------
Caused by: java.lang.OutOfMemoryError: nativeGetNewTLA
at org.apache.batik.css.parser.Scanner.getStringValue(Unknown Source)
at org.apache.batik.css.parser.Parser.number(Unknown Source)

We need to run at-least 16 reports in parallel in production. Any pointers on how to tune the code / configuration

Thanks

[Updated on: Mon, 26 September 2011 14:00]

Report message to a moderator

Re: OutOfMemory in birt engine [message #729620 is a reply to message #729551] Mon, 26 September 2011 16:02 Go to previous messageGo to next message
Jason Weathersby is currently offline Jason Weathersby
Messages: 9167
Registered: July 2009
Senior Member

What are your JVM settings? Do you have -XX:MaxPermSize=256M set?

Jason

On 9/26/2011 9:31 AM, sam wrote:
> Hello All
>
> We are doing load testing for our reporting engine. Here is our stack
>
> OS - Linux CPU's - 2 - 2GHz processors
> RAM - 3GB
> Weblogic - 10.0.2
> Java - jrockit_150_17
> BIRT - 3.7
>
> We are running simultaneously 5 huge reports. Failing with DIFFERENT out
> of memory errors.
>
> java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size:
> 353040, Num elements: 88256
> at java.util.ArrayList.ensureCapacity(ArrayList.java:169)
> -----------------------------
> Caused by: java.lang.OutOfMemoryError: nativeGetNewTLA
> at org.apache.batik.css.parser.Scanner.getStringValue(Unknown Source)
> at org.apache.batik.css.parser.Parser.number(Unknown Source)
>
> We need to run at-least 16 reports in parallel in production. Any
> pointers on how to tune the code / configuration
>
> Thanks
>
Re: OutOfMemory in birt engine [message #729816 is a reply to message #729620] Tue, 27 September 2011 03:01 Go to previous messageGo to next message
sam va is currently offline sam va
Messages: 121
Registered: July 2011
Senior Member
I have the following in JVM args

-XxcompressedRefs=0 -XXlargeObjectLimit:64k -XXtlaSize:min=64k,preferred=128k -XX:PermSize=128m -XX:MaxPermSize=512m

Thanks
Re: OutOfMemory in birt engine [message #730006 is a reply to message #729816] Tue, 27 September 2011 13:19 Go to previous messageGo to next message
sam va is currently offline sam va
Messages: 121
Registered: July 2011
Senior Member
More Info

Mem Info
MemTotal: 3735296 kB
MemFree: 311840 kB
Buffers: 119728 kB
Cached: 1340652 kB
SwapCached: 0 kB
Active: 2730400 kB
Inactive: 462120 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 3735296 kB
LowFree: 311840 kB
SwapTotal: 4194296 kB
SwapFree: 4194288 kB

CPU Info
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 29
model name : Intel(R) Xeon(R) CPU X7460 @ 2.66GHz
stepping : 8
cpu MHz : 2659.877

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 29
model name : Intel(R) Xeon(R) CPU X7460 @ 2.66GHz
stepping : 8
cpu MHz : 2659.877

Software
Weblogic - 10.0.2
Java - jrockit_150_17
BIRT - 3.7

[Updated on: Tue, 27 September 2011 13:20]

Report message to a moderator

Re: OutOfMemory in birt engine [message #730078 is a reply to message #730006] Tue, 27 September 2011 15:24 Go to previous messageGo to next message
Jason Weathersby is currently offline Jason Weathersby
Messages: 9167
Registered: July 2009
Senior Member

I am not certain this a bug, but we need to track this in bugzilla. Can
you post this info to a bugzilla entry and then post the bug id back here?

Jason

On 9/27/2011 9:19 AM, sam wrote:
> More Info
>
> Mem Info
> MemTotal: 3735296 kB
> MemFree: 311840 kB
> Buffers: 119728 kB
> Cached: 1340652 kB
> SwapCached: 0 kB
> Active: 2730400 kB
> Inactive: 462120 kB
> HighTotal: 0 kB
> HighFree: 0 kB
> LowTotal: 3735296 kB
> LowFree: 311840 kB
> SwapTotal: 4194296 kB
> SwapFree: 4194288 kB
>
> CPU Info
> processor : 0
> vendor_id : GenuineIntel
> cpu family : 6
> model : 29
> model name : Intel(R) Xeon(R) CPU X7460 @ 2.66GHz
> stepping : 8
> cpu MHz : 2659.877
>
> processor : 1
> vendor_id : GenuineIntel
> cpu family : 6
> model : 29
> model name : Intel(R) Xeon(R) CPU X7460 @ 2.66GHz
> stepping : 8
> cpu MHz : 2659.877
>
> Software
> Weblogic - 10.0.2
> Java - jrockit_150_17
> BIRT - 3.7
>
Re: OutOfMemory in birt engine [message #730252 is a reply to message #730078] Wed, 28 September 2011 01:59 Go to previous messageGo to next message
sam va is currently offline sam va
Messages: 121
Registered: July 2011
Senior Member
Logged buzilla entry

359162

Thanks
Re: OutOfMemory exceptions in birt engine [message #730563 is a reply to message #729551] Wed, 28 September 2011 16:02 Go to previous messageGo to next message
sam va is currently offline sam va
Messages: 121
Registered: July 2011
Senior Member
Any pointers if someone worked on similar stuff would be very helpful

Thanks
Re: OutOfMemory in birt engine [message #730610 is a reply to message #730252] Wed, 28 September 2011 17:16 Go to previous messageGo to next message
Jason Weathersby is currently offline Jason Weathersby
Messages: 9167
Registered: July 2009
Senior Member

If you are using the API, have a look at slide 20 in the BirtMetal ppt
in this dev share entry:
http://www.birt-exchange.org/org/devshare/deploying-birt-reports/1187-eclipsecon-2010-birt-metal-ppt/

Jason

On 9/27/2011 9:59 PM, sam wrote:
> Logged buzilla entry
>
> 359162
>
> Thanks
Re: OutOfMemory exceptions in birt engine [message #731884 is a reply to message #729551] Mon, 03 October 2011 11:49 Go to previous messageGo to next message
sam va is currently offline sam va
Messages: 121
Registered: July 2011
Senior Member
we tried all the options in the BIRTMetal.ppt

Ran out of all the options but still getting OutOfMemory for various reasons

Is BIRT scalable?

Thanks
Re: OutOfMemory exceptions in birt engine [message #731945 is a reply to message #731884] Mon, 03 October 2011 14:29 Go to previous messageGo to next message
Jason Weathersby is currently offline Jason Weathersby
Messages: 9167
Registered: July 2009
Senior Member

There are a lot of people using BIRT in large scale apps. I will check
with the dev team on this bug.

Jason

On 10/3/2011 7:49 AM, sam wrote:
> we tried all the options in the BIRTMetal.ppt
>
> Ran out of all the options but still getting OutOfMemory for various
> reasons
>
> Is BIRT scalable?
>
> Thanks
Re: OutOfMemory exceptions in birt engine [message #732005 is a reply to message #731945] Mon, 03 October 2011 16:38 Go to previous messageGo to next message
sam va is currently offline sam va
Messages: 121
Registered: July 2011
Senior Member
I will try to put more info on the errors we are getting by t'rrow.

Thanks
Re: OutOfMemory exceptions in birt engine [message #733620 is a reply to message #732005] Wed, 05 October 2011 08:39 Go to previous messageGo to next message
sam va is currently offline sam va
Messages: 121
Registered: July 2011
Senior Member
I get the following error

java.lang.OutOfMemoryError: nativeGetNewTLA
at org.eclipse.birt.data.engine.executor.cache.ResultObjectUtil.readData(ResultObjectUtil.java:175)
at org.eclipse.birt.data.engine.executor.cache.disk.DataFileReader.read(DataFileReader.java:104)
at org.eclipse.birt.data.engine.executor.cache.disk.RowFile.readRowFromFile(RowFile.java:248)
at org.eclipse.birt.data.engine.executor.cache.disk.RowFile.fetch(RowFile.java:209)
at org.eclipse.birt.data.engine.executor.cache.disk.DiskCacheResultSet.nextRow(DiskCacheResultSet.java:102)
at org.eclipse.birt.data.engine.executor.cache.disk.DiskCache.next(DiskCache.java:174)
at org.eclipse.birt.data.engine.executor.cache.SmartCache.next(SmartCache.java:155)
at org.eclipse.birt.data.engine.executor.transform.CachedResultSet.next(CachedResultSet.java:436)
at org.eclipse.birt.data.engine.impl.ResultIterator.hasNextRow(ResultIterator.java:569)
at org.eclipse.birt.data.engine.impl.ResultIterator.nextRow(ResultIterator.java:510)
at org.eclipse.birt.data.engine.impl.ResultIterator.next(ResultIterator.java:460)
at org.eclipse.birt.data.engine.impl.ResultIterator.close(ResultIterator.java:969)
at org.eclipse.birt.data.engine.impl.ResultIterator$1.dataEngineShutdown(ResultIterator.java:200)
at org.eclipse.birt.data.engine.impl.DataEngineImpl.shutdown(DataEngineImpl.java:565)
at org.eclipse.birt.report.data.adapter.impl.DataRequestSessionImpl.shutdown(DataRequestSessionImpl.java:509)
at org.eclipse.birt.report.engine.data.dte.AbstractDataEngine.shutdown(AbstractDataEngine.java:348)
at org.eclipse.birt.report.engine.data.dte.DataGenerationEngine.shutdown(DataGenerationEngine.java:151)
at org.eclipse.birt.report.engine.executor.ExecutionContext.close(ExecutionContext.java:480)
at org.eclipse.birt.report.engine.api.impl.EngineTask.close(EngineTask.java:1540)
at org.eclipse.birt.report.engine.api.impl.RunTask.close(RunTask.java:309)

And I have set the following code to the task (as mentioned in the BirtMetal.ppt)

task.getAppContext().put(DataEngine.MEMORY_USAGE, DataEngine.MEMORY_USAGE_CONSERVATIVE);
task.getAppContext().put(DataEngine.DATA_SET_CACHE_ROW_LIMIT, "0");
task.getAppContext().put(DataEngine.MEMORY_BUFFER_SIZE, "2");

Similarly, I set the options for EngineConfig also...
Starting the test with 2 reports in parallel. And will slowly run more reports if everything goes fine.
One report is running fine without any issues. But takes 1hr. FYI, we generate the report and PDF/Excel along with it.


Thanks

[Updated on: Wed, 05 October 2011 11:40]

Report message to a moderator

Re: OutOfMemory exceptions in birt engine [message #733700 is a reply to message #733620] Wed, 05 October 2011 13:09 Go to previous messageGo to next message
sam va is currently offline sam va
Messages: 121
Registered: July 2011
Senior Member
2 reports completed successfully. But took 1hr 45m. For 25000 records (report generation/pdf/excel).
will run 3 reports in parallel and see.

Thanks
Re: OutOfMemory exceptions in birt engine [message #733745 is a reply to message #733700] Wed, 05 October 2011 14:36 Go to previous messageGo to next message
Jason Weathersby is currently offline Jason Weathersby
Messages: 9167
Registered: July 2009
Senior Member

Can you add this data to the bugzilla entry?

Jason

On 10/5/2011 9:09 AM, sam wrote:
> 2 reports completed successfully. But took 1hr 45m. For 25000 records
> (report generation/pdf/excel).
> will run 3 reports in parallel and see.
>
> Thanks
Re: OutOfMemory exceptions in birt engine [message #753514 is a reply to message #729551] Thu, 27 October 2011 14:38 Go to previous message
sam va is currently offline sam va
Messages: 121
Registered: July 2011
Senior Member
I think we have got a handle on to this.
We tried a sample report with 60K records and the rptDocument took around 20MB. The actual report with which we load tested was taking around 500MB. Looks clearly that the report was not generated efficiently. One issue could be developing the report in 2.6.2 and running in 3.7. Although I am not sure if this is the reason, I found this as a data point. So we re-developed the report on 3.7 and the rptdocument is not crossing 80MB for 60K rows.

And now we successfully generated 15 reports in parallel. and still going. this is in lower env. Would like to reach the limit and will update all. Just as an FYI, our current reporting env is running max 16 reports in production.

Thanks Jason and the DEV team in helping me out

BIRT ROKCS!!!

Thanks again
Previous Topic:BIRT Report Engine Installation Problem
Next Topic:BIRT not working with IE8 with tomcat, Object expected error
Goto Forum:
  


Current Time: Sat Sep 20 14:11:58 GMT 2014

Powered by FUDForum. Page generated in 0.11016 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software