Eclipse Community Forums: BIRT » OutOfMemory exceptions in birt engine

Home » Archived » BIRT » OutOfMemory exceptions in birt engine

OutOfMemory exceptions in birt engine [message #729551]

Mon, 26 September 2011 09:31

Eclipse User

Hello All

We are doing load testing for our reporting engine. Here is our stack

OS - Linux
CPU's - 2 - 2GHz processors
RAM - 3GB
Weblogic - 10.0.2
Java - jrockit_150_17
BIRT - 3.7

We are running simultaneously 5 huge reports (30,000 records each. rptDocument, pdf, excel). Failing with DIFFERENT out of memory errors.

java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size: 353040, Num elements: 88256
at java.util.ArrayList.ensureCapacity(ArrayList.java:169)
-----------------------------
Caused by: java.lang.OutOfMemoryError: nativeGetNewTLA
at org.apache.batik.css.parser.Scanner.getStringValue(Unknown Source)
at org.apache.batik.css.parser.Parser.number(Unknown Source)

We need to run at-least 16 reports in parallel in production. Any pointers on how to tune the code / configuration

Thanks

[Updated on: Mon, 26 September 2011 10:00] by Moderator

Re: OutOfMemory in birt engine [message #729620 is a reply to message #729551]

Mon, 26 September 2011 12:02

Eclipse User

What are your JVM settings? Do you have -XX:MaxPermSize=256M set?

Jason

On 9/26/2011 9:31 AM, sam wrote:
> Hello All
>
> We are doing load testing for our reporting engine. Here is our stack
>
> OS - Linux CPU's - 2 - 2GHz processors
> RAM - 3GB
> Weblogic - 10.0.2
> Java - jrockit_150_17
> BIRT - 3.7
>
> We are running simultaneously 5 huge reports. Failing with DIFFERENT out
> of memory errors.
>
> java.lang.OutOfMemoryError: allocLargeObjectOrArray - Object size:
> 353040, Num elements: 88256
> at java.util.ArrayList.ensureCapacity(ArrayList.java:169)
> -----------------------------
> Caused by: java.lang.OutOfMemoryError: nativeGetNewTLA
> at org.apache.batik.css.parser.Scanner.getStringValue(Unknown Source)
> at org.apache.batik.css.parser.Parser.number(Unknown Source)
>
> We need to run at-least 16 reports in parallel in production. Any
> pointers on how to tune the code / configuration
>
> Thanks
>

Re: OutOfMemory in birt engine [message #729816 is a reply to message #729620]

Mon, 26 September 2011 23:01

Eclipse User

I have the following in JVM args

-XxcompressedRefs=0 -XXlargeObjectLimit:64k -XXtlaSize:min=64k,preferred=128k -XX:PermSize=128m -XX:MaxPermSize=512m

Thanks

Re: OutOfMemory in birt engine [message #730006 is a reply to message #729816]

Tue, 27 September 2011 09:19

Eclipse User

More Info

Mem Info
MemTotal: 3735296 kB
MemFree: 311840 kB
Buffers: 119728 kB
Cached: 1340652 kB
SwapCached: 0 kB
Active: 2730400 kB
Inactive: 462120 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 3735296 kB
LowFree: 311840 kB
SwapTotal: 4194296 kB
SwapFree: 4194288 kB

CPU Info
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 29
model name : Intel(R) Xeon(R) CPU X7460 @ 2.66GHz
stepping : 8
cpu MHz : 2659.877

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 29
model name : Intel(R) Xeon(R) CPU X7460 @ 2.66GHz
stepping : 8
cpu MHz : 2659.877

Software
Weblogic - 10.0.2
Java - jrockit_150_17
BIRT - 3.7

[Updated on: Tue, 27 September 2011 09:20] by Moderator

Re: OutOfMemory in birt engine [message #730078 is a reply to message #730006]

Tue, 27 September 2011 11:24

Eclipse User

I am not certain this a bug, but we need to track this in bugzilla. Can
you post this info to a bugzilla entry and then post the bug id back here?

Jason

On 9/27/2011 9:19 AM, sam wrote:
> More Info
>
> Mem Info
> MemTotal: 3735296 kB
> MemFree: 311840 kB
> Buffers: 119728 kB
> Cached: 1340652 kB
> SwapCached: 0 kB
> Active: 2730400 kB
> Inactive: 462120 kB
> HighTotal: 0 kB
> HighFree: 0 kB
> LowTotal: 3735296 kB
> LowFree: 311840 kB
> SwapTotal: 4194296 kB
> SwapFree: 4194288 kB
>
> CPU Info
> processor : 0
> vendor_id : GenuineIntel
> cpu family : 6
> model : 29
> model name : Intel(R) Xeon(R) CPU X7460 @ 2.66GHz
> stepping : 8
> cpu MHz : 2659.877
>
> processor : 1
> vendor_id : GenuineIntel
> cpu family : 6
> model : 29
> model name : Intel(R) Xeon(R) CPU X7460 @ 2.66GHz
> stepping : 8
> cpu MHz : 2659.877
>
> Software
> Weblogic - 10.0.2
> Java - jrockit_150_17
> BIRT - 3.7
>

Re: OutOfMemory in birt engine [message #730252 is a reply to message #730078]

Tue, 27 September 2011 21:59

Eclipse User

Logged buzilla entry

359162

Thanks

Re: OutOfMemory exceptions in birt engine [message #730563 is a reply to message #729551]

Wed, 28 September 2011 12:02

Eclipse User

Any pointers if someone worked on similar stuff would be very helpful

Thanks

Re: OutOfMemory in birt engine [message #730610 is a reply to message #730252]

Wed, 28 September 2011 13:16

Eclipse User

If you are using the API, have a look at slide 20 in the BirtMetal ppt
in this dev share entry:
http://www.birt-exchange.org/org/devshare/deploying-birt-reports/1187-eclipsecon-2010-birt-metal-ppt/

Jason

On 9/27/2011 9:59 PM, sam wrote:
> Logged buzilla entry
>
> 359162
>
> Thanks

Re: OutOfMemory exceptions in birt engine [message #731884 is a reply to message #729551]

Mon, 03 October 2011 07:49

Eclipse User

we tried all the options in the BIRTMetal.ppt

Ran out of all the options but still getting OutOfMemory for various reasons

Is BIRT scalable?

Thanks

Re: OutOfMemory exceptions in birt engine [message #731945 is a reply to message #731884]

Mon, 03 October 2011 10:29

Eclipse User

There are a lot of people using BIRT in large scale apps. I will check
with the dev team on this bug.

Jason

On 10/3/2011 7:49 AM, sam wrote:
> we tried all the options in the BIRTMetal.ppt
>
> Ran out of all the options but still getting OutOfMemory for various
> reasons
>
> Is BIRT scalable?
>
> Thanks

Re: OutOfMemory exceptions in birt engine [message #732005 is a reply to message #731945]

Mon, 03 October 2011 12:38

Eclipse User

I will try to put more info on the errors we are getting by t'rrow.

Thanks

Re: OutOfMemory exceptions in birt engine [message #733620 is a reply to message #732005]

Wed, 05 October 2011 04:39

Eclipse User

I get the following error

java.lang.OutOfMemoryError: nativeGetNewTLA
at org.eclipse.birt.data.engine.executor.cache.ResultObjectUtil.readData(ResultObjectUtil.java:175)
at org.eclipse.birt.data.engine.executor.cache.disk.DataFileReader.read(DataFileReader.java:104)
at org.eclipse.birt.data.engine.executor.cache.disk.RowFile.readRowFromFile(RowFile.java:248)
at org.eclipse.birt.data.engine.executor.cache.disk.RowFile.fetch(RowFile.java:209)
at org.eclipse.birt.data.engine.executor.cache.disk.DiskCacheResultSet.nextRow(DiskCacheResultSet.java:102)
at org.eclipse.birt.data.engine.executor.cache.disk.DiskCache.next(DiskCache.java:174)
at org.eclipse.birt.data.engine.executor.cache.SmartCache.next(SmartCache.java:155)
at org.eclipse.birt.data.engine.executor.transform.CachedResultSet.next(CachedResultSet.java:436)
at org.eclipse.birt.data.engine.impl.ResultIterator.hasNextRow(ResultIterator.java:569)
at org.eclipse.birt.data.engine.impl.ResultIterator.nextRow(ResultIterator.java:510)
at org.eclipse.birt.data.engine.impl.ResultIterator.next(ResultIterator.java:460)
at org.eclipse.birt.data.engine.impl.ResultIterator.close(ResultIterator.java:969)
at org.eclipse.birt.data.engine.impl.ResultIterator$1.dataEngineShutdown(ResultIterator.java:200)
at org.eclipse.birt.data.engine.impl.DataEngineImpl.shutdown(DataEngineImpl.java:565)
at org.eclipse.birt.report.data.adapter.impl.DataRequestSessionImpl.shutdown(DataRequestSessionImpl.java:509)
at org.eclipse.birt.report.engine.data.dte.AbstractDataEngine.shutdown(AbstractDataEngine.java:348)
at org.eclipse.birt.report.engine.data.dte.DataGenerationEngine.shutdown(DataGenerationEngine.java:151)
at org.eclipse.birt.report.engine.executor.ExecutionContext.close(ExecutionContext.java:480)
at org.eclipse.birt.report.engine.api.impl.EngineTask.close(EngineTask.java:1540)
at org.eclipse.birt.report.engine.api.impl.RunTask.close(RunTask.java:309)

And I have set the following code to the task (as mentioned in the BirtMetal.ppt)

task.getAppContext().put(DataEngine.MEMORY_USAGE, DataEngine.MEMORY_USAGE_CONSERVATIVE);
task.getAppContext().put(DataEngine.DATA_SET_CACHE_ROW_LIMIT, "0");
task.getAppContext().put(DataEngine.MEMORY_BUFFER_SIZE, "2");

Similarly, I set the options for EngineConfig also...
Starting the test with 2 reports in parallel. And will slowly run more reports if everything goes fine.
One report is running fine without any issues. But takes 1hr. FYI, we generate the report and PDF/Excel along with it.

Thanks

[Updated on: Wed, 05 October 2011 07:40] by Moderator

Re: OutOfMemory exceptions in birt engine [message #733700 is a reply to message #733620]

Wed, 05 October 2011 09:09

Eclipse User

2 reports completed successfully. But took 1hr 45m. For 25000 records (report generation/pdf/excel).
will run 3 reports in parallel and see.

Thanks

Re: OutOfMemory exceptions in birt engine [message #733745 is a reply to message #733700]

Wed, 05 October 2011 10:36

Eclipse User

Can you add this data to the bugzilla entry?

Jason

On 10/5/2011 9:09 AM, sam wrote:
> 2 reports completed successfully. But took 1hr 45m. For 25000 records
> (report generation/pdf/excel).
> will run 3 reports in parallel and see.
>
> Thanks

Re: OutOfMemory exceptions in birt engine [message #753514 is a reply to message #729551]

Thu, 27 October 2011 10:38

Eclipse User

I think we have got a handle on to this.
We tried a sample report with 60K records and the rptDocument took around 20MB. The actual report with which we load tested was taking around 500MB. Looks clearly that the report was not generated efficiently. One issue could be developing the report in 2.6.2 and running in 3.7. Although I am not sure if this is the reason, I found this as a data point. So we re-developed the report on 3.7 and the rptdocument is not crossing 80MB for 60K rows.

And now we successfully generated 15 reports in parallel. and still going. this is in lower env. Would like to reach the limit and will update all. Just as an FYI, our current reporting env is running max 16 reports in production.

Thanks Jason and the DEV team in helping me out

BIRT ROKCS!!!

Thanks again

Previous Topic:	BIRT Report Engine Installation Problem
Next Topic:	BIRT not working with IE8 with tomcat, Object expected error

Goto Forum:

-=] Back to Top [=-

Current Time: Fri Jul 25 09:14:56 EDT 2025

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter