Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » EMF » [CDO] Encounter java.lang.OutOfMemoryError: GC overhead limit exceeded when commit from 4 clients
[CDO] Encounter java.lang.OutOfMemoryError: GC overhead limit exceeded when commit from 4 clients [message #893639] Thu, 05 July 2012 00:13 Go to next message
Apirom Na Nakorn is currently offline Apirom Na Nakorn
Messages: 14
Registered: July 2012
Junior Member
Hi,

I try to commit data(65535 elements for each client and the size of each element is 1KB) from 4 clients simultaneously to 4 resource path under the same repository in CDO Server (4.1), but I get IOTimeoutException so I set the new timout in session.options().getNet4jProtocol().setTimeout() to ISignalProtocol.NO_TIMEOUT and run again. This time I get an exception as follow :

Caused by: org.eclipse.emf.cdo.util.CommitException: Rollback in DBStore: java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.lang.StringCoding$StringDecoder.decode(Unknown Source)
at java.lang.StringCoding.decode(Unknown Source)
at java.lang.String.<init>(Unknown Source)
at java.lang.String.<init>(Unknown Source)
at com.mysql.jdbc.PreparedStatement.asSql(PreparedStatement.java:754)
at com.mysql.jdbc.PreparedStatement.asSql(PreparedStatement.java:707)
at com.mysql.jdbc.PreparedStatement.toString(PreparedStatement.java:4718)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.ObjectTypeTable.putObjectType(ObjectTypeTable.java:116)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.DelegatingObjectTypeMapper.putObjectType(DelegatingObjectTypeMapper.java:68)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.AbstractHorizontalMappingStrategy.putObjectType(AbstractHorizontalMappingStrategy.java:81)
at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.HorizontalBranchingClassMapping.writeRevision(HorizontalBranchingClassMapping.java:763)
at org.eclipse.emf.cdo.server.internal.db.DBStoreAccessor.writeRevision(DBStoreAccessor.java:583)
at org.eclipse.emf.cdo.server.internal.db.DBStoreAccessor.writeRevisions(DBStoreAccessor.java:563)
at org.eclipse.emf.cdo.spi.server.StoreAccessor.doWrite(StoreAccessor.java:94)
at org.eclipse.emf.cdo.spi.server.StoreAccessorBase.write(StoreAccessorBase.java:149)

How to solve this problem ?

Thanks in advance

Neung

[Updated on: Thu, 05 July 2012 00:17]

Report message to a moderator

Re: CDO] Encounter java.lang.OutOfMemoryError: GC overhead limit exceeded when commit from 4 clients [message #893642 is a reply to message #893639] Thu, 05 July 2012 00:57 Go to previous messageGo to next message
Eike Stepper is currently offline Eike Stepper
Messages: 5480
Registered: July 2009
Senior Member
Am 05.07.2012 06:13, schrieb Apirom Na Nakorn:
> Hi,
>
> I try to commit data(65535 elements for each client and the size of each element is 1KB) from 4 clients
> simultaneously to 4 resource path under the same repository in CDO Server (4.1), but I get IOTimeoutException so I set
> the new timout in session.options().getNet4jProtocol().setTimeout() to ISignalProtocol.NO_TIMEOUT and run again.
Hmm, I'd have expected a call setCommitTimeout() fixes it.

> This time I get an exception as follow :
>
> Caused by: org.eclipse.emf.cdo.util.CommitException: Rollback in DBStore: java.lang.OutOfMemoryError: GC overhead
> limit exceeded
I've never seen this exception before but Google brings up answers like this:
http://stackoverflow.com/questions/5839359/java-lang-outofmemoryerror-gc-overhead-limit-exceeded

What's the exact command line you're starting the server with?

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper


> at java.lang.StringCoding$StringDecoder.decode(Unknown Source)
> at java.lang.StringCoding.decode(Unknown Source)
> at java.lang.String.<init>(Unknown Source)
> at java.lang.String.<init>(Unknown Source)
> at com.mysql.jdbc.PreparedStatement.asSql(PreparedStatement.java:754)
> at com.mysql.jdbc.PreparedStatement.asSql(PreparedStatement.java:707)
> at com.mysql.jdbc.PreparedStatement.toString(PreparedStatement.java:4718)
> at org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.ObjectTypeTable.putObjectType(ObjectTypeTable.java:116)
> at
> org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.DelegatingObjectTypeMapper.putObjectType(DelegatingObjectTypeMapper.java:68)
> at
> org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.AbstractHorizontalMappingStrategy.putObjectType(AbstractHorizontalMappingStrategy.java:81)
> at
> org.eclipse.emf.cdo.server.internal.db.mapping.horizontal.HorizontalBranchingClassMapping.writeRevision(HorizontalBranchingClassMapping.java:763)
> at org.eclipse.emf.cdo.server.internal.db.DBStoreAccessor.writeRevision(DBStoreAccessor.java:583)
> at org.eclipse.emf.cdo.server.internal.db.DBStoreAccessor.writeRevisions(DBStoreAccessor.java:563)
> at org.eclipse.emf.cdo.spi.server.StoreAccessor.doWrite(StoreAccessor.java:94)
> at org.eclipse.emf.cdo.spi.server.StoreAccessorBase.write(StoreAccessorBase.java:149)
>
> How to solve this problem ?
>
> Thanks in advance
>
> Neung
Re: [CDO]Encounter java.lang.OutOfMemoryError: GC overhead limit exceeded when commit from 4 clients [message #893644 is a reply to message #893642] Thu, 05 July 2012 01:09 Go to previous messageGo to next message
Apirom Na Nakorn is currently offline Apirom Na Nakorn
Messages: 14
Registered: July 2012
Junior Member
Dear Eike

I use the following options to start CDO Server on Mac OSX (with JVM 64bits):

-debug
-console
-consoleLog
-Xms40m
-Xmx6144m
-XstartOnFirstThread
-Dorg.eclipse.emf.cdo.server.browser.port=7777
-Dorg.osgi.service.http.port=8080
-Dosgi.console.enable.builtin=true
-Ddebug=true
-Dorg.eclipse.swt.internal.carbon.smallFonts

I run CDO server on Mac OSX with the following java :

java version "1.6.0_33"
Java(TM) SE Runtime Environment (build 1.6.0_33-b03-424-11M3720)
Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode)

Thank you

Best Regards

Neung

[Updated on: Thu, 05 July 2012 01:09]

Report message to a moderator

Re: [CDO]Encounter java.lang.OutOfMemoryError: GC overhead limit exceeded when commit from 4 clients [message #893663 is a reply to message #893644] Thu, 05 July 2012 03:32 Go to previous messageGo to next message
Eike Stepper is currently offline Eike Stepper
Messages: 5480
Registered: July 2009
Senior Member
Am 05.07.2012 07:09, schrieb Apirom Na Nakorn:
> Dear Eike
>
> I use the following options to start CDO Server on Mac OSX (with JVM 64bits):
>
> -debug
> -console
> -consoleLog
> -Xms40m
> -Xmx6144m
> -XstartOnFirstThread
> -Dorg.eclipse.emf.cdo.server.browser.port=7777
> -Dorg.osgi.service.http.port=8080
> -Dosgi.console.enable.builtin=true
> -Ddebug=true
> -Dorg.eclipse.swt.internal.carbon.smallFonts
>
> I run CDO server on Mac OSX with the following java :
>
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b03-424-11M3720)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03-424, mixed mode)
>
> Thank you
Have you tried the advice from the article I linked in my previous post?

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Re: [CDO]Encounter java.lang.OutOfMemoryError: GC overhead limit exceeded when commit from 4 clients [message #893667 is a reply to message #893644] Thu, 05 July 2012 04:05 Go to previous messageGo to next message
Apirom Na Nakorn is currently offline Apirom Na Nakorn
Messages: 14
Registered: July 2012
Junior Member
Dear Eike

I start CDO Server with option -XX:-UseGCOverheadLimit and wait for the server to run. Now, it's about 2 hours and the server is running without throwing the exception, but 1 client already died because out of memory error. BTW, the accept solution for the post you mention suggest that

1. Specify more memory like you mentioned, try something in between like -Xmx512m first
2. Work with smaller batches of HashMap objects to process at once if possible
3. If you have a lot of duplicate strings, use intern() on them before putting them into the HashMap
4. Use the HashMap(int initialCapacity, float loadFactor) constructor to tune for your case

for no.1 I already done with 6GB, but for the rest I'm not sure how to do it. So, I go with the option -XX:-UseGCOverheadLimit

Thanks in advance

Neung
Re: [CDO]Encounter java.lang.OutOfMemoryError: GC overhead limit exceeded when commit from 4 clients [message #893675 is a reply to message #893667] Thu, 05 July 2012 04:25 Go to previous messageGo to next message
Eike Stepper is currently offline Eike Stepper
Messages: 5480
Registered: July 2009
Senior Member
Am 05.07.2012 10:05, schrieb Apirom Na Nakorn:
> Dear Eike
>
> I start CDO Server with option -XX:-UseGCOverheadLimit and wait for the server to run. Now, it's about 2 hours and
> the server is running without throwing the exception, but 1 client already died because out of memory error. BTW, the
> accept solution for the post you mention suggest that
>
> 1. Specify more memory like you mentioned, try something in between like -Xmx512m first
> 2. Work with smaller batches of HashMap objects to process at once if possible
> 3. If you have a lot of duplicate strings, use intern() on them before putting them into the HashMap
> 4. Use the HashMap(int initialCapacity, float loadFactor) constructor to tune for your case
>
> for no.1 I already done with 6GB, but for the rest I'm not sure how to do it. So, I go with the option
> -XX:-UseGCOverheadLimit
I mostly wanted to refer to the -XX:-UseGCOverheadLimit option.

You should generally try to keep the commit size as small as possible, depending on the heap size that you assign.

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Re: [CDO]Encounter java.lang.OutOfMemoryError: GC overhead limit exceeded when commit from 4 clients [message #893690 is a reply to message #893675] Thu, 05 July 2012 05:43 Go to previous messageGo to next message
Apirom Na Nakorn is currently offline Apirom Na Nakorn
Messages: 14
Registered: July 2012
Junior Member
I will wait until CDO server finish.
You mean that if I have 65535 elements to commit then I should split them into many commits and each commit contains small number of elements.

Thank you

Neung
Re: [CDO]Encounter java.lang.OutOfMemoryError: GC overhead limit exceeded when commit from 4 clients [message #893693 is a reply to message #893690] Thu, 05 July 2012 05:56 Go to previous messageGo to next message
Apirom Na Nakorn is currently offline Apirom Na Nakorn
Messages: 14
Registered: July 2012
Junior Member
Dear Eike

After the long running, CDO Server throw the exception as follow :

monitor-timer [debug.signal] Progress of signal 10: totalWork=100, work=0.554
Exception in thread "monitor-timer" java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:632)
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:97)
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
at org.eclipse.internal.net4j.buffer.Buffer.<init>(Buffer.java:58)
at org.eclipse.internal.net4j.buffer.BufferFactory.doProvideBuffer(BufferFactory.java:37)
at org.eclipse.internal.net4j.buffer.BufferProvider.provideBuffer(BufferProvider.java:51)
at org.eclipse.internal.net4j.buffer.BufferPool.doProvideBuffer(BufferPool.java:137)
at org.eclipse.internal.net4j.buffer.BufferProvider.provideBuffer(BufferProvider.java:51)
at org.eclipse.spi.net4j.Connector.provideBuffer(Connector.java:294)
at org.eclipse.spi.net4j.Channel.provideBuffer(Channel.java:241)
at org.eclipse.net4j.signal.SignalProtocol$SignalOutputStream$1.provideBuffer(SignalProtocol.java:607)
at org.eclipse.net4j.buffer.BufferOutputStream.ensureBufferPrivate(BufferOutputStream.java:209)
at org.eclipse.net4j.buffer.BufferOutputStream.write(BufferOutputStream.java:105)
at java.io.DataOutputStream.writeInt(DataOutputStream.java:180)
at org.eclipse.net4j.signal.MonitorProgressRequest.requesting(MonitorProgressRequest.java:47)
at org.eclipse.net4j.signal.Request.doExtendedOutput(Request.java:65)
at org.eclipse.net4j.signal.Signal.doOutput(Signal.java:296)
at org.eclipse.net4j.signal.Request.doExecute(Request.java:57)
at org.eclipse.net4j.signal.SignalActor.execute(SignalActor.java:51)
at org.eclipse.net4j.signal.Signal.runSync(Signal.java:251)
at org.eclipse.net4j.signal.SignalProtocol.startSignal(SignalProtocol.java:433)
at org.eclipse.net4j.signal.Request.sendAsync(Request.java:51)
at org.eclipse.net4j.signal.IndicationWithMonitoring$ReportingMonitor.sendProgress(IndicationWithMonitoring.java:161)
at org.eclipse.net4j.signal.IndicationWithMonitoring$ReportingMonitor.access$0(IndicationWithMonitoring.java:157)
at org.eclipse.net4j.signal.IndicationWithMonitoring$ReportingMonitor$1.run(IndicationWithMonitoring.java:127)
at java.util.TimerThread.mainLoop(Timer.java:512)
at java.util.TimerThread.run(Timer.java:462)

Is it possible to solve this problem ?

Thank you

Neung
Re: [CDO]Encounter java.lang.OutOfMemoryError: GC overhead limit exceeded when commit from 4 clients [message #893695 is a reply to message #893690] Thu, 05 July 2012 06:00 Go to previous messageGo to next message
Eike Stepper is currently offline Eike Stepper
Messages: 5480
Registered: July 2009
Senior Member
Am 05.07.2012 11:43, schrieb Apirom Na Nakorn:
> I will wait until CDO server finish. You mean that if I have 65535 elements to commit then I should split them into
> many commits and each commit contains small number of elements.
Yes, in principle. The server does not dispose of any internal data that's accumulated during processing of a single
commit operation until the operation is finished. It's not so much a problem of one large commit but of several
concurrent large commits. An adequate commit size depends on various things such as maximum number of concurrently
committing clients, heap space settings or the inability to break up the graph due without creating dangling or forward
refs.

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Re: [CDO]Encounter java.lang.OutOfMemoryError: GC overhead limit exceeded when commit from 4 clients [message #893700 is a reply to message #893693] Thu, 05 July 2012 06:09 Go to previous messageGo to next message
Eike Stepper is currently offline Eike Stepper
Messages: 5480
Registered: July 2009
Senior Member
Am 05.07.2012 11:56, schrieb Apirom Na Nakorn:
> Dear Eike
>
> After the long running, CDO Server throw the exception as follow :
>
> monitor-timer [debug.signal] Progress of signal 10: totalWork=100, work=0.554
> Exception in thread "monitor-timer" java.lang.OutOfMemoryError: Direct buffer memory
> at java.nio.Bits.reserveMemory(Bits.java:632)
> at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:97)
> at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
> at org.eclipse.internal.net4j.buffer.Buffer.<init>(Buffer.java:58)
> at org.eclipse.internal.net4j.buffer.BufferFactory.doProvideBuffer(BufferFactory.java:37)
> at org.eclipse.internal.net4j.buffer.BufferProvider.provideBuffer(BufferProvider.java:51)
> at org.eclipse.internal.net4j.buffer.BufferPool.doProvideBuffer(BufferPool.java:137)
> at org.eclipse.internal.net4j.buffer.BufferProvider.provideBuffer(BufferProvider.java:51)
> at org.eclipse.spi.net4j.Connector.provideBuffer(Connector.java:294)
> at org.eclipse.spi.net4j.Channel.provideBuffer(Channel.java:241)
> at org.eclipse.net4j.signal.SignalProtocol$SignalOutputStream$1.provideBuffer(SignalProtocol.java:607)
> at org.eclipse.net4j.buffer.BufferOutputStream.ensureBufferPrivate(BufferOutputStream.java:209)
> at org.eclipse.net4j.buffer.BufferOutputStream.write(BufferOutputStream.java:105)
> at java.io.DataOutputStream.writeInt(DataOutputStream.java:180)
> at org.eclipse.net4j.signal.MonitorProgressRequest.requesting(MonitorProgressRequest.java:47)
> at org.eclipse.net4j.signal.Request.doExtendedOutput(Request.java:65)
> at org.eclipse.net4j.signal.Signal.doOutput(Signal.java:296)
> at org.eclipse.net4j.signal.Request.doExecute(Request.java:57)
> at org.eclipse.net4j.signal.SignalActor.execute(SignalActor.java:51)
> at org.eclipse.net4j.signal.Signal.runSync(Signal.java:251)
> at org.eclipse.net4j.signal.SignalProtocol.startSignal(SignalProtocol.java:433)
> at org.eclipse.net4j.signal.Request.sendAsync(Request.java:51)
> at org.eclipse.net4j.signal.IndicationWithMonitoring$ReportingMonitor.sendProgress(IndicationWithMonitoring.java:161)
> at org.eclipse.net4j.signal.IndicationWithMonitoring$ReportingMonitor.access$0(IndicationWithMonitoring.java:157)
> at org.eclipse.net4j.signal.IndicationWithMonitoring$ReportingMonitor$1.run(IndicationWithMonitoring.java:127)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
>
> Is it possible to solve this problem ?
I fear not without analyzing your server with a memory profiler ;-(

You could change the BufferPool to keep preallocated buffers around. How many buffers are in your pool when it goes on
strike?

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Re: [CDO]Encounter java.lang.OutOfMemoryError: GC overhead limit exceeded when commit from 4 clients [message #893708 is a reply to message #893700] Thu, 05 July 2012 06:34 Go to previous messageGo to next message
Apirom Na Nakorn is currently offline Apirom Na Nakorn
Messages: 14
Registered: July 2012
Junior Member
Dear Eike

I am quite new to CDO and Net4J, how can I find this number of preallocated buffers for BufferPool ?

I grab some of the log before the exception as follow:

monitor-timer [debug.channel] Handling buffer: Buffer@32,500[PUTTING] --> Channel[1, SERVER, cdo]
monitor-timer [debug.signal] ================ Requesting: Signal[protocol=cdo, id=-3, name=MonitorProgressRequest, correlation=15,849]
monitor-timer [debug.signal] Progress of signal 10: totalWork=100, work=0.554
Exception in thread "monitor-timer" java.lang.OutOfMemoryError: Direct buffer memory

Is 32,500 be the number of buffers just before the exception ?

Thanks in advance

Neung
Re: [CDO]Encounter java.lang.OutOfMemoryError: GC overhead limit exceeded when commit from 4 clients [message #893716 is a reply to message #893708] Thu, 05 July 2012 06:52 Go to previous messageGo to next message
Eike Stepper is currently offline Eike Stepper
Messages: 5480
Registered: July 2009
Senior Member
Am 05.07.2012 12:34, schrieb Apirom Na Nakorn:
> Dear Eike
>
> I am quite new to CDO and Net4J, how can I find this number of preallocated buffers for BufferPool ?
>
> I grab some of the log before the exception as follow:
>
> monitor-timer [debug.channel] Handling buffer: Buffer@32,500[PUTTING] --> Channel[1, SERVER, cdo]
> monitor-timer [debug.signal] ================ Requesting: Signal[protocol=cdo, id=-3, name=MonitorProgressRequest,
> correlation=15,849]
> monitor-timer [debug.signal] Progress of signal 10: totalWork=100, work=0.554
> Exception in thread "monitor-timer" java.lang.OutOfMemoryError: Direct buffer memory
>
> Is 32,500 be the number of buffers just before the exception ?
No. You need to find out the value of org.eclipse.internal.net4j.buffer.BufferPool.pooledBuffers before or when the
exception occurs, or continuously.

Googling for "java.lang.OutOfMemoryError: Direct buffer memory" immediately gave me a hint about this JVM option:
-XX:MaxDirectMemorySize=<size>

Do you know Google? :P

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Re: [CDO]Encounter java.lang.OutOfMemoryError: GC overhead limit exceeded when commit from 4 clients [message #893728 is a reply to message #893716] Thu, 05 July 2012 07:19 Go to previous message
Apirom Na Nakorn is currently offline Apirom Na Nakorn
Messages: 14
Registered: July 2012
Junior Member
Dear Eike

Thank you, I will try that option

Neung Smile
Previous Topic:Model Change Notification, Adapters, GEF, ArrayStoreException
Next Topic:[emf] don't use other edit plug-ins as delegated resource locators
Goto Forum:
  


Current Time: Wed Jul 23 05:54:13 EDT 2014

Powered by FUDForum. Page generated in 0.02675 seconds