Home » Modeling » EMF » [CDO] DBStoreAccessor, SmartPreparedStatementCache
|
Re: [CDO] DBStoreAccessor, SmartPreparedStatementCache [message #898647 is a reply to message #898594] |
Fri, 27 July 2012 04:18 |
|
Am 26.07.2012 20:26, schrieb Per Sterner:
> Hello,
>
> I am still searching for our problems :)
What exactly are those? Are there any OutOfMemoryErrors being thrown?
> On my server I have after some days 3637 instances from the class
> 'org.eclipse.emf.cdo.server.internal.db.DBStoreAccessor' and some of these contain/reference alot of data. The mat -
> MemoryAnalyzer tells me that these instances contain/reference 350MB of data.
> There are only 2 CDOSessions open to the CDOServer, one client is in the same VM and the other client is operating
> remote. The CDOClient on the server is doing alot transactions and queries, but all in one thread.
>
> These classes have same instance count (in the CDOServer VM):
> org.eclipse.emf.cdo.server.internal.db.DBStoreAccessor
> org.eclipse.emf.cdo.server.internal.db.DBStoreAccessor$ConnectionKeepAliveTask
> org.eclipse.emf.cdo.server.internal.db.SmartPreparedStatementCache
> (The instances are connected to 'java.util.TimerTask')
What do you mean by "connected"? Do you have a screenshot from MAT?
Stefan, do you think we should pool the store accessors less aggressively? What about a limit on the
StoreAccessorPool.accessors queue? Or a periodic cleanup?
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
|
Re: [CDO] DBStoreAccessor, SmartPreparedStatementCache [message #898705 is a reply to message #898647] |
Fri, 27 July 2012 09:22 |
Stefan Winkler Messages: 307 Registered: July 2009 Location: Germany |
Senior Member |
|
|
Hi,
> Stefan, do you think we should pool the store accessors less
> aggressively? What about a limit on the StoreAccessorPool.accessors
> queue? Or a periodic cleanup?
Currently, we assume a rather constant use of concurrent operations
(which are carried out by StoreAccessors). Therefore, the pool in which
StoreAccessors live is not automatically cleaned up. If you have a usage
pattern, where you have peaks of activity and periods of low activity,
you'd have a pool of inactive StoreAccessors which is not cleaned up.
We couls implement a smarter pooling strategy which has a defined
regular amount of elements. If a peek requires more accessors, these are
created and pooled, but if they are inactive for some time, they could
be removed from the pool and freed up until the regular pool level is
reached again.
@Per: If this sounds good, please file a bugzilla for this request.
Cheers,
Stefan
Am 27.07.12 06:18, schrieb Eike Stepper:
> Am 26.07.2012 20:26, schrieb Per Sterner:
>> Hello,
>>
>> I am still searching for our problems :)
> What exactly are those? Are there any OutOfMemoryErrors being thrown?
>
>> On my server I have after some days 3637 instances from the class
>> 'org.eclipse.emf.cdo.server.internal.db.DBStoreAccessor' and some of
>> these contain/reference alot of data. The mat - MemoryAnalyzer tells
>> me that these instances contain/reference 350MB of data.
>> There are only 2 CDOSessions open to the CDOServer, one client is in
>> the same VM and the other client is operating remote. The CDOClient on
>> the server is doing alot transactions and queries, but all in one thread.
>>
>> These classes have same instance count (in the CDOServer VM):
>> org.eclipse.emf.cdo.server.internal.db.DBStoreAccessor
>> org.eclipse.emf.cdo.server.internal.db.DBStoreAccessor$ConnectionKeepAliveTask
>>
>> org.eclipse.emf.cdo.server.internal.db.SmartPreparedStatementCache
>> (The instances are connected to 'java.util.TimerTask')
> What do you mean by "connected"? Do you have a screenshot from MAT?
>
> Stefan, do you think we should pool the store accessors less
> aggressively? What about a limit on the StoreAccessorPool.accessors
> queue? Or a periodic cleanup?
>
> Cheers
> /Eike
>
> ----
> http://www.esc-net.de
> http://thegordian.blogspot.com
> http://twitter.com/eikestepper
>
>
|
|
|
Re: [CDO] DBStoreAccessor, SmartPreparedStatementCache [message #898744 is a reply to message #898705] |
Fri, 27 July 2012 12:34 |
Per Sterner Messages: 91 Registered: October 2011 |
Member |
|
|
Hi,
>> I am still searching for our problems
>What exactly are those? Are there any OutOfMemoryErrors being thrown?
Actually the cpu load goes up (~100%, Garbage collector) and therefore (?) a lot TimeoutException occur on transactions.
(I think 'OutOfMemoryErrors' would come later)
INFO | jvm 1 | 2012/07/27 08:03:50 | Caused by: org.eclipse.net4j.signal.RemoteException: org.eclipse.net4j.util.om.monitor.MonitorCanceledException: org.eclipse.net4j.util.concurrent.TimeoutRuntimeException: Timeout after 10001 millis
INFO | jvm 1 | 2012/07/27 08:03:50 | at org.eclipse.net4j.signal.RequestWithConfirmation.getRemoteException(RequestWithConfirmation.java:141)
INFO | jvm 1 | 2012/07/27 08:03:50 | at org.eclipse.net4j.signal.RequestWithConfirmation.setRemoteException(RequestWithConfirmation.java:130)
INFO | jvm 1 | 2012/07/27 08:03:50 | at org.eclipse.net4j.signal.SignalProtocol.handleRemoteException(SignalProtocol.java:467)
INFO | jvm 1 | 2012/07/27 08:03:50 | at org.eclipse.net4j.signal.RemoteExceptionIndication.indicating(RemoteExceptionIndication.java:66)
INFO | jvm 1 | 2012/07/27 08:03:50 | at org.eclipse.net4j.signal.Indication.doExtendedInput(Indication.java:57)
INFO | jvm 1 | 2012/07/27 08:03:50 | at
>> (The instances are connected to 'java.util.TimerTask')
>What do you mean by "connected"? Do you have a screenshot from MAT?
I have attached 3 screenshots.
>Currently, we assume a rather constant use of concurrent operations
>(which are carried out by StoreAccessors). Therefore, the pool in which
>StoreAccessors live is not automatically cleaned up. If you have a usage
>pattern, where you have peaks of activity and periods of low activity,
>you'd have a pool of inactive StoreAccessors which is not cleaned up.
>
We are syncing some data every 10 seconds (yes polling :-/). But only if new data arrives, this is commited in CDO (But there are always some changes).
>We couls implement a smarter pooling strategy which has a defined
>regular amount of elements. If a peek requires more accessors, these are
>created and pooled, but if they are inactive for some time, they could
>be removed from the pool and freed up until the regular pool level is
>reached again.
>
I think a smarter cleanup of the pool would solve our problem (screenshots)?
>@Per: If this sounds good, please file a bugzilla for this request.
Yes, this sounds good, on my way
Regards,
Per
Thanks for this project'
-
Attachment: mat_001.png
(Size: 129.93KB, Downloaded 145 times) -
Attachment: mat_002.png
(Size: 148.14KB, Downloaded 143 times) -
Attachment: mat_003.png
(Size: 146.38KB, Downloaded 128 times)
|
|
| | |
Re: [CDO] DBStoreAccessor, SmartPreparedStatementCache [message #900568 is a reply to message #900530] |
Tue, 07 August 2012 14:54 |
|
Hi Per,
This is a follow-up effect of the store accessors being pooled but never released in phases of low load.These
transaction must be inactive but still referenced via StoreAccessorBase.context.
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Am 07.08.2012 14:12, schrieb Per Sterner:
> Hello again,
>
> In our system I modified the cdo server to use the 'org.eclipse.emf.cdo.server.internal.db.NullPreparedStatementCache'.
>
> Then I let it run.
>
> 1. At the beginning after a lot queries the Memory looks like this (mat_001_startup.png) (server and one client run in the same VM)
>
> 2. After some time (a lot of queries and commits) it looks like this (mat_002.png).
> In my logfile there were many exceptions like this:
>
>
> org.eclipse.net4j.signal.RemoteException: org.eclipse.net4j.util.om.monitor.MonitorCanceledException: org.eclipse.net4j.util.concurrent.TimeoutRuntimeException: Timeout after 11040 millis
> at org.eclipse.net4j.signal.RequestWithConfirmation.getRemoteException(RequestWithConfirmation.java:141)
> at org.eclipse.net4j.signal.RequestWithConfirmation.setRemoteException(RequestWithConfirmation.java:130)
> at org.eclipse.net4j.signal.SignalProtocol.handleRemoteException(SignalProtocol.java:467)
> at org.eclipse.net4j.signal.RemoteExceptionIndication.indicating(RemoteExceptionIndication.java:66)
> at org.eclipse.net4j.signal.Indication.doExtendedInput(Indication.java:57)
> at org.eclipse.net4j.signal.Signal.doInput(Signal.java:328)
> at org.eclipse.net4j.signal.Indication.execute(Indication.java:51)
> at org.eclipse.net4j.signal.Signal.runSync(Signal.java:253)
> at org.eclipse.net4j.signal.Signal.run(Signal.java:149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
>
> org.eclipse.net4j.util.om.monitor.MonitorCanceledException: org.eclipse.net4j.util.concurrent.TimeoutRuntimeException: Timeout after 11040 millis
> at org.eclipse.net4j.util.om.monitor.Monitor.checkCanceled(Monitor.java:56)
> at org.eclipse.net4j.util.om.monitor.TimeoutMonitor.checkCanceled(TimeoutMonitor.java:116)
> at org.eclipse.net4j.util.om.monitor.NestedMonitor.checkCanceled(NestedMonitor.java:55)
> at org.eclipse.net4j.util.om.monitor.NestedMonitor.checkCanceled(NestedMonitor.java:55)
> at org.eclipse.net4j.util.om.monitor.AbstractMonitor.hasBegun(AbstractMonitor.java:36)
> at org.eclipse.net4j.util.om.monitor.AbstractMonitor.checkBegun(AbstractMonitor.java:141)
> at org.eclipse.net4j.util.om.monitor.AbstractMonitor.fork(AbstractMonitor.java:65)
> at org.eclipse.net4j.util.om.monitor.ProgressDistributor.run(ProgressDistributor.java:86)
> at org.eclipse.emf.cdo.server.internal.net4j.protocol.CommitTransactionIndication.indicatingCommit(CommitTransactionIndication.java:262)
> at org.eclipse.emf.cdo.server.internal.net4j.protocol.CommitTransactionIndication.indicating(CommitTransactionIndication.java:96)
> at org.eclipse.emf.cdo.server.internal.net4j.protocol.CDOServerIndicationWithMonitoring.indicating(CDOServerIndicationWithMonitoring.java:109)
> at org.eclipse.net4j.signal.IndicationWithMonitoring.indicating(IndicationWithMonitoring.java:86)
> at org.eclipse.net4j.signal.IndicationWithResponse.doExtendedInput(IndicationWithResponse.java:92)
> at org.eclipse.net4j.signal.Signal.doInput(Signal.java:328)
> at org.eclipse.net4j.signal.IndicationWithResponse.execute(IndicationWithResponse.java:65)
> at org.eclipse.net4j.signal.IndicationWithMonitoring.execute(IndicationWithMonitoring.java:65)
> at org.eclipse.net4j.signal.Signal.runSync(Signal.java:253)
> at org.eclipse.net4j.signal.Signal.run(Signal.java:149)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
>
> org.eclipse.net4j.util.concurrent.TimeoutRuntimeException: Timeout after 11040 millis
> at org.eclipse.net4j.util.om.monitor.TimeoutMonitor.handleTimeout(TimeoutMonitor.java:121)
> at org.eclipse.net4j.util.om.monitor.TimeoutMonitor$1.handleTimeout(TimeoutMonitor.java:61)
> at org.eclipse.net4j.util.concurrent.Timeouter$1.run(Timeouter.java:88)
> at java.util.TimerThread.mainLoop(Timer.java:512)
> at java.util.TimerThread.run(Timer.java:462)
>
>
> The h2-DB has grown a bit (1.5GB), perhaps the exceptions are okay. But the instance count of 'org.eclipse.emf.cdo.internal.server.Transaction' is too high? Are they not closed?
>
> I looked in my code, the Transactions on the client side are closed on an exception.
>
> regards,
>
> Per Sterner
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
|
Re: [CDO] DBStoreAccessor, SmartPreparedStatementCache [message #901516 is a reply to message #900568] |
Mon, 13 August 2012 08:34 |
Christian Mohr Messages: 34 Registered: June 2012 |
Member |
|
|
Hi,
as Eike suggested i tried to look into the DBStoreAccessor, but i have still some questions.
As Per wrote the ConnectionKeepAliveTask is one of the classes not being cleaned up. So i wanted to ask if thats because the DBStoreAccessor is pooled? Is this task stopped when the Accessor gets removed from the pool?
I tried to comment this part(DBStoreAccessor.doActivate) out:
@Override
protected void doActivate() throws Exception
{
DBStore store = getStore();
connection = store.getConnection();
// connectionKeepAliveTask = new ConnectionKeepAliveTask();
// long keepAlivePeriod = ConnectionKeepAliveTask.EXECUTION_PERIOD;
// long keepAlivePeriod = 1000L * 60L;
// Map<String, String> storeProps = store.getProperties();
// if (storeProps != null)
// {
// String value = storeProps.get(IDBStore.Props.CONNECTION_KEEPALIVE_PERIOD);
// if (value != null)
// {
// keepAlivePeriod = Long.parseLong(value) * 60L * 1000L;
// }
// }
// store.getConnectionKeepAliveTimer().schedule(connectionKeepAliveTask);
// TODO - make this configurable?
statementCache = CDODBUtil.createStatementCache();
statementCache.setConnection(connection);
LifecycleUtil.activate(statementCache);
}
As result all classes were removed correctly and didn't leak(DBAccessor, etc.). So I got the feeling that this ConnectionKeepAliveTask prevents the DBStoreAccessor from being garbage collected? (or did i get something wrong?)
Second, i wanted to ask for some hints where to look for the "Pooling-Strategy". I found the pools itself but not how and where they are cleaned up.
Thanks,
Christian
PS: I wrote it here because i think it doesn't belong into the Feature-Request
[Updated on: Mon, 13 August 2012 08:38] Report message to a moderator
|
|
| |
Re: [CDO] DBStoreAccessor, SmartPreparedStatementCache [message #901534 is a reply to message #901516] |
Mon, 13 August 2012 09:32 |
|
Hi Christian,
Please note that I've resolved https://bugs.eclipse.org/386289 in the meantime. Please feel free to reopen if you have
additional suggestions ;-)
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Am 13.08.2012 10:34, schrieb Christian Mohr:
> Hi,
>
> as Eike suggested i tried to look into the DBStoreAccessor, but i have still some questions.
>
> As Per wrote the ConnectionKeepAliveTask is one of the classes not being cleaned up. So i wanted to ask if thats
> because the DBStoreAccessor is pooled? Is this task stopped when the Accessor gets removed from the pool?
> I tried to comment this part(DBStoreAccessor.doActivate) out:
>
>
>
> @Override
> protected void doActivate() throws Exception
> {
> DBStore store = getStore();
> connection = store.getConnection();
> // connectionKeepAliveTask = new ConnectionKeepAliveTask();
>
> // long keepAlivePeriod = ConnectionKeepAliveTask.EXECUTION_PERIOD;
> // long keepAlivePeriod = 1000L * 60L;
> // Map<String, String> storeProps = store.getProperties();
> // if (storeProps != null)
> // {
> // String value = storeProps.get(IDBStore.Props.CONNECTION_KEEPALIVE_PERIOD);
> // if (value != null)
> // {
> // keepAlivePeriod = Long.parseLong(value) * 60L * 1000L;
> // }
> // }
>
> // store.getConnectionKeepAliveTimer().schedule(connectionKeepAliveTask);
>
> // TODO - make this configurable?
> statementCache = CDODBUtil.createStatementCache();
> statementCache.setConnection(connection);
> LifecycleUtil.activate(statementCache);
> }
>
> As result all classes were removed correctly and didn't leak(DBAccessor, etc.). So I got the feeling that this
> ConnectionKeepAliveTask prevents the DBStoreAccessor from being garbage collected. (or did i get something wrong?)
>
> Thanks,
>
> Christian
>
> PS: I wrote it here because i thinnk it doesn't belong into the Feature-Request
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
| | | |
Goto Forum:
Current Time: Mon Sep 23 14:00:39 GMT 2024
Powered by FUDForum. Page generated in 0.04395 seconds
|