Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » EMF » [CDO] Transaction commit & Threads
[CDO] Transaction commit & Threads [message #988811] Mon, 03 December 2012 05:19 Go to next message
Martin Blühweis is currently offline Martin Blühweis
Messages: 11
Registered: February 2012
Location: Innsbruck, Austria
Junior Member
hi,

i found some exceptions in my application log -

 org.eclipse.emf.cdo.util.CommitException: java.lang.OutOfMemoryError: unable to create new native thread
	at org.eclipse.emf.internal.cdo.transaction.CDOTransactionImpl.commit(CDOTransactionImpl.java:1148)
	at org.eclipse.emf.internal.cdo.transaction.CDOTransactionImpl.commit(CDOTransactionImpl.java:1154)
	at mypack.MyClass.commit(MyClass.java:357)
	... 26 more
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
	at java.lang.Thread.start0(Native Method)
	at java.lang.Thread.start(Unknown Source)
	at org.eclipse.net4j.util.concurrent.Worker.doActivate(Worker.java:96)
	at org.eclipse.net4j.util.concurrent.QueueWorker.doActivate(QueueWorker.java:109)
	at org.eclipse.net4j.util.lifecycle.Lifecycle.activate(Lifecycle.java:76)
	at org.eclipse.emf.internal.cdo.view.CDOViewImpl.getInvalidationRunner(CDOViewImpl.java:852)
	at org.eclipse.emf.internal.cdo.view.CDOViewImpl.invalidate(CDOViewImpl.java:794)
	at org.eclipse.emf.internal.cdo.session.CDOSessionImpl.invalidateView(CDOSessionImpl.java:1170)
	at org.eclipse.emf.internal.cdo.session.CDOSessionImpl.invalidateOrdered(CDOSessionImpl.java:1152)
	at org.eclipse.emf.internal.cdo.session.CDOSessionImpl.invalidate(CDOSessionImpl.java:1109)
	at org.eclipse.emf.internal.cdo.transaction.CDOTransactionImpl$CDOCommitContextImpl.postCommit(CDOTransactionImpl.java:2809)
	at org.eclipse.emf.internal.cdo.transaction.CDOSingleTransactionStrategyImpl.commit(CDOSingleTransactionStrategyImpl.java:74)
	at org.eclipse.emf.internal.cdo.transaction.CDOTransactionImpl.commit(CDOTransactionImpl.java:1134)


at first glance a memory leak in my application - but what makes me suspicious is, that it happens only on cdo transaction commits ...

after the failed commit everything works as expected again (commits also) until it happens again a few hours or days later.

(its a vaadin-webapp running within apache tomcat on a windows server)

any ideas?

thanks in advance,
martin
Re: [CDO] Transaction commit & Threads [message #988971 is a reply to message #988811] Tue, 04 December 2012 01:28 Go to previous messageGo to next message
Eike Stepper is currently offline Eike Stepper
Messages: 5503
Registered: July 2009
Senior Member
Am 03.12.2012 11:19, schrieb Martin Bl:
> hi,
>
> i found some exceptions in my application log -
>
> org.eclipse.emf.cdo.util.CommitException: java.lang.OutOfMemoryError: unable to create new native thread
> at org.eclipse.emf.internal.cdo.transaction.CDOTransactionImpl.commit(CDOTransactionImpl.java:1148)
> at org.eclipse.emf.internal.cdo.transaction.CDOTransactionImpl.commit(CDOTransactionImpl.java:1154)
> at mypack.MyClass.commit(MyClass.java:357)
> ... 26 more
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Unknown Source)
> at org.eclipse.net4j.util.concurrent.Worker.doActivate(Worker.java:96)
> at org.eclipse.net4j.util.concurrent.QueueWorker.doActivate(QueueWorker.java:109)
> at org.eclipse.net4j.util.lifecycle.Lifecycle.activate(Lifecycle.java:76)
> at org.eclipse.emf.internal.cdo.view.CDOViewImpl.getInvalidationRunner(CDOViewImpl.java:852)
> at org.eclipse.emf.internal.cdo.view.CDOViewImpl.invalidate(CDOViewImpl.java:794)
> at org.eclipse.emf.internal.cdo.session.CDOSessionImpl.invalidateView(CDOSessionImpl.java:1170)
> at org.eclipse.emf.internal.cdo.session.CDOSessionImpl.invalidateOrdered(CDOSessionImpl.java:1152)
> at org.eclipse.emf.internal.cdo.session.CDOSessionImpl.invalidate(CDOSessionImpl.java:1109)
> at
> org.eclipse.emf.internal.cdo.transaction.CDOTransactionImpl$CDOCommitContextImpl.postCommit(CDOTransactionImpl.java:2809)
> at
> org.eclipse.emf.internal.cdo.transaction.CDOSingleTransactionStrategyImpl.commit(CDOSingleTransactionStrategyImpl.java:74)
> at org.eclipse.emf.internal.cdo.transaction.CDOTransactionImpl.commit(CDOTransactionImpl.java:1134)
>
>
> at first glance a memory leak in my application - but what makes me suspicious is, that it happens only on cdo
> transaction commits ...
>
> after the failed commit everything works as expected again (commits also) until it happens again a few hours or days
> later.
>
> (its a vaadin-webapp running within apache tomcat on a windows server)
>
> any ideas?
Not really without more infos. Is it possible that you capture a list of the active threads when this happens?

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Re: [CDO] Transaction commit & Threads [message #989243 is a reply to message #988971] Wed, 05 December 2012 05:39 Go to previous messageGo to next message
Martin Blühweis is currently offline Martin Blühweis
Messages: 11
Registered: February 2012
Location: Innsbruck, Austria
Junior Member
Eike Stepper wrote on Tue, 04 December 2012 07:28
Not really without more infos. Is it possible that you capture a list of the active threads when this happens?

btw. i'm working with 4.1.0

i added the threads to my logs and got interesting results.

after starting my application i always edit the same field of the same entity and commit my transaction.
after the 1st commit i get 17 "InvalidationRunner-View ${count}" - threads
2nd: 34 (+17)
3rd: 52 (+18)
4th: 71 (+19)
5th: 91 (+20)
6th: 112 (+21)
...

the number of "pool-1-thread-${count2}" threads is increased by 1 with every commit.

all other threads seem to be constantly - i compared names and counts.

hope this helps,
cheers,
martin

[Updated on: Wed, 05 December 2012 05:40]

Report message to a moderator

Re: [CDO] Transaction commit & Threads [message #989415 is a reply to message #989243] Thu, 06 December 2012 02:20 Go to previous messageGo to next message
Eike Stepper is currently offline Eike Stepper
Messages: 5503
Registered: July 2009
Senior Member
Am 05.12.2012 11:39, schrieb Martin Bl:
> Eike Stepper wrote on Tue, 04 December 2012 07:28
>> Not really without more infos. Is it possible that you capture a list of the active threads when this happens?
>
> btw. i'm working with 4.1.0
>
> i added the threads to my logs and got interesting results.
>
> after starting my application i always edit the same field of the same entity and commit my transaction.
> after the 1st commit i get 17 "InvalidationRunner-View ${count}" - threads
> 2nd: 34 (+17)
> 3rd: 52 (+18)
> 4th: 71 (+19)
> 5th: 91 (+20)
> 6th: 112 (+21)
> ..
>
> the number of "pool-1-thread-${count2}" threads is increases by 1 with every commit.
Is it possible that you open a new CDOTransaction for each commit? Why is that?

And if so, is t possible that you don't close() it properly?

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Re: [CDO] Transaction commit & Threads [message #989436 is a reply to message #989415] Thu, 06 December 2012 03:56 Go to previous messageGo to next message
Martin Blühweis is currently offline Martin Blühweis
Messages: 11
Registered: February 2012
Location: Innsbruck, Austria
Junior Member
Eike Stepper wrote on Thu, 06 December 2012 08:20

Is it possible that you open a new CDOTransaction for each commit? Why is that?

And if so, is t possible that you don't close() it properly?

no.

every user gets exactly 1 transaction, which is closed when he closes the browser (tab).

so i extend my logging and repeat my tests:

INFO  12-06 09:28:59 try to commit transaction Transaction 1  (EhsCdoResource.java:359) 
INFO  12-06 09:28:59 Running threads:              65  (EhsCdoResource.java:385) 
INFO  12-06 09:28:59 Running invalidation threads: 24  (EhsCdoResource.java:386) 
INFO  12-06 09:28:59 Difference:                   41  (EhsCdoResource.java:387) 
INFO  12-06 09:29:16 try to commit transaction Transaction 1  (EhsCdoResource.java:359) 
INFO  12-06 09:29:16 Running threads:              90  (EhsCdoResource.java:385) 
INFO  12-06 09:29:16 Running invalidation threads: 48  (EhsCdoResource.java:386) 
INFO  12-06 09:29:16 Difference:                   42  (EhsCdoResource.java:387) 
INFO  12-06 09:29:37 try to commit transaction Transaction 1  (EhsCdoResource.java:359) 
INFO  12-06 09:29:37 Running threads:              116  (EhsCdoResource.java:385) 
INFO  12-06 09:29:37 Running invalidation threads: 73  (EhsCdoResource.java:386) 
INFO  12-06 09:29:37 Difference:                   43  (EhsCdoResource.java:387) 
INFO  12-06 09:29:52 try to commit transaction Transaction 1  (EhsCdoResource.java:359) 
INFO  12-06 09:29:53 Running threads:              143  (EhsCdoResource.java:385) 
INFO  12-06 09:29:53 Running invalidation threads: 99  (EhsCdoResource.java:386) 
INFO  12-06 09:29:53 Difference:                   44  (EhsCdoResource.java:387) 
INFO  12-06 09:30:08 try to commit transaction Transaction 1  (EhsCdoResource.java:359) 
INFO  12-06 09:30:08 Running threads:              171  (EhsCdoResource.java:385) 
INFO  12-06 09:30:08 Running invalidation threads: 126  (EhsCdoResource.java:386) 
INFO  12-06 09:30:08 Difference:                   45  (EhsCdoResource.java:387) 
INFO  12-06 09:30:15 try to commit transaction Transaction 1  (EhsCdoResource.java:359) 
INFO  12-06 09:30:15 Running threads:              199  (EhsCdoResource.java:385) 
INFO  12-06 09:30:15 Running invalidation threads: 154  (EhsCdoResource.java:386) 
INFO  12-06 09:30:15 Difference:                   45  (EhsCdoResource.java:387) 


now i close my browser tab ...
INFO  12-06 09:34:26 closing Transaction 1 from Session2 [MARTIN]  (EhsCdoResource.java:662) 

(the session was closed too)

... and open a new one to change and commit some data:

INFO  12-06 09:37:01 try to commit transaction Transaction 1  (EhsCdoResource.java:359) 
INFO  12-06 09:37:02 Running threads:              75  (EhsCdoResource.java:385) 
INFO  12-06 09:37:02 Running invalidation threads: 30  (EhsCdoResource.java:386) 
INFO  12-06 09:37:02 Difference:                   45  (EhsCdoResource.java:387) 
INFO  12-06 09:37:20 try to commit transaction Transaction 1  (EhsCdoResource.java:359) 
INFO  12-06 09:37:20 Running threads:              105  (EhsCdoResource.java:385) 
INFO  12-06 09:37:20 Running invalidation threads: 60  (EhsCdoResource.java:386) 
INFO  12-06 09:37:20 Difference:                   45  (EhsCdoResource.java:387) 
INFO  12-06 09:37:32 try to commit transaction Transaction 1  (EhsCdoResource.java:359) 
INFO  12-06 09:37:32 Running threads:              136  (EhsCdoResource.java:385) 
INFO  12-06 09:37:32 Running invalidation threads: 91  (EhsCdoResource.java:386) 
INFO  12-06 09:37:32 Difference:                   45  (EhsCdoResource.java:387) 
INFO  12-06 09:37:37 try to commit transaction Transaction 1  (EhsCdoResource.java:359) 
INFO  12-06 09:37:37 Running threads:              168  (EhsCdoResource.java:385) 
INFO  12-06 09:37:37 Running invalidation threads: 123  (EhsCdoResource.java:386) 
INFO  12-06 09:37:37 Difference:                   45  (EhsCdoResource.java:387) 


now a close the tab once more to do the same again:

INFO  12-06 09:41:45 closing Transaction 1 from Session3 [MARTIN]  (EhsCdoResource.java:662) 
INFO  12-06 09:43:10 try to commit transaction Transaction 1  (EhsCdoResource.java:359) 
INFO  12-06 09:43:10 Running threads:              79  (EhsCdoResource.java:385) 
INFO  12-06 09:43:10 Running invalidation threads: 34  (EhsCdoResource.java:386) 
INFO  12-06 09:43:10 Difference:                   45  (EhsCdoResource.java:387) 
INFO  12-06 09:43:26 try to commit transaction Transaction 1  (EhsCdoResource.java:359) 
INFO  12-06 09:43:26 Running threads:              113  (EhsCdoResource.java:385) 
INFO  12-06 09:43:26 Running invalidation threads: 68  (EhsCdoResource.java:386) 
INFO  12-06 09:43:26 Difference:                   45  (EhsCdoResource.java:387) 


as you can see, the number of invalidation threads is increasing and increasing.
closing transaction and session stops many threads, but not all of them.
so the outofmemory exceptions seems to be unavoidable after a while.

maybe it would be better to use 1 session for all my users - what do you think?

but i'm not sure if this can solve my problem ...

cheers, martin

p.s.
the "pool-1-thread-${count2}" do not seem to be a problem, looks like the GC cares about. but i didn't investigate that more closely.
Re: [CDO] Transaction commit & Threads [message #989453 is a reply to message #989436] Thu, 06 December 2012 04:47 Go to previous messageGo to next message
Eike Stepper is currently offline Eike Stepper
Messages: 5503
Registered: July 2009
Senior Member
Am 06.12.2012 09:56, schrieb Martin Bl:
> Eike Stepper wrote on Thu, 06 December 2012 08:20
>> Is it possible that you open a new CDOTransaction for each commit? Why is that?
>>
>> And if so, is t possible that you don't close() it properly?
>
> no.
> every user gets exactly 1 transaction, which is closed when he closes the browser (tab).
That sounds ok.

>
> so i extend my logging and repeat my tests:
Sorry, the below is really not readable in my news client ;-(

>
>
> INFO 12-06 09:28:59 try to commit transaction Transaction 1 (EhsCdoResource.java:359) INFO 12-06 09:28:59 Running
> threads: 65 (EhsCdoResource.java:385) INFO 12-06 09:28:59 Running invalidation threads: 24
> (EhsCdoResource.java:386) INFO 12-06 09:28:59 Difference: 41 (EhsCdoResource.java:387) INFO 12-06
> 09:29:16 try to commit transaction Transaction 1 (EhsCdoResource.java:359) INFO 12-06 09:29:16 Running
> threads: 90 (EhsCdoResource.java:385) INFO 12-06 09:29:16 Running invalidation threads: 48
> (EhsCdoResource.java:386) INFO 12-06 09:29:16 Difference: 42 (EhsCdoResource.java:387) INFO 12-06
> 09:29:37 try to commit transaction Transaction 1 (EhsCdoResource.java:359) INFO 12-06 09:29:37 Running
> threads: 116 (EhsCdoResource.java:385) INFO 12-06 09:29:37 Running invalidation threads: 73
> (EhsCdoResource.java:386) INFO 12-06 09:29:37 Difference: 43 (EhsCdoResource.java:387) INFO 12-06
> 09:29:52 try to commit transaction Transaction 1 (EhsCdoResource.java:359) INFO 12-06 09:29:53 Running
> threads: 143 (EhsCdoResource.java:385) INFO 12-06 09:29:53 Running invalidation threads: 99
> (EhsCdoResource.java:386) INFO 12-06 09:29:53 Difference: 44 (EhsCdoResource.java:387) INFO 12-06
> 09:30:08 try to commit transaction Transaction 1 (EhsCdoResource.java:359) INFO 12-06 09:30:08 Running
> threads: 171 (EhsCdoResource.java:385) INFO 12-06 09:30:08 Running invalidation threads: 126
> (EhsCdoResource.java:386) INFO 12-06 09:30:08 Difference: 45 (EhsCdoResource.java:387) INFO 12-06
> 09:30:15 try to commit transaction Transaction 1 (EhsCdoResource.java:359) INFO 12-06 09:30:15 Running
> threads: 199 (EhsCdoResource.java:385) INFO 12-06 09:30:15 Running invalidation threads: 154
> (EhsCdoResource.java:386) INFO 12-06 09:30:15 Difference: 45 (EhsCdoResource.java:387)
> now i close my browser tab ...
>
> INFO 12-06 09:34:26 closing Transaction 1 from Session2 [MARTIN] (EhsCdoResource.java:662) (the session was closed too)
>
> .. and open a new one to change and commit some data:
>
>
> INFO 12-06 09:37:01 try to commit transaction Transaction 1 (EhsCdoResource.java:359) INFO 12-06 09:37:02 Running
> threads: 75 (EhsCdoResource.java:385) INFO 12-06 09:37:02 Running invalidation threads: 30
> (EhsCdoResource.java:386) INFO 12-06 09:37:02 Difference: 45 (EhsCdoResource.java:387) INFO 12-06
> 09:37:20 try to commit transaction Transaction 1 (EhsCdoResource.java:359) INFO 12-06 09:37:20 Running
> threads: 105 (EhsCdoResource.java:385) INFO 12-06 09:37:20 Running invalidation threads: 60
> (EhsCdoResource.java:386) INFO 12-06 09:37:20 Difference: 45 (EhsCdoResource.java:387) INFO 12-06
> 09:37:32 try to commit transaction Transaction 1 (EhsCdoResource.java:359) INFO 12-06 09:37:32 Running
> threads: 136 (EhsCdoResource.java:385) INFO 12-06 09:37:32 Running invalidation threads: 91
> (EhsCdoResource.java:386) INFO 12-06 09:37:32 Difference: 45 (EhsCdoResource.java:387) INFO 12-06
> 09:37:37 try to commit transaction Transaction 1 (EhsCdoResource.java:359) INFO 12-06 09:37:37 Running
> threads: 168 (EhsCdoResource.java:385) INFO 12-06 09:37:37 Running invalidation threads: 123
> (EhsCdoResource.java:386) INFO 12-06 09:37:37 Difference: 45 (EhsCdoResource.java:387)
>
> now a close the tab once more to do the same again:
>
>
> INFO 12-06 09:41:45 closing Transaction 1 from Session3 [MARTIN] (EhsCdoResource.java:662) INFO 12-06 09:43:10 try
> to commit transaction Transaction 1 (EhsCdoResource.java:359) INFO 12-06 09:43:10 Running threads: 79
> (EhsCdoResource.java:385) INFO 12-06 09:43:10 Running invalidation threads: 34 (EhsCdoResource.java:386) INFO 12-06
> 09:43:10 Difference: 45 (EhsCdoResource.java:387) INFO 12-06 09:43:26 try to commit transaction
> Transaction 1 (EhsCdoResource.java:359) INFO 12-06 09:43:26 Running threads: 113
> (EhsCdoResource.java:385) INFO 12-06 09:43:26 Running invalidation threads: 68 (EhsCdoResource.java:386) INFO 12-06
> 09:43:26 Difference: 45 (EhsCdoResource.java:387)
> as you can see, the number of invalidation threads is increasing and increasing. closing transaction and session stops
> many threads, but not all of them.
> so the outofmemory exceptions seems to be unavoidable after a while.
Is it possible that you set a breakpoint in CDOViewImpl.doDeactivate() and try to find out what this code does for you:

@Override
protected void doDeactivate() throws Exception
{
CDOViewRegistryImpl.INSTANCE.deregister(this);

if (invalidationRunner != null)
{
LifecycleUtil.deactivate(invalidationRunner, OMLogger.Level.WARN);
invalidationRunner = null;
}


>
> maybe it would be better to use 1 session for all my users - what do you think?
Yeah, that would save a lot of the heap space! The session with the revision cache is probably the most expensive client
facility. But note that currently the user information (if used at all) is bound to the session. By reusing the session
all transactions would persist that userID (if used). This could probably be made more flexible with a feature request.

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Re: [CDO] Transaction commit & Threads [message #989455 is a reply to message #989453] Thu, 06 December 2012 04:56 Go to previous messageGo to next message
Eike Stepper is currently offline Eike Stepper
Messages: 5503
Registered: July 2009
Senior Member
Am 06.12.2012 10:47, schrieb Eike Stepper:
>
>>
>> maybe it would be better to use 1 session for all my users - what do you think?
> Yeah, that would save a lot of the heap space! [...]
BTW. I don't believe that reusing the session would have an impact on the threading problem because the thread in
question is maintained by the transactions.

Have you thought about this different design:

Use a single read-only CDOView for all users (and their read operations) and keep it open until you close the session.
For every user interaction that involves modifications to the model open a new transaction, do all the changes there,
then commit and close it.

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Re: [CDO] Transaction commit & Threads [message #989520 is a reply to message #989455] Thu, 06 December 2012 10:29 Go to previous messageGo to next message
Martin Blühweis is currently offline Martin Blühweis
Messages: 11
Registered: February 2012
Location: Innsbruck, Austria
Junior Member
thank you eike for guiding me in the right direction Wink

after debugging for a while i could find the problem, i'll try to describe:
i modified my code as suggested to have just 1 session for all users. was done quickly and worked well, but i was surprised when i repeated my tests.
after the first application session (which does not mean cdosession) transaction #1 was closed, after the 2nd something like transaction #30.

but by debugging into the code i found out, that no unexpected transactions were created meanwhile, just one for the new application session.

but what my app did, was creating views (to show, which user created data at which date) ... and yes, these views were not closed ... (in this case i was relieing on the GC)

so i'll fix that, and also think about your proposal to use one view for all users and starting a transaction when needed.

thanks a lot,
martin
Re: [CDO] Transaction commit & Threads [message #989533 is a reply to message #989520] Thu, 06 December 2012 11:26 Go to previous messageGo to next message
Martin Blühweis is currently offline Martin Blühweis
Messages: 11
Registered: February 2012
Location: Innsbruck, Austria
Junior Member
fixed the bug and tested again (still having one open transaction for every application session)

for every open transaction i see one invalidation thread which stops when the transaction is closed. correct?
Re: [CDO] Transaction commit & Threads [message #989544 is a reply to message #989533] Thu, 06 December 2012 11:44 Go to previous message
Eike Stepper is currently offline Eike Stepper
Messages: 5503
Registered: July 2009
Senior Member
Am 06.12.2012 17:26, schrieb Martin Bl:
> fixed the bug and tested again (still having one open transaction for every application session)
>
> for every open transaction i see one invalidation thread which stops when the transaction is closed. correct?
Correct ;-)

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Previous Topic:[CDO] org.eclipse.emf.cdo.common.util.PartialCollectionLoadingNotSupportedException
Next Topic:[CDO] HibernateStore UnsupportedOperationException
Goto Forum:
  


Current Time: Sat Aug 30 02:29:41 EDT 2014

Powered by FUDForum. Page generated in 0.05363 seconds