Deadlock in cache under load [message #491205] |
Tue, 13 October 2009 11:59  |
Eclipse User |
|
|
|
We are seeing dead locks occur under load with EclipseLink 1.1.1 and 1.1.2. We started seeing this issue when we moved from the JBoss transaction manager to a Spring based transaction manager. We took the Srping transaction manager for TopLink and changed the packages so it works with EclipseLink. We are seeing this issue on Jetty 6.1.20 as well as our previous version of JBoss (4.0.5) using the Spring based transaction manager. The deadlock is occurring in the cache on reads.
I have the thread dumps attached (gzipped). I have tried to attach them, but the attachment stuff isn't working on the forum. You can download the gzipped text file here: http://jefmsmit.googlepages.com/server-out.txt.gz
Thanks,
Adam
[Updated on: Tue, 13 October 2009 12:02] by Moderator
|
|
|
|
|
Re: Deadlock in cache under load [message #504067 is a reply to message #496677] |
Wed, 16 December 2009 20:04   |
Eclipse User |
|
|
|
I'm having the same problem.
I'm running an application on Sun GlassFish Enterprise Server v2.1 (9.1.1) (build b60e-fcs) with EclipseLink 1.1.2 as my JPA implementation. Our application has been getting a lot more load lately, and I've frequently encountered a situation in which all of my HTTP worker threads are locked waiting at
org.eclipse.persistence.internal.helper.ConcurrencyManager.a cquire(ConcurrencyManager.java:89)
Except for one, which is stuck in a loop at
org.eclipse.persistence.internal.helper.ConcurrencyManager.r eleaseDeferredLock(ConcurrencyManager.java:454)
I can provide a full thread dump of the locked state.
I've followed some of the advice from this thread:
http://forums.oracle.com/forums/thread.jspa?threadID=851676
Particularly setting
DeferredLockManager.SHOULD_USE_DEFERRED_LOCKS = false;
didn't seem to help at all. And setting "eclipselink.cache.shared.default"="false" causes several of my unit tests to fail. From what I understand setting that option isolates the read cache from the write cache, which seems to be having adverse side effects on my application.
I am not using join fetching, but I am using QueryHint.REFRESH_CASCADE. Do these have similar behavior?
Moreover, how do I resolve my problem? I've looked at the code for ConcurrencyManager in the more recent releases and I see no changes in this part of the code, so if it's a known bug, it doesn't appear to have been addressed.
In my case, it's not a true deadlock, but an infinite loop. Here is a code snip from ConcurrencyManager.releaseDeferredLock()
// Thread have three stages, one where they are doing work (i.e. building objects)
// two where they are done their own work but may be waiting on other threads to finish their work,
// and a third when they and all the threads they are waiting on are done.
// This is essentially a busy wait to determine if all the other threads are done.
while (true) {
// 2612538 - the default size of Map (32) is appropriate
Map recursiveSet = new IdentityHashMap();
if (isBuildObjectOnThreadComplete(currentThread, recursiveSet)) {// Thread job done.
lockManager.releaseActiveLocksOnThread();
removeDeferredLockManager(currentThread);
AbstractSessionLog.getLog().log(SessionLog.FINER, "deferred_locks_released", currentThread.getName());
return;
} else {// Not done yet, wait and check again.
try {
Thread.sleep(1);
} catch (InterruptedException ignoreAndContinue) {
}
}
}
For some reason isBuildObjectOnThreadComplete(currentThread, recursiveSet) always returns false, so that thread stays in that loop, and the concurrency manager never issues a notify() to release the waiting threads.
Cheers
|
|
|
|
|
|
|
Re: Deadlock in cache under load [message #1843580 is a reply to message #504430] |
Mon, 09 August 2021 08:25  |
Eclipse User |
|
|
|
Hello Everyone,
I know this is an old forum, but we recently faced this issue, when the application load was increased.
We have similar traces in the thread dumps, as are shared by different contributors of this forum.
We have already raised one service request with the Oracle team but they are asking for a reproducible test case.
We have been working on the test case, but yet we haven't got any success in reproducing it.
It will be a great help if someone can help in providing few suggestions on how to reproduce the issue.
Thanks in advance !!
Best Regards,
Nainsy Gupta
|
|
|
Powered by
FUDForum. Page generated in 0.03175 seconds