Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » EclipseLink » CacheKey getMutex() is not thread-safe(Suspected source of application deadlocks)
CacheKey getMutex() is not thread-safe [message #654960] Thu, 17 February 2011 13:47 Go to next message
Jason Schulz is currently offline Jason Schulz
Messages: 4
Registered: February 2011
Junior Member
The getMutex method in CacheKey uses double-check locking to instantiate member variable 'mutex'. Note that this ConcurrencyManager object is not declared as 'volatile'.

This code is the suspected cause in a series of application deadlocks. A set of partial thread dumps from a recent deadlock incident follows:


"WMQJCAResourceAdapter : 93" daemon prio=3 tid=0x0000000102543800 nid=0xa1e waiting on condition [0xfffffffd48dfb000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.eclipse.persistence.internal.helper.ConcurrencyManager.r eleaseDeferredLock(ConcurrencyManager.java:454)
at org.eclipse.persistence.internal.identitymaps.CacheKey.relea seDeferredLock(CacheKey.java:426)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.b uildObject(ObjectBuilder.java:779)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.b uildWorkingCopyCloneNormally(ObjectBuilder.java:583)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.b uildObjectInUnitOfWork(ObjectBuilder.java:552)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.b uildObject(ObjectBuilder.java:492)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.b uildObject(ObjectBuilder.java:444)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildOb ject(ObjectLevelReadQuery.java:635)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.conform IndividualResult(ObjectLevelReadQuery.java:799)
at org.eclipse.persistence.queries.ReadAllQuery.conformResult(R eadAllQuery.java:335)
at org.eclipse.persistence.queries.ReadAllQuery.registerResultI nUnitOfWork(ReadAllQuery.java:821)
at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLe velReadQuery(ReadAllQuery.java:464)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute DatabaseQuery(ObjectLevelReadQuery.java:997)
at org.eclipse.persistence.queries.DatabaseQuery.execute(Databa seQuery.java:675)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute (ObjectLevelReadQuery.java:958)
at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAll Query.java:432)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute InUnitOfWork(ObjectLevelReadQuery.java:1021)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.int ernalExecuteQuery(UnitOfWorkImpl.java:2898)
at org.eclipse.persistence.internal.sessions.AbstractSession.ex ecuteQuery(AbstractSession.java:1225)
at org.eclipse.persistence.internal.sessions.AbstractSession.ex ecuteQuery(AbstractSession.java:1207)
at org.eclipse.persistence.internal.sessions.AbstractSession.ex ecuteQuery(AbstractSession.java:1167)



"WMQJCAResourceAdapter : 90" daemon prio=3 tid=0x0000000101fe5000 nid=0xa1a in Object.wait() [0xfffffffd46dfe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.eclipse.persistence.internal.helper.ConcurrencyManager.a cquireReadLock(ConcurrencyManager.java:242)
- locked <0xfffffffdd6be05b8> (a org.eclipse.persistence.internal.helper.ConcurrencyManager)
at org.eclipse.persistence.internal.helper.ConcurrencyManager.c heckReadLock(ConcurrencyManager.java:230)
at org.eclipse.persistence.internal.identitymaps.CacheKey.check ReadLock(CacheKey.java:180)
at org.eclipse.persistence.internal.identitymaps.IdentityMapMan ager.getFromIdentityMap(IdentityMapManager.java:702)
at org.eclipse.persistence.internal.identitymaps.IdentityMapMan ager.getFromIdentityMap(IdentityMapManager.java:667)
at org.eclipse.persistence.internal.sessions.ObjectChangeSet.ge tTargetVersionOfSourceObject(ObjectChangeSet.java:367)
at org.eclipse.persistence.internal.sessions.MergeManager.merge ChangesFromChangeSet(MergeManager.java:362)
at org.eclipse.persistence.sessions.coordination.MergeChangeSet Command.executeWithSession(MergeChangeSetCommand.java:82)
at org.eclipse.persistence.internal.sessions.AbstractSession.pr ocessCommand(AbstractSession.java:3305)



"WMQJCAResourceAdapter : 84" daemon prio=3 tid=0x0000000102fa2000 nid=0xa0a in Object.wait() [0xfffffffd428fb000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.get ObjectFromSharedCacheForMerge(UnitOfWorkImpl.java:2674)
- locked <0xfffffffd8b912520> (a org.eclipse.persistence.internal.helper.ConcurrencyManager)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.get OriginalVersionOfObjectOrNull(UnitOfWorkImpl.java:2555)
at org.eclipse.persistence.internal.sessions.MergeManager.merge ChangesOfWorkingCopyIntoOriginal(MergeManager.java:582)
at org.eclipse.persistence.internal.sessions.MergeManager.merge Changes(MergeManager.java:263)
at org.eclipse.persistence.mappings.CollectionMapping.mergeInto Object(CollectionMapping.java:1404)
- locked <0xfffffffdd6c985b8> (a java.util.ArrayList)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.m ergeIntoObject(ObjectBuilder.java:2678)
at org.eclipse.persistence.internal.sessions.MergeManager.merge ChangesOfWorkingCopyIntoOriginal(MergeManager.java:597)
at org.eclipse.persistence.internal.sessions.MergeManager.merge Changes(MergeManager.java:263)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.mer geChangesIntoParent(UnitOfWorkImpl.java:3282)
at org.eclipse.persistence.internal.sessions.RepeatableWriteUni tOfWork.mergeChangesIntoParent(RepeatableWriteUnitOfWork.jav a:293)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.mer geClonesAfterCompletion(UnitOfWorkImpl.java:3414)
at org.eclipse.persistence.transaction.AbstractSynchronizationL istener.afterCompletion(AbstractSynchronizationListener.java :213)
at org.eclipse.persistence.transaction.JTASynchronizationListen er.afterCompletion(JTASynchronizationListener.java:79)

Report message to a moderator

Re: CacheKey getMutex() is not thread-safe [message #654974 is a reply to message #654960] Thu, 17 February 2011 15:32 Go to previous messageGo to next message
Chris Delahunt is currently offline Chris Delahunt
Messages: 863
Registered: July 2009
Senior Member
Hello,

I'm not sure what is causing the issue, but the thread dump shows that 3 threads are waiting because they need an object that another process should be building. Please try calling session.getIdentityMapAccessor().printIdentityMapLocks() to see which threads might contain the locks when you encounter this situation. This should help with the full thread dump to figure out whihc thread is building the object the others are waiting on, and figure out why it might get stuck - as it isn't apparent these 3 threads are stuck in a deadlock.

Also, what version of EclipseLink are you using? If possible, please try a later version, as this code has been refactored quite a bit. It is the quickest way to rule out fixed bugs, and makes diagnosing the issue easier.

Best Regards,
Chris

Report message to a moderator

Re: CacheKey getMutex() is not thread-safe [message #655006 is a reply to message #654974] Thu, 17 February 2011 18:25 Go to previous messageGo to next message
Jason Schulz is currently offline Jason Schulz
Messages: 4
Registered: February 2011
Junior Member
We're currently using version 2.0.2 of EclipseLink, although I noticed that the double-checked locking code also exists in the 2.1.2 version of CacheKey. So far, this problem has yet to be reproduced outside of our production environment where we cannot simply upgrade to a more recent version or print the identity map locks.

Thread dump lines match up to source in eclipselink-src-2.0.2.v20100323-r6872.zip.

Report message to a moderator

Re: CacheKey getMutex() is not thread-safe [message #656848 is a reply to message #655006] Mon, 28 February 2011 14:00 Go to previous message
James Sutherland is currently offline James Sutherland
Messages: 1844
Registered: July 2009
Senior Member
CacheKey getMutex cannot be concurrently called, so there is no issue with this code. The mutex is always created and acquired before the CacheKey is put into the cache.

Your deadlock is probably related to other issues, not this code.

I would recommend upgrading to the latest EclipseLink version as there have been some fixes in this area.

Resolving deadlocks is normally an involved process, so if upgrading does not resolve the issue, logging a bug or contacting support if you have a support contract would be a good next step. (from your stack this looks like an issue that was fixed).

James : Wiki : Book : Blog

Report message to a moderator

Previous Topic:Foreign Key Constraint Issue Involving One-To-Many and Many-To-One Relationship in EcliplseLink2.1.
Next Topic:CriteriaQuery with ManyToMany eager join fetch
Goto Forum:
  

[ Syndicate this forum (XML) ] [ RSS ]

Current Time: Fri May 24 06:06:47 EDT 2013

Powered by FUDForum. Page generated in 0.06828 seconds