Eclipse Community Forums: EclipseLink » CacheKey getMutex() is not thread-safe

Home » Eclipse Projects » EclipseLink » CacheKey getMutex() is not thread-safe(Suspected source of application deadlocks)

CacheKey getMutex() is not thread-safe [message #654960]

Thu, 17 February 2011 13:47

Eclipse User

The getMutex method in CacheKey uses double-check locking to instantiate member variable 'mutex'. Note that this ConcurrencyManager object is not declared as 'volatile'.

This code is the suspected cause in a series of application deadlocks. A set of partial thread dumps from a recent deadlock incident follows:

"WMQJCAResourceAdapter : 93" daemon prio=3 tid=0x0000000102543800 nid=0xa1e waiting on condition [0xfffffffd48dfb000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.eclipse.persistence.internal.helper.ConcurrencyManager.r eleaseDeferredLock(ConcurrencyManager.java:454)
at org.eclipse.persistence.internal.identitymaps.CacheKey.relea seDeferredLock(CacheKey.java:426)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.b uildObject(ObjectBuilder.java:779)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.b uildWorkingCopyCloneNormally(ObjectBuilder.java:583)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.b uildObjectInUnitOfWork(ObjectBuilder.java:552)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.b uildObject(ObjectBuilder.java:492)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.b uildObject(ObjectBuilder.java:444)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildOb ject(ObjectLevelReadQuery.java:635)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.conform IndividualResult(ObjectLevelReadQuery.java:799)
at org.eclipse.persistence.queries.ReadAllQuery.conformResult(R eadAllQuery.java:335)
at org.eclipse.persistence.queries.ReadAllQuery.registerResultI nUnitOfWork(ReadAllQuery.java:821)
at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLe velReadQuery(ReadAllQuery.java:464)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute DatabaseQuery(ObjectLevelReadQuery.java:997)
at org.eclipse.persistence.queries.DatabaseQuery.execute(Databa seQuery.java:675)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute (ObjectLevelReadQuery.java:958)
at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAll Query.java:432)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute InUnitOfWork(ObjectLevelReadQuery.java:1021)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.int ernalExecuteQuery(UnitOfWorkImpl.java:2898)
at org.eclipse.persistence.internal.sessions.AbstractSession.ex ecuteQuery(AbstractSession.java:1225)
at org.eclipse.persistence.internal.sessions.AbstractSession.ex ecuteQuery(AbstractSession.java:1207)
at org.eclipse.persistence.internal.sessions.AbstractSession.ex ecuteQuery(AbstractSession.java:1167)

"WMQJCAResourceAdapter : 90" daemon prio=3 tid=0x0000000101fe5000 nid=0xa1a in Object.wait() [0xfffffffd46dfe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.eclipse.persistence.internal.helper.ConcurrencyManager.a cquireReadLock(ConcurrencyManager.java:242)
- locked <0xfffffffdd6be05b8> (a org.eclipse.persistence.internal.helper.ConcurrencyManager)
at org.eclipse.persistence.internal.helper.ConcurrencyManager.c heckReadLock(ConcurrencyManager.java:230)
at org.eclipse.persistence.internal.identitymaps.CacheKey.check ReadLock(CacheKey.java:180)
at org.eclipse.persistence.internal.identitymaps.IdentityMapMan ager.getFromIdentityMap(IdentityMapManager.java:702)
at org.eclipse.persistence.internal.identitymaps.IdentityMapMan ager.getFromIdentityMap(IdentityMapManager.java:667)
at org.eclipse.persistence.internal.sessions.ObjectChangeSet.ge tTargetVersionOfSourceObject(ObjectChangeSet.java:367)
at org.eclipse.persistence.internal.sessions.MergeManager.merge ChangesFromChangeSet(MergeManager.java:362)
at org.eclipse.persistence.sessions.coordination.MergeChangeSet Command.executeWithSession(MergeChangeSetCommand.java:82)
at org.eclipse.persistence.internal.sessions.AbstractSession.pr ocessCommand(AbstractSession.java:3305)

"WMQJCAResourceAdapter : 84" daemon prio=3 tid=0x0000000102fa2000 nid=0xa0a in Object.wait() [0xfffffffd428fb000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.get ObjectFromSharedCacheForMerge(UnitOfWorkImpl.java:2674)
- locked <0xfffffffd8b912520> (a org.eclipse.persistence.internal.helper.ConcurrencyManager)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.get OriginalVersionOfObjectOrNull(UnitOfWorkImpl.java:2555)
at org.eclipse.persistence.internal.sessions.MergeManager.merge ChangesOfWorkingCopyIntoOriginal(MergeManager.java:582)
at org.eclipse.persistence.internal.sessions.MergeManager.merge Changes(MergeManager.java:263)
at org.eclipse.persistence.mappings.CollectionMapping.mergeInto Object(CollectionMapping.java:1404)
- locked <0xfffffffdd6c985b8> (a java.util.ArrayList)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.m ergeIntoObject(ObjectBuilder.java:2678)
at org.eclipse.persistence.internal.sessions.MergeManager.merge ChangesOfWorkingCopyIntoOriginal(MergeManager.java:597)
at org.eclipse.persistence.internal.sessions.MergeManager.merge Changes(MergeManager.java:263)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.mer geChangesIntoParent(UnitOfWorkImpl.java:3282)
at org.eclipse.persistence.internal.sessions.RepeatableWriteUni tOfWork.mergeChangesIntoParent(RepeatableWriteUnitOfWork.jav a:293)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.mer geClonesAfterCompletion(UnitOfWorkImpl.java:3414)
at org.eclipse.persistence.transaction.AbstractSynchronizationL istener.afterCompletion(AbstractSynchronizationListener.java :213)
at org.eclipse.persistence.transaction.JTASynchronizationListen er.afterCompletion(JTASynchronizationListener.java:79)

Re: CacheKey getMutex() is not thread-safe [message #654974 is a reply to message #654960]

Thu, 17 February 2011 15:32

Eclipse User

Hello,

I'm not sure what is causing the issue, but the thread dump shows that 3 threads are waiting because they need an object that another process should be building. Please try calling session.getIdentityMapAccessor().printIdentityMapLocks() to see which threads might contain the locks when you encounter this situation. This should help with the full thread dump to figure out whihc thread is building the object the others are waiting on, and figure out why it might get stuck - as it isn't apparent these 3 threads are stuck in a deadlock.

Also, what version of EclipseLink are you using? If possible, please try a later version, as this code has been refactored quite a bit. It is the quickest way to rule out fixed bugs, and makes diagnosing the issue easier.

Best Regards,
Chris

Re: CacheKey getMutex() is not thread-safe [message #655006 is a reply to message #654974]

Thu, 17 February 2011 18:25

Eclipse User

We're currently using version 2.0.2 of EclipseLink, although I noticed that the double-checked locking code also exists in the 2.1.2 version of CacheKey. So far, this problem has yet to be reproduced outside of our production environment where we cannot simply upgrade to a more recent version or print the identity map locks.

Thread dump lines match up to source in eclipselink-src-2.0.2.v20100323-r6872.zip.

Re: CacheKey getMutex() is not thread-safe [message #656848 is a reply to message #655006]

Mon, 28 February 2011 14:00

Eclipse User

CacheKey getMutex cannot be concurrently called, so there is no issue with this code. The mutex is always created and acquired before the CacheKey is put into the cache.

Your deadlock is probably related to other issues, not this code.

I would recommend upgrading to the latest EclipseLink version as there have been some fixes in this area.

Resolving deadlocks is normally an involved process, so if upgrading does not resolve the issue, logging a bug or contacting support if you have a support contract would be a good next step. (from your stack this looks like an issue that was fixed).

Previous Topic:	Foreign Key Constraint Issue Involving One-To-Many and Many-To-One Relationship in EcliplseLink2.1.
Next Topic:	CriteriaQuery with ManyToMany eager join fetch

Goto Forum:

-=] Back to Top [=-

Current Time: Wed Jul 23 13:46:08 EDT 2025

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter