Eclipse Community Forums: EclipseLink » CacheKey getMutex() is not thread-safe

Home » Eclipse Projects » EclipseLink » CacheKey getMutex() is not thread-safe(Suspected source of application deadlocks)

Show: Today's Messages :: Show Polls :: Message Navigator

CacheKey getMutex() is not thread-safe [message #654960]

Thu, 17 February 2011 18:47

Jason Schulz

Messages: 4
Registered: February 2011

Junior Member

The getMutex method in CacheKey uses double-check locking to instantiate member variable 'mutex'. Note that this ConcurrencyManager object is not declared as 'volatile'.

This code is the suspected cause in a series of application deadlocks. A set of partial thread dumps from a recent deadlock incident follows:

"WMQJCAResourceAdapter : 93" daemon prio=3 tid=0x0000000102543800 nid=0xa1e waiting on condition [0xfffffffd48dfb000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.eclipse.persistence.internal.helper.ConcurrencyManager.r eleaseDeferredLock(ConcurrencyManager.java:454)
at org.eclipse.persistence.internal.identitymaps.CacheKey.relea seDeferredLock(CacheKey.java:426)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.b uildObject(ObjectBuilder.java:779)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.b uildWorkingCopyCloneNormally(ObjectBuilder.java:583)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.b uildObjectInUnitOfWork(ObjectBuilder.java:552)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.b uildObject(ObjectBuilder.java:492)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.b uildObject(ObjectBuilder.java:444)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildOb ject(ObjectLevelReadQuery.java:635)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.conform IndividualResult(ObjectLevelReadQuery.java:799)
at org.eclipse.persistence.queries.ReadAllQuery.conformResult(R eadAllQuery.java:335)
at org.eclipse.persistence.queries.ReadAllQuery.registerResultI nUnitOfWork(ReadAllQuery.java:821)
at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLe velReadQuery(ReadAllQuery.java:464)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute DatabaseQuery(ObjectLevelReadQuery.java:997)
at org.eclipse.persistence.queries.DatabaseQuery.execute(Databa seQuery.java:675)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute (ObjectLevelReadQuery.java:958)
at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAll Query.java:432)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute InUnitOfWork(ObjectLevelReadQuery.java:1021)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.int ernalExecuteQuery(UnitOfWorkImpl.java:2898)
at org.eclipse.persistence.internal.sessions.AbstractSession.ex ecuteQuery(AbstractSession.java:1225)
at org.eclipse.persistence.internal.sessions.AbstractSession.ex ecuteQuery(AbstractSession.java:1207)
at org.eclipse.persistence.internal.sessions.AbstractSession.ex ecuteQuery(AbstractSession.java:1167)

"WMQJCAResourceAdapter : 90" daemon prio=3 tid=0x0000000101fe5000 nid=0xa1a in Object.wait() [0xfffffffd46dfe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.eclipse.persistence.internal.helper.ConcurrencyManager.a cquireReadLock(ConcurrencyManager.java:242)
- locked <0xfffffffdd6be05b8> (a org.eclipse.persistence.internal.helper.ConcurrencyManager)
at org.eclipse.persistence.internal.helper.ConcurrencyManager.c heckReadLock(ConcurrencyManager.java:230)
at org.eclipse.persistence.internal.identitymaps.CacheKey.check ReadLock(CacheKey.java:180)
at org.eclipse.persistence.internal.identitymaps.IdentityMapMan ager.getFromIdentityMap(IdentityMapManager.java:702)
at org.eclipse.persistence.internal.identitymaps.IdentityMapMan ager.getFromIdentityMap(IdentityMapManager.java:667)
at org.eclipse.persistence.internal.sessions.ObjectChangeSet.ge tTargetVersionOfSourceObject(ObjectChangeSet.java:367)
at org.eclipse.persistence.internal.sessions.MergeManager.merge ChangesFromChangeSet(MergeManager.java:362)
at org.eclipse.persistence.sessions.coordination.MergeChangeSet Command.executeWithSession(MergeChangeSetCommand.java:82)
at org.eclipse.persistence.internal.sessions.AbstractSession.pr ocessCommand(AbstractSession.java:3305)

"WMQJCAResourceAdapter : 84" daemon prio=3 tid=0x0000000102fa2000 nid=0xa0a in Object.wait() [0xfffffffd428fb000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.get ObjectFromSharedCacheForMerge(UnitOfWorkImpl.java:2674)
- locked <0xfffffffd8b912520> (a org.eclipse.persistence.internal.helper.ConcurrencyManager)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.get OriginalVersionOfObjectOrNull(UnitOfWorkImpl.java:2555)
at org.eclipse.persistence.internal.sessions.MergeManager.merge ChangesOfWorkingCopyIntoOriginal(MergeManager.java:582)
at org.eclipse.persistence.internal.sessions.MergeManager.merge Changes(MergeManager.java:263)
at org.eclipse.persistence.mappings.CollectionMapping.mergeInto Object(CollectionMapping.java:1404)
- locked <0xfffffffdd6c985b8> (a java.util.ArrayList)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.m ergeIntoObject(ObjectBuilder.java:2678)
at org.eclipse.persistence.internal.sessions.MergeManager.merge ChangesOfWorkingCopyIntoOriginal(MergeManager.java:597)
at org.eclipse.persistence.internal.sessions.MergeManager.merge Changes(MergeManager.java:263)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.mer geChangesIntoParent(UnitOfWorkImpl.java:3282)
at org.eclipse.persistence.internal.sessions.RepeatableWriteUni tOfWork.mergeChangesIntoParent(RepeatableWriteUnitOfWork.jav a:293)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.mer geClonesAfterCompletion(UnitOfWorkImpl.java:3414)
at org.eclipse.persistence.transaction.AbstractSynchronizationL istener.afterCompletion(AbstractSynchronizationListener.java :213)
at org.eclipse.persistence.transaction.JTASynchronizationListen er.afterCompletion(JTASynchronizationListener.java:79)

Report message to a moderator

Re: CacheKey getMutex() is not thread-safe [message #654974 is a reply to message #654960]

Thu, 17 February 2011 20:32

Chris Delahunt

Messages: 1389
Registered: July 2009

Senior Member

Hello,

I'm not sure what is causing the issue, but the thread dump shows that 3 threads are waiting because they need an object that another process should be building. Please try calling session.getIdentityMapAccessor().printIdentityMapLocks() to see which threads might contain the locks when you encounter this situation. This should help with the full thread dump to figure out whihc thread is building the object the others are waiting on, and figure out why it might get stuck - as it isn't apparent these 3 threads are stuck in a deadlock.

Also, what version of EclipseLink are you using? If possible, please try a later version, as this code has been refactored quite a bit. It is the quickest way to rule out fixed bugs, and makes diagnosing the issue easier.

Best Regards,
Chris

Report message to a moderator

Re: CacheKey getMutex() is not thread-safe [message #655006 is a reply to message #654974]

Thu, 17 February 2011 23:25

Jason Schulz

Messages: 4
Registered: February 2011

Junior Member

We're currently using version 2.0.2 of EclipseLink, although I noticed that the double-checked locking code also exists in the 2.1.2 version of CacheKey. So far, this problem has yet to be reproduced outside of our production environment where we cannot simply upgrade to a more recent version or print the identity map locks.

Thread dump lines match up to source in eclipselink-src-2.0.2.v20100323-r6872.zip.

Report message to a moderator

Re: CacheKey getMutex() is not thread-safe [message #656848 is a reply to message #655006]

Mon, 28 February 2011 19:00

James Sutherland

Messages: 1939
Registered: July 2009
Location: Ottawa, Canada

Senior Member

CacheKey getMutex cannot be concurrently called, so there is no issue with this code. The mutex is always created and acquired before the CacheKey is put into the cache.

Your deadlock is probably related to other issues, not this code.

I would recommend upgrading to the latest EclipseLink version as there have been some fixes in this area.

Resolving deadlocks is normally an involved process, so if upgrading does not resolve the issue, logging a bug or contacting support if you have a support contract would be a good next step. (from your stack this looks like an issue that was fixed).

James : Wiki : Book : Blog : Twitter

Report message to a moderator

Previous Topic:	Foreign Key Constraint Issue Involving One-To-Many and Many-To-One Relationship in EcliplseLink2.1.
Next Topic:	CriteriaQuery with ManyToMany eager join fetch

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Fri Apr 19 01:31:54 GMT 2024

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter