Eclipse Community Forums: EclipseLink

Help

Home

Home » Eclipse Projects » EclipseLink » Hung threads

Show: Today's Messages :: Show Polls :: Message Navigator

Hung threads [message #1129321]

Tue, 08 October 2013 14:13

Amita Mehta

Messages: 25
Registered: May 2013

Junior Member

We are using EclipseLink 2.5.1.v20130918-f2b9fc5 with WAS8

We are getting hung threads and a lot of time is being spent in the getChildObject call. Sometimes it happens for a one to one relationship and sometimes for a one to many.

We have removed all the joins.
We were having this problem initially with TopLink and so upgraded to EclipseLink but get the same same problem.
Any suggestions and solutions are appreciated.
Thanks.

Log

at org/eclipse/persistence/internal/helper/ConcurrencyManager.acquire(ConcurrencyManager.java:80(Compiled Code))
at org/eclipse/persistence/internal/identitymaps/CacheKey.acquire(CacheKey.java:132(Compiled Code))
at org/eclipse/persistence/internal/identitymaps/AbstractIdentityMap.acquireLock(AbstractIdentityMap.java:108(Compiled Code))
at org/eclipse/persistence/internal/identitymaps/IdentityMapManager.acquireLock(IdentityMapManager.java:157(Compiled Code))
at org/eclipse/persistence/internal/sessions/IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:99(Compiled Code))
at org/eclipse/persistence/internal/sessions/IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:90(Compiled Code))
at org/eclipse/persistence/internal/sessions/AbstractSession.retrieveCacheKey(AbstractSession.java:5143(Compiled Code))
at org/eclipse/persistence/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:964(Compiled Code))
at org/eclipse/persistence/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:736(Compiled Code))
at org/eclipse/persistence/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:688(Compiled Code))
at org/eclipse/persistence/queries/ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795(Compiled Code))
at org/eclipse/persistence/queries/ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:554(Compiled Code))
at org/eclipse/persistence/queries/ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168(Compiled Code))
at org/eclipse/persistence/queries/DatabaseQuery.execute(DatabaseQuery.java:899(Compiled Code))
at org/eclipse/persistence/queries/ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127(Compiled Code))
at org/eclipse/persistence/queries/ReadObjectQuery.execute(ReadObjectQuery.java:431(Compiled Code))
at org/eclipse/persistence/internal/sessions/AbstractSession.internalExecuteQuery(AbstractSession.java:3203(Compiled Code))
at org/eclipse/persistence/internal/sessions/AbstractSession.executeQuery(AbstractSession.java:1793(Compiled Code))
at org/eclipse/persistence/internal/sessions/AbstractSession.executeQuery(AbstractSession.java:1775(Compiled Code))
at org/eclipse/persistence/internal/indirection/NoIndirectionPolicy.valueFromQuery(NoIndirectionPolicy.java:326(Compiled Code))
at org/eclipse/persistence/mappings/ForeignReferenceMapping.valueFromRowInternal(ForeignReferenceMapping.java:2294(Compiled Code))
at org/eclipse/persistence/mappings/OneToOneMapping.valueFromRowInternal(OneToOneMapping.java:1795(Compiled Code))
at org/eclipse/persistence/mappings/ForeignReferenceMapping.valueFromRow(ForeignReferenceMapping.java:2144(Compiled Code))
at org/eclipse/persistence/mappings/ForeignReferenceMapping.readFromRowIntoObject(ForeignReferenceMapping.java:1471(Compiled Code))
at org/eclipse/persistence/internal/descriptors/ObjectBuilder.buildAttributesIntoObject(ObjectBuilder.java:461(Compiled Code))
at org/eclipse/persistence/internal/descriptors/ObjectBuilder.refreshObjectIfRequired(ObjectBuilder.java:4300(Compiled Code))
at org/eclipse/persistence/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:1039(Compiled Code))
at org/eclipse/persistence/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:736(Compiled Code))
at org/eclipse/persistence/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:688(Compiled Code))
at org/eclipse/persistence/queries/ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795(Compiled Code))
at org/eclipse/persistence/queries/ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:554(Compiled Code))
at org/eclipse/persistence/queries/ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168(Compiled Code))
at org/eclipse/persistence/queries/DatabaseQuery.execute(DatabaseQuery.java:899(Compiled Code))
at org/eclipse/persistence/queries/ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127(Compiled Code))
at org/eclipse/persistence/queries/ReadObjectQuery.execute(ReadObjectQuery.java:431(Compiled Code))
at org/eclipse/persistence/internal/sessions/AbstractSession.internalExecuteQuery(AbstractSession.java:3203(Compiled Code))
at org/eclipse/persistence/internal/sessions/AbstractSession.executeQuery(AbstractSession.java:1793(Compiled Code))
at org/eclipse/persistence/internal/sessions/AbstractSession.executeQuery(AbstractSession.java:1775(Compiled Code))
at org/eclipse/persistence/internal/indirection/QueryBasedValueHolder.instantiate(QueryBasedValueHolder.java:129(Compiled Code))
at org/eclipse/persistence/internal/indirection/QueryBasedValueHolder.instantiate(QueryBasedValueHolder.java:116(Compiled Code))

Report message to a moderator

Re: Hung threads [message #1129432 is a reply to message #1129321]

Tue, 08 October 2013 16:16

Chris Delahunt

Messages: 1379
Registered: July 2009

Senior Member

You will need to obtain the full thread dump to track down which threads are involved in issue, and also call session.getIdentityMapAccessor().printIdentityMapLocks() to see which threads own which locks for the objects involved. See http://wiki.eclipse.org/EclipseLink/FAQ/JPA#How_to_diagnose_and_resolve_hangs_and_deadlocks.3F for potential workarounds, and contact support if you are able to. The stack trace shown does not use indirection on the OneToOne mapping on the object being read in, so this might be a good place to start if you are able to make changes. It helps to isolate the object involved and see if this is a true deadlock or just a severe bottleneck caused by thread contention on a single object.

Best Regards,
Chris

Report message to a moderator

Re: Hung threads [message #1130768 is a reply to message #1129432]

Wed, 09 October 2013 22:58

Amita Mehta

Messages: 25
Registered: May 2013

Junior Member

On further analysis I see that I am fetching the 1-m child records by an expression query. The problems lies with setting refreshIdentityMapResult on the query.
If I comment out the line - query.refreshIdentityMapResult();
I am not seeing hung threads so far atleast in the dozen times I ran my unit test. I still have to test it more since with TopLink the hung thread was very sporadic.

What does refreshIdentityMapResult do to lock multiple threads?
What are the possible problems I need to look out if I do not call refreshIdentityMapResult?

Thanks, Amita

Report message to a moderator

Re: Hung threads [message #1131359 is a reply to message #1130768]

Thu, 10 October 2013 12:56

Chris Delahunt

Messages: 1379
Registered: July 2009

Senior Member

refreshIdentityMapResult does just what it says - it refreshes the objects being read in with the data from the database. It must lock the objects though so that any other threads that might be trying to access those objects (such as for other reads or writes) wait until it is done updating them, otherwise the data would be inconsistent and cause concurrency issues.

Care must always be taken with the refresh settings, but they are commonly used to avoid stale data. Problems that might occur from removing refreshing could be a greater occurrence of optimistic lock exceptions.

Report message to a moderator

Re: Hung threads [message #1131474 is a reply to message #1131359]

Thu, 10 October 2013 15:05

Amita Mehta

Messages: 25
Registered: May 2013

Junior Member

Thanks Chris for explanation.

The problem is that the locks never get released. I have let the process run over 2 days and the locks are still not released.
Another curious thing is that I have named queries that have this setting set in the workbench but I haven't seen the named queries hang so far. Is the locking different for the different types of queries?
We migrated from TopLink to EclipseLink for this hung thread problems and are not using JPA.

Report message to a moderator

Re: Hung threads [message #1131830 is a reply to message #1129432]

Thu, 10 October 2013 20:47

Amita Mehta

Messages: 25
Registered: May 2013

Junior Member

How does EclipseLink support work? Do we use our existing Oracle support?
Thanks

Report message to a moderator

Re: Hung threads [message #1499857 is a reply to message #1129321]

Fri, 05 December 2014 13:23

Pawel Zak

Messages: 8
Registered: November 2012

Junior Member

These locks appeared on our customer environment - we cannot reproduce this on our side.
They sent us few thread dumps and we can see that 14 threads have been waiting for more than six days.

We have 12 threads which are locked and waiting on the same object (all of them try to execute the same query). Here is one of them:

"ORBacus:ThreadPool-0:Dispatcher-13" nid=91 state=WAITING
    - waiting on <0x6f47f9ad> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)
    - locked <0x6f47f9ad> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:503)
    at org.eclipse.persistence.internal.helper.ConcurrencyManager.acquire(ConcurrencyManager.java:80)
    at org.eclipse.persistence.internal.identitymaps.CacheKey.acquire(CacheKey.java:132)
    at org.eclipse.persistence.internal.identitymaps.AbstractIdentityMap.acquireLock(AbstractIdentityMap.java:108)
    at org.eclipse.persistence.internal.identitymaps.IdentityMapManager.acquireLock(IdentityMapManager.java:157)
    at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:99)
    at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:90)
    at org.eclipse.persistence.internal.sessions.AbstractSession.retrieveCacheKey(AbstractSession.java:5143)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:964)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneFromRow(ObjectBuilder.java:2239)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:847)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:734)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:688)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795)
    at org.eclipse.persistence.queries.ReadObjectQuery.registerResultInUnitOfWork(ReadObjectQuery.java:885)
    at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:552)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
    at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:431)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2910)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1793)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1775)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1726)
    at org.eclipse.persistence.internal.sessions.AbstractSession.checkAndRefreshInvalidObject(AbstractSession.java:1219)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:1013)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:969)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getAndCloneCacheKeyFromParent(UnitOfWorkIdentityMapAccessor.java:209)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getFromIdentityMap(UnitOfWorkIdentityMapAccessor.java:137)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3956)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3908)
    at org.eclipse.persistence.mappings.ObjectReferenceMapping.buildUnitofWorkCloneForPartObject(ObjectReferenceMapping.java:109)
    at org.eclipse.persistence.mappings.ObjectReferenceMapping.buildCloneForPartObject(ObjectReferenceMapping.java:71)
    at org.eclipse.persistence.internal.indirection.NoIndirectionPolicy.cloneAttribute(NoIndirectionPolicy.java:79)
    at org.eclipse.persistence.mappings.ForeignReferenceMapping.buildCloneFromRow(ForeignReferenceMapping.java:337)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildAttributesIntoWorkingCopyClone(ObjectBuilder.java:1996)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneFromRow(ObjectBuilder.java:2249)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:847)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:734)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:688)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795)
    at org.eclipse.persistence.queries.ReadAllQuery.registerResultInUnitOfWork(ReadAllQuery.java:890)
    at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:509)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
    at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:403)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2910)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1793)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1775)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1740)
    at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:258)
    at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:469)

Query which is being executed here:
Query query = em.createQuery("SELECT entity1 FROM Entity1 entity1 WHERE entity1.en2 = :en2 AND entity1.en3 = :en3)"
query.setParameter("en2", en2);
query.setParameter("en3", en3);
query.setHint(QueryHints.CACHE_USAGE, CacheUsage.ConformResultsInUnitOfWork);
query.getResultList();

Some relations from Entity1 class:
@OneToOne
@JoinColumn(name = "en2_id")
protected Entity2 en2

@ManyToOne(cascade = {CascadeType.PERSIST}, fetch = FetchType.LAZY)
@JoinColumn(name = "en3_id")
protected Entity3 en3;

And we have 2 threads which are locked and waiting on another object (but both on the same):

First thread:

"ORBacus:ThreadPool-0:Dispatcher-5" nid=83 state=WAITING
    - waiting on <0x1ee40373> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)
    - locked <0x1ee40373> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:503)
    at org.eclipse.persistence.internal.helper.ConcurrencyManager.acquire(ConcurrencyManager.java:80)
    at org.eclipse.persistence.internal.identitymaps.CacheKey.acquire(CacheKey.java:132)
    at org.eclipse.persistence.internal.identitymaps.AbstractIdentityMap.acquireLock(AbstractIdentityMap.java:108)
    at org.eclipse.persistence.internal.identitymaps.IdentityMapManager.acquireLock(IdentityMapManager.java:157)
    at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:99)
    at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:90)
    at org.eclipse.persistence.internal.sessions.AbstractSession.retrieveCacheKey(AbstractSession.java:5143)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:964)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneFromRow(ObjectBuilder.java:2239)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:847)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:734)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:688)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795)
    at org.eclipse.persistence.queries.ReadObjectQuery.registerResultInUnitOfWork(ReadObjectQuery.java:885)
    at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:552)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
    at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:431)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2910)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1793)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1775)
    at org.eclipse.persistence.internal.indirection.QueryBasedValueHolder.instantiate(QueryBasedValueHolder.java:129)
    at org.eclipse.persistence.internal.indirection.QueryBasedValueHolder.instantiate(QueryBasedValueHolder.java:116)
    at org.eclipse.persistence.internal.indirection.DatabaseValueHolder.getValue(DatabaseValueHolder.java:89)
    at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiateImpl(UnitOfWorkValueHolder.java:173)
    at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiate(UnitOfWorkValueHolder.java:234)
    at org.eclipse.persistence.internal.indirection.DatabaseValueHolder.getValue(DatabaseValueHolder.java:89)
      - locked org.eclipse.persistence.internal.indirection.UnitOfWorkQueryValueHolder@51456c1d
    at org.eclipse.persistence.internal.indirection.BasicIndirectionPolicy.getRealAttributeValueFromObject(BasicIndirectionPolicy.java:280)
    at org.eclipse.persistence.internal.indirection.WeavedObjectBasicIndirectionPolicy.getRealAttributeValueFromObject(WeavedObjectBasicIndirectionPolicy.java:73)
    at org.eclipse.persistence.internal.indirection.IndirectionPolicy.instantiateObject(IndirectionPolicy.java:220)
    at org.eclipse.persistence.mappings.ForeignReferenceMapping.buildCloneFromRow(ForeignReferenceMapping.java:345)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildAttributesIntoWorkingCopyClone(ObjectBuilder.java:1996)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneFromRow(ObjectBuilder.java:2249)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:847)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:734)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:688)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795)
    at org.eclipse.persistence.queries.ReadObjectQuery.registerResultInUnitOfWork(ReadObjectQuery.java:885)
    at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:552)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
    at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:431)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2910)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1793)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1775)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1726)
    at org.eclipse.persistence.internal.sessions.AbstractSession.checkAndRefreshInvalidObject(AbstractSession.java:1219)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:1013)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:969)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getAndCloneCacheKeyFromParent(UnitOfWorkIdentityMapAccessor.java:209)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getFromIdentityMap(UnitOfWorkIdentityMapAccessor.java:137)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3956)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3908)
    at org.eclipse.persistence.mappings.ObjectReferenceMapping.buildUnitofWorkCloneForPartObject(ObjectReferenceMapping.java:109)
    at org.eclipse.persistence.mappings.ObjectReferenceMapping.buildCloneForPartObject(ObjectReferenceMapping.java:71)
    at org.eclipse.persistence.internal.indirection.NoIndirectionPolicy.cloneAttribute(NoIndirectionPolicy.java:79)
    at org.eclipse.persistence.mappings.ForeignReferenceMapping.buildCloneFromRow(ForeignReferenceMapping.java:337)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildAttributesIntoWorkingCopyClone(ObjectBuilder.java:1996)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneFromRow(ObjectBuilder.java:2249)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:847)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:734)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:688)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795)
    at org.eclipse.persistence.queries.ReadAllQuery.registerResultInUnitOfWork(ReadAllQuery.java:890)
    at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:509)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
    at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:403)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2910)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1793)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1775)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1740)
    at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:258)
    at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:469)

Query which is being executed in here:
Query query = em.createQuery("SELECT entity3 FROM Entity3 entity3 WHERE entity3.entity4.id = " + entity4_id);
query.getResultList()

This relation in Entity3 class:

@OneToOne(fetch = FetchType.LAZY, cascade = CascadeType.PERSIST)
@JoinColumn(name = "entity4_id")
private Entity4 entity4;

Second thread:

"ORBacus:ThreadPool-0:Dispatcher-7" nid=85 state=WAITING
    - waiting on <0x1ee40373> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)
    - locked <0x1ee40373> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:503)
    at org.eclipse.persistence.internal.helper.ConcurrencyManager.acquire(ConcurrencyManager.java:80)
    at org.eclipse.persistence.internal.identitymaps.CacheKey.acquire(CacheKey.java:132)
    at org.eclipse.persistence.internal.identitymaps.AbstractIdentityMap.acquireLock(AbstractIdentityMap.java:108)
    at org.eclipse.persistence.internal.identitymaps.IdentityMapManager.acquireLock(IdentityMapManager.java:157)
    at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:99)
    at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:90)
    at org.eclipse.persistence.internal.sessions.AbstractSession.retrieveCacheKey(AbstractSession.java:5143)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:964)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneFromRow(ObjectBuilder.java:2239)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:847)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:734)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:688)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795)
    at org.eclipse.persistence.queries.ReadObjectQuery.registerResultInUnitOfWork(ReadObjectQuery.java:885)
    at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:552)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
    at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:431)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2910)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1793)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1775)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1726)
    at org.eclipse.persistence.internal.sessions.AbstractSession.checkAndRefreshInvalidObject(AbstractSession.java:1219)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:1013)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:969)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getAndCloneCacheKeyFromParent(UnitOfWorkIdentityMapAccessor.java:209)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getFromIdentityMap(UnitOfWorkIdentityMapAccessor.java:137)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3956)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3908)
    at org.eclipse.persistence.mappings.ObjectReferenceMapping.buildUnitofWorkCloneForPartObject(ObjectReferenceMapping.java:109)
    at org.eclipse.persistence.mappings.ObjectReferenceMapping.buildCloneForPartObject(ObjectReferenceMapping.java:71)
    at org.eclipse.persistence.internal.indirection.UnitOfWorkQueryValueHolder.buildCloneFor(UnitOfWorkQueryValueHolder.java:60)
    at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiateImpl(UnitOfWorkValueHolder.java:173)
    at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiate(UnitOfWorkValueHolder.java:234)
    at org.eclipse.persistence.internal.indirection.DatabaseValueHolder.getValue(DatabaseValueHolder.java:89)
    at com.package.Entity5._persistence_get_entity6(Entity5.java)

This relation in Entity5 class:

@OneToOne(fetch = FetchType.LAZY)
@JoinColumn(name = "entity6_id")
private Entity6 entity6;

Do you have any ideas how to fix this issue?

EclipseLink version: 2.5.1

Persistence Unit properties:
<property name="eclipselink.weaving" value="static"/>
<property name="eclipselink.weaving.internal" value="false"/>
<property name="eclipselink.weaving.fetchgroups" value="false"/>

Report message to a moderator

Re: Hung threads [message #1667960 is a reply to message #1129321]

Thu, 12 March 2015 09:09

Pawel Zak

Messages: 8
Registered: November 2012

Junior Member

Problem still appears on 2.5.2.

It seems that we are able to reproduce this after some time (10hours - few days) of intensive tests.

Properties and settings:

<property name="eclipselink.weaving" value="static"/>
<property name="eclipselink.weaving.internal" value="false"/>
<property name="eclipselink.weaving.fetchgroups" value="false"/>
ConcurrencyManager.setShouldTrackStack(true);
DeferredLockManager.SHOULD_USE_DEFERRED_LOCKS = false;
JoinFetching is not used
EntityManager is not shared between threads

After deadlock appeared and stayed for this case:

Name: ORBacus:ThreadPool-0:Dispatcher-0
State: WAITING on org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey@5a010513
Total blocked: 14 727  Total waited: 14 882

Stack trace: 
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
org.eclipse.persistence.internal.helper.ConcurrencyManager.acquire(ConcurrencyManager.java:80)
org.eclipse.persistence.internal.identitymaps.CacheKey.acquire(CacheKey.java:132)
org.eclipse.persistence.internal.identitymaps.AbstractIdentityMap.acquireLock(AbstractIdentityMap.java:108)
org.eclipse.persistence.internal.identitymaps.IdentityMapManager.acquireLock(IdentityMapManager.java:157)
org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:99)
org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:90)
org.eclipse.persistence.internal.sessions.AbstractSession.retrieveCacheKey(AbstractSession.java:5155)
org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:964)
org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneFromRow(ObjectBuilder.java:2239)
org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:847)
org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:734)
org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:688)
org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795)
org.eclipse.persistence.queries.ReadObjectQuery.registerResultInUnitOfWork(ReadObjectQuery.java:885)
org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:552)
org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168)
org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:431)
org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2910)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1804)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1786)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1737)
org.eclipse.persistence.internal.jpa.EntityManagerImpl.executeQuery(EntityManagerImpl.java:900)
org.eclipse.persistence.internal.jpa.EntityManagerImpl.refresh(EntityManagerImpl.java:1040)
org.eclipse.persistence.internal.jpa.EntityManagerImpl.refresh(EntityManagerImpl.java:929)

I printed locks and here is the result:

Current object locks:

End of locked objects.

So no lock objects in this case. I can patch eclipselink and add more logging to find out what is wrong, but please advice: what and where?

Report message to a moderator

Re: Hung threads [message #1706212 is a reply to message #1667960]

Tue, 25 August 2015 01:11

Sanjath Shringeri

Messages: 2
Registered: August 2015

Junior Member

Any update on this? We are also having similar issue and I do see a deadlock on the DB at that time. But, the deadlock seems to be caused by couple of SQLs initiated by eclipse link (setting null values to 'order' column)

Report message to a moderator

Re: Hung threads [message #1706421 is a reply to message #1706212]

Wed, 26 August 2015 18:51

Chris Delahunt

Messages: 1379
Registered: July 2009

Senior Member

Similar issues can occur all the time, but it is not necessarily a bug or a true deadlock. If one thread is building an object, any other threads must wait until that thread is done. Depending on your model, there are any number of reasons this might cause a bottleneck, such as if you have a large model constantly being refreshed as just one example. True database deadlocks also cause issues, but generally with writes, not reads.

In the thread dump provided by Pawel, the issue still is not obvious as it does not show the thread that holds the lock, only the ones backing up waiting for it. The first set of 12 threads are building an entity1 instance and trying to build one of the references that is eagerly fetched - presumably entity2 instances. A quick solution would be to make this relationship lazy and these 12 threads would be able to skip having to wait for the lock, but it does not show an actual deadlock situation. This is why the session.getIdentityMapAccessor().printIdentityMapLocks() I suggested earlier is important, as it will show the threads that hold each lock, allowing you to correlate what each thread in the dump is doing to the locks it holds. From there, you can identify which thread is causing the deadlock (or bottleneck) and decide what should be done to prevent it.

Best Regards,
Chris

Report message to a moderator

Re: Hung threads [message #1706616 is a reply to message #1129321]

Fri, 28 August 2015 07:46

Sanjath Shringeri

Messages: 2
Registered: August 2015

Junior Member

Deadlock we observed may be unrelated, we could see deadlock even in good runs which seems to get rolledback and our application code has retry mechanism. So, that may have been false alarm. However, when we set the default caching to 'isolated' level (eclipselink.cache.shared.default=false) seems to stop the problem. But, that is anyway related to the location where it is getting hung. By making the cache to be isolated the code path is completely avoided. My suspicion is that in some strange combination, the object lock acquired in org.eclipse.persistence.internal.identitymaps.FullIdentityMap.remove() method isn't released ( I dont see any of the release methods in the finally block)

In our thread stack we did not see any other thread that may be holding on the lock.

Stack trace when the object lock is hung is this (version: 2.6.0)
"main" #1 prio=5 os_prio=0 tid=0x00007f5ab4009800 nid=0xce4f in Object.wait() [0x00007f5abacef000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.eclipse.persistence.internal.helper.ConcurrencyManager.acquire(ConcurrencyManager.java:80)
- locked <0x000000067f5c4608> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)
at org.eclipse.persistence.internal.identitymaps.CacheKey.acquire(CacheKey.java:132)
at org.eclipse.persistence.internal.identitymaps.FullIdentityMap.remove(FullIdentityMap.java:212)
at org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap.remove(HardCacheWeakIdentityMap.java:88)
at org.eclipse.persistence.internal.identitymaps.AbstractIdentityMap.remove(AbstractIdentityMap.java:421)
at org.eclipse.persistence.internal.identitymaps.IdentityMapManager.removeFromIdentityMap(IdentityMapManager.java:1491)
at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.removeFromIdentityMap(IdentityMapAccessor.java:1150)
at org.eclipse.persistence.internal.sessions.IsolatedClientSessionIdentityMapAccessor.removeFromIdentityMap(IsolatedClientSessionIdentityMapAccessor.java:692)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.postMergeChanges(UnitOfWorkImpl.java:3699)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.mergeChangesIntoParent(UnitOfWorkImpl.java:3313)
at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.mergeChangesIntoParent(RepeatableWriteUnitOfWork.java:379)
at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:291)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)
at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)

Report message to a moderator

Re: Hung threads [message #1707418 is a reply to message #1706616]

Fri, 04 September 2015 15:13

Chris Delahunt

Messages: 1379
Registered: July 2009

Senior Member

Locks in this stack are obtained for the transaction merge process, and should be released only when the entire commit process completes. You will need to use the printidentitymaps call mentioned previously to find what other threads might have eclipselink locks, and use the thread dump and logs to track what they might have been doing.

Report message to a moderator

Previous Topic:	Entity is not updated
Next Topic:	Entity Update throws NPE for @ElementColection of Map - Eclipselink 2.5.2

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Thu Feb 25 16:49:51 GMT 2021

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter