Eclipse Community Forums: EclipseLink

Home » Eclipse Projects » EclipseLink » Hung threads

Tue, 08 October 2013 10:13

Eclipse User

We are using EclipseLink 2.5.1.v20130918-f2b9fc5 with WAS8

We are getting hung threads and a lot of time is being spent in the getChildObject call. Sometimes it happens for a one to one relationship and sometimes for a one to many.

We have removed all the joins.
We were having this problem initially with TopLink and so upgraded to EclipseLink but get the same same problem.
Any suggestions and solutions are appreciated.
Thanks.

Log

at org/eclipse/persistence/internal/helper/ConcurrencyManager.acquire(ConcurrencyManager.java:80(Compiled Code))
at org/eclipse/persistence/internal/identitymaps/CacheKey.acquire(CacheKey.java:132(Compiled Code))
at org/eclipse/persistence/internal/identitymaps/AbstractIdentityMap.acquireLock(AbstractIdentityMap.java:108(Compiled Code))
at org/eclipse/persistence/internal/identitymaps/IdentityMapManager.acquireLock(IdentityMapManager.java:157(Compiled Code))
at org/eclipse/persistence/internal/sessions/IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:99(Compiled Code))
at org/eclipse/persistence/internal/sessions/IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:90(Compiled Code))
at org/eclipse/persistence/internal/sessions/AbstractSession.retrieveCacheKey(AbstractSession.java:5143(Compiled Code))
at org/eclipse/persistence/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:964(Compiled Code))
at org/eclipse/persistence/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:736(Compiled Code))
at org/eclipse/persistence/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:688(Compiled Code))
at org/eclipse/persistence/queries/ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795(Compiled Code))
at org/eclipse/persistence/queries/ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:554(Compiled Code))
at org/eclipse/persistence/queries/ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168(Compiled Code))
at org/eclipse/persistence/queries/DatabaseQuery.execute(DatabaseQuery.java:899(Compiled Code))
at org/eclipse/persistence/queries/ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127(Compiled Code))
at org/eclipse/persistence/queries/ReadObjectQuery.execute(ReadObjectQuery.java:431(Compiled Code))
at org/eclipse/persistence/internal/sessions/AbstractSession.internalExecuteQuery(AbstractSession.java:3203(Compiled Code))
at org/eclipse/persistence/internal/sessions/AbstractSession.executeQuery(AbstractSession.java:1793(Compiled Code))
at org/eclipse/persistence/internal/sessions/AbstractSession.executeQuery(AbstractSession.java:1775(Compiled Code))
at org/eclipse/persistence/internal/indirection/NoIndirectionPolicy.valueFromQuery(NoIndirectionPolicy.java:326(Compiled Code))
at org/eclipse/persistence/mappings/ForeignReferenceMapping.valueFromRowInternal(ForeignReferenceMapping.java:2294(Compiled Code))
at org/eclipse/persistence/mappings/OneToOneMapping.valueFromRowInternal(OneToOneMapping.java:1795(Compiled Code))
at org/eclipse/persistence/mappings/ForeignReferenceMapping.valueFromRow(ForeignReferenceMapping.java:2144(Compiled Code))
at org/eclipse/persistence/mappings/ForeignReferenceMapping.readFromRowIntoObject(ForeignReferenceMapping.java:1471(Compiled Code))
at org/eclipse/persistence/internal/descriptors/ObjectBuilder.buildAttributesIntoObject(ObjectBuilder.java:461(Compiled Code))
at org/eclipse/persistence/internal/descriptors/ObjectBuilder.refreshObjectIfRequired(ObjectBuilder.java:4300(Compiled Code))
at org/eclipse/persistence/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:1039(Compiled Code))
at org/eclipse/persistence/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:736(Compiled Code))
at org/eclipse/persistence/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:688(Compiled Code))
at org/eclipse/persistence/queries/ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795(Compiled Code))
at org/eclipse/persistence/queries/ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:554(Compiled Code))
at org/eclipse/persistence/queries/ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168(Compiled Code))
at org/eclipse/persistence/queries/DatabaseQuery.execute(DatabaseQuery.java:899(Compiled Code))
at org/eclipse/persistence/queries/ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127(Compiled Code))
at org/eclipse/persistence/queries/ReadObjectQuery.execute(ReadObjectQuery.java:431(Compiled Code))
at org/eclipse/persistence/internal/sessions/AbstractSession.internalExecuteQuery(AbstractSession.java:3203(Compiled Code))
at org/eclipse/persistence/internal/sessions/AbstractSession.executeQuery(AbstractSession.java:1793(Compiled Code))
at org/eclipse/persistence/internal/sessions/AbstractSession.executeQuery(AbstractSession.java:1775(Compiled Code))
at org/eclipse/persistence/internal/indirection/QueryBasedValueHolder.instantiate(QueryBasedValueHolder.java:129(Compiled Code))
at org/eclipse/persistence/internal/indirection/QueryBasedValueHolder.instantiate(QueryBasedValueHolder.java:116(Compiled Code))

Re: Hung threads [message #1129432 is a reply to message #1129321]

Tue, 08 October 2013 12:16

Eclipse User

You will need to obtain the full thread dump to track down which threads are involved in issue, and also call session.getIdentityMapAccessor().printIdentityMapLocks() to see which threads own which locks for the objects involved. See http://wiki.eclipse.org/EclipseLink/FAQ/JPA#How_to_diagnose_and_resolve_hangs_and_deadlocks.3F for potential workarounds, and contact support if you are able to. The stack trace shown does not use indirection on the OneToOne mapping on the object being read in, so this might be a good place to start if you are able to make changes. It helps to isolate the object involved and see if this is a true deadlock or just a severe bottleneck caused by thread contention on a single object.

Best Regards,
Chris

Re: Hung threads [message #1130768 is a reply to message #1129432]

Wed, 09 October 2013 18:58

Eclipse User

On further analysis I see that I am fetching the 1-m child records by an expression query. The problems lies with setting refreshIdentityMapResult on the query.
If I comment out the line - query.refreshIdentityMapResult();
I am not seeing hung threads so far atleast in the dozen times I ran my unit test. I still have to test it more since with TopLink the hung thread was very sporadic.

What does refreshIdentityMapResult do to lock multiple threads?
What are the possible problems I need to look out if I do not call refreshIdentityMapResult?

Thanks, Amita

Re: Hung threads [message #1131359 is a reply to message #1130768]

Thu, 10 October 2013 08:56

Eclipse User

refreshIdentityMapResult does just what it says - it refreshes the objects being read in with the data from the database. It must lock the objects though so that any other threads that might be trying to access those objects (such as for other reads or writes) wait until it is done updating them, otherwise the data would be inconsistent and cause concurrency issues.

Care must always be taken with the refresh settings, but they are commonly used to avoid stale data. Problems that might occur from removing refreshing could be a greater occurrence of optimistic lock exceptions.

Re: Hung threads [message #1131474 is a reply to message #1131359]

Thu, 10 October 2013 11:05

Eclipse User

Thanks Chris for explanation.

The problem is that the locks never get released. I have let the process run over 2 days and the locks are still not released.
Another curious thing is that I have named queries that have this setting set in the workbench but I haven't seen the named queries hang so far. Is the locking different for the different types of queries?
We migrated from TopLink to EclipseLink for this hung thread problems and are not using JPA.

Re: Hung threads [message #1131830 is a reply to message #1129432]

Thu, 10 October 2013 16:47

Eclipse User

How does EclipseLink support work? Do we use our existing Oracle support?
Thanks

Re: Hung threads [message #1499857 is a reply to message #1129321]

Fri, 05 December 2014 08:23

Eclipse User

These locks appeared on our customer environment - we cannot reproduce this on our side.
They sent us few thread dumps and we can see that 14 threads have been waiting for more than six days.

We have 12 threads which are locked and waiting on the same object (all of them try to execute the same query). Here is one of them:

"ORBacus:ThreadPool-0:Dispatcher-13" nid=91 state=WAITING
    - waiting on <0x6f47f9ad> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)
    - locked <0x6f47f9ad> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:503)
    at org.eclipse.persistence.internal.helper.ConcurrencyManager.acquire(ConcurrencyManager.java:80)
    at org.eclipse.persistence.internal.identitymaps.CacheKey.acquire(CacheKey.java:132)
    at org.eclipse.persistence.internal.identitymaps.AbstractIdentityMap.acquireLock(AbstractIdentityMap.java:108)
    at org.eclipse.persistence.internal.identitymaps.IdentityMapManager.acquireLock(IdentityMapManager.java:157)
    at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:99)
    at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:90)
    at org.eclipse.persistence.internal.sessions.AbstractSession.retrieveCacheKey(AbstractSession.java:5143)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:964)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneFromRow(ObjectBuilder.java:2239)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:847)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:734)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:688)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795)
    at org.eclipse.persistence.queries.ReadObjectQuery.registerResultInUnitOfWork(ReadObjectQuery.java:885)
    at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:552)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
    at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:431)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2910)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1793)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1775)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1726)
    at org.eclipse.persistence.internal.sessions.AbstractSession.checkAndRefreshInvalidObject(AbstractSession.java:1219)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:1013)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:969)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getAndCloneCacheKeyFromParent(UnitOfWorkIdentityMapAccessor.java:209)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getFromIdentityMap(UnitOfWorkIdentityMapAccessor.java:137)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3956)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3908)
    at org.eclipse.persistence.mappings.ObjectReferenceMapping.buildUnitofWorkCloneForPartObject(ObjectReferenceMapping.java:109)
    at org.eclipse.persistence.mappings.ObjectReferenceMapping.buildCloneForPartObject(ObjectReferenceMapping.java:71)
    at org.eclipse.persistence.internal.indirection.NoIndirectionPolicy.cloneAttribute(NoIndirectionPolicy.java:79)
    at org.eclipse.persistence.mappings.ForeignReferenceMapping.buildCloneFromRow(ForeignReferenceMapping.java:337)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildAttributesIntoWorkingCopyClone(ObjectBuilder.java:1996)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneFromRow(ObjectBuilder.java:2249)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:847)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:734)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:688)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795)
    at org.eclipse.persistence.queries.ReadAllQuery.registerResultInUnitOfWork(ReadAllQuery.java:890)
    at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:509)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
    at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:403)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2910)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1793)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1775)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1740)
    at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:258)
    at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:469)

Query which is being executed here:
Query query = em.createQuery("SELECT entity1 FROM Entity1 entity1 WHERE entity1.en2 = :en2 AND entity1.en3 = :en3)"
query.setParameter("en2", en2);
query.setParameter("en3", en3);
query.setHint(QueryHints.CACHE_USAGE, CacheUsage.ConformResultsInUnitOfWork);
query.getResultList();

Some relations from Entity1 class:
@OneToOne
@JoinColumn(name = "en2_id")
protected Entity2 en2

@ManyToOne(cascade = {CascadeType.PERSIST}, fetch = FetchType.LAZY)
@JoinColumn(name = "en3_id")
protected Entity3 en3;

And we have 2 threads which are locked and waiting on another object (but both on the same):

First thread:

"ORBacus:ThreadPool-0:Dispatcher-5" nid=83 state=WAITING
    - waiting on <0x1ee40373> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)
    - locked <0x1ee40373> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:503)
    at org.eclipse.persistence.internal.helper.ConcurrencyManager.acquire(ConcurrencyManager.java:80)
    at org.eclipse.persistence.internal.identitymaps.CacheKey.acquire(CacheKey.java:132)
    at org.eclipse.persistence.internal.identitymaps.AbstractIdentityMap.acquireLock(AbstractIdentityMap.java:108)
    at org.eclipse.persistence.internal.identitymaps.IdentityMapManager.acquireLock(IdentityMapManager.java:157)
    at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:99)
    at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:90)
    at org.eclipse.persistence.internal.sessions.AbstractSession.retrieveCacheKey(AbstractSession.java:5143)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:964)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneFromRow(ObjectBuilder.java:2239)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:847)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:734)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:688)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795)
    at org.eclipse.persistence.queries.ReadObjectQuery.registerResultInUnitOfWork(ReadObjectQuery.java:885)
    at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:552)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
    at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:431)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2910)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1793)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1775)
    at org.eclipse.persistence.internal.indirection.QueryBasedValueHolder.instantiate(QueryBasedValueHolder.java:129)
    at org.eclipse.persistence.internal.indirection.QueryBasedValueHolder.instantiate(QueryBasedValueHolder.java:116)
    at org.eclipse.persistence.internal.indirection.DatabaseValueHolder.getValue(DatabaseValueHolder.java:89)
    at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiateImpl(UnitOfWorkValueHolder.java:173)
    at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiate(UnitOfWorkValueHolder.java:234)
    at org.eclipse.persistence.internal.indirection.DatabaseValueHolder.getValue(DatabaseValueHolder.java:89)
      - locked org.eclipse.persistence.internal.indirection.UnitOfWorkQueryValueHolder@51456c1d
    at org.eclipse.persistence.internal.indirection.BasicIndirectionPolicy.getRealAttributeValueFromObject(BasicIndirectionPolicy.java:280)
    at org.eclipse.persistence.internal.indirection.WeavedObjectBasicIndirectionPolicy.getRealAttributeValueFromObject(WeavedObjectBasicIndirectionPolicy.java:73)
    at org.eclipse.persistence.internal.indirection.IndirectionPolicy.instantiateObject(IndirectionPolicy.java:220)
    at org.eclipse.persistence.mappings.ForeignReferenceMapping.buildCloneFromRow(ForeignReferenceMapping.java:345)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildAttributesIntoWorkingCopyClone(ObjectBuilder.java:1996)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneFromRow(ObjectBuilder.java:2249)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:847)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:734)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:688)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795)
    at org.eclipse.persistence.queries.ReadObjectQuery.registerResultInUnitOfWork(ReadObjectQuery.java:885)
    at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:552)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
    at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:431)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2910)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1793)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1775)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1726)
    at org.eclipse.persistence.internal.sessions.AbstractSession.checkAndRefreshInvalidObject(AbstractSession.java:1219)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:1013)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:969)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getAndCloneCacheKeyFromParent(UnitOfWorkIdentityMapAccessor.java:209)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getFromIdentityMap(UnitOfWorkIdentityMapAccessor.java:137)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3956)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3908)
    at org.eclipse.persistence.mappings.ObjectReferenceMapping.buildUnitofWorkCloneForPartObject(ObjectReferenceMapping.java:109)
    at org.eclipse.persistence.mappings.ObjectReferenceMapping.buildCloneForPartObject(ObjectReferenceMapping.java:71)
    at org.eclipse.persistence.internal.indirection.NoIndirectionPolicy.cloneAttribute(NoIndirectionPolicy.java:79)
    at org.eclipse.persistence.mappings.ForeignReferenceMapping.buildCloneFromRow(ForeignReferenceMapping.java:337)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildAttributesIntoWorkingCopyClone(ObjectBuilder.java:1996)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneFromRow(ObjectBuilder.java:2249)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:847)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:734)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:688)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795)
    at org.eclipse.persistence.queries.ReadAllQuery.registerResultInUnitOfWork(ReadAllQuery.java:890)
    at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:509)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
    at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:403)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2910)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1793)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1775)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1740)
    at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:258)
    at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:469)

Query which is being executed in here:
Query query = em.createQuery("SELECT entity3 FROM Entity3 entity3 WHERE entity3.entity4.id = " + entity4_id);
query.getResultList()

This relation in Entity3 class:

@OneToOne(fetch = FetchType.LAZY, cascade = CascadeType.PERSIST)
@JoinColumn(name = "entity4_id")
private Entity4 entity4;

Second thread:

"ORBacus:ThreadPool-0:Dispatcher-7" nid=85 state=WAITING
    - waiting on <0x1ee40373> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)
    - locked <0x1ee40373> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:503)
    at org.eclipse.persistence.internal.helper.ConcurrencyManager.acquire(ConcurrencyManager.java:80)
    at org.eclipse.persistence.internal.identitymaps.CacheKey.acquire(CacheKey.java:132)
    at org.eclipse.persistence.internal.identitymaps.AbstractIdentityMap.acquireLock(AbstractIdentityMap.java:108)
    at org.eclipse.persistence.internal.identitymaps.IdentityMapManager.acquireLock(IdentityMapManager.java:157)
    at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:99)
    at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:90)
    at org.eclipse.persistence.internal.sessions.AbstractSession.retrieveCacheKey(AbstractSession.java:5143)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:964)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneFromRow(ObjectBuilder.java:2239)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:847)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:734)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:688)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795)
    at org.eclipse.persistence.queries.ReadObjectQuery.registerResultInUnitOfWork(ReadObjectQuery.java:885)
    at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:552)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
    at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:431)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2910)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1793)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1775)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1726)
    at org.eclipse.persistence.internal.sessions.AbstractSession.checkAndRefreshInvalidObject(AbstractSession.java:1219)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:1013)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:969)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getAndCloneCacheKeyFromParent(UnitOfWorkIdentityMapAccessor.java:209)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getFromIdentityMap(UnitOfWorkIdentityMapAccessor.java:137)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3956)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3908)
    at org.eclipse.persistence.mappings.ObjectReferenceMapping.buildUnitofWorkCloneForPartObject(ObjectReferenceMapping.java:109)
    at org.eclipse.persistence.mappings.ObjectReferenceMapping.buildCloneForPartObject(ObjectReferenceMapping.java:71)
    at org.eclipse.persistence.internal.indirection.UnitOfWorkQueryValueHolder.buildCloneFor(UnitOfWorkQueryValueHolder.java:60)
    at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiateImpl(UnitOfWorkValueHolder.java:173)
    at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiate(UnitOfWorkValueHolder.java:234)
    at org.eclipse.persistence.internal.indirection.DatabaseValueHolder.getValue(DatabaseValueHolder.java:89)
    at com.package.Entity5._persistence_get_entity6(Entity5.java)

This relation in Entity5 class:

@OneToOne(fetch = FetchType.LAZY)
@JoinColumn(name = "entity6_id")
private Entity6 entity6;

Do you have any ideas how to fix this issue?

EclipseLink version: 2.5.1

Persistence Unit properties:
<property name="eclipselink.weaving" value="static"/>
<property name="eclipselink.weaving.internal" value="false"/>
<property name="eclipselink.weaving.fetchgroups" value="false"/>

Re: Hung threads [message #1667960 is a reply to message #1129321]

Thu, 12 March 2015 05:09

Eclipse User

Problem still appears on 2.5.2.

It seems that we are able to reproduce this after some time (10hours - few days) of intensive tests.

Properties and settings:

<property name="eclipselink.weaving" value="static"/>
<property name="eclipselink.weaving.internal" value="false"/>
<property name="eclipselink.weaving.fetchgroups" value="false"/>
ConcurrencyManager.setShouldTrackStack(true);
DeferredLockManager.SHOULD_USE_DEFERRED_LOCKS = false;
JoinFetching is not used
EntityManager is not shared between threads

After deadlock appeared and stayed for this case:

Name: ORBacus:ThreadPool-0:Dispatcher-0
State: WAITING on org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey@5a010513
Total blocked: 14 727  Total waited: 14 882

Stack trace: 
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
org.eclipse.persistence.internal.helper.ConcurrencyManager.acquire(ConcurrencyManager.java:80)
org.eclipse.persistence.internal.identitymaps.CacheKey.acquire(CacheKey.java:132)
org.eclipse.persistence.internal.identitymaps.AbstractIdentityMap.acquireLock(AbstractIdentityMap.java:108)
org.eclipse.persistence.internal.identitymaps.IdentityMapManager.acquireLock(IdentityMapManager.java:157)
org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:99)
org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:90)
org.eclipse.persistence.internal.sessions.AbstractSession.retrieveCacheKey(AbstractSession.java:5155)
org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:964)
org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneFromRow(ObjectBuilder.java:2239)
org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:847)
org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:734)
org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:688)
org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795)
org.eclipse.persistence.queries.ReadObjectQuery.registerResultInUnitOfWork(ReadObjectQuery.java:885)
org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:552)
org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168)
org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:431)
org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2910)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1804)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1786)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1737)
org.eclipse.persistence.internal.jpa.EntityManagerImpl.executeQuery(EntityManagerImpl.java:900)
org.eclipse.persistence.internal.jpa.EntityManagerImpl.refresh(EntityManagerImpl.java:1040)
org.eclipse.persistence.internal.jpa.EntityManagerImpl.refresh(EntityManagerImpl.java:929)

I printed locks and here is the result:

Current object locks:

End of locked objects.

So no lock objects in this case. I can patch eclipselink and add more logging to find out what is wrong, but please advice: what and where?

Re: Hung threads [message #1706212 is a reply to message #1667960]

Mon, 24 August 2015 21:11

Eclipse User

Any update on this? We are also having similar issue and I do see a deadlock on the DB at that time. But, the deadlock seems to be caused by couple of SQLs initiated by eclipse link (setting null values to 'order' column)

Re: Hung threads [message #1706421 is a reply to message #1706212]

Wed, 26 August 2015 14:51

Eclipse User

Similar issues can occur all the time, but it is not necessarily a bug or a true deadlock. If one thread is building an object, any other threads must wait until that thread is done. Depending on your model, there are any number of reasons this might cause a bottleneck, such as if you have a large model constantly being refreshed as just one example. True database deadlocks also cause issues, but generally with writes, not reads.

In the thread dump provided by Pawel, the issue still is not obvious as it does not show the thread that holds the lock, only the ones backing up waiting for it. The first set of 12 threads are building an entity1 instance and trying to build one of the references that is eagerly fetched - presumably entity2 instances. A quick solution would be to make this relationship lazy and these 12 threads would be able to skip having to wait for the lock, but it does not show an actual deadlock situation. This is why the session.getIdentityMapAccessor().printIdentityMapLocks() I suggested earlier is important, as it will show the threads that hold each lock, allowing you to correlate what each thread in the dump is doing to the locks it holds. From there, you can identify which thread is causing the deadlock (or bottleneck) and decide what should be done to prevent it.

Best Regards,
Chris

Re: Hung threads [message #1706616 is a reply to message #1129321]

Fri, 28 August 2015 03:46

Eclipse User

Deadlock we observed may be unrelated, we could see deadlock even in good runs which seems to get rolledback and our application code has retry mechanism. So, that may have been false alarm. However, when we set the default caching to 'isolated' level (eclipselink.cache.shared.default=false) seems to stop the problem. But, that is anyway related to the location where it is getting hung. By making the cache to be isolated the code path is completely avoided. My suspicion is that in some strange combination, the object lock acquired in org.eclipse.persistence.internal.identitymaps.FullIdentityMap.remove() method isn't released ( I dont see any of the release methods in the finally block)

In our thread stack we did not see any other thread that may be holding on the lock.

Stack trace when the object lock is hung is this (version: 2.6.0)
"main" #1 prio=5 os_prio=0 tid=0x00007f5ab4009800 nid=0xce4f in Object.wait() [0x00007f5abacef000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.eclipse.persistence.internal.helper.ConcurrencyManager.acquire(ConcurrencyManager.java:80)
- locked <0x000000067f5c4608> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)
at org.eclipse.persistence.internal.identitymaps.CacheKey.acquire(CacheKey.java:132)
at org.eclipse.persistence.internal.identitymaps.FullIdentityMap.remove(FullIdentityMap.java:212)
at org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap.remove(HardCacheWeakIdentityMap.java:88)
at org.eclipse.persistence.internal.identitymaps.AbstractIdentityMap.remove(AbstractIdentityMap.java:421)
at org.eclipse.persistence.internal.identitymaps.IdentityMapManager.removeFromIdentityMap(IdentityMapManager.java:1491)
at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.removeFromIdentityMap(IdentityMapAccessor.java:1150)
at org.eclipse.persistence.internal.sessions.IsolatedClientSessionIdentityMapAccessor.removeFromIdentityMap(IsolatedClientSessionIdentityMapAccessor.java:692)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.postMergeChanges(UnitOfWorkImpl.java:3699)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.mergeChangesIntoParent(UnitOfWorkImpl.java:3313)
at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.mergeChangesIntoParent(RepeatableWriteUnitOfWork.java:379)
at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:291)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)
at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)

Re: Hung threads [message #1707418 is a reply to message #1706616]

Fri, 04 September 2015 11:13

Eclipse User

Locks in this stack are obtained for the transaction merge process, and should be released only when the entire commit process completes. You will need to use the printidentitymaps call mentioned previously to find what other threads might have eclipselink locks, and use the thread dump and logs to track what they might have been doing.

Previous Topic:	Entity is not updated
Next Topic:	Entity Update throws NPE for @ElementColection of Map - Eclipselink 2.5.2

Goto Forum:

-=] Back to Top [=-

Current Time: Fri Jul 04 16:21:58 EDT 2025

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter