Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » EclipseLink » Hung threads
Hung threads [message #1129321] Tue, 08 October 2013 14:13 Go to next message
Amita Mehta is currently offline Amita MehtaFriend
Messages: 18
Registered: May 2013
Junior Member
We are using EclipseLink 2.5.1.v20130918-f2b9fc5 with WAS8

We are getting hung threads and a lot of time is being spent in the getChildObject call. Sometimes it happens for a one to one relationship and sometimes for a one to many.

We have removed all the joins.
We were having this problem initially with TopLink and so upgraded to EclipseLink but get the same same problem.
Any suggestions and solutions are appreciated.
Thanks.

Log

at org/eclipse/persistence/internal/helper/ConcurrencyManager.acquire(ConcurrencyManager.java:80(Compiled Code))
at org/eclipse/persistence/internal/identitymaps/CacheKey.acquire(CacheKey.java:132(Compiled Code))
at org/eclipse/persistence/internal/identitymaps/AbstractIdentityMap.acquireLock(AbstractIdentityMap.java:108(Compiled Code))
at org/eclipse/persistence/internal/identitymaps/IdentityMapManager.acquireLock(IdentityMapManager.java:157(Compiled Code))
at org/eclipse/persistence/internal/sessions/IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:99(Compiled Code))
at org/eclipse/persistence/internal/sessions/IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:90(Compiled Code))
at org/eclipse/persistence/internal/sessions/AbstractSession.retrieveCacheKey(AbstractSession.java:5143(Compiled Code))
at org/eclipse/persistence/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:964(Compiled Code))
at org/eclipse/persistence/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:736(Compiled Code))
at org/eclipse/persistence/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:688(Compiled Code))
at org/eclipse/persistence/queries/ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795(Compiled Code))
at org/eclipse/persistence/queries/ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:554(Compiled Code))
at org/eclipse/persistence/queries/ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168(Compiled Code))
at org/eclipse/persistence/queries/DatabaseQuery.execute(DatabaseQuery.java:899(Compiled Code))
at org/eclipse/persistence/queries/ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127(Compiled Code))
at org/eclipse/persistence/queries/ReadObjectQuery.execute(ReadObjectQuery.java:431(Compiled Code))
at org/eclipse/persistence/internal/sessions/AbstractSession.internalExecuteQuery(AbstractSession.java:3203(Compiled Code))
at org/eclipse/persistence/internal/sessions/AbstractSession.executeQuery(AbstractSession.java:1793(Compiled Code))
at org/eclipse/persistence/internal/sessions/AbstractSession.executeQuery(AbstractSession.java:1775(Compiled Code))
at org/eclipse/persistence/internal/indirection/NoIndirectionPolicy.valueFromQuery(NoIndirectionPolicy.java:326(Compiled Code))
at org/eclipse/persistence/mappings/ForeignReferenceMapping.valueFromRowInternal(ForeignReferenceMapping.java:2294(Compiled Code))
at org/eclipse/persistence/mappings/OneToOneMapping.valueFromRowInternal(OneToOneMapping.java:1795(Compiled Code))
at org/eclipse/persistence/mappings/ForeignReferenceMapping.valueFromRow(ForeignReferenceMapping.java:2144(Compiled Code))
at org/eclipse/persistence/mappings/ForeignReferenceMapping.readFromRowIntoObject(ForeignReferenceMapping.java:1471(Compiled Code))
at org/eclipse/persistence/internal/descriptors/ObjectBuilder.buildAttributesIntoObject(ObjectBuilder.java:461(Compiled Code))
at org/eclipse/persistence/internal/descriptors/ObjectBuilder.refreshObjectIfRequired(ObjectBuilder.java:4300(Compiled Code))
at org/eclipse/persistence/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:1039(Compiled Code))
at org/eclipse/persistence/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:736(Compiled Code))
at org/eclipse/persistence/internal/descriptors/ObjectBuilder.buildObject(ObjectBuilder.java:688(Compiled Code))
at org/eclipse/persistence/queries/ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795(Compiled Code))
at org/eclipse/persistence/queries/ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:554(Compiled Code))
at org/eclipse/persistence/queries/ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168(Compiled Code))
at org/eclipse/persistence/queries/DatabaseQuery.execute(DatabaseQuery.java:899(Compiled Code))
at org/eclipse/persistence/queries/ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127(Compiled Code))
at org/eclipse/persistence/queries/ReadObjectQuery.execute(ReadObjectQuery.java:431(Compiled Code))
at org/eclipse/persistence/internal/sessions/AbstractSession.internalExecuteQuery(AbstractSession.java:3203(Compiled Code))
at org/eclipse/persistence/internal/sessions/AbstractSession.executeQuery(AbstractSession.java:1793(Compiled Code))
at org/eclipse/persistence/internal/sessions/AbstractSession.executeQuery(AbstractSession.java:1775(Compiled Code))
at org/eclipse/persistence/internal/indirection/QueryBasedValueHolder.instantiate(QueryBasedValueHolder.java:129(Compiled Code))
at org/eclipse/persistence/internal/indirection/QueryBasedValueHolder.instantiate(QueryBasedValueHolder.java:116(Compiled
Code))


Re: Hung threads [message #1129432 is a reply to message #1129321] Tue, 08 October 2013 16:16 Go to previous messageGo to next message
Chris Delahunt is currently offline Chris DelahuntFriend
Messages: 1085
Registered: July 2009
Senior Member
You will need to obtain the full thread dump to track down which threads are involved in issue, and also call session.getIdentityMapAccessor().printIdentityMapLocks() to see which threads own which locks for the objects involved. See http://wiki.eclipse.org/EclipseLink/FAQ/JPA#How_to_diagnose_and_resolve_hangs_and_deadlocks.3F for potential workarounds, and contact support if you are able to. The stack trace shown does not use indirection on the OneToOne mapping on the object being read in, so this might be a good place to start if you are able to make changes. It helps to isolate the object involved and see if this is a true deadlock or just a severe bottleneck caused by thread contention on a single object.

Best Regards,
Chris
Re: Hung threads [message #1130768 is a reply to message #1129432] Wed, 09 October 2013 22:58 Go to previous messageGo to next message
Amita Mehta is currently offline Amita MehtaFriend
Messages: 18
Registered: May 2013
Junior Member
On further analysis I see that I am fetching the 1-m child records by an expression query. The problems lies with setting refreshIdentityMapResult on the query.
If I comment out the line - query.refreshIdentityMapResult();
I am not seeing hung threads so far atleast in the dozen times I ran my unit test. I still have to test it more since with TopLink the hung thread was very sporadic.

What does refreshIdentityMapResult do to lock multiple threads?
What are the possible problems I need to look out if I do not call refreshIdentityMapResult?

Thanks, Amita
Re: Hung threads [message #1131359 is a reply to message #1130768] Thu, 10 October 2013 12:56 Go to previous messageGo to next message
Chris Delahunt is currently offline Chris DelahuntFriend
Messages: 1085
Registered: July 2009
Senior Member
refreshIdentityMapResult does just what it says - it refreshes the objects being read in with the data from the database. It must lock the objects though so that any other threads that might be trying to access those objects (such as for other reads or writes) wait until it is done updating them, otherwise the data would be inconsistent and cause concurrency issues.

Care must always be taken with the refresh settings, but they are commonly used to avoid stale data. Problems that might occur from removing refreshing could be a greater occurrence of optimistic lock exceptions.
Re: Hung threads [message #1131474 is a reply to message #1131359] Thu, 10 October 2013 15:05 Go to previous messageGo to next message
Amita Mehta is currently offline Amita MehtaFriend
Messages: 18
Registered: May 2013
Junior Member
Thanks Chris for explanation.

The problem is that the locks never get released. I have let the process run over 2 days and the locks are still not released.
Another curious thing is that I have named queries that have this setting set in the workbench but I haven't seen the named queries hang so far. Is the locking different for the different types of queries?
We migrated from TopLink to EclipseLink for this hung thread problems and are not using JPA.
Re: Hung threads [message #1131830 is a reply to message #1129432] Thu, 10 October 2013 20:47 Go to previous messageGo to next message
Amita Mehta is currently offline Amita MehtaFriend
Messages: 18
Registered: May 2013
Junior Member
How does EclipseLink support work? Do we use our existing Oracle support?
Thanks
Re: Hung threads [message #1499857 is a reply to message #1129321] Fri, 05 December 2014 13:23 Go to previous messageGo to next message
Pawel Zak is currently offline Pawel ZakFriend
Messages: 8
Registered: November 2012
Junior Member
These locks appeared on our customer environment - we cannot reproduce this on our side.
They sent us few thread dumps and we can see that 14 threads have been waiting for more than six days.

We have 12 threads which are locked and waiting on the same object (all of them try to execute the same query). Here is one of them:

"ORBacus:ThreadPool-0:Dispatcher-13" nid=91 state=WAITING
    - waiting on <0x6f47f9ad> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)
    - locked <0x6f47f9ad> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:503)
    at org.eclipse.persistence.internal.helper.ConcurrencyManager.acquire(ConcurrencyManager.java:80)
    at org.eclipse.persistence.internal.identitymaps.CacheKey.acquire(CacheKey.java:132)
    at org.eclipse.persistence.internal.identitymaps.AbstractIdentityMap.acquireLock(AbstractIdentityMap.java:108)
    at org.eclipse.persistence.internal.identitymaps.IdentityMapManager.acquireLock(IdentityMapManager.java:157)
    at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:99)
    at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:90)
    at org.eclipse.persistence.internal.sessions.AbstractSession.retrieveCacheKey(AbstractSession.java:5143)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:964)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneFromRow(ObjectBuilder.java:2239)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:847)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:734)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:688)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795)
    at org.eclipse.persistence.queries.ReadObjectQuery.registerResultInUnitOfWork(ReadObjectQuery.java:885)
    at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:552)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
    at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:431)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2910)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1793)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1775)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1726)
    at org.eclipse.persistence.internal.sessions.AbstractSession.checkAndRefreshInvalidObject(AbstractSession.java:1219)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:1013)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:969)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getAndCloneCacheKeyFromParent(UnitOfWorkIdentityMapAccessor.java:209)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getFromIdentityMap(UnitOfWorkIdentityMapAccessor.java:137)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3956)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3908)
    at org.eclipse.persistence.mappings.ObjectReferenceMapping.buildUnitofWorkCloneForPartObject(ObjectReferenceMapping.java:109)
    at org.eclipse.persistence.mappings.ObjectReferenceMapping.buildCloneForPartObject(ObjectReferenceMapping.java:71)
    at org.eclipse.persistence.internal.indirection.NoIndirectionPolicy.cloneAttribute(NoIndirectionPolicy.java:79)
    at org.eclipse.persistence.mappings.ForeignReferenceMapping.buildCloneFromRow(ForeignReferenceMapping.java:337)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildAttributesIntoWorkingCopyClone(ObjectBuilder.java:1996)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneFromRow(ObjectBuilder.java:2249)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:847)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:734)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:688)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795)
    at org.eclipse.persistence.queries.ReadAllQuery.registerResultInUnitOfWork(ReadAllQuery.java:890)
    at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:509)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
    at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:403)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2910)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1793)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1775)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1740)
    at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:258)
    at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:469)

Query which is being executed here:
Query query = em.createQuery("SELECT entity1 FROM Entity1 entity1 WHERE entity1.en2 = :en2 AND entity1.en3 = :en3)"
query.setParameter("en2", en2);
query.setParameter("en3", en3);
query.setHint(QueryHints.CACHE_USAGE, CacheUsage.ConformResultsInUnitOfWork);
query.getResultList();

Some relations from Entity1 class:
@OneToOne
@JoinColumn(name = "en2_id")
protected Entity2 en2

@ManyToOne(cascade = {CascadeType.PERSIST}, fetch = FetchType.LAZY)
@JoinColumn(name = "en3_id")
protected Entity3 en3;



And we have 2 threads which are locked and waiting on another object (but both on the same):

First thread:
"ORBacus:ThreadPool-0:Dispatcher-5" nid=83 state=WAITING
    - waiting on <0x1ee40373> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)
    - locked <0x1ee40373> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:503)
    at org.eclipse.persistence.internal.helper.ConcurrencyManager.acquire(ConcurrencyManager.java:80)
    at org.eclipse.persistence.internal.identitymaps.CacheKey.acquire(CacheKey.java:132)
    at org.eclipse.persistence.internal.identitymaps.AbstractIdentityMap.acquireLock(AbstractIdentityMap.java:108)
    at org.eclipse.persistence.internal.identitymaps.IdentityMapManager.acquireLock(IdentityMapManager.java:157)
    at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:99)
    at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:90)
    at org.eclipse.persistence.internal.sessions.AbstractSession.retrieveCacheKey(AbstractSession.java:5143)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:964)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneFromRow(ObjectBuilder.java:2239)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:847)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:734)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:688)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795)
    at org.eclipse.persistence.queries.ReadObjectQuery.registerResultInUnitOfWork(ReadObjectQuery.java:885)
    at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:552)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
    at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:431)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2910)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1793)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1775)
    at org.eclipse.persistence.internal.indirection.QueryBasedValueHolder.instantiate(QueryBasedValueHolder.java:129)
    at org.eclipse.persistence.internal.indirection.QueryBasedValueHolder.instantiate(QueryBasedValueHolder.java:116)
    at org.eclipse.persistence.internal.indirection.DatabaseValueHolder.getValue(DatabaseValueHolder.java:89)
    at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiateImpl(UnitOfWorkValueHolder.java:173)
    at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiate(UnitOfWorkValueHolder.java:234)
    at org.eclipse.persistence.internal.indirection.DatabaseValueHolder.getValue(DatabaseValueHolder.java:89)
      - locked org.eclipse.persistence.internal.indirection.UnitOfWorkQueryValueHolder@51456c1d
    at org.eclipse.persistence.internal.indirection.BasicIndirectionPolicy.getRealAttributeValueFromObject(BasicIndirectionPolicy.java:280)
    at org.eclipse.persistence.internal.indirection.WeavedObjectBasicIndirectionPolicy.getRealAttributeValueFromObject(WeavedObjectBasicIndirectionPolicy.java:73)
    at org.eclipse.persistence.internal.indirection.IndirectionPolicy.instantiateObject(IndirectionPolicy.java:220)
    at org.eclipse.persistence.mappings.ForeignReferenceMapping.buildCloneFromRow(ForeignReferenceMapping.java:345)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildAttributesIntoWorkingCopyClone(ObjectBuilder.java:1996)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneFromRow(ObjectBuilder.java:2249)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:847)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:734)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:688)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795)
    at org.eclipse.persistence.queries.ReadObjectQuery.registerResultInUnitOfWork(ReadObjectQuery.java:885)
    at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:552)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
    at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:431)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2910)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1793)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1775)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1726)
    at org.eclipse.persistence.internal.sessions.AbstractSession.checkAndRefreshInvalidObject(AbstractSession.java:1219)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:1013)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:969)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getAndCloneCacheKeyFromParent(UnitOfWorkIdentityMapAccessor.java:209)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getFromIdentityMap(UnitOfWorkIdentityMapAccessor.java:137)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3956)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3908)
    at org.eclipse.persistence.mappings.ObjectReferenceMapping.buildUnitofWorkCloneForPartObject(ObjectReferenceMapping.java:109)
    at org.eclipse.persistence.mappings.ObjectReferenceMapping.buildCloneForPartObject(ObjectReferenceMapping.java:71)
    at org.eclipse.persistence.internal.indirection.NoIndirectionPolicy.cloneAttribute(NoIndirectionPolicy.java:79)
    at org.eclipse.persistence.mappings.ForeignReferenceMapping.buildCloneFromRow(ForeignReferenceMapping.java:337)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildAttributesIntoWorkingCopyClone(ObjectBuilder.java:1996)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneFromRow(ObjectBuilder.java:2249)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:847)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:734)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:688)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795)
    at org.eclipse.persistence.queries.ReadAllQuery.registerResultInUnitOfWork(ReadAllQuery.java:890)
    at org.eclipse.persistence.queries.ReadAllQuery.executeObjectLevelReadQuery(ReadAllQuery.java:509)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
    at org.eclipse.persistence.queries.ReadAllQuery.execute(ReadAllQuery.java:403)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2910)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1793)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1775)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1740)
    at org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:258)
    at org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:469)

Query which is being executed in here:
Query query = em.createQuery("SELECT entity3 FROM Entity3 entity3 WHERE entity3.entity4.id = " + entity4_id);
query.getResultList()

This relation in Entity3 class:

@OneToOne(fetch = FetchType.LAZY, cascade = CascadeType.PERSIST)
@JoinColumn(name = "entity4_id")
private Entity4 entity4;



Second thread:
"ORBacus:ThreadPool-0:Dispatcher-7" nid=85 state=WAITING
    - waiting on <0x1ee40373> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)
    - locked <0x1ee40373> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)
    at java.lang.Object.wait(Native Method)
    at java.lang.Object.wait(Object.java:503)
    at org.eclipse.persistence.internal.helper.ConcurrencyManager.acquire(ConcurrencyManager.java:80)
    at org.eclipse.persistence.internal.identitymaps.CacheKey.acquire(CacheKey.java:132)
    at org.eclipse.persistence.internal.identitymaps.AbstractIdentityMap.acquireLock(AbstractIdentityMap.java:108)
    at org.eclipse.persistence.internal.identitymaps.IdentityMapManager.acquireLock(IdentityMapManager.java:157)
    at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:99)
    at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:90)
    at org.eclipse.persistence.internal.sessions.AbstractSession.retrieveCacheKey(AbstractSession.java:5143)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:964)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneFromRow(ObjectBuilder.java:2239)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:847)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:734)
    at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:688)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795)
    at org.eclipse.persistence.queries.ReadObjectQuery.registerResultInUnitOfWork(ReadObjectQuery.java:885)
    at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:552)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168)
    at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
    at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:431)
    at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2910)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1793)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1775)
    at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1726)
    at org.eclipse.persistence.internal.sessions.AbstractSession.checkAndRefreshInvalidObject(AbstractSession.java:1219)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:1013)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.cloneAndRegisterObject(UnitOfWorkImpl.java:969)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getAndCloneCacheKeyFromParent(UnitOfWorkIdentityMapAccessor.java:209)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkIdentityMapAccessor.getFromIdentityMap(UnitOfWorkIdentityMapAccessor.java:137)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3956)
    at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.registerExistingObject(UnitOfWorkImpl.java:3908)
    at org.eclipse.persistence.mappings.ObjectReferenceMapping.buildUnitofWorkCloneForPartObject(ObjectReferenceMapping.java:109)
    at org.eclipse.persistence.mappings.ObjectReferenceMapping.buildCloneForPartObject(ObjectReferenceMapping.java:71)
    at org.eclipse.persistence.internal.indirection.UnitOfWorkQueryValueHolder.buildCloneFor(UnitOfWorkQueryValueHolder.java:60)
    at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiateImpl(UnitOfWorkValueHolder.java:173)
    at org.eclipse.persistence.internal.indirection.UnitOfWorkValueHolder.instantiate(UnitOfWorkValueHolder.java:234)
    at org.eclipse.persistence.internal.indirection.DatabaseValueHolder.getValue(DatabaseValueHolder.java:89)
    at com.package.Entity5._persistence_get_entity6(Entity5.java)



This relation in Entity5 class:

@OneToOne(fetch = FetchType.LAZY)
@JoinColumn(name = "entity6_id")
private Entity6 entity6;



Do you have any ideas how to fix this issue?


EclipseLink version: 2.5.1

Persistence Unit properties:
<property name="eclipselink.weaving" value="static"/>
<property name="eclipselink.weaving.internal" value="false"/>
<property name="eclipselink.weaving.fetchgroups" value="false"/>
Re: Hung threads [message #1667960 is a reply to message #1129321] Thu, 12 March 2015 09:09 Go to previous messageGo to next message
Pawel Zak is currently offline Pawel ZakFriend
Messages: 8
Registered: November 2012
Junior Member
Problem still appears on 2.5.2.

It seems that we are able to reproduce this after some time (10hours - few days) of intensive tests.

Properties and settings:
<property name="eclipselink.weaving" value="static"/>
<property name="eclipselink.weaving.internal" value="false"/>
<property name="eclipselink.weaving.fetchgroups" value="false"/>
ConcurrencyManager.setShouldTrackStack(true);
DeferredLockManager.SHOULD_USE_DEFERRED_LOCKS = false;
JoinFetching is not used
EntityManager is not shared between threads


After deadlock appeared and stayed for this case:

Name: ORBacus:ThreadPool-0:Dispatcher-0
State: WAITING on org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey@5a010513
Total blocked: 14 727  Total waited: 14 882

Stack trace: 
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
org.eclipse.persistence.internal.helper.ConcurrencyManager.acquire(ConcurrencyManager.java:80)
org.eclipse.persistence.internal.identitymaps.CacheKey.acquire(CacheKey.java:132)
org.eclipse.persistence.internal.identitymaps.AbstractIdentityMap.acquireLock(AbstractIdentityMap.java:108)
org.eclipse.persistence.internal.identitymaps.IdentityMapManager.acquireLock(IdentityMapManager.java:157)
org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:99)
org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:90)
org.eclipse.persistence.internal.sessions.AbstractSession.retrieveCacheKey(AbstractSession.java:5155)
org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:964)
org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneFromRow(ObjectBuilder.java:2239)
org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:847)
org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:734)
org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:688)
org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:795)
org.eclipse.persistence.queries.ReadObjectQuery.registerResultInUnitOfWork(ReadObjectQuery.java:885)
org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:552)
org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1168)
org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:899)
org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1127)
org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:431)
org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1215)
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2910)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1804)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1786)
org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1737)
org.eclipse.persistence.internal.jpa.EntityManagerImpl.executeQuery(EntityManagerImpl.java:900)
org.eclipse.persistence.internal.jpa.EntityManagerImpl.refresh(EntityManagerImpl.java:1040)
org.eclipse.persistence.internal.jpa.EntityManagerImpl.refresh(EntityManagerImpl.java:929)


I printed locks and here is the result:

Current object locks:

End of locked objects.



So no lock objects in this case. I can patch eclipselink and add more logging to find out what is wrong, but please advice: what and where?
Re: Hung threads [message #1706212 is a reply to message #1667960] Tue, 25 August 2015 01:11 Go to previous messageGo to next message
Sanjath Shringeri is currently offline Sanjath ShringeriFriend
Messages: 2
Registered: August 2015
Junior Member
Any update on this? We are also having similar issue and I do see a deadlock on the DB at that time. But, the deadlock seems to be caused by couple of SQLs initiated by eclipse link (setting null values to 'order' column)
Re: Hung threads [message #1706421 is a reply to message #1706212] Wed, 26 August 2015 18:51 Go to previous messageGo to next message
Chris Delahunt is currently offline Chris DelahuntFriend
Messages: 1085
Registered: July 2009
Senior Member
Similar issues can occur all the time, but it is not necessarily a bug or a true deadlock. If one thread is building an object, any other threads must wait until that thread is done. Depending on your model, there are any number of reasons this might cause a bottleneck, such as if you have a large model constantly being refreshed as just one example. True database deadlocks also cause issues, but generally with writes, not reads.

In the thread dump provided by Pawel, the issue still is not obvious as it does not show the thread that holds the lock, only the ones backing up waiting for it. The first set of 12 threads are building an entity1 instance and trying to build one of the references that is eagerly fetched - presumably entity2 instances. A quick solution would be to make this relationship lazy and these 12 threads would be able to skip having to wait for the lock, but it does not show an actual deadlock situation. This is why the session.getIdentityMapAccessor().printIdentityMapLocks() I suggested earlier is important, as it will show the threads that hold each lock, allowing you to correlate what each thread in the dump is doing to the locks it holds. From there, you can identify which thread is causing the deadlock (or bottleneck) and decide what should be done to prevent it.

Best Regards,
Chris
Re: Hung threads [message #1706616 is a reply to message #1129321] Fri, 28 August 2015 07:46 Go to previous message
Sanjath Shringeri is currently offline Sanjath ShringeriFriend
Messages: 2
Registered: August 2015
Junior Member
Deadlock we observed may be unrelated, we could see deadlock even in good runs which seems to get rolledback and our application code has retry mechanism. So, that may have been false alarm. However, when we set the default caching to 'isolated' level (eclipselink.cache.shared.default=false) seems to stop the problem. But, that is anyway related to the location where it is getting hung. By making the cache to be isolated the code path is completely avoided. My suspicion is that in some strange combination, the object lock acquired in org.eclipse.persistence.internal.identitymaps.FullIdentityMap.remove() method isn't released ( I dont see any of the release methods in the finally block)

In our thread stack we did not see any other thread that may be holding on the lock.

Stack trace when the object lock is hung is this (version: 2.6.0)
"main" #1 prio=5 os_prio=0 tid=0x00007f5ab4009800 nid=0xce4f in Object.wait() [0x00007f5abacef000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at org.eclipse.persistence.internal.helper.ConcurrencyManager.acquire(ConcurrencyManager.java:80)
- locked <0x000000067f5c4608> (a org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap$ReferenceCacheKey)
at org.eclipse.persistence.internal.identitymaps.CacheKey.acquire(CacheKey.java:132)
at org.eclipse.persistence.internal.identitymaps.FullIdentityMap.remove(FullIdentityMap.java:212)
at org.eclipse.persistence.internal.identitymaps.HardCacheWeakIdentityMap.remove(HardCacheWeakIdentityMap.java:88)
at org.eclipse.persistence.internal.identitymaps.AbstractIdentityMap.remove(AbstractIdentityMap.java:421)
at org.eclipse.persistence.internal.identitymaps.IdentityMapManager.removeFromIdentityMap(IdentityMapManager.java:1491)
at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.removeFromIdentityMap(IdentityMapAccessor.java:1150)
at org.eclipse.persistence.internal.sessions.IsolatedClientSessionIdentityMapAccessor.removeFromIdentityMap(IsolatedClientSessionIdentityMapAccessor.java:692)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.postMergeChanges(UnitOfWorkImpl.java:3699)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.mergeChangesIntoParent(UnitOfWorkImpl.java:3313)
at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.mergeChangesIntoParent(RepeatableWriteUnitOfWork.java:379)
at org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.commitRootUnitOfWork(RepeatableWriteUnitOfWork.java:291)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.commitAndResume(UnitOfWorkImpl.java:1169)
at org.eclipse.persistence.internal.jpa.transaction.EntityTransactionImpl.commit(EntityTransactionImpl.java:134)
Previous Topic:Need to extend existing ComplexType defined in XSD to accept empty strings from XML
Goto Forum:
  


Current Time: Fri Aug 28 12:48:36 GMT 2015

Powered by FUDForum. Page generated in 0.03326 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software