Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » EclipseLink » Eclipselink - dead locks in aqcuire and release deferred locks - Eclipselink 2.4.2 and above
Eclipselink - dead locks in aqcuire and release deferred locks - Eclipselink 2.4.2 and above [message #1801870] Mon, 28 January 2019 19:02 Go to next message
Nuno G. de M. is currently offline Nuno G. de M.Friend
Messages: 33
Registered: September 2012
Member
Hi,

In an ongoing project running on Weblogic 12.1.2, which bundles eclipselink 2.4.2, every two or three weeks, the hole system crashes beacuse of what appears to be dead locks in the ConcurrencyManager / CacheKeys of eclipselink.


Pretty much, the system goes into a situation that everything dies Singleton EJBs are forever, deltaspike locks are forever locked, etc... because ultimately threads at the eclipselink layer are forever stopped either doing:
- Aqcuire Lock
- Release Deferred Lock (and it is interesteding that a method called release deferred lock can actually cause thread to be waiting forever).

In the last thread dump I have looked at, we had the interesting scenario.
11 THREADS were STUCK with stack traces similar to the one bellow.
NOTE: - This Eclipselink 2.4.2 - The lines of code in the concurrency manager of 2.6.4 would be elsewhere.


------------------------ ACQUIRE LOCK STUCH THREAD START SNIPPET ------------------------------
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.eclipse.persistence.internal.helper.ConcurrencyManager.acquire(ConcurrencyManager.java:94)
- locked <0x00000007119e05e8> (a org.eclipse.persistence.internal.helper.ConcurrencyManager)
at org.eclipse.persistence.internal.identitymaps.CacheKey.acquire(CacheKey.java:133)
at org.eclipse.persistence.internal.identitymaps.AbstractIdentityMap.acquireLock(AbstractIdentityMap.java:122)
at org.eclipse.persistence.internal.identitymaps.IdentityMapManager.acquireLock(IdentityMapManager.java:150)
at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:93)
at org.eclipse.persistence.internal.sessions.IdentityMapAccessor.acquireLock(IdentityMapAccessor.java:84)
at org.eclipse.persistence.internal.sessions.AbstractSession.retrieveCacheKey(AbstractSession.java:4834)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:782)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:611)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:564)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:777)
at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:462)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1150)
at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:852)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1109)
at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:421)
at org.eclipse.persistence.internal.sessions.AbstractSession.internalExecuteQuery(AbstractSession.java:2977)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1607)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1589)
at org.eclipse.persistence.internal.indirection.NoIndirectionPolicy.valueFromQuery(NoIndirectionPolicy.java:326)
at org.eclipse.persistence.mappings.ForeignReferenceMapping.valueFromRowInternal(ForeignReferenceMapping.java:2162)
at org.eclipse.persistence.mappings.OneToOneMapping.valueFromRowInternal(OneToOneMapping.java:1717)
at org.eclipse.persistence.mappings.ForeignReferenceMapping.valueFromRow(ForeignReferenceMapping.java:2051)
at org.eclipse.persistence.mappings.ForeignReferenceMapping.readFromRowIntoObject(ForeignReferenceMapping.java:1386)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildAttributesIntoObject(ObjectBuilder.java:448)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.refreshObjectIfRequired(ObjectBuilder.java:3706)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:842)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:611)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:564)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:777)
at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:462)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1150)
at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:852)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1109)
at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:421)
at org.eclipse.persistence.internal.sessions.AbstractSession.internalExecuteQuery(AbstractSession.java:2977)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1607)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1589)
at org.eclipse.persistence.internal.indirection.NoIndirectionPolicy.valueFromQuery(NoIndirectionPolicy.java:326)
at org.eclipse.persistence.mappings.ForeignReferenceMapping.valueFromRowInternal(ForeignReferenceMapping.java:2162)
at org.eclipse.persistence.mappings.OneToOneMapping.valueFromRowInternal(OneToOneMapping.java:1717)
at org.eclipse.persistence.mappings.ForeignReferenceMapping.valueFromRow(ForeignReferenceMapping.java:2051)



------------------------ ACQUIRE LOCK STUCK THREAD END SNIPPET ------------------------------



In the next snippet I will put the code of eclipse link running this acquire:

---------------------------------------------
ECLIPSELINK CODE:
---------------------------------------
/**
* Wait for all threads except the active thread.
* If the active thread just increment the depth.
* This should be called before entering a critical section.
* called with true from the merge process, if true then the refresh will not refresh the object
*/
public synchronized void acquire(boolean forMerge) throws ConcurrencyException {
while (((this.activeThread != null) || (this.numberOfReaders > 0)) && (this.activeThread != Thread.currentThread())) {
// This must be in a while as multiple threads may be released, or another thread may rush the acquire after one is released.
try {
this.numberOfWritersWaiting++;
wait(); <-------------------------------------------------------------- THE THREAD IS HERE IN AN ETERNAL LOOP AND NOT GOING ANYWHERE (most likely because of a dead lock)
this.numberOfWritersWaiting--;
} catch (InterruptedException exception) {
throw ConcurrencyException.waitWasInterrupted(exception.getMessage());
}
}
if (this.activeThread == null) {
this.activeThread = Thread.currentThread();
if (shouldTrackStack){
this.stack = new Exception();
}
}
this.lockedByMergeManager = forMerge;
this.depth++;
}
----------------------------------------------------------------------------


Ok the code above has a couple of things that I do not really understand.
1 - We are doing a TIMELESS WAIT inside of a WHILE.
I get the point while the condition of the while keeps returning TRUE the thread cannot be allowed to go forward and believe it has acuired the cache key.
Fine.
But if the code is not DEAD LOCK free - which I am guessing it is not - it should at least be DEAD LOCK resistent.
In the sense that if after TOO LONG we are still inside of this WHILE.
It is time to interrupt the transition, roll it back.,
Releas any of the acquired LOCKs by the thread.
And let other threads go forward with their transactions.
IN this manner - if transaction Synchronization is GOOD - which it appears to be - but not bullet proof - which appears to be not - at least if threads are given as RULE max allowed time to acquire a lock, we can be certain that a dead lock does not run eternal.
After a while threads will start abroding and releasing their locks.

This does not happen.


Point 2.
The code above looks wrong to me.
At least I do not understand why the thread - if it is interrupted does not reset the writers count.

Look at this:

wait();
this.numberOfWritersWaiting--;
} catch (InterruptedException exception) {
throw ConcurrencyException.waitWasInterrupted(exception.getMessage());
}

Ok - So if my WAIT is interrupted:
this.numberOfWritersWaiting-- will never be called.

So the Cache Key will forever have a GHOST Writer.
Or am I missing something?


------------ END of ACQUIRE STACK TRACE COMMENTS ------------------


The other part of the story in this dead lock are the 47 threads that apparently managed to get STUCK forever doing a RELEASE DEFERRED LOCKS.

The stack traces of these threads look all very similar to this:

------------------------ BEGIN SNIPPET RELEASE DEFFERRED LOCK STACK TRACE ------------------------------------
[STUCK] ExecuteThread: '130' for queue: 'weblogic.kernel.Default (self-tuning)' - priority:2 - threadId:0x000000002b07e800 - nativeId:0x2380 - nativeId (decimal):9088 - state:TIMED_WAITING
stackTrace:
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.eclipse.persistence.internal.helper.ConcurrencyManager.releaseDeferredLock(ConcurrencyManager.java:466) CASE 1
at org.eclipse.persistence.internal.identitymaps.CacheKey.releaseDeferredLock(CacheKey.java:419)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:872)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildWorkingCopyCloneNormally(ObjectBuilder.java:723)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObjectInUnitOfWork(ObjectBuilder.java:676)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:609)
at org.eclipse.persistence.internal.descriptors.ObjectBuilder.buildObject(ObjectBuilder.java:564)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.buildObject(ObjectLevelReadQuery.java:777)
at org.eclipse.persistence.queries.ReadObjectQuery.conformResult(ReadObjectQuery.java:355)
at org.eclipse.persistence.queries.ReadObjectQuery.registerResultInUnitOfWork(ReadObjectQuery.java:783)
at org.eclipse.persistence.queries.ReadObjectQuery.executeObjectLevelReadQuery(ReadObjectQuery.java:460)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeDatabaseQuery(ObjectLevelReadQuery.java:1150)
at org.eclipse.persistence.queries.DatabaseQuery.execute(DatabaseQuery.java:852)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.execute(ObjectLevelReadQuery.java:1109)
at org.eclipse.persistence.queries.ReadObjectQuery.execute(ReadObjectQuery.java:421)
at org.eclipse.persistence.queries.ObjectLevelReadQuery.executeInUnitOfWork(ObjectLevelReadQuery.java:1197)
at org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.internalExecuteQuery(UnitOfWorkImpl.java:2879)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1607)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1589)
at org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(AbstractSession.java:1540)
at org.eclipse.persistence.internal.jpa.EntityManagerImpl.executeQuery(EntityManagerImpl.java:838)
at org.eclipse.persistence.internal.jpa.EntityManagerImpl.findInternal(EntityManagerImpl.java:778)
at org.eclipse.persistence.internal.jpa.EntityManagerImpl.find(EntityManagerImpl.java:671)
at org.eclipse.persistence.internal.jpa.EntityManagerImpl.find(EntityManagerImpl.java:543)
at sun.reflect.GeneratedMethodAccessor249.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at weblogic.persistence.BasePersistenceContextProxyImpl.invoke(BasePersistenceContextProxyImpl.java:111)
at weblogic.persistence.TransactionalEntityManagerProxyImpl.invoke(TransactionalEntityManagerProxyImpl.java:82)
at weblogic.persistence.BasePersistenceContextProxyImpl.invoke(BasePersistenceContextProxyImpl.java:92)



------------------------ END SNIPPET RELEASE DEFFERRED LOCK STACK TRACE ------------------------------------


----------- ECLIPSELINK CODE RELATED TO RELEASE DEFFERRED LOCK ----------------
/**
* Release the deferred lock.
* This uses a deadlock detection and resolution algorithm to avoid cache deadlocks.
* The deferred lock manager keeps track of the lock for a thread, so that other
* thread know when a deadlock has occurred and can resolve it.
*/
public void releaseDeferredLock() throws ConcurrencyException {
Thread currentThread = Thread.currentThread();
DeferredLockManager lockManager = getDeferredLockManager(currentThread);
if (lockManager == null) {
return;
}
int depth = lockManager.getThreadDepth();

if (depth > 1) {
lockManager.decrementDepth();
return;
}

// If the set is null or empty, means there is no deferred lock for this thread, return.
if (!lockManager.hasDeferredLock()) {
lockManager.releaseActiveLocksOnThread();
removeDeferredLockManager(currentThread);
return;
}

lockManager.setIsThreadComplete(true);

// Thread have three stages, one where they are doing work (i.e. building objects)
// two where they are done their own work but may be waiting on other threads to finish their work,
// and a third when they and all the threads they are waiting on are done.
// This is essentially a busy wait to determine if all the other threads are done.
while (true) {
try{
// 2612538 - the default size of Map (32) is appropriate
Map recursiveSet = new IdentityHashMap();
if (isBuildObjectOnThreadComplete(currentThread, recursiveSet)) {// Thread job done.
lockManager.releaseActiveLocksOnThread();
removeDeferredLockManager(currentThread);
AbstractSessionLog.getLog().log(SessionLog.FINER, SessionLog.CACHE, "deferred_locks_released", currentThread.getName());
return;
} else {// Not done yet, wait and check again.
try {
Thread.sleep(1); <-------------------- This is where threads are stuck forever
} catch (InterruptedException interrupted) {
AbstractSessionLog.getLog().logThrowable(SessionLog.SEVERE, SessionLog.CACHE, interrupted);
lockManager.releaseActiveLocksOnThread();
removeDeferredLockManager(currentThread);
throw ConcurrencyException.waitWasInterrupted(interrupted.getMessage());
}
}
} catch (Error error) {
AbstractSessionLog.getLog().logThrowable(SessionLog.SEVERE, SessionLog.CACHE, error);
lockManager.releaseActiveLocksOnThread();
removeDeferredLockManager(currentThread);
throw error;
}
}
}
--------------------------------------------------------------------------

I actually prefer the code of the release deferred lock to that of the acquire.
In this release deferred lock the thread is not given a TIMELESS WAIT.

I deslike still the fact that the code in this method suffers from the same problem as the acuire. If you have a problem - and the threads are stuck forever in the eclipselink layer of code they:
(a) Will never get out of it
(b) They will never tell you what locks they have aquired, what cache keys they did not acquire that somebody else has acuired etc...

I would imagine that if a thread is stuck in RELEASED deferred lock for too long.
It should be possible to build a graph stating:
- This thread is owning Cache Key A,B,C and wants cache key D.
Bu cache KEY D is owned by thead 2. And thread 2 is trying to acquire cache KEY A.
HEre is your dead lock.

No. We look at a stack trace and we have literaly no Idea what entity cache key is currently being the root of our problem.
We have no Idea of knowing what resources each THREAD is currently owning.
And what resources it cannot acquire.
We have none of this information.

IT would already be a very big help if at least even without this information, if threads get into a cyclic dead lock they they can explode via an exception out of this.


NOTE:
For the time being, to try to survive the ongoing problem, what I have done was attack the DEFERRED lock source code.
Since the deferred lock source code seems to not suffer from the cuncrrency exception bug of the acquire lock source code that forgets to decrement thenumber of writers.

And in the RELEASE DEFERRED lock, right now, if the process is in that while(true) for more than 40 seconds - configurable by system property - I make sure that I simulate a concurrency exception.
This in hope that with the simulated exception the code:
lockManager.releaseActiveLocksOnThread();
removeDeferredLockManager(currentThread);

Will run and solve my dead lock.



---------------------

Question:
- Since eclipselink 2.4.2 have there been improvements to the concurrency mangament?
- Have any dead lock risks been resolved/mitigated since then?
- Have any features to explode threads out of dead loclks been implemented?

- Is it not a BUG that the acquire lock, if INTERRUPTED, does not do the same thing that the releaseDeferredLock code is doing, namely, the thread is not going to be releasing the active locks on the thread. Which menas - the exception handling of an interrupted thread in releasing a deferred lock is logically correct, but that of the acquire lock will break the cache leaving the cache key corrupted with the counters of writters incremented and the locks aquired by the thread still blocked by the thread that was interrupted?


NOTE:
I have seen a very similar thread dump occuring in an eclipselink 2.6.4 running weblogic 12.2.1.2.
So I am guessing that depending on the degree of concurrency and number of JDBC connections allowed by an application, the risk for dead locks is also present in 2.6.4. But if 2.6.4 has improvements in relation to 2.4.2 - it would be a good Idea, in my oppinion to down port the concurrency manager imprvements to 2.4.X branch as well...
Would this be possible? Are there any improvements?



Are there any plans of improving the concurrency manager to ensure that the system dead lock TOLERANT by, say, aborting transacitons that are stuck for too long try to aquire release locks?


A third and useful improvement, would be to make a dead lock Explanation tool. I am guessing as well that for someone knowing eclipselink lock management system, that it is is not that hard to make a search algorithm that starts an Apparently stuck thread and sees what locks it cannot aquire but wants to aquire. Expans the thread that is currently owning the lock and does the same to ID until it finally finds out a lock already expanded in the past.
The report could say:
- (a) Here is the set of threads in a dead lock
- (b) Here is the Locks each has acquired that create a circular dependency.

Could this be done?
Are there any plans to do this?


If we know which threads got stuck together, and by looking at the stack trace we know what business logic these threads are triggering, perhaps there would be a possibility to serialize these processes and from the business side mitigate the risk of dead lock.

Can anybody comment on this?

Both in regards to the version 2.4.2 of WLS 12.1.2 and 2.6.4 of WLS 12.2.1.2?


What recommendations would you have to mitigate the dead lock problem?

The thread dumps say tones about hte problem but at the same time they say very little.
To have any chance of really understanding the dead lock one:
- Needs a Heap dump that we can look at stuck thread memory state and figure out what cache key is getting us burned to aquire
- Needs to understand the domain model of eclipselink in depth to understand how these deferred locks are supposed to work and be managed
- Needs a lot of luck as well, because some information is simply not having a level of detail necessary.
Such as a counter of Numbers of Reads/Writters.
This is not very helpful to know that there are 4 Readers on a cache key.
Because a counter of readers is just an integer, it tells us nothing about the threads that are currently being counted as readers for example.
So I consider it boderline impossible for a non eclipselink expert to have any chance of determining the reason for the dead lock.
The answer to the question should come from eclipselink code itsef - after the system is percieved to start mal functioning on the DB layer.
Could something be done to improve this?



Thanks for any help.
Thanks for any comments and suggestons.

For the time being - I am going with a bazuka approach. The release deferred lock in the elcipselink we are using is tuned to if the method does not finish in 40 seocnds - assume the dead lock is in place and run the interrupt exception logic that releases all of the deferred locks.
Would there be any better approach as a work around?


Kindest regards

[Updated on: Tue, 29 January 2019 07:33]

Report message to a moderator

Re: Eclipselink - dead locks in aqcuire and release deferred locks - Eclipselink 2.4.2 and above [message #1801944 is a reply to message #1801870] Tue, 29 January 2019 17:31 Go to previous messageGo to next message
Nuno G. de M. is currently offline Nuno G. de M.Friend
Messages: 33
Registered: September 2012
Member
In the case of hacking eclipselink 2.6.4.

This is what the HACKED class currently looks like:

/*******************************************************************************
 * Copyright (c) 1998, 2013 Oracle and/or its affiliates. All rights reserved.
 * This program and the accompanying materials are made available under the 
 * terms of the Eclipse Public License v1.0 and Eclipse Distribution License v. 1.0 
 * which accompanies this distribution. 
 * The Eclipse Public License is available at http://www.eclipse.org/legal/epl-v10.html
 * and the Eclipse Distribution License is available at 
 * http://www.eclipse.org/org/documents/edl-v10.php.
 *
 * Contributors:
 *     Oracle - initial API and implementation from Oracle TopLink
 ******************************************************************************/  
package org.eclipse.persistence.internal.helper;

import java.io.*;
import java.util.*;
import java.util.concurrent.ConcurrentHashMap;

import org.eclipse.persistence.config.SystemProperties;
import org.eclipse.persistence.exceptions.*;
import org.eclipse.persistence.internal.localization.*;
import org.eclipse.persistence.internal.identitymaps.CacheKey;
import org.eclipse.persistence.logging.*;

/**
 * INTERNAL:
 * <p>
 * <b>Purpose</b>: To maintain concurrency for a particular task.
 * It is a wrappers of a semaphore that allows recursive waits by a single thread.
 * <p>
 * <b>Responsibilities</b>:
 * <ul>
 * <li> Keep track of the active thread.
 * <li> Wait all other threads until the first thread is done.
 * <li> Maintain the depth of the active thread.
 * </ul>
 */
public class ConcurrencyManager implements Serializable {
    
    protected int numberOfReaders;
    protected int depth;
    protected int numberOfWritersWaiting;
    protected volatile transient Thread activeThread;
    public static Map<Thread, DeferredLockManager> deferredLockManagers = initializeDeferredLockManagers();
    protected boolean lockedByMergeManager;
    
    protected static boolean shouldTrackStack = System.getProperty(SystemProperties.RECORD_STACK_ON_LOCK) != null;
    protected Exception stack;

    /**
     * Initialize the newly allocated instance of this class.
     * Set the depth to zero.
     */
    public ConcurrencyManager() {
        this.depth = 0;
        this.numberOfReaders = 0;
        this.numberOfWritersWaiting = 0;
    }

    /**
     * Wait for all threads except the active thread.
     * If the active thread just increment the depth.
     * This should be called before entering a critical section.
     */
    public void acquire() throws ConcurrencyException {
        this.acquire(false);
    }

    /**
     * Wait for all threads except the active thread.
     * If the active thread just increment the depth.
     * This should be called before entering a critical section.
     * called with true from the merge process, if true then the refresh will not refresh the object
     */
    public synchronized void acquire(boolean forMerge) throws ConcurrencyException {
     // FIX - APQ-2473 - Flag the time when we start the while loop
        final Date whileStartDate = new Date();
        Thread currentThread = Thread.currentThread();
        DeferredLockManager lockManager = getDeferredLockManager(currentThread);
        
        while (((this.activeThread != null) || (this.numberOfReaders > 0)) && (this.activeThread != Thread.currentThread())) {
            // This must be in a while as multiple threads may be released, or another thread may rush the acquire after one is released.
            try {
                this.numberOfWritersWaiting++;
                
                // FIX - APQ-2473 - Do not wait forever - max allowed time to wait is 10 second and then check conditions again
                // wait();
                wait(10000l);
                
                
                // FIX - APQ-2473 -
                // Run a method that will fire up an exception if we  having been sleeping for too long
                HackingEclipseHelperUtil.SINGLETON.determineIfReleaseDeferredLockAppearsToBeDeadLocked(this, whileStartDate, lockManager);
                
                // FIX - APQ-2473 -  Moved to finally block to ensure it always runs
                // this.numberOfWritersWaiting--;
            } catch (InterruptedException exception) {
                // FIX - APQ-2473 - If the thread is interrupted we want to make sure we release all of the locks the thread was owning
                releaseAllLocksAquiredByThread(lockManager);
                
                throw ConcurrencyException.waitWasInterrupted(exception.getMessage());
            } finally {
                // FIX - APQ-2473 -
                // Since above we incremente the number of writers
                // whether or not the thread is exploded by an interrupt 
                // we need to make sure we decrement the number of writer to not allow the code to be corrupeted 
                this.numberOfWritersWaiting--;
            }
        }
        if (this.activeThread == null) {
            this.activeThread = Thread.currentThread();
            if (shouldTrackStack){
                this.stack = new Exception();
            }
        }
        this.lockedByMergeManager = forMerge;
        this.depth++;
    }

    /**
     * If the lock is not acquired already acquire it and return true.
     * If it has been acquired already return false
     * Added for CR 2317
     */
    public boolean acquireNoWait() throws ConcurrencyException {
        return acquireNoWait(false);
    }

    /**
     * If the lock is not acquired already acquire it and return true.
     * If it has been acquired already return false
     * Added for CR 2317
     * called with true from the merge process, if true then the refresh will not refresh the object
     */
    public synchronized boolean acquireNoWait(boolean forMerge) throws ConcurrencyException {
        if ((this.activeThread == null && this.numberOfReaders == 0) || (this.activeThread == Thread.currentThread())) {
            //if I own the lock increment depth
            acquire(forMerge);
            return true;
        } else {
            return false;
        }
    }

    /**
     * If the lock is not acquired already acquire it and return true.
     * If it has been acquired already return false
     * Added for CR 2317
     * called with true from the merge process, if true then the refresh will not refresh the object
     */
    public synchronized boolean acquireWithWait(boolean forMerge, int wait) throws ConcurrencyException {
        if ((this.activeThread == null && this.numberOfReaders == 0) || (this.activeThread == Thread.currentThread())) {
            //if I own the lock increment depth
            acquire(forMerge);
            return true;
        } else {
            try {
                wait(10000l);
            } catch (InterruptedException e) {
                return false;
            }
            if ((this.activeThread == null && this.numberOfReaders == 0) || (this.activeThread == Thread.currentThread())){
                acquire(forMerge);
                return true;
            }
            return false;
        }
    }

    /**
     * If the activeThread is not set, acquire it and return true.
     * If the activeThread is set, it has been acquired already, return false.
     * Added for Bug 5840635
     * Call with true from the merge process, if true then the refresh will not refresh the object.
     */
    public synchronized boolean acquireIfUnownedNoWait(boolean forMerge) throws ConcurrencyException {
        // Only acquire lock if active thread is null. Do not check current thread. 
        if (this.activeThread == null && this.numberOfReaders == 0) {
             // if lock is unowned increment depth
            acquire(forMerge);
            return true;
        } else {
            return false;
        }
    }

    /**
     * Add deferred lock into a hashtable to avoid deadlock
     */
    public void acquireDeferredLock() throws ConcurrencyException {
        Thread currentThread = Thread.currentThread();
        DeferredLockManager lockManager = getDeferredLockManager(currentThread);
        if (lockManager == null) {
            lockManager = new DeferredLockManager();
            putDeferredLock(currentThread, lockManager);
        }
        lockManager.incrementDepth();
        synchronized (this) {
            // FIX - APQ-2473 - Flag the time when we start the while loop
            final Date whileStartDate = new Date();
            
            while (this.numberOfReaders != 0) {
                // There are readers of this object, wait until they are done before determining if
                //there are any other writers.  If not we will wait on the readers for acquire.  If another
                //thread is also waiting on the acquire then a deadlock could occur.  See bug 3049635
                //We could release all active locks before releasing deferred but the object may not be finished building
                //we could make the readers get a hard lock, but then we would just build a deferred lock even though
                //the object is not being built.
                try {
                    this.numberOfWritersWaiting++;
                  
                    // FIX - APQ-2473 - Do not wait forever - max allowed time to wait is 10 second and then check conditions again
                    // wait();
                    wait(1000l);
                    
                    // FIX - APQ-2473 -  Moved to finally block to ensure it always runs
                    // this.numberOfWritersWaiting--;
                    

                    // FIX - APQ-2473 -
                    // Run a method that will fire up an exception if we  having been sleeping for too long
                    HackingEclipseHelperUtil.SINGLETON.determineIfReleaseDeferredLockAppearsToBeDeadLocked(this, whileStartDate, lockManager);                    
                } catch (InterruptedException exception) {   
                    // FIX - APQ-2473 - If the thread is interrupted we want to make sure we release all of the locks the thread was owning
                    releaseAllLocksAquiredByThread(lockManager);
                    
                    
                    throw ConcurrencyException.waitWasInterrupted(exception.getMessage());
                } finally {
                    // FIX - APQ-2473 -
                    // Since above we incremente the number of writers
                    // whether or not the thread is exploded by an interrupt 
                    // we need to make sure we decrement the number of writer to not allow the code to be corrupeted 
                    this.numberOfWritersWaiting--;
                }
            }
            if ((this.activeThread == currentThread) || (!isAcquired())) {
                lockManager.addActiveLock(this);
                acquire();
            } else {
                lockManager.addDeferredLock(this);
                if (AbstractSessionLog.getLog().shouldLog(SessionLog.FINER) && this instanceof CacheKey) {
                    AbstractSessionLog.getLog().log(SessionLog.FINER, SessionLog.CACHE, "acquiring_deferred_lock", ((CacheKey)this).getObject(), currentThread.getName());
                }
            }
        }
    }
        
    /**
     * Check the lock state, if locked, acquire and release a deferred lock.
     * This optimizes out the normal deferred-lock check if not locked.
     */
    public void checkDeferredLock() throws ConcurrencyException {
        // If it is not locked, then just return.
        if (this.activeThread == null) {
            return;
        }
        acquireDeferredLock();
        releaseDeferredLock();
    }
    
    /**
     * Check the lock state, if locked, acquire and release a read lock.
     * This optimizes out the normal read-lock check if not locked.
     */
    public void checkReadLock() throws ConcurrencyException {
        // If it is not locked, then just return.
        if (this.activeThread == null) {
            return;
        }
        acquireReadLock();
        releaseReadLock();
    }
    
    /**
     * Wait on any writer.
     * Allow concurrent reads.
     */
    public synchronized void acquireReadLock() throws ConcurrencyException {
        // Cannot check for starving writers as will lead to deadlocks.
        while ((this.activeThread != null) && (this.activeThread != Thread.currentThread())) {
            try {
                wait();
            } catch (InterruptedException exception) {
                throw ConcurrencyException.waitWasInterrupted(exception.getMessage());
            }
        }
        this.numberOfReaders++;
    }

    /**
     * If this is acquired return false otherwise acquire readlock and return true
     */
    public synchronized boolean acquireReadLockNoWait() {
        if ((this.activeThread == null) || (this.activeThread == Thread.currentThread())) {
            acquireReadLock();
            return true;
        } else {
            return false;
        }
    }

    /**
     * Return the active thread.
     */
    public Thread getActiveThread() {
        return activeThread;
    }

    /**
     * Return the deferred lock manager from the thread
     */
    public static DeferredLockManager getDeferredLockManager(Thread thread) {
        return getDeferredLockManagers().get(thread);
    }

    /**
     * Return the deferred lock manager hashtable (thread - DeferredLockManager).
     */
    protected static Map<Thread, DeferredLockManager> getDeferredLockManagers() {
        return deferredLockManagers;
    }
    
    /**
     * Init the deferred lock managers (thread - DeferredLockManager).
     */
    protected static Map initializeDeferredLockManagers() {
        return new ConcurrentHashMap();
    }

    /**
     * Return the current depth of the active thread.
     */
    public int getDepth() {
        return depth;
    }

    /**
     * Number of writer that want the lock.
     * This is used to ensure that a writer is not starved.
     */
    public int getNumberOfReaders() {
        return numberOfReaders;
    }

    /**
     * Number of writers that want the lock.
     * This is used to ensure that a writer is not starved.
     */
    public int getNumberOfWritersWaiting() {
        return numberOfWritersWaiting;
    }
    
    /**
     * Return if a thread has acquire this manager.
     */
    public boolean isAcquired() {
        return depth > 0;
    }

    /**
     * INTERNAL:
     * Used byt the refresh process to determine if this concurrency manager is locked by
     * the merge process.  If it is then the refresh should not refresh the object
     */
    public boolean isLockedByMergeManager() {
        return this.lockedByMergeManager;
    }

    /**
     * Check if the deferred locks of a thread are all released
     */
    public static boolean isBuildObjectOnThreadComplete(Thread thread, Map recursiveSet) {
        if (recursiveSet.containsKey(thread)) {
            return true;
        }
        recursiveSet.put(thread, thread);

        DeferredLockManager lockManager = getDeferredLockManager(thread);
        if (lockManager == null) {
            return true;
        }

        Vector deferredLocks = lockManager.getDeferredLocks();
        for (Enumeration deferredLocksEnum = deferredLocks.elements();
                 deferredLocksEnum.hasMoreElements();) {
            ConcurrencyManager deferedLock = (ConcurrencyManager)deferredLocksEnum.nextElement();
            Thread activeThread = null;
            if (deferedLock.isAcquired()) {
                activeThread = deferedLock.getActiveThread();

                // the active thread may be set to null at anypoint
                // if added for CR 2330 
                if (activeThread != null) {
                    DeferredLockManager currentLockManager = getDeferredLockManager(activeThread);
                    if (currentLockManager == null) {
                        return false;
                    } else if (currentLockManager.isThreadComplete()) {
                        activeThread = deferedLock.getActiveThread();
                        // The lock may suddenly finish and no longer have an active thread.
                        if (activeThread != null) {
                            if (!isBuildObjectOnThreadComplete(activeThread, recursiveSet)) {
                                return false;
                            }
                        }
                    } else {
                        return false;
                    }
                }
            }
        }
        return true;
    }

    /**
     * Return if this manager is within a nested acquire.
     */
    public boolean isNested() {
        return depth > 1;
    }

    public void putDeferredLock(Thread thread, DeferredLockManager lockManager) {
        getDeferredLockManagers().put(thread, lockManager);
    }

    /**
     * Decrement the depth for the active thread.
     * Assume the current thread is the active one.
     * Raise an error if the depth become < 0.
     * The notify will release the first thread waiting on the object,
     * if no threads are waiting it will do nothing.
     */
    public synchronized void release() throws ConcurrencyException {
        if (this.depth == 0) {
            throw ConcurrencyException.signalAttemptedBeforeWait();
        } else {
            this.depth--;
        }
        if (this.depth == 0) {
            this.activeThread = null;
            if (shouldTrackStack){
                this.stack = null;
            }
            this.lockedByMergeManager = false;
            notifyAll();
        }
    }

    /**
     * Release the deferred lock.
     * This uses a deadlock detection and resolution algorithm to avoid cache deadlocks.
     * The deferred lock manager keeps track of the lock for a thread, so that other
     * thread know when a deadlock has occurred and can resolve it.
     */
    public void releaseDeferredLock() throws ConcurrencyException {
        Thread currentThread = Thread.currentThread();
        DeferredLockManager lockManager = getDeferredLockManager(currentThread);
        if (lockManager == null) {
            return;
        }
        int depth = lockManager.getThreadDepth();

        if (depth > 1) {
            lockManager.decrementDepth();
            return;
        }

        // If the set is null or empty, means there is no deferred lock for this thread, return.
        if (!lockManager.hasDeferredLock()) {
            lockManager.releaseActiveLocksOnThread();
            removeDeferredLockManager(currentThread);
            return;
        }

        lockManager.setIsThreadComplete(true);
        
        // FIX - APQ-2473 - start a date time
        // Thread have three stages, one where they are doing work (i.e. building objects)
        // two where they are done their own work but may be waiting on other threads to finish their work,
        // and a third when they and all the threads they are waiting on are done.
        // This is essentially a busy wait to determine if all the other threads are done.
        final Date whileStartDate = new Date();

        // Thread have three stages, one where they are doing work (i.e. building objects)
        // two where they are done their own work but may be waiting on other threads to finish their work,
        // and a third when they and all the threads they are waiting on are done.
        // This is essentially a busy wait to determine if all the other threads are done.
        while (true) {
            try{
                // 2612538 - the default size of Map (32) is appropriate
                Map recursiveSet = new IdentityHashMap();
                if (isBuildObjectOnThreadComplete(currentThread, recursiveSet)) {// Thread job done.
                    lockManager.releaseActiveLocksOnThread();
                    removeDeferredLockManager(currentThread);
                    AbstractSessionLog.getLog().log(SessionLog.FINER, SessionLog.CACHE, "deferred_locks_released", currentThread.getName());
                    return;
                } else {// Not done yet, wait and check again.
                    try {
                        Thread.sleep(1);
                        
                        // FIX - APQ-2473 -
                        // Run a method that will fire up an exception if we  having been sleeping for too long
                        HackingEclipseHelperUtil.SINGLETON.determineIfReleaseDeferredLockAppearsToBeDeadLocked(this, whileStartDate, lockManager);
                        
                    } catch (InterruptedException interrupted) {
                        AbstractSessionLog.getLog().logThrowable(SessionLog.SEVERE, SessionLog.CACHE, interrupted);
                        releaseAllLocksAquiredByThread(lockManager);
                        throw ConcurrencyException.waitWasInterrupted(interrupted.getMessage());
                    }
                }
            } catch (Error error) {
                AbstractSessionLog.getLog().logThrowable(SessionLog.SEVERE, SessionLog.CACHE, error);
                releaseAllLocksAquiredByThread(lockManager);
                throw error;
            }
        }
    }
    
    

    /**
     * Decrement the number of readers.
     * Used to allow concurrent reads.
     */
    public synchronized void releaseReadLock() throws ConcurrencyException {
        if (this.numberOfReaders == 0) {
            throw ConcurrencyException.signalAttemptedBeforeWait();
        } else {
            this.numberOfReaders--;
        }
        if (this.numberOfReaders == 0) {
            notifyAll();
        }
    }

    /**
     * Remove the deferred lock manager for the thread
     */
    public static DeferredLockManager removeDeferredLockManager(Thread thread) {
        return getDeferredLockManagers().remove(thread);
    }

    /**
     * Set the active thread.
     */
    public void setActiveThread(Thread activeThread) {
        this.activeThread = activeThread;
    }

    /**
     * Set the current depth of the active thread.
     */
    protected void setDepth(int depth) {
        this.depth = depth;
    }

    /**
     * INTERNAL:
     * Used by the mergemanager to let the read know not to refresh this object as it is being
     * loaded by the merge process.
     */
    public void setIsLockedByMergeManager(boolean state) {
        this.lockedByMergeManager = state;
    }

    /**
     * Track the number of readers.
     */
    protected void setNumberOfReaders(int numberOfReaders) {
        this.numberOfReaders = numberOfReaders;
    }

    /**
     * Number of writers that want the lock.
     * This is used to ensure that a writer is not starved.
     */
    protected void setNumberOfWritersWaiting(int numberOfWritersWaiting) {
        this.numberOfWritersWaiting = numberOfWritersWaiting;
    }
    
    public synchronized void transitionToDeferredLock() {
        Thread currentThread = Thread.currentThread();
        DeferredLockManager lockManager = getDeferredLockManager(currentThread);
        if (lockManager == null) {
            lockManager = new DeferredLockManager();
            putDeferredLock(currentThread, lockManager);
        }
        lockManager.incrementDepth();
        lockManager.addActiveLock(this);
    }

    /**
     * Print the nested depth.
     */
    public String toString() {
        Object[] args = { Integer.valueOf(getDepth()) };
        return Helper.getShortClassName(getClass()) + ToStringLocalization.buildMessage("nest_level", args);
    }

    public Exception getStack() {
        return stack;
    }

    public void setStack(Exception stack) {
        this.stack = stack;
    }

    public static boolean shouldTrackStack() {
        return shouldTrackStack;
    }

    /**
     * INTERNAL:
     * This can be set during debugging to record the stacktrace when a lock is acquired.
     * Then once IdentityMapAccessor.printIdentityMapLocks() is called the stack call for each
     * lock will be printed as well.  Because locking issues are usually quite time sensitive setting 
     * this flag may inadvertently remove the deadlock because of the change in timings.
     * 
     * There is also a system level property for this setting. "eclipselink.cache.record-stack-on-lock"
     * @param shouldTrackStack
     */
    public static void setShouldTrackStack(boolean shouldTrackStack) {
        ConcurrencyManager.shouldTrackStack = shouldTrackStack;
    }
    
    // Hacking APIs
    /**
     * For the thread to release all of its locks.
     * @param lockManager
     *      the deferred lock manager
     */
    // FIX - APQ-2473 - Put in one place the code that frees up the locks acuired by the current thread
    protected void releaseAllLocksAquiredByThread(DeferredLockManager lockManager) {
        Thread currentThread = Thread.currentThread();
        lockManager.releaseActiveLocksOnThread();
        removeDeferredLockManager(currentThread);
        
        // Log some information in the LOG
        String cacheKeyToString = HackingEclipseHelperUtil.SINGLETON.createToStringExplainingOwnedCacheKey(this);
        String errMsg = "releaseAllLocksAquiredByThread has been invoked. On  " + cacheKeyToString;
        AbstractSessionLog.getLog().log(SessionLog.SEVERE, SessionLog.CACHE, "deferred_locks_released", currentThread.getName());
        
        // Question:
        // Would it be safe to send a NOTIFY ALL?
        // probably not. But there might be threads waiting for the release of the lock
        // Normally the Notifyall is necessary when locks are released. But in this case we are even dealing with not one but ALL
        // of hte locks owned by thread so there would be multiple objects notify - we probably cannot do this
    }
    
}




And then there is this helper UTIL:

/*******************************************************************************
 * Copyright (c) 1998, 2013 Oracle and/or its affiliates. All rights reserved.
 * This program and the accompanying materials are made available under the 
 * terms of the Eclipse Public License v1.0 and Eclipse Distribution License v. 1.0 
 * which accompanies this distribution. 
 * The Eclipse Public License is available at http://www.eclipse.org/legal/epl-v10.html
 * and the Eclipse Distribution License is available at 
 * http://www.eclipse.org/org/documents/edl-v10.php.
 *
 * Contributors:
 *     Oracle - initial API and implementation from Oracle TopLink
 ******************************************************************************/  
package org.eclipse.persistence.internal.helper;

import java.io.*;
import java.util.*;
import java.util.concurrent.ConcurrentHashMap;

import org.eclipse.persistence.config.SystemProperties;
import org.eclipse.persistence.exceptions.*;
import org.eclipse.persistence.internal.localization.*;
import org.eclipse.persistence.internal.identitymaps.CacheKey;
import org.eclipse.persistence.logging.*;

/**
 * Helper class to help us build diagnostic strings
 */
public class HackingEclipseHelperUtil implements Serializable {
    
    public static final HackingEclipseHelperUtil SINGLETON = new HackingEclipseHelperUtil();
    
   
    /**
     * @return check if the user has specified a system property to control how long we are willing to wait before firing up an exception
     */
    public long getMaxAllowedSleepTimeMs() {
        long defaultMaxAllowedSleepTimeMs = 40000;
        
        // (a) Check if a system property was provided by the user to control how long we tolerate before exploding thread waiting to release
        // deferred locks
        String eclipseLinkConcurrencyManagerMaxAllowedSleepTimeMs = System.getProperty("eclipseLinkConcurrencyManagerMaxAllowedSleepTimeMs");
        if(eclipseLinkConcurrencyManagerMaxAllowedSleepTimeMs != null) {
            try {
                return Long.parseLong(eclipseLinkConcurrencyManagerMaxAllowedSleepTimeMs.trim());
            } catch(Exception ignoreE) {
                return defaultMaxAllowedSleepTimeMs ;
            }            
        }
        return defaultMaxAllowedSleepTimeMs;
    }
    
    /**
     * Create a print of the ACTIVE locks associated to the DeferredLockManager
     */
    public String createStringWithSummaryOfActiveLocksOnThread(DeferredLockManager lockManager) {
        StringBuilder sb = new StringBuilder();
        sb.append("DeferredLockManager - Listing of all Deferred Locks.");
        sb.append("\n\n");
        sb.append("Thread Name: ").append(Thread.currentThread().getName());
        sb.append("\n\n");
        
        // Loop over al of the active locks and print them
        long activeLock = 0;
        Vector activeLocks = lockManager.getActiveLocks();
        if (!activeLocks.isEmpty()) {
            for (Enumeration activeLocksEnum = activeLocks.elements();
                     activeLocksEnum.hasMoreElements();) {
                activeLock++;
                ConcurrencyManager manager = (ConcurrencyManager)activeLocksEnum.nextElement();
                String concurrencyManagerActiveLock = SINGLETON.createToStringExplainingOwnedCacheKey(manager);
                sb.append("ACTIVE LOCK NR: ").append("" + activeLock).append("ConcurrencyManager: ").append(concurrencyManagerActiveLock);
                sb.append("\n\n");
            }
        }
        return sb.toString();        
    }
    
    /**
     * 
     * @return A to string of the owned cache key
     */
    public String createToStringExplainingOwnedCacheKey(ConcurrencyManager concurrencyManager) {
        if(concurrencyManager instanceof CacheKey) {
            CacheKey cacheKey = (CacheKey) concurrencyManager;
            Object cacheKeykey = cacheKey.getKey();
            Object cacheKeyObject = cacheKey.getObject();
            String canonicalName = cacheKeyObject != null? cacheKeyObject.getClass().getCanonicalName():null;
            return String.format("ConcurrencyManager-CacheKey: %1$s ownerCacheKey (key,object, canonicalName) = (%2$s, %3$s, %4$s).", this, cacheKeykey, cacheKeyObject, canonicalName);
        } else {
            return String.format("ConcurrencyManager: %1$s. .", this);
        }
        
    }
    
    /**
     * Throw an interrupted exception if appears that eclipse link code is taking too long to release a deferred lock. 
     * @param whileStartDate
     *      the start date of the while tru loop for releasing a deferred lock 
     * @throws InterruptedException
     *  we fire an interupted exception to ensure that the code  blows up and releases all of the locks it had.
     */
    public void determineIfReleaseDeferredLockAppearsToBeDeadLocked(ConcurrencyManager concurrencyManager, final Date whileStartDate, DeferredLockManager lockManager) throws InterruptedException {
        // Determine if we believe to be dealing with a dead lock
        Thread currentThread = Thread.currentThread();
        final long maxAllowedSleepTime40ThousandMs = HackingEclipseHelperUtil.SINGLETON.getMaxAllowedSleepTimeMs();
        Date whileCurrentDate = new Date();
        long elpasedTime = whileCurrentDate.getTime() - whileStartDate.getTime();
        if (elpasedTime > maxAllowedSleepTime40ThousandMs) {
            // We believe this is a dead lock so now we will log some information
            String ownedCacheKey = createToStringExplainingOwnedCacheKey(concurrencyManager);
           
            String errorMessageBase = String.format(
                    "RELEASE DEFERRED LOCK PROBLEM:  The release deffered log process has not managed to finish in: %1$s ms.%n"
                            + " (ownerCacheKey) = (%2$s). Current thread: %3$s. %n",
                    elpasedTime, ownedCacheKey, currentThread.getName());
            String errorMessageExplainingActiveLocksOnThread = HackingEclipseHelperUtil.SINGLETON.createStringWithSummaryOfActiveLocksOnThread(lockManager);
            String errorMessage = errorMessageBase +  errorMessageExplainingActiveLocksOnThread;

            AbstractSessionLog.getLog().log(SessionLog.SEVERE, SessionLog.CACHE, errorMessage,
                    currentThread.getName());
            
            throw new InterruptedException(errorMessage);
        }
    }
    
}



Re: Eclipselink - dead locks in aqcuire and release deferred locks - Eclipselink 2.4.2 and above [message #1801945 is a reply to message #1801944] Tue, 29 January 2019 17:34 Go to previous messageGo to next message
Nuno G. de M. is currently offline Nuno G. de M.Friend
Messages: 33
Registered: September 2012
Member
In the above hacked code.

The modifications can all be found simply by searching for:
// FIX - APQ-2473

And it will be pretty clear that what I am doing to the code is trying to ensure that there are not ETERNAL WAITS until someone remembers to notifY().

That every wait has a time limt.
And if we are waiting for too long the determineIfReleaseDeferredLockAppearsToBeDeadLocked -- will fire some fake thread interrupted exception.

And this leads to try to release all of the Locks.

I think that something like this - put properly implemented - is needed very direly.
And should ideally be back merged far back.

So that we can patch our concurrency managers.

Re: Eclipselink - dead locks in aqcuire and release deferred locks - Eclipselink 2.4.2 and above [message #1803447 is a reply to message #1801945] Thu, 28 February 2019 19:04 Go to previous message
Chris Delahunt is currently offline Chris DelahuntFriend
Messages: 1326
Registered: July 2009
Senior Member
There is a lot of information here, making it hard to understand what you are after. The locking scenario in EclipseLink is rather complex, but there is a good reason -Processes like updates, refreshes and the initial queries that bring objects into the cache must prevent other reading level processes from returning those objects until they are completed. A deferred lock is just a way that allows a reading thread to register and get in line for a notification that it can return and use those objects. In applications that use a shared cache and are heavy on the refreshes and updates on key/root components in an object graph, these root objects the graph might cause a bottleneck for the system. The way to identify this is to look at the threads involved in any apparent 'deadlock', and identify what they are doing; the object graphs they modify and access. If you know that, you can then modify your access or the model itself to remove those key spots of contention.

Commonly, mixing lazy and eager access in large graph will obfuscate the problem, as accessing a reference will force a query which goes to the cache and might load more objects then might be needed, causing other threads to build up on a slower write process. Putting in lazy everywhere possible sometimes helps, so these threads can do their thing without having to wait to fetch an entire object graph.

I don't disagree that some configurable level of detection on the ConcurrencyManager would be ideal, only that such changes usually have performance consequences. Unlike databases which have deadlock detection, EclipseLink does not control the container in which it runs and relies on the JVM to monitor its threads. If you discover such a situation, you might instrument your code to call printIdentityMaps which might show the objects involved in various EclipseLink locks, as mentioned here: https://www.eclipse.org/forums/index.php?t=msg&th=389907&goto=934039&#msg_934039

For others, the steps outlined here are also a good staring point before going into changing EclipseLink concurrency managers: http://wiki.eclipse.org/EclipseLink/FAQ/JPA#How_to_diagnose_and_resolve_hangs_and_deadlocks.3F
Previous Topic:EclipseLink does not see created tables in H2 DB
Next Topic:ExpressionBuilder left outer join with on clause
Goto Forum:
  


Current Time: Sat Aug 24 21:55:38 GMT 2019

Powered by FUDForum. Page generated in 0.02279 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top