Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » EMF » [CDO] lock(long timeout) not timeout-ing...(In some cases, method lock() lasts longer than given timeout (ie : forever!))
[CDO] lock(long timeout) not timeout-ing... [message #1694764] Thu, 07 May 2015 15:43 Go to next message
Alexandre Borgoltz is currently offline Alexandre BorgoltzFriend
Messages: 31
Registered: July 2009
Location: France
Member
Hey everybody,

I am facing a strange problem with CDO write locking. I reproduced it with CDO 4.3 and CDO 4.4

I am having some concurrent threads simply trying to add objects in a list.

The case
For example (Model1), say a Company has many Categories.
Each thread iterates N times.
At each iteration, each thread

  1. creates a new session and a new transaction in that session.
    CDOSession session = openSession();
    CDOTransaction transaction = session.openTransaction();

  2. retreives the company
    Company company = transaction.getObject(...);
    CDOObject cdoObject = CDOUtil.getCDOObject(company);

  3. write-lock the company - notice the "LOCK_TIMEOUT" parameter
    CDOLock lock = cdoObject.cdoWriteLock();
    
    for (int retry = 0; retry < RETRIES; ++retry)
    {
      try
      {              
        lock.lock(LOCK_TIMEOUT);
        break;
      }
      catch (TimeoutException ex)
      {
        if (retry == RETRIES - 1)
        {
          return;
        }
      }
    }

  4. creates a new Category
    Category category = getModel1Factory().createCategory();

  5. adds the category to the company's list of categories
    company.getCategories().add(category);

  6. commit
    transaction.commit()

  7. Both session and transaction are closed at the end of the iteration.
    finally
    {
      transaction.close();
      transaction.getSession().close();
    }



Note 1:
AutoReleaseLocks is enabled.

Note 2:
This test case is highly inspired by the "LockingSequenceTest" from CDO's test suite. The only noticeable differences are that
[*] my test does create a new session (+transaction obviously) at each iteration
[*] instead of modifying a simple EAttribute, I add item into an EReference

The problem we encounter is that after a -short- moment, no thread seem to be able to acquire the lock : they all seem to wait for the lock forever! It does not happen with the modification of the EAttribute, only when adding into an EReference!!!


Note 3: .lock() method does not respect the timeout (?)
After some time analyzing, I found out that the problem shows up when one thread gets stuck in the call of .lock(long timeout) : eventhough I set the timeout parameter to -say- 1000L, the method never returns...
More precisely, the lock() method ends up calling CDOViewImpl.lockObjects(...) wich sends a "lock" message to the server. The result of the message contains a timestamp. lockObjects then waitForUpdate(theTimestamp), with no timeout defined.

I guess it is not a normal behaviour that the awaited update never comes in (maybe something went wrong on the server side and/or the timestamp is wrong?). And I think that it is the real problem.
But the fact is that lock(N) should actually timeout after N milliseconds anyway. I guess using waitForUpdate(long timestamp, long timeout) would help... of course it's not that simple because it would mean that either:
[*] lock returns "successfully" but within an out-of-date transaction
[*] lock throws a TimeoutException but having actually locked the write.

In brief, I guess we'd have to replace the
 waitForUpdate(requiredTimestamp);
in CDOViewImpl. lockObjects(...) with something like
if(!waitForUpdate(requiredTimestamp, timeout) {
  // unlock
  //throw a timeoutexception ?
}


Note 4:
In the unit test I am attaching with this message, I have added a mechanism to detect hanging .lock() calls : I use a custom Thread that I start just before the call to .lock() and that I stop right after .lock() as returned. In the meantime, the thread measures time and logs when the theoritical max duration has been exceeded:
TimerThread timerThread = new TimerThread(LOCK_TIMEOUT);
try
{
  timerThread.start();
  lock.lock(LOCK_TIMEOUT);
  break;
}
catch (TimeoutException ex)
{
  // ...
}
finally 
{
  timerThread.done()
}

The corresponding log makes it obvious that at some point one call blocks eternally and then all the other threads exhaust their lock retrials until they die...

Note 5:
The problem does not appear with a small number of threads/additions, depending on the machine. On a slow comuter, I could reproduce it with 5 Threads and 2 additions per threads. On a faster one, it needs 10 threads...

Conclusion

In the end, I don't know why the waitForUpdate never returns in this very special case. Am I doing something wrong?

Congratulations for being brave enough to read my whole message and thank you very much by advance for your help!

--
Alexandre
Re: [CDO] lock(long timeout) not timeout-ing... [message #1695123 is a reply to message #1694764] Tue, 12 May 2015 07:47 Go to previous messageGo to next message
Alexandre Borgoltz is currently offline Alexandre BorgoltzFriend
Messages: 31
Registered: July 2009
Location: France
Member
1400+ reads and no answer... I must face the reality : my question is either unclear or too long...? Or have I gone insane, finding problems that do not exist? Wink

Is there anything I should elaborate on (or summarize!!) to help readers help me Wink ?

Did anybody run the unit test? Can they confirm it fails?

Any hint what I should look into?

Thank you in advance!

--
Alexandre
Re: [CDO] lock(long timeout) not timeout-ing... [message #1695134 is a reply to message #1695123] Tue, 12 May 2015 08:39 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
Am 12.05.2015 um 09:47 schrieb Alexandre Borgoltz:
> 1400+ reads and no answer... I must face the reality : my question is either unclear or too long...? Or have I gone
> insane, finding problems that do not exist? ;)
>
> Is there anything I should elaborate on (or summarize!!) to help readers help me ;) ?
>
> Did anybody run the unit test?
I've left your post with an unread marker so that I don't forget to investigate your problem when I'm done with my
current task ;-)

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper


> Can they confirm it fails?
>
> Any hint what I should look into?
>
> Thank you in advance!
>


Re: [CDO] lock(long timeout) not timeout-ing... [message #1695144 is a reply to message #1695134] Tue, 12 May 2015 09:37 Go to previous messageGo to next message
Alexandre Borgoltz is currently offline Alexandre BorgoltzFriend
Messages: 31
Registered: July 2009
Location: France
Member
> I've left your post with an unread marker so that
> I don't forget to investigate your problem
> when I'm done with my current task Wink
Super great! Thank you Eike.
Let me know if I can do anything to help!

--
Alexandre
Re: [CDO] lock(long timeout) not timeout-ing... [message #1701976 is a reply to message #1695144] Fri, 17 July 2015 09:09 Go to previous message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6682
Registered: July 2009
Senior Member
Am 12.05.2015 um 11:37 schrieb Alexandre Borgoltz:
>> I've left your post with an unread marker so that I don't forget to investigate your problem when I'm done with my
>> current task ;)
> Super great! Thank you Eike.
> Let me know if I can do anything to help!
As expected this was really awful to debug. But after three days I could identify the problems and fix them:

472924: Problems with explicit locking and high-frequency session open/close
https://bugs.eclipse.org/bugs/show_bug.cgi?id=472924

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper


Previous Topic:[CDO] Multi-threaded access and CommitConflicts
Next Topic:[CDO] Logging Commits with a Repository handler
Goto Forum:
  


Current Time: Sat Apr 20 04:24:11 GMT 2024

Powered by FUDForum. Page generated in 0.03390 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top