Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » EMF » Concurrent access on EMF model
Concurrent access on EMF model [message #1464731] Fri, 07 November 2014 20:48 Go to next message
Alan DW is currently offline Alan DWFriend
Messages: 119
Registered: March 2012
Senior Member
Hello everyone,

we are building a web application that uses Ecore for its data model. All clients operate on the same data model instance that is held in-memory on the server.

Since we have multiple clients, we also have parallel access to the model. In our case, it will be 90% read access - so reads need to be really fast. We have our own locking mechanisms for write operations. However, if one client acquires the write lock on an element, we do not want to prevent all other clients from reading it (our domain does not permit this) - i.e. we have to permit "dirty reads".

The problem is as follows: imagine a multiplicity-many EReference somewhere in the model. A client "A" is currently reading (i.e. iterating over) it. A second client "B" acquires the write lock and changes the value of the EReference. This will inevitably throw a ConcurrentModificationException on client "A".

Long story short: what's the simplest way of preventing that ConcurrentModificationException? In a normal Java model, I would replace the collections with "Copy-On-Write" collections, but Ecore does not allow that. Is it really necessary to switch to a full-fledged repository like CDO just to prevent this ConcurrentModificationException?


I'd be very grateful for any advice.


Alan
Re: Concurrent access on EMF model [message #1465226 is a reply to message #1464731] Sat, 08 November 2014 06:31 Go to previous messageGo to next message
Ed Merks is currently offline Ed MerksFriend
Messages: 28968
Registered: July 2009
Senior Member
Alan,

Comments below.

On 07/11/2014 9:48 PM, Alan DW wrote:
> Hello everyone,
>
> we are building a web application that uses Ecore for its data model.
> All clients operate on the same data model instance that is held
> in-memory on the server.
>
> Since we have multiple clients, we also have parallel access to the
> model. In our case, it will be 90% read access - so reads need to be
> really fast. We have our own locking mechanisms for write operations.
> However, if one client acquires the write lock on an element, we do
> not want to prevent all other clients from reading it (our domain does
> not permit this) - i.e. we have to permit "dirty reads".
Well, dirty reads is one thing, but without any kind of atomicity in the
writes relative to the reads, this still seems problematic. E.g., any
update to any containment list or bidirectional reference involves
toggling the state of several objects so even if dirty reads are okay,
reads that see a partially modified state will always be problematic...
>
> The problem is as follows: imagine a multiplicity-many EReference
> somewhere in the model. A client "A" is currently reading (i.e.
> iterating over) it. A second client "B" acquires the write lock and
> changes the value of the EReference. This will inevitably throw a
> ConcurrentModificationException on client "A".
Suppose the client is expecting there to be a next element, but that one
is removed, you'd get an index out of bounds exception...
>
> Long story short: what's the simplest way of preventing that
> ConcurrentModificationException?
I think that's just one symptom of many you might see. And it's one
that generally helps you realize there are likely to be many other
problems that crop up...
> In a normal Java model, I would replace the collections with
> "Copy-On-Write" collections,
But even this, without any kind of synchronization will be problematic.
And even single valued containment/bidirectional references will be
problematic...
> but Ecore does not allow that.
As long as you return the right list implementation, that's okay (though
not entirely expected). You can even use
org.eclipse.emf.ecore.util.EcoreEList.UnmodifiableEList.UnmodifiableEList(InternalEObject,
EStructuralFeature, int, Object[]) to prevent the client from modifying
the list.
> Is it really necessary to switch to a full-fledged repository like CDO
> just to prevent this ConcurrentModificationException?
>
>
> I'd be very grateful for any advice.
Depending on whether the writers are doing very restricted/limited kinds
of updates, I don't think you can assuming that other than CMEs, the
readers will generally work fine in the face of multi-threaded updates.
>
>
> Alan
Re: Concurrent access on EMF model [message #1465296 is a reply to message #1465226] Sat, 08 November 2014 07:53 Go to previous messageGo to next message
Ed Willink is currently offline Ed WillinkFriend
Messages: 5468
Registered: July 2009
Senior Member
Hi

Ecore provides many powerful capabilities and if you need them all you
will need to add another. You might try to ensure that the collection
implementation used in all get'List' methods is a 'ConcurrentEList'.

However not all of Ecore's capabilities are needed by all applications
so perhaps something simpler will suit you better.

Ignoring Ecore's limitations, how might an ideal solution work? Well
obviously your primitive storage elements will be thread safe supporting
concurrent dirty reads. Just not what Ecore does.

There are a variety of MtoT tools within (and outside) Eclipse, so you
might be much better off writing a transformation from your Ecore
metamodel to your own code structures that exhibit all the
characteristics that you need and none that you do not. When what you
need is relatively simple, auto-generating the code is also relatively
simple, certainly much easier than persuading Ecore to do what it was
not intended to do. (You can use the normal Ecore generation as a
single-threaded test harness oracle to validate your custom code
generation.)

Regards

Ed Willink


On 08/11/2014 06:31, Ed Merks wrote:
> Alan,
>>> I'd be very grateful for any advice.
> Depending on whether the writers are doing very restricted/limited kinds
> of updates, I don't think you can assuming that other than CMEs, the
> readers will generally work fine in the face of multi-threaded updates.
Re: Concurrent access on EMF model [message #1466619 is a reply to message #1465296] Sun, 09 November 2014 10:42 Go to previous messageGo to next message
Alan DW is currently offline Alan DWFriend
Messages: 119
Registered: March 2012
Senior Member
Hi,

first of all: thank you for your responses.

@Ed Merks: I can clearly see the issue with dirty reads where an object reads an inconsistent state. This is precisely what is causing my nightmares at the moment. I'm trying hard to think of cases where this can actually happen. Our current solution is to wrap each and every EObject such that its methods return a Copy-On-Write wrapper around the actual EList. That wrapper is thread-safe in all of its operations, it automatically wraps all contained EObjects it returns in its methods (such that they in turn can again return wrapped ELists), and furthermore allows clients to acquire a lock on the entire list, if absolutely required. That being said, the individual elements of our model are actually quite simple. We do not have (or need) any opposite references, and containment references are not as problematic either, because essentially we have one big root container object and a lot of objects that are contained - but that's it. We might have additions and removals from that container, but we never have things like moving one EObject to a different parent. Generally speaking, our "object graph" is rather simple: a container with containment references to its elements, and the elements have normal (non-opposite) references to each other.

@Ed Willink: You do have a point here. We could certainly take Ecore out of our data model implementation and roll our own. From a certain perspective, this would make total sense and we were discussing it internally as well. However, there is one thing about Ecore that we really need and cannot do without: dynamic ecore. We need the capabilities to create EClasses at runtime and instantiate them on the fly without having to do any code generation or Java compilation in between. And that's something that Java on its own just is not good at doing. You could do some nasty bytecode manipulation stuff, but we really do not want to go that way. If we were to switch from Ecore to our own data model implementation, we would basically have to re-implement the entire reflective layer (EClass, EStructuralFeature, eGet(), eSet() and so on).


Thank you both for your input, it certainly gave me more food for thought.
Re: Concurrent access on EMF model [message #1467705 is a reply to message #1466619] Mon, 10 November 2014 08:30 Go to previous messageGo to next message
Ed Willink is currently offline Ed WillinkFriend
Messages: 5468
Registered: July 2009
Senior Member
Hi

On 09/11/2014 10:42, Alan DW wrote:
> We need the
> capabilities to create EClasses at runtime and instantiate them on the
> fly without having to do any code generation or Java compilation in
> between.

Certainly more challenging, but the equivalent of DynamicEObject is not
that hard.

Even more challenging to make Ecore thread safe. Until recently the
Ecore philosophy was that metamodel changes at run-time were illegal, so
any multi-threaded creation of EClasses requires more detailed
thread-safe consideration of how superclass relationships are being
maintained and of the stability of feature ids. Ecore does now permit
some metamodel evolution but I rather doubt that those changes were made
with thread safety in mind.

Regards

Ed Willink
Re: Concurrent access on EMF model [message #1467729 is a reply to message #1467705] Mon, 10 November 2014 08:58 Go to previous messageGo to next message
Ed Merks is currently offline Ed MerksFriend
Messages: 28968
Registered: July 2009
Senior Member
Ed,

Comments below.

On 10/11/2014 9:30 AM, Ed Willink wrote:
> Hi
>
> On 09/11/2014 10:42, Alan DW wrote:
>> We need the
>> capabilities to create EClasses at runtime and instantiate them on the
>> fly without having to do any code generation or Java compilation in
>> between.
>
> Certainly more challenging, but the equivalent of DynamicEObject is
> not that hard.
>
> Even more challenging to make Ecore thread safe.
That isn't Alan's goal. His goal is to make instances thread safe,
although the exactly meaning of that isn't entirely clear. That's a
major problem with thread safety discussions, particularly in the face
of data structure modifications. It's typically the case that thread
safe primitive data structures don't result in thread safe course
grained code, e.g., the following isn't thread safe even if the list is:

int index = list.indexOf(x);
if (index != -1)
{
list.remove(index):
}

> Until recently the Ecore philosophy was that metamodel changes at
> run-time were illegal, so any multi-threaded creation of EClasses
> requires more detailed thread-safe consideration of how superclass
> relationships are being maintained and of the stability of feature ids.
That's still the general limitation. It's not so different from Java's
hot-replace where you can't add/remove fields/methods from the
already-created instances.
> Ecore does now permit some metamodel evolution but I rather doubt that
> those changes were made with thread safety in mind.
>
> Regards
>
> Ed Willink
>
Re: Concurrent access on EMF model [message #1468499 is a reply to message #1467729] Mon, 10 November 2014 22:27 Go to previous messageGo to next message
Alan DW is currently offline Alan DWFriend
Messages: 119
Registered: March 2012
Senior Member
Hello again,


@Ed Willink: We had a discussion in our team today about implementing our own data model. Right now, it would be simply too much effort, mainly due to the tight integration of Ecore structures in our UI code. But we will keep that option in mind for the future.

@Ed Merks: You nailed the problem with your example code snippet. "Check then act" is the bane of thread safety. We do have a classical 3-layer-model in our application architecture: UI, business and persistence. In the business layer, we hold the shared data model. Now, the thing is: my feeling tells me that access synchronization to the data model really is no job for the UI layer, wouldn't you agree? At least I would not expect to find "synchronized" blocks or try-finally blocks with java.util.concurrent.Lock in clean UI code. But then again, where else? Any level lower than the UI and you wind up with code snippets such as yours, which you just *can't* synchronize properly without knowing the entire code block that requires access, and that would clearly be in the UI. I can see that in a "normal" scenario, one would employ some kind of data store or repository to lazily fetch the elements the UI currently needs (providing each user thread with its own copy) and let that data store handle the heavy lifting about synchronization. However, our application demands that we execute queries that work globally on the entire model, and with 300.000+ model elements, you just can't instantiate it 50 times for 50 users on a central server, that's just not feasible in terms of RAM. On some occasions, we even need ALL elements, which pretty much defeats the purpose of lazy loading. Do you have any ideas that come to your mind?
On a personal note: I'm a young software developer and in the business for about 2 years now. I have never ever seen any library as awesome as EMF and Ecore. As far as I know, you have been one of the masterminds behind it - I just would like to say a big, heartfelt "thanks" Smile
Re: Concurrent access on EMF model [message #1469004 is a reply to message #1468499] Tue, 11 November 2014 08:05 Go to previous messageGo to next message
Ed Merks is currently offline Ed MerksFriend
Messages: 28968
Registered: July 2009
Senior Member
Alan,

Comments below

On 10/11/2014 11:27 PM, Alan DW wrote:
> Hello again,
>
>
> @Ed Willink: We had a discussion in our team today about implementing
> our own data model. Right now, it would be simply too much effort,
> mainly due to the tight integration of Ecore structures in our UI
> code. But we will keep that option in mind for the future.
>
> @Ed Merks: You nailed the problem with your example code snippet.
> "Check then act" is the bane of thread safety.
Yes, it's my standard example back from my corporate days where there
were armies of misguided people who firmly believed that sprinkling the
EMF framework with some magical synchronization dust would solve their
high level threading problems.
> We do have a classical 3-layer-model in our application architecture:
> UI, business and persistence. In the business layer, we hold the
> shared data model. Now, the thing is: my feeling tells me that access
> synchronization to the data model really is no job for the UI layer,
> wouldn't you agree?
The EMF transaction framework tries to provide a kind of shared
read/exclusive write model. The Xtext framework takes a different but
somewhat analogous approach. Certainly there needs to be some type of
high-level policy for how threads should interact...
> At least I would not expect to find "synchronized" blocks or
> try-finally blocks with java.util.concurrent.Lock in clean UI code.
Well, hopefully not logs of hand-written such things.
> But then again, where else?
Given that overall thread safety cannot be achieved simply by using
thread safe primitives, on generally does need a thread-based policy
with course grained locking...
> Any level lower than the UI and you wind up with code snippets such as
> yours, which you just *can't* synchronize properly without knowing the
> entire code block that requires access, and that would clearly be in
> the UI. I can see that in a "normal" scenario, one would employ some
> kind of data store or repository to lazily fetch the elements the UI
> currently needs (providing each user thread with its own copy) and let
> that data store handle the heavy lifting about synchronization.
> However, our application demands that we execute queries that work
> globally on the entire model, and with 300.000+ model elements, you
> just can't instantiate it 50 times for 50 users on a central server,
> that's just not feasible in terms of RAM.
No.
> On some occasions, we even need ALL elements, which pretty much
> defeats the purpose of lazy loading.
Yes, life is a tradeoff.
> Do you have any ideas that come to your mind?
The only type of approach that seems generally consistent in this sort
of case is a multiple-readers exclusive writers model. To use such an
approach with a general viewer, I could imagine delegating content/label
providers that do the high level read-locking and that via the command
stack one does the high-level write locking...
> On a personal note: I'm a young software developer and in the business
> for about 2 years now. I have never ever seen any library as awesome
> as EMF and Ecore.
Flattery gets you far in life. :-P
> As far as I know, you have been one of the masterminds behind it - I
> just would like to say a big, heartfelt "thanks" :)
Re: Concurrent access on EMF model [message #1470333 is a reply to message #1469004] Wed, 12 November 2014 08:56 Go to previous messageGo to next message
Felix Dorner is currently offline Felix DornerFriend
Messages: 295
Registered: March 2012
Senior Member
On 11/11/2014 09:05, Ed Merks wrote
> The EMF transaction framework tries to provide a kind of shared
> read/exclusive write model.

I don't think it's shared read. Transactional access is entirely serial,
be it read or write. There may only be one reader or writer at a given
time, so there's 0 concurrency.

Felix
Re: Concurrent access on EMF model [message #1470351 is a reply to message #1470333] Wed, 12 November 2014 09:10 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 6325
Registered: July 2009
Senior Member
Hi,

You might want to have a look at my article "Concurrent Access to Models":

http://thegordian.blogspot.de/2011/07/concurrent-access-to-models.html

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper


Am 12.11.2014 um 09:56 schrieb Felix Dorner:
> On 11/11/2014 09:05, Ed Merks wrote
>> The EMF transaction framework tries to provide a kind of shared
>> read/exclusive write model.
>
> I don't think it's shared read. Transactional access is entirely serial, be it read or write. There may only be one
> reader or writer at a given time, so there's 0 concurrency.
>
> Felix
Re: Concurrent access on EMF model [message #1470399 is a reply to message #1470351] Wed, 12 November 2014 09:55 Go to previous messageGo to next message
Felix Dorner is currently offline Felix DornerFriend
Messages: 295
Registered: March 2012
Senior Member
On 12/11/2014 10:10, Eike Stepper wrote:
> Hi,
>
> You might want to have a look at my article "Concurrent Access to Models":
>
> http://thegordian.blogspot.de/2011/07/concurrent-access-to-models.html

Nice! This point here, is such a tough one:

* It is intrusive because each single access to the model must be wrapped.

I don't know how often I sit here reading code that doesn't wrap reads
correctly :_(.

Felix
Re: Concurrent access on EMF model [message #1470603 is a reply to message #1470333] Wed, 12 November 2014 13:24 Go to previous message
Christian W. Damus is currently offline Christian W. DamusFriend
Messages: 1161
Registered: July 2009
Location: Canada
Senior Member

Hi, Felix,

Yes, read access is exclusive because reading EMF objects can produce
concrete side-effects (proxy resolution, resource loading, etc.)
However, readers can cooperatively yield access to other readers that
may be waiting, if they know that they are doing a lot of work in a
read-only transaction. It's not automatic/preemptive, a throw-back to
the 1980s, but it's there.

Christian


On 2014-11-12 08:56:45 +0000, Felix Dorner said:

> On 11/11/2014 09:05, Ed Merks wrote
>> The EMF transaction framework tries to provide a kind of shared
>> read/exclusive write model.
>
> I don't think it's shared read. Transactional access is entirely
> serial, be it read or write. There may only be one reader or writer at
> a given time, so there's 0 concurrency.
>
> Felix
Previous Topic:[CDO] ClassCastException in CDODeltaNotification for Enum-based attribute
Next Topic:EMF Edit ItemProvider getText method always returns a default string?
Goto Forum:
  


Current Time: Mon Nov 20 17:33:36 GMT 2017

Powered by FUDForum. Page generated in 0.02365 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software