Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » EMF » [CDO] non ordered list
[CDO] non ordered list [message #1000708] Tue, 15 January 2013 09:55 Go to next message
Christophe MOINE is currently offline Christophe MOINEFriend
Messages: 34
Registered: February 2012
Member
Hello everybody,

I use 0..* references with containment property set to set, and tried to set the option "ordered" to false but it doesn't seem to be handled, is that correct ?

I use references between two big tables (let's say A and B) for benchmarking purposes: 1.000.000 that refers to 50.000.000 rows Smile

unfortunatly I now reach timeouts Smile I'm afraid that the table that links both (A_B_list table) is slowing down everything. I really don't want to bother with the order between the two tables: is it necessarily required by CDO ? Or is there any mean to skip this ?

thank you a lot in advance,

Still on the way to make CDO as efficient as raw SQL Wink

PS: I developped a raw DB inserted to allocate hudge amount of cdo_objects (related to http://www.eclipse.org/forums/index.php/t/441261/), it seems to stabilizes over time, I don't if any of you are intesrested, please let me know Smile I gonna post my 'patch' right now Wink)

Thank you in advance,

regards,

Cheers.

/Christophe.
Re: [CDO] non ordered list [message #1000738 is a reply to message #1000708] Tue, 15 January 2013 11:00 Go to previous messageGo to next message
Eike Stepper is currently offline Eike StepperFriend
Messages: 5586
Registered: July 2009
Senior Member
Am 15.01.2013 10:55, schrieb Christophe MOINE:
> Hello everybody,
>
> I use 0..* references with containment property set to set, and tried to set the option "ordered" to false but it
> doesn't seem to be handled, is that correct ?
Mostly correct. There are some optimizations in the DBStore (not in the core). Some bugs are related:

272475: [DB] Increase DBStore performance for isMany() attributes [UMBRELLA]
https://bugs.eclipse.org/bugs/show_bug.cgi?id=272475

390283: [DB] Incorrect handling of operations on unordered ELists [FIXED]
https://bugs.eclipse.org/bugs/show_bug.cgi?id=390283

272478: Better support for attributes having isMany() == true and isOrdered() == false [FIXED]
https://bugs.eclipse.org/bugs/show_bug.cgi?id=272478

The major problem is many mechansims in CDO depend heavily on the element index (e.g. removal messages). We'd certainly
appreciate enhancements in this entire area but I fear it's not that simple. A matter of effort and funding in the end.
Or good contributions ;-)

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper


>
> I use references between two big tables (let's say A and B) for benchmarking purposes: 1.000.000 that refers to
> 50.000.000 rows :)
>
> unfortunatly I now reach timeouts :) I'm afraid that the table that links both (A_B_list table) is slowing down
> everything. I really don't want to bother with the order between the two tables: is it necessarily required by CDO ?
> Or is there any mean to skip this ?
>
> thank you a lot in advance,
>
> Still on the way to make CDO as efficient as raw SQL ;)
>
> PS: I developped a raw DB inserted to allocate hudge amount of cdo_objects (related to
> http://www.eclipse.org/forums/index.php/t/441261/), it seems to stabilizes over time, I don't if any of you are
> intesrested, please let me know :) I gonna post my 'patch' right now ;))
>
> Thank you in advance,
>
> regards,
>
> Cheers.
>
> /Christophe.
Re: [CDO] non ordered list [message #1000862 is a reply to message #1000738] Tue, 15 January 2013 15:05 Go to previous messageGo to next message
Christophe MOINE is currently offline Christophe MOINEFriend
Messages: 34
Registered: February 2012
Member
Thank you Eike.

My first feeling on this is to embed information in table B in a column of table A: it is not very beautiful from a DB point of view, I don't know if I can afford to do that in production yet... :/

I'll read the other issues to see if there are other solutions,

Regards,

/Christophe.
Re: [CDO] non ordered list [message #1064737 is a reply to message #1000738] Thu, 20 June 2013 17:46 Go to previous messageGo to next message
Alain Picard is currently offline Alain PicardFriend
Messages: 181
Registered: July 2009
Senior Member
Eike,

We have important multi-threading issues when multiple threads are
trying to add or modify a list feature. But since those lists are all
unordered, operations shouldn't care about the index and should simply
ask the back-end server and db to append to the list.

I've noticed that starting right in EMF in DelegatingNotifyingListImpl
that an add(Collection) is changed under the cover to an add(size(),
collection).

Next CDO seems to also assume that all additions will be supplying the
index, like CDOAddFeatureDeltaImpl, etc.

Also because of all those index expectations, it seems that a conflict
resolver isn't of much help and please correct me if I'm wrong.

Has any progress been made to correctly support list append operation in
CDO (and consequently in EMF)?

Thanks
Alain



On 1/15/2013 6:00 AM, Eike Stepper wrote:
> Am 15.01.2013 10:55, schrieb Christophe MOINE:
>> Hello everybody,
>>
>> I use 0..* references with containment property set to set, and tried
>> to set the option "ordered" to false but it doesn't seem to be
>> handled, is that correct ?
> Mostly correct. There are some optimizations in the DBStore (not in the
> core). Some bugs are related:
>
> 272475: [DB] Increase DBStore performance for isMany() attributes
> [UMBRELLA]
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=272475
>
> 390283: [DB] Incorrect handling of operations on unordered ELists [FIXED]
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=390283
>
> 272478: Better support for attributes having isMany() == true and
> isOrdered() == false [FIXED]
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=272478
>
> The major problem is many mechansims in CDO depend heavily on the
> element index (e.g. removal messages). We'd certainly appreciate
> enhancements in this entire area but I fear it's not that simple. A
> matter of effort and funding in the end. Or good contributions ;-)
>
> Cheers
> /Eike
>
> ----
> http://www.esc-net.de
> http://thegordian.blogspot.com
> http://twitter.com/eikestepper
>
>
>>
>> I use references between two big tables (let's say A and B) for
>> benchmarking purposes: 1.000.000 that refers to 50.000.000 rows :)
>>
>> unfortunatly I now reach timeouts :) I'm afraid that the table that
>> links both (A_B_list table) is slowing down everything. I really don't
>> want to bother with the order between the two tables: is it
>> necessarily required by CDO ? Or is there any mean to skip this ?
>>
>> thank you a lot in advance,
>>
>> Still on the way to make CDO as efficient as raw SQL ;)
>>
>> PS: I developped a raw DB inserted to allocate hudge amount of
>> cdo_objects (related to
>> http://www.eclipse.org/forums/index.php/t/441261/), it seems to
>> stabilizes over time, I don't if any of you are intesrested, please
>> let me know :) I gonna post my 'patch' right now ;))
>>
>> Thank you in advance,
>>
>> regards,
>>
>> Cheers.
>>
>> /Christophe.
>
Re: [CDO] non ordered list [message #1064754 is a reply to message #1064737] Thu, 20 June 2013 20:06 Go to previous messageGo to next message
Ed Merks is currently offline Ed MerksFriend
Messages: 26278
Registered: July 2009
Senior Member
Alain,

At the EMF level no, there are no efforts to do such things. Lists
aren't thread safe so adds must be sequentialized somehow and lists are
inherently ordered. All reflective access to models assume multi-valued
references are lists...


On 20/06/2013 7:46 PM, Alain Picard wrote:
> Eike,
>
> We have important multi-threading issues when multiple threads are
> trying to add or modify a list feature. But since those lists are all
> unordered, operations shouldn't care about the index and should simply
> ask the back-end server and db to append to the list.
>
> I've noticed that starting right in EMF in DelegatingNotifyingListImpl
> that an add(Collection) is changed under the cover to an add(size(),
> collection).
>
> Next CDO seems to also assume that all additions will be supplying the
> index, like CDOAddFeatureDeltaImpl, etc.
>
> Also because of all those index expectations, it seems that a conflict
> resolver isn't of much help and please correct me if I'm wrong.
>
> Has any progress been made to correctly support list append operation
> in CDO (and consequently in EMF)?
>
> Thanks
> Alain
>
>
>
> On 1/15/2013 6:00 AM, Eike Stepper wrote:
>> Am 15.01.2013 10:55, schrieb Christophe MOINE:
>>> Hello everybody,
>>>
>>> I use 0..* references with containment property set to set, and tried
>>> to set the option "ordered" to false but it doesn't seem to be
>>> handled, is that correct ?
>> Mostly correct. There are some optimizations in the DBStore (not in the
>> core). Some bugs are related:
>>
>> 272475: [DB] Increase DBStore performance for isMany() attributes
>> [UMBRELLA]
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=272475
>>
>> 390283: [DB] Incorrect handling of operations on unordered ELists
>> [FIXED]
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=390283
>>
>> 272478: Better support for attributes having isMany() == true and
>> isOrdered() == false [FIXED]
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=272478
>>
>> The major problem is many mechansims in CDO depend heavily on the
>> element index (e.g. removal messages). We'd certainly appreciate
>> enhancements in this entire area but I fear it's not that simple. A
>> matter of effort and funding in the end. Or good contributions ;-)
>>
>> Cheers
>> /Eike
>>
>> ----
>> http://www.esc-net.de
>> http://thegordian.blogspot.com
>> http://twitter.com/eikestepper
>>
>>
>>>
>>> I use references between two big tables (let's say A and B) for
>>> benchmarking purposes: 1.000.000 that refers to 50.000.000 rows :)
>>>
>>> unfortunatly I now reach timeouts :) I'm afraid that the table that
>>> links both (A_B_list table) is slowing down everything. I really don't
>>> want to bother with the order between the two tables: is it
>>> necessarily required by CDO ? Or is there any mean to skip this ?
>>>
>>> thank you a lot in advance,
>>>
>>> Still on the way to make CDO as efficient as raw SQL ;)
>>>
>>> PS: I developped a raw DB inserted to allocate hudge amount of
>>> cdo_objects (related to
>>> http://www.eclipse.org/forums/index.php/t/441261/), it seems to
>>> stabilizes over time, I don't if any of you are intesrested, please
>>> let me know :) I gonna post my 'patch' right now ;))
>>>
>>> Thank you in advance,
>>>
>>> regards,
>>>
>>> Cheers.
>>>
>>> /Christophe.
>>
>
>
Re: [CDO] non ordered list [message #1064917 is a reply to message #1064754] Fri, 21 June 2013 16:04 Go to previous messageGo to next message
Alain Picard is currently offline Alain PicardFriend
Messages: 181
Registered: July 2009
Senior Member
Ed,

Nice talking to you again.

While I agree that lists are inherently ordered and not thread safe, the
issue is a bit more targeted.

In list an add(int, object) is documented as an insert and add(object)
is documented as an append. It is the respect of those list contracts
that I'm looking for.

From a MOF multiplicity standpoint, isOrdered specifies that the order
of the elements is important, while it is not important or guaranteed
otherwise.

So whenever a multi-valued reference is used and flagged as isOrdered
false, then we could take advantage of that to improve performance.

For example imagine a M-V ref with 3 refs: A,B,C

Now I have 2 transactions with CDO that ask to add new elements, One D
and the other E. Since those are add(D) and add(E) there should be no
confusion as they should just be appended. We might end up with A,B,C,D,
E or A,B,C,E,D but that is Ok since we specified that order is not
important.

Let's take another scenario, where here transaction 1 asks to remove
item 0 (1st) and the other to add(D). Since the add gets translated to
add(3,D), if the commits of transact 1 end up ahead the list will be
left with B,C. But D is looking to be added at 3 which is past the end
of the list. This results in an artificial problem since we simply
wanted to append to the list.

So why is it that list appends are not respecting the list contract?

Cheers,
Alain



On 6/20/2013 4:06 PM, Ed Merks wrote:
> Alain,
>
> At the EMF level no, there are no efforts to do such things. Lists
> aren't thread safe so adds must be sequentialized somehow and lists are
> inherently ordered. All reflective access to models assume multi-valued
> references are lists...
>
>
> On 20/06/2013 7:46 PM, Alain Picard wrote:
>> Eike,
>>
>> We have important multi-threading issues when multiple threads are
>> trying to add or modify a list feature. But since those lists are all
>> unordered, operations shouldn't care about the index and should simply
>> ask the back-end server and db to append to the list.
>>
>> I've noticed that starting right in EMF in DelegatingNotifyingListImpl
>> that an add(Collection) is changed under the cover to an add(size(),
>> collection).
>>
>> Next CDO seems to also assume that all additions will be supplying the
>> index, like CDOAddFeatureDeltaImpl, etc.
>>
>> Also because of all those index expectations, it seems that a conflict
>> resolver isn't of much help and please correct me if I'm wrong.
>>
>> Has any progress been made to correctly support list append operation
>> in CDO (and consequently in EMF)?
>>
>> Thanks
>> Alain
>>
>>
>>
>> On 1/15/2013 6:00 AM, Eike Stepper wrote:
>>> Am 15.01.2013 10:55, schrieb Christophe MOINE:
>>>> Hello everybody,
>>>>
>>>> I use 0..* references with containment property set to set, and tried
>>>> to set the option "ordered" to false but it doesn't seem to be
>>>> handled, is that correct ?
>>> Mostly correct. There are some optimizations in the DBStore (not in the
>>> core). Some bugs are related:
>>>
>>> 272475: [DB] Increase DBStore performance for isMany() attributes
>>> [UMBRELLA]
>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=272475
>>>
>>> 390283: [DB] Incorrect handling of operations on unordered ELists
>>> [FIXED]
>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=390283
>>>
>>> 272478: Better support for attributes having isMany() == true and
>>> isOrdered() == false [FIXED]
>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=272478
>>>
>>> The major problem is many mechansims in CDO depend heavily on the
>>> element index (e.g. removal messages). We'd certainly appreciate
>>> enhancements in this entire area but I fear it's not that simple. A
>>> matter of effort and funding in the end. Or good contributions ;-)
>>>
>>> Cheers
>>> /Eike
>>>
>>> ----
>>> http://www.esc-net.de
>>> http://thegordian.blogspot.com
>>> http://twitter.com/eikestepper
>>>
>>>
>>>>
>>>> I use references between two big tables (let's say A and B) for
>>>> benchmarking purposes: 1.000.000 that refers to 50.000.000 rows :)
>>>>
>>>> unfortunatly I now reach timeouts :) I'm afraid that the table that
>>>> links both (A_B_list table) is slowing down everything. I really don't
>>>> want to bother with the order between the two tables: is it
>>>> necessarily required by CDO ? Or is there any mean to skip this ?
>>>>
>>>> thank you a lot in advance,
>>>>
>>>> Still on the way to make CDO as efficient as raw SQL ;)
>>>>
>>>> PS: I developped a raw DB inserted to allocate hudge amount of
>>>> cdo_objects (related to
>>>> http://www.eclipse.org/forums/index.php/t/441261/), it seems to
>>>> stabilizes over time, I don't if any of you are intesrested, please
>>>> let me know :) I gonna post my 'patch' right now ;))
>>>>
>>>> Thank you in advance,
>>>>
>>>> regards,
>>>>
>>>> Cheers.
>>>>
>>>> /Christophe.
>>>
>>
>>
>
Re: [CDO] non ordered list [message #1064923 is a reply to message #1064917] Fri, 21 June 2013 16:28 Go to previous message
Ed Merks is currently offline Ed MerksFriend
Messages: 26278
Registered: July 2009
Senior Member
Alain,

Comments below.

On 21/06/2013 6:04 PM, Alain Picard wrote:
> Ed,
>
> Nice talking to you again.
>
> While I agree that lists are inherently ordered and not thread safe,
> the issue is a bit more targeted.
>
> In list an add(int, object) is documented as an insert and add(object)
> is documented as an append. It is the respect of those list contracts
> that I'm looking for.
>
> From a MOF multiplicity standpoint, isOrdered specifies that the order
> of the elements is important, while it is not important or guaranteed
> otherwise.
>
> So whenever a multi-valued reference is used and flagged as isOrdered
> false, then we could take advantage of that to improve performance.
Yes, there is that possibility. And in any case, if there are multiple
threads adding things, and somehow those adds are properly
sequentialized, the final order would also be arbitrary.
>
> For example imagine a M-V ref with 3 refs: A,B,C
>
> Now I have 2 transactions with CDO that ask to add new elements, One D
> and the other E. Since those are add(D) and add(E) there should be no
> confusion as they should just be appended. We might end up with
> A,B,C,D, E or A,B,C,E,D but that is Ok since we specified that order
> is not important.
Yes, I see.
>
> Let's take another scenario, where here transaction 1 asks to remove
> item 0 (1st) and the other to add(D). Since the add gets translated to
> add(3,D), if the commits of transact 1 end up ahead the list will be
> left with B,C. But D is looking to be added at 3 which is past the end
> of the list. This results in an artificial problem since we simply
> wanted to append to the list.
>
> So why is it that list appends are not respecting the list contract?
So this is where the whole issue of threading kicks in. It's just not
"fair" to say the list contract isn't being respected because that
contract says nothing about multi-threading. It only says when you add
something, it ends up at the end of the list. CDO is of course free to
ignore indices for an unordered list, but keeping the entire framework
consistent when clients expect notifications that include indices isn't
a trivial undertaking; pretty much an impossible one I would say...

It's best I let Eike answer for what's possible with CDO and what's
needed in EMF to enable that...
>
> Cheers,
> Alain
>
>
>
> On 6/20/2013 4:06 PM, Ed Merks wrote:
>> Alain,
>>
>> At the EMF level no, there are no efforts to do such things. Lists
>> aren't thread safe so adds must be sequentialized somehow and lists are
>> inherently ordered. All reflective access to models assume multi-valued
>> references are lists...
>>
>>
>> On 20/06/2013 7:46 PM, Alain Picard wrote:
>>> Eike,
>>>
>>> We have important multi-threading issues when multiple threads are
>>> trying to add or modify a list feature. But since those lists are all
>>> unordered, operations shouldn't care about the index and should simply
>>> ask the back-end server and db to append to the list.
>>>
>>> I've noticed that starting right in EMF in DelegatingNotifyingListImpl
>>> that an add(Collection) is changed under the cover to an add(size(),
>>> collection).
>>>
>>> Next CDO seems to also assume that all additions will be supplying the
>>> index, like CDOAddFeatureDeltaImpl, etc.
>>>
>>> Also because of all those index expectations, it seems that a conflict
>>> resolver isn't of much help and please correct me if I'm wrong.
>>>
>>> Has any progress been made to correctly support list append operation
>>> in CDO (and consequently in EMF)?
>>>
>>> Thanks
>>> Alain
>>>
>>>
>>>
>>> On 1/15/2013 6:00 AM, Eike Stepper wrote:
>>>> Am 15.01.2013 10:55, schrieb Christophe MOINE:
>>>>> Hello everybody,
>>>>>
>>>>> I use 0..* references with containment property set to set, and tried
>>>>> to set the option "ordered" to false but it doesn't seem to be
>>>>> handled, is that correct ?
>>>> Mostly correct. There are some optimizations in the DBStore (not in
>>>> the
>>>> core). Some bugs are related:
>>>>
>>>> 272475: [DB] Increase DBStore performance for isMany() attributes
>>>> [UMBRELLA]
>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=272475
>>>>
>>>> 390283: [DB] Incorrect handling of operations on unordered ELists
>>>> [FIXED]
>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=390283
>>>>
>>>> 272478: Better support for attributes having isMany() == true and
>>>> isOrdered() == false [FIXED]
>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=272478
>>>>
>>>> The major problem is many mechansims in CDO depend heavily on the
>>>> element index (e.g. removal messages). We'd certainly appreciate
>>>> enhancements in this entire area but I fear it's not that simple. A
>>>> matter of effort and funding in the end. Or good contributions ;-)
>>>>
>>>> Cheers
>>>> /Eike
>>>>
>>>> ----
>>>> http://www.esc-net.de
>>>> http://thegordian.blogspot.com
>>>> http://twitter.com/eikestepper
>>>>
>>>>
>>>>>
>>>>> I use references between two big tables (let's say A and B) for
>>>>> benchmarking purposes: 1.000.000 that refers to 50.000.000 rows :)
>>>>>
>>>>> unfortunatly I now reach timeouts :) I'm afraid that the table that
>>>>> links both (A_B_list table) is slowing down everything. I really
>>>>> don't
>>>>> want to bother with the order between the two tables: is it
>>>>> necessarily required by CDO ? Or is there any mean to skip this ?
>>>>>
>>>>> thank you a lot in advance,
>>>>>
>>>>> Still on the way to make CDO as efficient as raw SQL ;)
>>>>>
>>>>> PS: I developped a raw DB inserted to allocate hudge amount of
>>>>> cdo_objects (related to
>>>>> http://www.eclipse.org/forums/index.php/t/441261/), it seems to
>>>>> stabilizes over time, I don't if any of you are intesrested, please
>>>>> let me know :) I gonna post my 'patch' right now ;))
>>>>>
>>>>> Thank you in advance,
>>>>>
>>>>> regards,
>>>>>
>>>>> Cheers.
>>>>>
>>>>> /Christophe.
>>>>
>>>
>>>
>>
>
>
Previous Topic:How to correctly compare EDataType?
Next Topic:Empty tags
Goto Forum:
  


Current Time: Wed Dec 17 23:21:48 GMT 2014

Powered by FUDForum. Page generated in 0.01800 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software