Home » Modeling » EMF » [CDO] non ordered list
| [CDO] non ordered list [message #1000708] |
Tue, 15 January 2013 09:55  |
 Christophe MOINE
Messages: 34
Registered: February 2012 |
Member |
|
|
Hello everybody,
I use 0..* references with containment property set to set, and tried to set the option "ordered" to false but it doesn't seem to be handled, is that correct ?
I use references between two big tables (let's say A and B) for benchmarking purposes: 1.000.000 that refers to 50.000.000 rows 
unfortunatly I now reach timeouts I'm afraid that the table that links both (A_B_list table) is slowing down everything. I really don't want to bother with the order between the two tables: is it necessarily required by CDO ? Or is there any mean to skip this ?
thank you a lot in advance,
Still on the way to make CDO as efficient as raw SQL 
PS: I developped a raw DB inserted to allocate hudge amount of cdo_objects (related to http://www.eclipse.org/forums/index.php/t/441261/), it seems to stabilizes over time, I don't if any of you are intesrested, please let me know I gonna post my 'patch' right now )
Thank you in advance,
regards,
Cheers.
/Christophe.
|
|
|
| Re: [CDO] non ordered list [message #1000738 is a reply to message #1000708] |
Tue, 15 January 2013 11:00  |
|
Am 15.01.2013 10:55, schrieb Christophe MOINE:
> Hello everybody,
>
> I use 0..* references with containment property set to set, and tried to set the option "ordered" to false but it
> doesn't seem to be handled, is that correct ?
Mostly correct. There are some optimizations in the DBStore (not in the core). Some bugs are related:
272475: [DB] Increase DBStore performance for isMany() attributes [UMBRELLA]
https://bugs.eclipse.org/bugs/show_bug.cgi?id=272475
390283: [DB] Incorrect handling of operations on unordered ELists [FIXED]
https://bugs.eclipse.org/bugs/show_bug.cgi?id=390283
272478: Better support for attributes having isMany() == true and isOrdered() == false [FIXED]
https://bugs.eclipse.org/bugs/show_bug.cgi?id=272478
The major problem is many mechansims in CDO depend heavily on the element index (e.g. removal messages). We'd certainly
appreciate enhancements in this entire area but I fear it's not that simple. A matter of effort and funding in the end.
Or good contributions ;-)
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
>
> I use references between two big tables (let's say A and B) for benchmarking purposes: 1.000.000 that refers to
> 50.000.000 rows :)
>
> unfortunatly I now reach timeouts :) I'm afraid that the table that links both (A_B_list table) is slowing down
> everything. I really don't want to bother with the order between the two tables: is it necessarily required by CDO ?
> Or is there any mean to skip this ?
>
> thank you a lot in advance,
>
> Still on the way to make CDO as efficient as raw SQL ;)
>
> PS: I developped a raw DB inserted to allocate hudge amount of cdo_objects (related to
> http://www.eclipse.org/forums/index.php/t/441261/), it seems to stabilizes over time, I don't if any of you are
> intesrested, please let me know :) I gonna post my 'patch' right now ;))
>
> Thank you in advance,
>
> regards,
>
> Cheers.
>
> /Christophe.
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
| |
| Re: [CDO] non ordered list [message #1064737 is a reply to message #1000738] |
Thu, 20 June 2013 17:46 |
Alain Picard
Messages: 266
Registered: July 2009 |
Senior Member |
|
|
Eike,
We have important multi-threading issues when multiple threads are
trying to add or modify a list feature. But since those lists are all
unordered, operations shouldn't care about the index and should simply
ask the back-end server and db to append to the list.
I've noticed that starting right in EMF in DelegatingNotifyingListImpl
that an add(Collection) is changed under the cover to an add(size(),
collection).
Next CDO seems to also assume that all additions will be supplying the
index, like CDOAddFeatureDeltaImpl, etc.
Also because of all those index expectations, it seems that a conflict
resolver isn't of much help and please correct me if I'm wrong.
Has any progress been made to correctly support list append operation in
CDO (and consequently in EMF)?
Thanks
Alain
On 1/15/2013 6:00 AM, Eike Stepper wrote:
> Am 15.01.2013 10:55, schrieb Christophe MOINE:
>> Hello everybody,
>>
>> I use 0..* references with containment property set to set, and tried
>> to set the option "ordered" to false but it doesn't seem to be
>> handled, is that correct ?
> Mostly correct. There are some optimizations in the DBStore (not in the
> core). Some bugs are related:
>
> 272475: [DB] Increase DBStore performance for isMany() attributes
> [UMBRELLA]
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=272475
>
> 390283: [DB] Incorrect handling of operations on unordered ELists [FIXED]
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=390283
>
> 272478: Better support for attributes having isMany() == true and
> isOrdered() == false [FIXED]
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=272478
>
> The major problem is many mechansims in CDO depend heavily on the
> element index (e.g. removal messages). We'd certainly appreciate
> enhancements in this entire area but I fear it's not that simple. A
> matter of effort and funding in the end. Or good contributions ;-)
>
> Cheers
> /Eike
>
> ----
> http://www.esc-net.de
> http://thegordian.blogspot.com
> http://twitter.com/eikestepper
>
>
>>
>> I use references between two big tables (let's say A and B) for
>> benchmarking purposes: 1.000.000 that refers to 50.000.000 rows :)
>>
>> unfortunatly I now reach timeouts :) I'm afraid that the table that
>> links both (A_B_list table) is slowing down everything. I really don't
>> want to bother with the order between the two tables: is it
>> necessarily required by CDO ? Or is there any mean to skip this ?
>>
>> thank you a lot in advance,
>>
>> Still on the way to make CDO as efficient as raw SQL ;)
>>
>> PS: I developped a raw DB inserted to allocate hudge amount of
>> cdo_objects (related to
>> http://www.eclipse.org/forums/index.php/t/441261/), it seems to
>> stabilizes over time, I don't if any of you are intesrested, please
>> let me know :) I gonna post my 'patch' right now ;))
>>
>> Thank you in advance,
>>
>> regards,
>>
>> Cheers.
>>
>> /Christophe.
>
|
|
|
| Re: [CDO] non ordered list [message #1064754 is a reply to message #1064737] |
Thu, 20 June 2013 20:06 |
Ed Merks
Messages: 33137
Registered: July 2009 |
Senior Member |
|
|
Alain,
At the EMF level no, there are no efforts to do such things. Lists
aren't thread safe so adds must be sequentialized somehow and lists are
inherently ordered. All reflective access to models assume multi-valued
references are lists...
On 20/06/2013 7:46 PM, Alain Picard wrote:
> Eike,
>
> We have important multi-threading issues when multiple threads are
> trying to add or modify a list feature. But since those lists are all
> unordered, operations shouldn't care about the index and should simply
> ask the back-end server and db to append to the list.
>
> I've noticed that starting right in EMF in DelegatingNotifyingListImpl
> that an add(Collection) is changed under the cover to an add(size(),
> collection).
>
> Next CDO seems to also assume that all additions will be supplying the
> index, like CDOAddFeatureDeltaImpl, etc.
>
> Also because of all those index expectations, it seems that a conflict
> resolver isn't of much help and please correct me if I'm wrong.
>
> Has any progress been made to correctly support list append operation
> in CDO (and consequently in EMF)?
>
> Thanks
> Alain
>
>
>
> On 1/15/2013 6:00 AM, Eike Stepper wrote:
>> Am 15.01.2013 10:55, schrieb Christophe MOINE:
>>> Hello everybody,
>>>
>>> I use 0..* references with containment property set to set, and tried
>>> to set the option "ordered" to false but it doesn't seem to be
>>> handled, is that correct ?
>> Mostly correct. There are some optimizations in the DBStore (not in the
>> core). Some bugs are related:
>>
>> 272475: [DB] Increase DBStore performance for isMany() attributes
>> [UMBRELLA]
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=272475
>>
>> 390283: [DB] Incorrect handling of operations on unordered ELists
>> [FIXED]
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=390283
>>
>> 272478: Better support for attributes having isMany() == true and
>> isOrdered() == false [FIXED]
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=272478
>>
>> The major problem is many mechansims in CDO depend heavily on the
>> element index (e.g. removal messages). We'd certainly appreciate
>> enhancements in this entire area but I fear it's not that simple. A
>> matter of effort and funding in the end. Or good contributions ;-)
>>
>> Cheers
>> /Eike
>>
>> ----
>> http://www.esc-net.de
>> http://thegordian.blogspot.com
>> http://twitter.com/eikestepper
>>
>>
>>>
>>> I use references between two big tables (let's say A and B) for
>>> benchmarking purposes: 1.000.000 that refers to 50.000.000 rows :)
>>>
>>> unfortunatly I now reach timeouts :) I'm afraid that the table that
>>> links both (A_B_list table) is slowing down everything. I really don't
>>> want to bother with the order between the two tables: is it
>>> necessarily required by CDO ? Or is there any mean to skip this ?
>>>
>>> thank you a lot in advance,
>>>
>>> Still on the way to make CDO as efficient as raw SQL ;)
>>>
>>> PS: I developped a raw DB inserted to allocate hudge amount of
>>> cdo_objects (related to
>>> http://www.eclipse.org/forums/index.php/t/441261/), it seems to
>>> stabilizes over time, I don't if any of you are intesrested, please
>>> let me know :) I gonna post my 'patch' right now ;))
>>>
>>> Thank you in advance,
>>>
>>> regards,
>>>
>>> Cheers.
>>>
>>> /Christophe.
>>
>
>
Ed Merks
Professional Support: https://www.macromodeling.com/
|
|
|
| Re: [CDO] non ordered list [message #1064917 is a reply to message #1064754] |
Fri, 21 June 2013 16:04 |
Alain Picard
Messages: 266
Registered: July 2009 |
Senior Member |
|
|
Ed,
Nice talking to you again.
While I agree that lists are inherently ordered and not thread safe, the
issue is a bit more targeted.
In list an add(int, object) is documented as an insert and add(object)
is documented as an append. It is the respect of those list contracts
that I'm looking for.
From a MOF multiplicity standpoint, isOrdered specifies that the order
of the elements is important, while it is not important or guaranteed
otherwise.
So whenever a multi-valued reference is used and flagged as isOrdered
false, then we could take advantage of that to improve performance.
For example imagine a M-V ref with 3 refs: A,B,C
Now I have 2 transactions with CDO that ask to add new elements, One D
and the other E. Since those are add(D) and add(E) there should be no
confusion as they should just be appended. We might end up with A,B,C,D,
E or A,B,C,E,D but that is Ok since we specified that order is not
important.
Let's take another scenario, where here transaction 1 asks to remove
item 0 (1st) and the other to add(D). Since the add gets translated to
add(3,D), if the commits of transact 1 end up ahead the list will be
left with B,C. But D is looking to be added at 3 which is past the end
of the list. This results in an artificial problem since we simply
wanted to append to the list.
So why is it that list appends are not respecting the list contract?
Cheers,
Alain
On 6/20/2013 4:06 PM, Ed Merks wrote:
> Alain,
>
> At the EMF level no, there are no efforts to do such things. Lists
> aren't thread safe so adds must be sequentialized somehow and lists are
> inherently ordered. All reflective access to models assume multi-valued
> references are lists...
>
>
> On 20/06/2013 7:46 PM, Alain Picard wrote:
>> Eike,
>>
>> We have important multi-threading issues when multiple threads are
>> trying to add or modify a list feature. But since those lists are all
>> unordered, operations shouldn't care about the index and should simply
>> ask the back-end server and db to append to the list.
>>
>> I've noticed that starting right in EMF in DelegatingNotifyingListImpl
>> that an add(Collection) is changed under the cover to an add(size(),
>> collection).
>>
>> Next CDO seems to also assume that all additions will be supplying the
>> index, like CDOAddFeatureDeltaImpl, etc.
>>
>> Also because of all those index expectations, it seems that a conflict
>> resolver isn't of much help and please correct me if I'm wrong.
>>
>> Has any progress been made to correctly support list append operation
>> in CDO (and consequently in EMF)?
>>
>> Thanks
>> Alain
>>
>>
>>
>> On 1/15/2013 6:00 AM, Eike Stepper wrote:
>>> Am 15.01.2013 10:55, schrieb Christophe MOINE:
>>>> Hello everybody,
>>>>
>>>> I use 0..* references with containment property set to set, and tried
>>>> to set the option "ordered" to false but it doesn't seem to be
>>>> handled, is that correct ?
>>> Mostly correct. There are some optimizations in the DBStore (not in the
>>> core). Some bugs are related:
>>>
>>> 272475: [DB] Increase DBStore performance for isMany() attributes
>>> [UMBRELLA]
>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=272475
>>>
>>> 390283: [DB] Incorrect handling of operations on unordered ELists
>>> [FIXED]
>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=390283
>>>
>>> 272478: Better support for attributes having isMany() == true and
>>> isOrdered() == false [FIXED]
>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=272478
>>>
>>> The major problem is many mechansims in CDO depend heavily on the
>>> element index (e.g. removal messages). We'd certainly appreciate
>>> enhancements in this entire area but I fear it's not that simple. A
>>> matter of effort and funding in the end. Or good contributions ;-)
>>>
>>> Cheers
>>> /Eike
>>>
>>> ----
>>> http://www.esc-net.de
>>> http://thegordian.blogspot.com
>>> http://twitter.com/eikestepper
>>>
>>>
>>>>
>>>> I use references between two big tables (let's say A and B) for
>>>> benchmarking purposes: 1.000.000 that refers to 50.000.000 rows :)
>>>>
>>>> unfortunatly I now reach timeouts :) I'm afraid that the table that
>>>> links both (A_B_list table) is slowing down everything. I really don't
>>>> want to bother with the order between the two tables: is it
>>>> necessarily required by CDO ? Or is there any mean to skip this ?
>>>>
>>>> thank you a lot in advance,
>>>>
>>>> Still on the way to make CDO as efficient as raw SQL ;)
>>>>
>>>> PS: I developped a raw DB inserted to allocate hudge amount of
>>>> cdo_objects (related to
>>>> http://www.eclipse.org/forums/index.php/t/441261/), it seems to
>>>> stabilizes over time, I don't if any of you are intesrested, please
>>>> let me know :) I gonna post my 'patch' right now ;))
>>>>
>>>> Thank you in advance,
>>>>
>>>> regards,
>>>>
>>>> Cheers.
>>>>
>>>> /Christophe.
>>>
>>
>>
>
|
|
|
| Re: [CDO] non ordered list [message #1064923 is a reply to message #1064917] |
Fri, 21 June 2013 16:28 |
Ed Merks
Messages: 33137
Registered: July 2009 |
Senior Member |
|
|
Alain,
Comments below.
On 21/06/2013 6:04 PM, Alain Picard wrote:
> Ed,
>
> Nice talking to you again.
>
> While I agree that lists are inherently ordered and not thread safe,
> the issue is a bit more targeted.
>
> In list an add(int, object) is documented as an insert and add(object)
> is documented as an append. It is the respect of those list contracts
> that I'm looking for.
>
> From a MOF multiplicity standpoint, isOrdered specifies that the order
> of the elements is important, while it is not important or guaranteed
> otherwise.
>
> So whenever a multi-valued reference is used and flagged as isOrdered
> false, then we could take advantage of that to improve performance.
Yes, there is that possibility. And in any case, if there are multiple
threads adding things, and somehow those adds are properly
sequentialized, the final order would also be arbitrary.
>
> For example imagine a M-V ref with 3 refs: A,B,C
>
> Now I have 2 transactions with CDO that ask to add new elements, One D
> and the other E. Since those are add(D) and add(E) there should be no
> confusion as they should just be appended. We might end up with
> A,B,C,D, E or A,B,C,E,D but that is Ok since we specified that order
> is not important.
Yes, I see.
>
> Let's take another scenario, where here transaction 1 asks to remove
> item 0 (1st) and the other to add(D). Since the add gets translated to
> add(3,D), if the commits of transact 1 end up ahead the list will be
> left with B,C. But D is looking to be added at 3 which is past the end
> of the list. This results in an artificial problem since we simply
> wanted to append to the list.
>
> So why is it that list appends are not respecting the list contract?
So this is where the whole issue of threading kicks in. It's just not
"fair" to say the list contract isn't being respected because that
contract says nothing about multi-threading. It only says when you add
something, it ends up at the end of the list. CDO is of course free to
ignore indices for an unordered list, but keeping the entire framework
consistent when clients expect notifications that include indices isn't
a trivial undertaking; pretty much an impossible one I would say...
It's best I let Eike answer for what's possible with CDO and what's
needed in EMF to enable that...
>
> Cheers,
> Alain
>
>
>
> On 6/20/2013 4:06 PM, Ed Merks wrote:
>> Alain,
>>
>> At the EMF level no, there are no efforts to do such things. Lists
>> aren't thread safe so adds must be sequentialized somehow and lists are
>> inherently ordered. All reflective access to models assume multi-valued
>> references are lists...
>>
>>
>> On 20/06/2013 7:46 PM, Alain Picard wrote:
>>> Eike,
>>>
>>> We have important multi-threading issues when multiple threads are
>>> trying to add or modify a list feature. But since those lists are all
>>> unordered, operations shouldn't care about the index and should simply
>>> ask the back-end server and db to append to the list.
>>>
>>> I've noticed that starting right in EMF in DelegatingNotifyingListImpl
>>> that an add(Collection) is changed under the cover to an add(size(),
>>> collection).
>>>
>>> Next CDO seems to also assume that all additions will be supplying the
>>> index, like CDOAddFeatureDeltaImpl, etc.
>>>
>>> Also because of all those index expectations, it seems that a conflict
>>> resolver isn't of much help and please correct me if I'm wrong.
>>>
>>> Has any progress been made to correctly support list append operation
>>> in CDO (and consequently in EMF)?
>>>
>>> Thanks
>>> Alain
>>>
>>>
>>>
>>> On 1/15/2013 6:00 AM, Eike Stepper wrote:
>>>> Am 15.01.2013 10:55, schrieb Christophe MOINE:
>>>>> Hello everybody,
>>>>>
>>>>> I use 0..* references with containment property set to set, and tried
>>>>> to set the option "ordered" to false but it doesn't seem to be
>>>>> handled, is that correct ?
>>>> Mostly correct. There are some optimizations in the DBStore (not in
>>>> the
>>>> core). Some bugs are related:
>>>>
>>>> 272475: [DB] Increase DBStore performance for isMany() attributes
>>>> [UMBRELLA]
>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=272475
>>>>
>>>> 390283: [DB] Incorrect handling of operations on unordered ELists
>>>> [FIXED]
>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=390283
>>>>
>>>> 272478: Better support for attributes having isMany() == true and
>>>> isOrdered() == false [FIXED]
>>>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=272478
>>>>
>>>> The major problem is many mechansims in CDO depend heavily on the
>>>> element index (e.g. removal messages). We'd certainly appreciate
>>>> enhancements in this entire area but I fear it's not that simple. A
>>>> matter of effort and funding in the end. Or good contributions ;-)
>>>>
>>>> Cheers
>>>> /Eike
>>>>
>>>> ----
>>>> http://www.esc-net.de
>>>> http://thegordian.blogspot.com
>>>> http://twitter.com/eikestepper
>>>>
>>>>
>>>>>
>>>>> I use references between two big tables (let's say A and B) for
>>>>> benchmarking purposes: 1.000.000 that refers to 50.000.000 rows :)
>>>>>
>>>>> unfortunatly I now reach timeouts :) I'm afraid that the table that
>>>>> links both (A_B_list table) is slowing down everything. I really
>>>>> don't
>>>>> want to bother with the order between the two tables: is it
>>>>> necessarily required by CDO ? Or is there any mean to skip this ?
>>>>>
>>>>> thank you a lot in advance,
>>>>>
>>>>> Still on the way to make CDO as efficient as raw SQL ;)
>>>>>
>>>>> PS: I developped a raw DB inserted to allocate hudge amount of
>>>>> cdo_objects (related to
>>>>> http://www.eclipse.org/forums/index.php/t/441261/), it seems to
>>>>> stabilizes over time, I don't if any of you are intesrested, please
>>>>> let me know :) I gonna post my 'patch' right now ;))
>>>>>
>>>>> Thank you in advance,
>>>>>
>>>>> regards,
>>>>>
>>>>> Cheers.
>>>>>
>>>>> /Christophe.
>>>>
>>>
>>>
>>
>
>
Ed Merks
Professional Support: https://www.macromodeling.com/
|
|
|
Goto Forum:
Current Time: Fri Apr 19 20:38:40 GMT 2024
Powered by FUDForum. Page generated in 0.04326 seconds |
] 
Search
Help
Register
Login
Home
