Eclipse Community Forums: EMF » [CDO] using the query interface

Home » Modeling » EMF » [CDO] using the query interface

[CDO] using the query interface [message #423342]

Mon, 29 September 2008 06:20

Eclipse User

Hi,

over the weekend I thought about ways to speed up my application.
One of the features of CDO is that it can load only the parts of a model
which are needed. However at the same time every access to a non-cached
object results in a DB query. I thought, I could speed up the process of
reading a larger part of a model by pre-caching it. E.g. if there was a
way to load an CDO object from the database and all of its containments
(children) up to a certain depth (IOW the object O and off of
O.getCildren() and all of O.getChildren().each { c -> c.getChildren() }
, I could pre-load a complete tree in one operation.
Then I came in this morning and I thought, well, there once was a
proposal for a CDO query feature and I found out that the bug is closed
already.

So,

1. Is the CDO query interface usable?
2. Which query language do you support?
3. Is the native mysql DBStore supported?
4. Does one query really lead to one DB-select operation (i.e. is the
query translated to SQL and executed at DB level) or is the query
evaluated and executed at the CDO server resulting in single
DB-operations for reading revisions one by one?

Thaks in advance for your answers!

Cheers,
Stefan

Re: [CDO] using the query interface [message #423355 is a reply to message #423342]

Mon, 29 September 2008 08:38

Eclipse User

Stefan Winkler schrieb:
> Hi,
>
> over the weekend I thought about ways to speed up my application.
> One of the features of CDO is that it can load only the parts of a
> model which are needed. However at the same time every access to a
> non-cached object results in a DB query. I thought, I could speed up
> the process of reading a larger part of a model by pre-caching it.
> E.g. if there was a way to load an CDO object from the database and
> all of its containments (children) up to a certain depth (IOW the
> object O and off of O.getCildren() and all of O.getChildren().each { c
> -> c.getChildren() } , I could pre-load a complete tree in one operation.
You might also want to look at some other new features to optimize the
characteristics. Have a look at the attached document in:

245654: Provide a "Tweak Performance" tutorial
https://bugs.eclipse.org/bugs/show_bug.cgi?id=245654

Maybe you're able to develop (and contribute?) a special
CDOFeatureAnalyzer...

> Then I came in this morning and I thought, well, there once was a
> proposal for a CDO query feature and I found out that the bug is
> closed already.
>
> So,
>
> 1. Is the CDO query interface usable?
If I take the question as it's written, I'd say yes. The API is new but
carefully chosen. But you'll have to provide a query language name that
is implemented either by the backend store (i.e. IStore of CDO) or the
repository itself. And currently only a test language is supported by
the MEMStore and a simple "query resources" language by all
repositories. In other words, the query feature is an extensible
framework rather than a complete set of implemented query languages. It
provides simple service provider interfaces for developers of backend
query languages or repository-level query languages (including an
extension point), as well as a convenient client API to work with
synchronous and asynchronous queries.

> 2. Which query language do you support?
See above. We are looking for help with the implementation of additional
query languages. There are currently three obvious ones on the wish list:

244169: [Hibernate] Store to support Query (HQL)
https://bugs.eclipse.org/bugs/show_bug.cgi?id=244169

248933: [DB] Support SQL as a CDOQuery language
https://bugs.eclipse.org/bugs/show_bug.cgi?id=248933

245658: [Query] Provide OCL query language
https://bugs.eclipse.org/bugs/show_bug.cgi?id=245658

> 3. Is the native mysql DBStore supported?
See above.

> 4. Does one query really lead to one DB-select operation (i.e. is the
> query translated to SQL and executed at DB level) or is the query
> evaluated and executed at the CDO server resulting in single
> DB-operations for reading revisions one by one?
It's left completely to the backend query language implementor how a
query translates into backend primitives. Since repository-level queries
are expected to run against the server-side RevisionManager it's likely
that they will not be optimized for certain backend types. If SQL was
provided as a DBStore-level query language I don't see a problem in
backing a query by a single SQL statement.

Cheers
/Eike

>
> Thaks in advance for your answers!
>
> Cheers,
> Stefan
>
>
>
>

Re: [CDO] using the query interface [message #423361 is a reply to message #423355]

Mon, 29 September 2008 10:28

Eclipse User

Hi Eike,
> You might also want to look at some other new features to optimize the
> characteristics. Have a look at the attached document in:
>
> 245654: Provide a "Tweak Performance" tutorial
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=245654
>
> Maybe you're able to develop (and contribute?) a special
> CDOFeatureAnalyzer...
Mhhh ... thanks for the pointer. Interesting read ...
But I think there is a basic problem around here:

org.eclipse.emf.cdo.internal.server.RevisionManager:

@Override
protected List<InternalCDORevision> loadRevisions(Collection<CDOID>
ids, int referenceChunk)
{
IStoreReader storeReader = StoreThreadLocal.getStoreReader();
List<InternalCDORevision> revisions = new
ArrayList<InternalCDORevision>();
for (CDOID id : ids)
{
InternalCDORevision revision =
(InternalCDORevision)storeReader.readRevision(id, referenceChunk);
revisions.add(revision);
}

return revisions;
}

In my opinion the IStoreReader should support reading multiple revisions
at once.
My guess is that prefetching revisions one by one does not provide much
performance gain (well, at least as long as I use the jvm-Connector...).
What really should drive performance would be reducing single database
accesses and batch them together.
So, instead of
- select ... from ... where cdo_id = 214 and cdo_version = 4
- read answer
- fill in revision
- select ... from ... where cdo_id = 215 and cdo_version = 4
- read answer
- fill in revision
- select ... from ... where cdo_id = 216 and cdo_version = 4
- read answer
- fill in revision
it should be much more efficient to implement a way to
- select ... from ... where (cdo_id = 214 and cdo_version = 4) or
(cdo_id = 215 and cdo_version = 4) or (cdo_id = 216 and cdo_version = 4)
- read answer
- fill in revisions

However, I see several challenges here :-(
- the current IDBStore implementation might be hard to enhance because
of the single-sql-string-builder
- the above works only for objects of the same type - what if we use
polymorphism in the list?
- ...

Anyway, do you want an enhancement request filed? "Support fetching of
multiple revisions at once in IStoreReader"?

Cheers,
Stefan

Re: [CDO] using the query interface [message #423363 is a reply to message #423361]

Mon, 29 September 2008 10:41

Eclipse User

Stefan Winkler schrieb:
> Hi Eike,
>> You might also want to look at some other new features to optimize
>> the characteristics. Have a look at the attached document in:
>>
>> 245654: Provide a "Tweak Performance" tutorial
>> https://bugs.eclipse.org/bugs/show_bug.cgi?id=245654
>>
>> Maybe you're able to develop (and contribute?) a special
>> CDOFeatureAnalyzer...
> Mhhh ... thanks for the pointer. Interesting read ...
> But I think there is a basic problem around here:
>
> org.eclipse.emf.cdo.internal.server.RevisionManager:
>
> @Override
> protected List<InternalCDORevision> loadRevisions(Collection<CDOID>
> ids, int referenceChunk)
> {
> IStoreReader storeReader = StoreThreadLocal.getStoreReader();
> List<InternalCDORevision> revisions = new
> ArrayList<InternalCDORevision>();
> for (CDOID id : ids)
> {
> InternalCDORevision revision =
> (InternalCDORevision)storeReader.readRevision(id, referenceChunk);
> revisions.add(revision);
> }
>
> return revisions;
> }
>
> In my opinion the IStoreReader should support reading multiple
> revisions at once.
> My guess is that prefetching revisions one by one does not provide
> much performance gain (well, at least as long as I use the
> jvm-Connector...). What really should drive performance would be
> reducing single database accesses and batch them together.
> So, instead of
> - select ... from ... where cdo_id = 214 and cdo_version = 4
> - read answer
> - fill in revision
> - select ... from ... where cdo_id = 215 and cdo_version = 4
> - read answer
> - fill in revision
> - select ... from ... where cdo_id = 216 and cdo_version = 4
> - read answer
> - fill in revision
> it should be much more efficient to implement a way to
> - select ... from ... where (cdo_id = 214 and cdo_version = 4) or
> (cdo_id = 215 and cdo_version = 4) or (cdo_id = 216 and cdo_version = 4)
> - read answer
> - fill in revisions
>
> However, I see several challenges here :-(
> - the current IDBStore implementation might be hard to enhance because
> of the single-sql-string-builder
This seems to be a minor technical challenge, but...
> - the above works only for objects of the same type - what if we use
> polymorphism in the list?
.... this one was also my first thought. A more conceptual issue and
probably a real problem ;-)
I have no idea how to solve this and a solution without considering
polymorphism is not appealing.

> - ...
>
> Anyway, do you want an enhancement request filed? "Support fetching of
> multiple revisions at once in IStoreReader"?
I think it would make sense to discuss if the protocol between
RevisionManager and IStoreReader can be optimized in general (still with
respect to reading multiple revisions at once). But I doubt that the
DBSore with its horizontal mapping strategy will particularly benefit
from it. Would you like to come up with a contribution for all this?

Cheers
/Eike

Re: [CDO] using the query interface [message #423416 is a reply to message #423363]

Tue, 30 September 2008 05:18

Eclipse User

Hi,
>>
>> Anyway, do you want an enhancement request filed? "Support fetching
>> of multiple revisions at once in IStoreReader"?
> I think it would make sense to discuss if the protocol between
> RevisionManager and IStoreReader can be optimized in general (still
> with respect to reading multiple revisions at once). But I doubt that
> the DBSore with its horizontal mapping strategy will particularly
> benefit from it. Would you like to come up with a contribution for all
> this?
I feared you would ask for this ..
Like? yes
Have time? regretfully no :-( - at least not right now :-(

Cheers
Stefan

Re: [CDO] using the query interface [message #423417 is a reply to message #423416]

Tue, 30 September 2008 05:27

Eclipse User

Stefan Winkler schrieb:
> Hi,
>>>
>>> Anyway, do you want an enhancement request filed? "Support fetching
>>> of multiple revisions at once in IStoreReader"?
>> I think it would make sense to discuss if the protocol between
>> RevisionManager and IStoreReader can be optimized in general (still
>> with respect to reading multiple revisions at once). But I doubt that
>> the DBSore with its horizontal mapping strategy will particularly
>> benefit from it. Would you like to come up with a contribution for
>> all this?
> I feared you would ask for this ..
> Like? yes
> Have time? regretfully no :-( - at least not right now :-(
I feared you would say for this .. ;-)

Cheers
/Eike

Re: [CDO] using the query interface [message #423653 is a reply to message #423417]

Sat, 04 October 2008 12:15

Eclipse User

Hi Eike,

against all odds I was able to find some time (and a good idea).
So please have a look at Bug 249681 (
https://bugs.eclipse.org/bugs/show_bug.cgi?id=249681 ).

Cheers,
Stefan

Eike Stepper schrieb:
> Stefan Winkler schrieb:
>> Hi,
>>>>
>>>> Anyway, do you want an enhancement request filed? "Support fetching
>>>> of multiple revisions at once in IStoreReader"?
>>> I think it would make sense to discuss if the protocol between
>>> RevisionManager and IStoreReader can be optimized in general (still
>>> with respect to reading multiple revisions at once). But I doubt that
>>> the DBSore with its horizontal mapping strategy will particularly
>>> benefit from it. Would you like to come up with a contribution for
>>> all this?
>> I feared you would ask for this ..
>> Like? yes
>> Have time? regretfully no :-( - at least not right now :-(
> I feared you would say for this .. ;-)
>
> Cheers
> /Eike

Previous Topic:	How to write an EMF model which has to implemment an external Java interface
Next Topic:	Scheme editor, ecore to gmf

Goto Forum:

-=] Back to Top [=-

Current Time: Sat Jul 12 06:54:24 EDT 2025

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter