Home » Modeling » EMF » [CDO] Performance Issues on loading large amount of data
 [CDO] Performance Issues on loading large amount of data [message #787330] |
Tue, 31 January 2012 13:13  |
 Christophe Moine
Messages: 14
Registered: July 2009 |
Junior Member |
|
|
Hello everybody,
This is my first post on this forum, so please be gentle. I am stuck on CDO performance issue, and did not find the solution on the forum, I hope I didn't miss a post...
In my application, I need to load on each startup 150.000 entries (that are kind of constants). I face 2 issues:
- Sometimes it can very quick (about 5 secs) but sometimes, it reaches the timeout 
I prevented any other thread to access CDO but nothing change: is there some cache data that gets invalid after a while, or something like that ?
- For each entry, the SQL Engine does the following:
SET OPTION SQL_SELECT_LIMIT=DEFAULT
SELECT cdo_class FROM cdo_objects WHERE cdo_id=16497
SELECT uri FROM cdo_external_refs WHERE id=-54
SET OPTION SQL_SELECT_LIMIT=1
SELECT cdo_version, cdo_created, cdo_revised, cdo_resource, cdo_container, cdo_feature, ... FROM ... WHERE cdo_id=16497
Moreover, when I JProfile my application, my application & the CDO server are very passive, it seems that the SQL server is completly flooded.
I read so many times the Wiki page on CDO performance, and tried many configurations, but nothing change
I don't know if the hints actually prevent 5 SQL statements for one entry ? Is there a way to drastically improve performance on my case ? (loading constants in the DB)
About my actual configuration, I use MySQL DbSotre, and use the following code the configuration my view:
view.options().setRevisionPrefetchingPolicy(CDOUtil.createRevisionPrefetchingPolicy(PREFETCH_VALUE));
view.options().setCacheReferenceType(ReferenceType.STRONG);
Any ideas ?
Thank you a lot in advance,
Christophe.
|
|
| |
| Re: [CDO] Performance Issues on loading large amount of data [message #794364 is a reply to message #787330] |
Thu, 09 February 2012 07:36  |
|
Am 31.01.2012 14:13, schrieb Christophe Moine:
> Hello everybody,
>
> This is my first post on this forum,
And I accidentally marked it as read when Thunderbird messed up my local newsgroup state the other week. Sorry!
> so please be gentle.
Ok ;-)
> I am stuck on CDO performance issue, and did not find the solution on the forum, I hope I didn't miss a post...
I've cc'ed Sefan for DBStore-specific concerns. But let's see, though...
>
> In my application, I need to load on each startup 150.000 entries (that are kind of constants). I face 2 issues:
>
> - Sometimes it can very quick (about 5 secs) but sometimes, it reaches the timeout :(
> I prevented any other thread to access CDO but nothing change: is there some cache data that gets invalid after a
> while, or something like that ?
> - For each entry, the SQL Engine does the following:
> SET OPTION SQL_SELECT_LIMIT=DEFAULT
Stefan, can we influence that? Maybe we should remember the current limit and only set it when it changes?
> SELECT cdo_class FROM cdo_objects WHERE cdo_id=16497
That's to find out in what table to look up the object identified by the given ID. The only ways to avoid this are:
a) Keep *all* object->type mappings in memory. Not feasible for large models.
b) Embed the object's type in all references to it (CDOIDs). That's a feasible way (and the HibernateStore takes it,
IIRC). But I know that not all users of CDO want this tremendous overhead in heap footprint plus storage and payload
sizes. Thus we'd have to make it very optimized and *optional* . Possible, though.
> SELECT uri FROM cdo_external_refs WHERE id=-54
According to the two maps in org.eclipse.emf.cdo.server.internal.db.MetaDataManager this should only ever happen at most
once per concrete EClass in your models. If it happens more than once you should submit a bugzilla.
> SET OPTION SQL_SELECT_LIMIT=1
I see. Here the limit is changed from the DEFAULT (see above). Let's see the next statement for why we're doing this...
> SELECT cdo_version, cdo_created, cdo_revised, cdo_resource, cdo_container, cdo_feature, ... FROM ... WHERE cdo_id=16497
*if*
there's a potential for more than 1 row in the result set
*and*
there's no ORDER BY clause
*then*
it can't be right to limit the result set to just any one row.
Stefan?
> Moreover, when I JProfile my application, my application & the CDO server are very passive, it seems that the SQL
> server is completly flooded.
I would expect that I/O intensive threads are blocked most of the time. Let's see, though, what we can do about the
number of different SQL statemets issued.
>
> I read so many times the Wiki page on CDO performance, and tried many configurations, but nothing change :(
Well, that page is mostly not about tuning for specific storage back-ends.
>
> I don't know if the hints actually prevent 5 SQL statements for one entry ?
No.
> Is there a way to drastically improve performance on my case ? (loading constants in the DB)
See above.
>
> About my actual configuration, I use MySQL DbSotre, and use the following code the configuration my view:
> view.options().setRevisionPrefetchingPolicy(CDOUtil.createRevisionPrefetchingPolicy(PREFETCH_VALUE));
That should help with the issue of 1+n round-trips when fetching a list.
> view.options().setCacheReferenceType(ReferenceType.STRONG);
That can lead to heap space problems because objects will no longer be garbage collected. Be sure that you know how many
different objects will ever be loaded by that view!
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
|
| Re: [CDO] Performance Issues on loading large amount of data [message #794730 is a reply to message #794364] |
Thu, 09 February 2012 16:32 |
Stefan Winkler
Messages: 307
Registered: July 2009
Location: Germany |
Senior Member |
|
|
Hi,
comments below.
Am 09.02.12 08:36, schrieb Eike Stepper:
> Am 31.01.2012 14:13, schrieb Christophe Moine:
>> Hello everybody,
>>
>> This is my first post on this forum,
> And I accidentally marked it as read when Thunderbird messed up my local
> newsgroup state the other week. Sorry!
>
>> so please be gentle.
> Ok ;-)
>
>> I am stuck on CDO performance issue, and did not find the solution on
>> the forum, I hope I didn't miss a post...
> I've cc'ed Sefan for DBStore-specific concerns. But let's see, though...
>
>>
>> In my application, I need to load on each startup 150.000 entries
>> (that are kind of constants). I face 2 issues:
>>
>> - Sometimes it can very quick (about 5 secs) but sometimes, it reaches
>> the timeout :(
>> I prevented any other thread to access CDO but nothing change: is
>> there some cache data that gets invalid after a while, or something
>> like that ?
>> - For each entry, the SQL Engine does the following:
>> SET OPTION SQL_SELECT_LIMIT=DEFAULT
> Stefan, can we influence that? Maybe we should remember the current
> limit and only set it when it changes?
I guess this command comes from the JDBC driver. It could be the case
that because of the SQL_SELECT_LIMIT=1 below, this line is added. What
we could do is add a select limit of 1 here as well, as the object type
mapping should also contain at most one line here.
>> SELECT cdo_class FROM cdo_objects WHERE cdo_id=16497
> That's to find out in what table to look up the object identified by the
> given ID. The only ways to avoid this are:
>
> a) Keep *all* object->type mappings in memory. Not feasible for large
> models.
There is a cache for the object (cdoid) to type (eclass) mapping. What
could be done is adding a way to pre-populate the cache (-> please file
a bug/enhancement request for that). But a cache is always limited to a
certain number of objects, as Eike pointed out.
> b) Embed the object's type in all references to it (CDOIDs). That's a
> feasible way (and the HibernateStore takes it, IIRC). But I know that
> not all users of CDO want this tremendous overhead in heap footprint
> plus storage and payload sizes. Thus we'd have to make it very optimized
> and *optional* . Possible, though.
but currently not implemented, as far as I know. So this would be an
enhancement request (I think there's already a bug for that, let me
check ... ah here it is:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=254567)
>> SELECT uri FROM cdo_external_refs WHERE id=-54
> According to the two maps in
> org.eclipse.emf.cdo.server.internal.db.MetaDataManager this should only
> ever happen at most once per concrete EClass in your models. If it
> happens more than once you should submit a bugzilla.
>
>> SET OPTION SQL_SELECT_LIMIT=1
> I see. Here the limit is changed from the DEFAULT (see above). Let's see
> the next statement for why we're doing this...
I think there should be only one row returned, because only one row
matching the criteria should exist.
>> SELECT cdo_version, cdo_created, cdo_revised, cdo_resource,
>> cdo_container, cdo_feature, ... FROM ... WHERE cdo_id=16497
> *if*
> there's a potential for more than 1 row in the result set
> *and*
> there's no ORDER BY clause
> *then*
> it can't be right to limit the result set to just any one row.
>
> Stefan?
>
>> Moreover, when I JProfile my application, my application & the CDO
>> server are very passive, it seems that the SQL server is completly
>> flooded.
> I would expect that I/O intensive threads are blocked most of the time.
> Let's see, though, what we can do about the number of different SQL
> statemets issued.
>
>>
>> I read so many times the Wiki page on CDO performance, and tried many
>> configurations, but nothing change :(
> Well, that page is mostly not about tuning for specific storage back-ends.
>
>>
>> I don't know if the hints actually prevent 5 SQL statements for one
>> entry ?
> No.
>
>> Is there a way to drastically improve performance on my case ?
>> (loading constants in the DB)
> See above.
Are you running MySQL with the default settings or have you adjusted the
different parameters? I am no MySQL performance expert but google might
help in detemining some options (e.g. cache sizes etc.) which you can
adjust.
Cheers,
Stefan
>>
>> About my actual configuration, I use MySQL DbSotre, and use the
>> following code the configuration my view:
>> view.options().setRevisionPrefetchingPolicy(CDOUtil.createRevisionPrefetchingPolicy(PREFETCH_VALUE));
>>
> That should help with the issue of 1+n round-trips when fetching a list.
>
>> view.options().setCacheReferenceType(ReferenceType.STRONG);
> That can lead to heap space problems because objects will no longer be
> garbage collected. Be sure that you know how many different objects will
> ever be loaded by that view!
>
> Cheers
> /Eike
>
> ----
> http://www.esc-net.de
> http://thegordian.blogspot.com
> http://twitter.com/eikestepper
>
>
|
|
|
| Re: [CDO] Performance Issues on loading large amount of data [message #794794 is a reply to message #794730] |
Thu, 09 February 2012 18:01 |
Christophe Moine
Messages: 14
Registered: July 2009 |
Junior Member |
|
|
Hello Eike and Stefan,
Thanks a lot for your answers.
there are very interesting points in your message:
Quote:> a) Keep *all* object->type mappings in memory. Not feasible for large
> models.
There is a cache for the object (cdoid) to type (eclass) mapping. What
could be done is adding a way to pre-populate the cache (-> please file
a bug/enhancement request for that). But a cache is always limited to a
certain number of objects, as Eike pointed out.
The meta model is quite light, but the model itself contains quite a lot of items, so I don't think it would be very efficient. Unless we could specify only for specific CDOIDs....
I've entered an enhancement: http s://bugs.eclipse.org/bugs/show_bug.cgi?id=371113
Quote:>> About my actual configuration, I use MySQL DbSotre, and use the
>> following code the configuration my view:
>> view.options().setRevisionPrefetchingPolicy(CDOUtil.createRevisionPrefetchingPolicy(PREFETCH_VALUE));
>>
> That should help with the issue of 1+n round-trips when fetching a list.
Sorry, but I don't understand "1+n round-trips"... Does it refer to something in the Wiki ?
Quote:>> view.options().setCacheReferenceType(ReferenceType.STRONG);
> That can lead to heap space problems because objects will no longer be
> garbage collected. Be sure that you know how many different objects will
> ever be loaded by that view!
It actually doesn't matter much in my case.
I just want to load constants that have names description and codes, and I want the user can find a constant simply typing part of the name, description, or code...
When the 150.000 objects are loaded, it goes quickly enough.... The object load is the problematic part.
Because this is constants, isn't possible to define a cache on the server side so that it wouldn't be necessary the query the SQLServer each time ? That would be great I think.
Thanks a lot in advance,
Christophe.
[Updated on: Thu, 09 February 2012 18:02] Report message to a moderator |
|
| |
| Re: [CDO] Performance Issues on loading large amount of data [message #795181 is a reply to message #794794] |
Fri, 10 February 2012 06:53 |
|
Am 09.02.2012 19:01, schrieb Christophe Moine:
> [...]
> Quote:
>> >> About my actual configuration, I use MySQL DbSotre, and use the
>> >> following code the configuration my view:
>> >> view.options().setRevisionPrefetchingPolicy(CDOUtil.createRevisionPrefetchingPolicy(PREFETCH_VALUE));
>> >>
>> > That should help with the issue of 1+n round-trips when fetching a list.
>
>
> Sorry, but I don't understand "1+n round-trips"... Does it refer to something in the Wiki ?
That means that first the object is loaded with all the CDOIDs of referenced objects (1 server round-trip). When then
the references are iterated and accessed there are additional round-trips, one for each of the n reference targets.
>
> Quote:
>> >> view.options().setCacheReferenceType(ReferenceType.STRONG);
>> > That can lead to heap space problems because objects will no longer be
>> > garbage collected. Be sure that you know how many different objects will
>> > ever be loaded by that view!
>
>
> It actually doesn't matter much in my case.
>
> I just want to load constants that have names description and codes, and I want the user can find a constant simply
> typing part of the name, description, or code...
> When the 150.000 objects are loaded, it goes quickly enough.... The object load is the problematic part.
When they're all in one object subtree you can prefetch them altogether with
subtree.cdoPrefetch(CDORevision.DEPTH_INFINITE);
You could also check whether you really need all those constant things as CDOObjects. You could also encode them all (or
groups of them) into a byte[] and store the groups in separate CDOObjects. In your model you would replace the getters
of the constant references by derived attributes.
>
> Because this is constants, isn't possible to define a cache on the server side so that it wouldn't be necessary the
> query the SQLServer each time ? That would be great I think.
That's certainly possible. All you need to do is register an IRepository.ReadAccessHandler, inspect the passed revisions
and store strong Java references to them in a strongly referenced collection. That would prevent the revisions from
being removed from CDO's cache. My aforementioned approach sounds more promising, though.
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
|
| Re: [CDO] Performance Issues on loading large amount of data [message #795465 is a reply to message #795181] |
Fri, 10 February 2012 13:54 |
Christophe Moine
Messages: 14
Registered: July 2009 |
Junior Member |
|
|
Hello Eike,
Again, thank you very much for your answer. I did some tests, but I am confused with results...
I tried:
myCdoResourceThatContainsConstants.cdoPrefetch(CDORevision.DEPTH_INFINITE);
At first I thought it became incredibly fast, but actually all the load time is taken by cdoPretech which seems pretty obvious.
Moreoever, because I am loading constants at once with the cdoPrefetch, I often reach the timeout. I tried to change the time on the client (sessionConfiguration.setSignalTimeout(SIGNAL_TIMEOUT);
getRepository().getSessionManager().addListener(new ContainerEventAdapter<ISession>() {
@Override
protected void onAdded(IContainer<ISession> container, ISession session) {
ISessionProtocol protocol=session.getProtocol();
if (protocol instanceof ISignalProtocol) {
ISignalProtocol<?> signalProtocol=(ISignalProtocol<?>) protocol;
signalProtocol.setTimeout(SESSION_TIMEOUT);
}
}
});
Anyway, I think we have definetly a different approach: you make optimization on the client side, but I definitely think I need an optimization on the server:
- when I JProfile my application, it seems the bottleneck is the SQL Queries (because of the numerous queries): the client and the server seemed to be almost idle during the loading, I conclude the time is taken with I/O of the queries.
- In my case, the server is almost always running, but the GUI application is not. I don't want that the GUI always generates those massive queries.
I had a look at IRepository.ReadAccessHandler, and I don't really see how to use it, and you discouraged me a little bit using it. The couple JDBC/Lucene looks again fine for me. Is that shocking to access to the DB directly for constants objects ?
Thanks,
Christophe.
[Updated on: Fri, 10 February 2012 13:54] Report message to a moderator |
|
|
| Re: [CDO] Performance Issues on loading large amount of data [message #795569 is a reply to message #795465] |
Fri, 10 February 2012 16:02 |
|
Am 10.02.2012 14:54, schrieb Christophe Moine:
> Hello Eike,
>
> Again, thank you very much for your answer. I did some tests, but I am confused with results...
>
> I tried:
> Quote:
>> myCdoResourceThatContainsConstants.cdoPrefetch(CDORevision.DEPTH_INFINITE);
>
>
> At first I thought it became incredibly fast, but actually all the load time is taken by cdoPretech which seems pretty
> obvious.
> Moreoever, because I am loading constants at once with the cdoPrefetch, I often reach the timeout. I tried to change
> the time on the client (sessionConfiguration.setSignalTimeout(SIGNAL_TIMEOUT);) and on the server:
> getRepository().getSessionManager().addListener(new ContainerEventAdapter<ISession>() {
> @Override
> protected void onAdded(IContainer<ISession> container, ISession session) {
> ISessionProtocol protocol=session.getProtocol();
> if (protocol instanceof ISignalProtocol) {
> ISignalProtocol<?> signalProtocol=(ISignalProtocol<?>) protocol;
> signalProtocol.setTimeout(SESSION_TIMEOUT);
> }
> }
> }); but no changes :( Maybe there is a bug out there ?
Why not CDONet4jSession.Options.getNet4jProtocol().setTimeout(...) ?
> Anyway, I think we have definetly a different approach: you make optimization on the client side, but I definitely
> think I need an optimization on the server:
> - when I JProfile my application, it seems the bottleneck is the SQL Queries (because of the numerous queries): the
> client and the server seemed to be almost idle during the loading, I conclude the time is taken with I/O of the queries.
I fully agree. Stefan has put it on his list.
But generally CDOObjects (i.e., instances of your modeled ECLasses) are the granule for lazy loading on demand. To me it
seems you should not model your constants as EClasses but rather as EAttributes. See my previous hint.
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
| |
| Re: [CDO] Performance Issues on loading large amount of data [message #797273 is a reply to message #797259] |
Mon, 13 February 2012 09:02 |
|
Am 13.02.2012 09:41, schrieb Christophe Moine:
> Quote:
>> To me it seems you should not model your constants as EClasses but rather as EAttributes. See my previous hint.
>
>
> This approach looks surprising for me. What I feel here is that I should use EAttribute instead of EClass because of
> technical constraint. But one of the main advantage of modelization is to abstract technical stuff.
It is. But noone says the result will perform well :P
> Moreover I prefer EClass instead of EAttribute because those constants have many attributes (name, several codes,
> etc...),
Still you can serialize their values to byte[] attributes because their characteristic is never to change after initial
creation.
> and several EObjects have references to those constants.
That can be solved with derived references.
> Additionnally, those constants can be dynamically updated from internet....
Updated? Then they're certainly not constants.
> (this are kind of new products...)
Or do you really mean that new ones can be added?
> It would be very complex to encapsulate this into EAttribute :(
No, you'd just have to create a new version of the object that contains the serialized value of all constants.
> What I've done last friday is a direct JDBC connection which allows to load all constants in a single query in about 3
> seconds, which seems reasonnable.
> It creates instance of my CDOObject, set the EAttribute I am interested in (just like an indexation), and set CDOID
> using setInternalCdoID...
Calling internal methods may lead to unexpected behaviour including repository corruption. We can't support problems the
result from doing so.
> For me they represent MockObjects because they are not connected to the DB through CDO at all because they are not
> connected to a resource. But I should be able to retrieve the correct object using the View.getObject method I guess
> then.
Sorry, I don't want to follow this route. If you want to load objects through a mechanism outside of CDO you should
consider to use a non-CDO resource for that.
> The problem remaining with this approach is that I need to add option to connection to CDO server PLUS the connection
> to SQL server. Because the CDO server knows where the SQL server is located, is the client can query the CDO server to
> know where the SQL server is located ? Or is is possible for the CDO server to forward queries somehow ?
Sure, you could create/register your own IQueryHandler with the repository. Have a look at SQLQueryHandler.
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
Cheers
/Eike
----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper
|
|
|
| Re: [CDO] Performance Issues on loading large amount of data [message #798369 is a reply to message #797273] |
Tue, 14 February 2012 15:47 |
Christophe Moine
Messages: 14
Registered: July 2009 |
Junior Member |
|
|
Quote:> Additionnally, those constants can be dynamically updated from internet....
Updated? Then they're certainly not constants.
> (this are kind of new products...)
Or do you really mean that new ones can be added?
Sorry I was not very clear on that point: this is constants (products never change (except for an active state...)), but new constants (products) can be added every one and again (just like software updates).
You seem to say it is very easy to encapsulate byte [] attributes into EClass... well in my understanding, I need to write an 'unmarshaller' (byte [] => object) and 'marshaller' ( object => byte []). My questions are:
- Are the object unmarshalled are POJO, or CDOObject ? I can't see how they can be CDOObject actually.
- My question is rude, but what is the point in usually CDO if I have to handle serialization/deserialization to byte [] ? I think I can then handle it very well with files ? (thus with EMF / XMI)
Quote:Sorry, I don't want to follow this route. If you want to load objects through a mechanism outside of CDO you should
consider to use a non-CDO resource for that.
Sure, I understand your point. It seems to work anyway, since the CDOIDs seem to be the core of CDO, I don't see why it wouldn't be supported later on...
Quote:Sure, you could create/register your own IQueryHandler with the repository. Have a look at SQLQueryHandler.
That is just great ! I'll have a look at it,
thank you very much.
Cheers,
Christophe.
|
|
|
Goto Forum:
Current Time: Tue Apr 16 09:59:51 GMT 2024
Powered by FUDForum. Page generated in 0.29972 seconds |
] 
Search
Help
Register
Login
Home
