Home » Modeling » EMF » [EMF] [Teneo] : benchmarks ? And Return of Experience ?
| [EMF] [Teneo] : benchmarks ? And Return of Experience ? [message #423625] |
Fri, 03 October 2008 14:48  |
 Tristan Faure
Messages: 460
Registered: July 2009 |
Senior Member |
|
|
Hi
where can i found some information about EMF benchmarks ?
Do you know if there is a model limit size (using XMI) ?
For a teneo use is there some speed benchmarks ?
Is somebody try to use EMF and Teneo for very huge models (Actually i
can't quantify what is the limit between models and huge models ;)) ?
what kind of problem do you have ? Do you need to apply a process to
avoid performance problem ?
|
|
|
| Re: [EMF] [Teneo] : benchmarks ? And Return of Experience ? [message #423629 is a reply to message #423625] |
Fri, 03 October 2008 15:15  |
Ed Merks
Messages: 33137
Registered: July 2009 |
Senior Member |
|
|
This is a multi-part message in MIME format.
--------------090005060204000809040208
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Tristan,
Comments below.
Tristan FAURE wrote:
> Hi
>
> where can i found some information about EMF benchmarks ?
We used to run micro benchmark performance tests, but we've not been
doing that for a while. Loading performance depends a lot on options,
the complexity of the model, not just the size, i.e,. lots of cross
references will make things slower.
> Do you know if there is a model limit size (using XMI) ?
The size of the heap?
> For a teneo use is there some speed benchmarks ?
>
> Is somebody try to use EMF and Teneo for very huge models
Searching the newsgroups will turn up previous discussions (the second
search both lower down):
http://www.eclipse.org/newsportal/thread.php?group=eclipse.t ools.emf
http://www.eclipse.org/newsportal/thread.php?group=eclipse.t echnology.emft
> (Actually i can't quantify what is the limit between models and huge
> models ;)) ?
Exactly. It's relative.
> what kind of problem do you have ?
Eventually you run out of memory.
> Do you need to apply a process to avoid performance problem ?
>
>
> /EMF Performance Tips/
> < http://www.google.ca/url?sa=t&source=web&ct=res& cd=1&url=http%3A%2F%2Fwww.eclipse.org%2Fmodeling%2Femf%2 Fdocs%2Fperformance%2FEMFPerformanceTips.html&ei=-jbmSMT PJIqO8wTn5735Cw&usg=AFQjCNH8hcsWZ-j-8MfFwYjl_reru_wU_w&a mp;sig2=7pxIPBkobe-P3D0nz3jK4Q>
>
--------------090005060204000809040208
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Tristan,<br>
<br>
Comments below.<br>
<br>
Tristan FAURE wrote:
<blockquote cite="mid:gc5bbl$1hc$1@build.eclipse.org" type="cite">Hi
<br>
<br>
where can i found some information about EMF benchmarks ?
<br>
</blockquote>
We used to run micro benchmark performance tests, but we've not been
doing that for a while. Loading performance depends a lot on options,
the complexity of the model, not just the size, i.e,. lots of cross
references will make things slower.<br>
<blockquote cite="mid:gc5bbl$1hc$1@build.eclipse.org" type="cite">Do
you know if there is a model limit size (using XMI) ?
<br>
</blockquote>
The size of the heap?<br>
<blockquote cite="mid:gc5bbl$1hc$1@build.eclipse.org" type="cite">For a
teneo use is there some speed benchmarks ?
<br>
<br>
Is somebody try to use EMF and Teneo for very huge models </blockquote>
Searching the newsgroups will turn up previous discussions (the second
search both lower down):<br>
<blockquote><a
href=" http://www.eclipse.org/newsportal/thread.php?group=eclipse.t ools.emf"> http://www.eclipse.org/newsportal/thread.php?group=eclipse.t ools.emf</a><br>
<a
href=" http://www.eclipse.org/newsportal/thread.php?group=eclipse.t echnology.emft"> http://www.eclipse.org/newsportal/thread.php?group=eclipse.t echnology.emft</a><br>
</blockquote>
<br>
<blockquote cite="mid:gc5bbl$1hc$1@build.eclipse.org" type="cite">(Actually
i can't quantify what is the limit between models and huge models ;)) ?
</blockquote>
Exactly. It's relative.<br>
<blockquote cite="mid:gc5bbl$1hc$1@build.eclipse.org" type="cite">what
kind of problem do you have ?</blockquote>
Eventually you run out of memory.<br>
<blockquote cite="mid:gc5bbl$1hc$1@build.eclipse.org" type="cite">Do
you need to apply a process to avoid performance problem ?
<br>
<h3 class="r"><a
href="http://www.google.ca/url?sa=t& ;source=web&ct=res&cd=1&url=http%3A% 2F%2Fwww.eclipse.org%2Fmodeling%2Femf%2Fdocs%2Fperformance%2 FEMFPerformanceTips.html&ei=-jbmSMTPJIqO8wTn5735Cw&a mp;amp;usg=AFQjCNH8hcsWZ-j-8MfFwYjl_reru_wU_w&sig2=7 pxIPBkobe-P3D0nz3jK4Q "
class="l"
onmousedown="return rwt(this,'','','res','1','AFQjCNH8hcsWZ-j-8MfFwYjl_reru_wU_w ','&sig2=7pxIPBkobe-P3D0nz3jK4Q') "><em>EMF
Performance Tips</em></a></h3>
</blockquote>
</body>
</html>
--------------090005060204000809040208--
Ed Merks
Professional Support: https://www.macromodeling.com/
|
|
| |
| Re: [EMF] [Teneo] : benchmarks ? And Return of Experience ? [message #423632 is a reply to message #423630] |
Fri, 03 October 2008 15:28 |
Ed Merks
Messages: 33137
Registered: July 2009 |
Senior Member |
|
|
Tristan,
IBM uses UML which is based on EMF to load massive models. Of course
clients feed in models until the system runs out of memory. But that
happens for massive models. Sometimes I joke that Java doesn't scale
because while I have 4G of physical RAM, I can't build more than a 2G
heap...
EMF's XML deserializer loads an entire resource at once, but cross
resources references are loaded on demand as you visit the data.
Nothing will automatically unload them if you stop using them. The fact
that ResourceSet, Resource, and EObject form a containment tree with two
way references means that general a reference to any EObject or Resource
will keep the entire tree in memory...
Tristan FAURE wrote:
> tahnk you ed for you remarks and maybe sorry for my question a little
> bit too evasive
>
> I'll check your links and if anybody has some experience return he is
> welcome.
>
> And about model limit size it doens't depends on the physical size ?
> Does EMF loads models incrementally ?
>
> Ed Merks a écrit :
>> Tristan,
>>
>> Comments below.
>>
>> Tristan FAURE wrote:
>>> Hi
>>>
>>> where can i found some information about EMF benchmarks ?
>> We used to run micro benchmark performance tests, but we've not been
>> doing that for a while. Loading performance depends a lot on
>> options, the complexity of the model, not just the size, i.e,. lots
>> of cross references will make things slower.
>>> Do you know if there is a model limit size (using XMI) ?
>> The size of the heap?
>>> For a teneo use is there some speed benchmarks ?
>>>
>>> Is somebody try to use EMF and Teneo for very huge models
>> Searching the newsgroups will turn up previous discussions (the
>> second search both lower down):
>>
>> http://www.eclipse.org/newsportal/thread.php?group=eclipse.t ools.emf
>>
>> http://www.eclipse.org/newsportal/thread.php?group=eclipse.t echnology.emft
>>
>>
>>
>>> (Actually i can't quantify what is the limit between models and huge
>>> models ;)) ?
>> Exactly. It's relative.
>>> what kind of problem do you have ?
>> Eventually you run out of memory.
>>> Do you need to apply a process to avoid performance problem ?
>>>
>>>
>>> /EMF Performance Tips/
>>>
>>> < http://www.google.ca/url?sa=t&source=web&ct=res& cd=1&url=http%3A%2F%2Fwww.eclipse.org%2Fmodeling%2Femf%2 Fdocs%2Fperformance%2FEMFPerformanceTips.html&ei=-jbmSMT PJIqO8wTn5735Cw&usg=AFQjCNH8hcsWZ-j-8MfFwYjl_reru_wU_w&a mp;sig2=7pxIPBkobe-P3D0nz3jK4Q>
>>>
>>>
Ed Merks
Professional Support: https://www.macromodeling.com/
|
|
|
| Re: [EMF] [Teneo] : benchmarks ? And Return of Experience ? [message #423635 is a reply to message #423630] |
Fri, 03 October 2008 15:58 |
Simon Mc Duff
Messages: 596
Registered: July 2009 |
Senior Member |
|
|
Maybe it can help but I know some technology are built to access big graphs.
With CDO, you can load a big graph as big as you want. While you are
crawling the graph, objects that are not used anymore will be evict from the
cache automatically. This is possible because objects are never linked
together. Garbage collector can do its work.
I'm able to load 4 gig of data even if my JVM maxsize is at 260 megs !!!
What it is nice is while you are crawling.. you don't need to do anything
special. (e.g.: clean objects, remove object from cache, etc...)
Simon
"Tristan FAURE" <tristan.faure@atosorigin.com> wrote in message
news:48E6386E.8000608@atosorigin.com...
> tahnk you ed for you remarks and maybe sorry for my question a little bit
> too evasive
>
> I'll check your links and if anybody has some experience return he is
> welcome.
>
> And about model limit size it doens't depends on the physical size ? Does
> EMF loads models incrementally ?
>
> Ed Merks a écrit :
>> Tristan,
>>
>> Comments below.
>>
>> Tristan FAURE wrote:
>>> Hi
>>>
>>> where can i found some information about EMF benchmarks ?
>> We used to run micro benchmark performance tests, but we've not been
>> doing that for a while. Loading performance depends a lot on options,
>> the complexity of the model, not just the size, i.e,. lots of cross
>> references will make things slower.
>>> Do you know if there is a model limit size (using XMI) ?
>> The size of the heap?
>>> For a teneo use is there some speed benchmarks ?
>>>
>>> Is somebody try to use EMF and Teneo for very huge models
>> Searching the newsgroups will turn up previous discussions (the second
>> search both lower down):
>>
>> http://www.eclipse.org/newsportal/thread.php?group=eclipse.t ools.emf
>>
>> http://www.eclipse.org/newsportal/thread.php?group=eclipse.t echnology.emft
>>
>>
>>> (Actually i can't quantify what is the limit between models and huge
>>> models ;)) ?
>> Exactly. It's relative.
>>> what kind of problem do you have ?
>> Eventually you run out of memory.
>>> Do you need to apply a process to avoid performance problem ?
>>>
>>>
>>> /EMF Performance Tips/
>>>
>>> < http://www.google.ca/url?sa=t&source=web&ct=res& cd=1&url=http%3A%2F%2Fwww.eclipse.org%2Fmodeling%2Femf%2 Fdocs%2Fperformance%2FEMFPerformanceTips.html&ei=-jbmSMT PJIqO8wTn5735Cw&usg=AFQjCNH8hcsWZ-j-8MfFwYjl_reru_wU_w&a mp;sig2=7pxIPBkobe-P3D0nz3jK4Q>
>>>
|
|
|
| Re: [EMF] [Teneo] : benchmarks ? And Return of Experience ? [message #423636 is a reply to message #423630] |
Fri, 03 October 2008 15:59 |
Martin Taal
Messages: 5468
Registered: July 2009 |
Senior Member |
|
|
Hi Tristan,
As Ed pointed out, it always depends. You should use one of the later versions of Teneo as the 1.0.0
release had a performance issue with large sets of data (10000s of objects in mem).
If you have performance issues then I would definitely enable hibernate sql logging so you can see
what queries are fired by hibernate at which point. This has helped me a lot to find issues here and
there.
EMF uses EMF proxying to prevent eager loading. However Teneo/Hibernate does not use that at the
moment, Teneo/hibernate uses the standard lazy loading mechanism of hibernate (cglib and lazy loaded
lists). So both native EMF and EMF/Teneo/Hibernate support incremental loading.
gr. Martin
Tristan FAURE wrote:
> tahnk you ed for you remarks and maybe sorry for my question a little
> bit too evasive
>
> I'll check your links and if anybody has some experience return he is
> welcome.
>
> And about model limit size it doens't depends on the physical size ?
> Does EMF loads models incrementally ?
>
> Ed Merks a écrit :
>> Tristan,
>>
>> Comments below.
>>
>> Tristan FAURE wrote:
>>> Hi
>>>
>>> where can i found some information about EMF benchmarks ?
>> We used to run micro benchmark performance tests, but we've not been
>> doing that for a while. Loading performance depends a lot on options,
>> the complexity of the model, not just the size, i.e,. lots of cross
>> references will make things slower.
>>> Do you know if there is a model limit size (using XMI) ?
>> The size of the heap?
>>> For a teneo use is there some speed benchmarks ?
>>>
>>> Is somebody try to use EMF and Teneo for very huge models
>> Searching the newsgroups will turn up previous discussions (the second
>> search both lower down):
>>
>> http://www.eclipse.org/newsportal/thread.php?group=eclipse.t ools.emf
>>
>> http://www.eclipse.org/newsportal/thread.php?group=eclipse.t echnology.emft
>>
>>
>>
>>> (Actually i can't quantify what is the limit between models and huge
>>> models ;)) ?
>> Exactly. It's relative.
>>> what kind of problem do you have ?
>> Eventually you run out of memory.
>>> Do you need to apply a process to avoid performance problem ?
>>>
>>>
>>> /EMF Performance Tips/
>>>
>>> < http://www.google.ca/url?sa=t&source=web&ct=res& cd=1&url=http%3A%2F%2Fwww.eclipse.org%2Fmodeling%2Femf%2 Fdocs%2Fperformance%2FEMFPerformanceTips.html&ei=-jbmSMT PJIqO8wTn5735Cw&usg=AFQjCNH8hcsWZ-j-8MfFwYjl_reru_wU_w&a mp;sig2=7pxIPBkobe-P3D0nz3jK4Q>
>>>
>>>
--
With Regards, Martin Taal
Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Cell: +31 (0)6 288 48 943
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
|
|
|
| Re: [EMF] [Teneo] : benchmarks ? And Return of Experience ? [message #423646 is a reply to message #423636] |
Sat, 04 October 2008 00:59 |
Andrew H
Messages: 117
Registered: July 2009 |
Senior Member |
|
|
We have a pretty large model ( > 800 classifiers). I've been playing
around with persistence via teneo this last week. I've just now got the
basic thing working and need to look at some of the bottlenecks I've
hit. Most I think revolve around eager fetching.
The DB schema that teneo / hibernate have generated has > 500 tables
which is pretty large model in my book.
This is currently hurting on start up as hibernate evaluates what needs
to change. I'm assuming hibernate has a way to turn that off but haven't
pursued that yet.
On the performance front I'm about to look into tweaking a few things
such as:
- selectively turning off eager fetching of containment on queries. In
general I want it to do this as each object we will pull out of the db
is a pretty large (and diverse) tree. If I needed to specify what to
fetch in each query I would get a very large amount of querying code.
- However, on some operations (particularly when I persist new
versions of these objects) I need to navigate some one to many
containment relationships to add this one to the list. I think it is
eager fetching all the data on every version at the moment which is
causing it to grind. I think hibernate allows you to set fetch
strategies on individual criteria.
- large inheritance trees. I have one class in particular - Product -
that has many subtypes. Hibernate is doing a query on product that joins
on to every subtype. Since its only ever one of these this is wasteful.
I set the MAX_FETCH_DEPTH to 0 to try to stop this but it still does it.
I suspect I may need custom mappings on Product somehow to turn this off.
- Denormalising some classes. I believe there is an Embeddable concept
in EMF that may help this. Also I may choose different inheritance
strategies for different hierarchies.
Hopefully next week I will have more info on my experiences working with
a large model
cheers
Andrew
Martin Taal wrote:
> Hi Tristan,
> As Ed pointed out, it always depends. You should use one of the later
> versions of Teneo as the 1.0.0 release had a performance issue with
> large sets of data (10000s of objects in mem).
>
> If you have performance issues then I would definitely enable hibernate
> sql logging so you can see what queries are fired by hibernate at which
> point. This has helped me a lot to find issues here and there.
>
> EMF uses EMF proxying to prevent eager loading. However Teneo/Hibernate
> does not use that at the moment, Teneo/hibernate uses the standard lazy
> loading mechanism of hibernate (cglib and lazy loaded lists). So both
> native EMF and EMF/Teneo/Hibernate support incremental loading.
>
> gr. Martin
>
> Tristan FAURE wrote:
>> tahnk you ed for you remarks and maybe sorry for my question a little
>> bit too evasive
>>
>> I'll check your links and if anybody has some experience return he is
>> welcome.
>>
>> And about model limit size it doens't depends on the physical size ?
>> Does EMF loads models incrementally ?
>>
>> Ed Merks a écrit :
>>> Tristan,
>>>
>>> Comments below.
>>>
>>> Tristan FAURE wrote:
>>>> Hi
>>>>
>>>> where can i found some information about EMF benchmarks ?
>>> We used to run micro benchmark performance tests, but we've not been
>>> doing that for a while. Loading performance depends a lot on
>>> options, the complexity of the model, not just the size, i.e,. lots
>>> of cross references will make things slower.
>>>> Do you know if there is a model limit size (using XMI) ?
>>> The size of the heap?
>>>> For a teneo use is there some speed benchmarks ?
>>>>
>>>> Is somebody try to use EMF and Teneo for very huge models
>>> Searching the newsgroups will turn up previous discussions (the
>>> second search both lower down):
>>>
>>> http://www.eclipse.org/newsportal/thread.php?group=eclipse.t ools.emf
>>>
>>> http://www.eclipse.org/newsportal/thread.php?group=eclipse.t echnology.emft
>>>
>>>
>>>
>>>> (Actually i can't quantify what is the limit between models and huge
>>>> models ;)) ?
>>> Exactly. It's relative.
>>>> what kind of problem do you have ?
>>> Eventually you run out of memory.
>>>> Do you need to apply a process to avoid performance problem ?
>>>>
>>>>
>>>> /EMF Performance Tips/
>>>>
>>>> < http://www.google.ca/url?sa=t&source=web&ct=res& cd=1&url=http%3A%2F%2Fwww.eclipse.org%2Fmodeling%2Femf%2 Fdocs%2Fperformance%2FEMFPerformanceTips.html&ei=-jbmSMT PJIqO8wTn5735Cw&usg=AFQjCNH8hcsWZ-j-8MfFwYjl_reru_wU_w&a mp;sig2=7pxIPBkobe-P3D0nz3jK4Q>
>>>>
>>>>
>
>
|
|
|
| Re: [EMF] [Teneo] : benchmarks ? And Return of Experience ? [message #423647 is a reply to message #423636] |
Sat, 04 October 2008 01:00 |
Andrew H
Messages: 117
Registered: July 2009 |
Senior Member |
|
|
We have a pretty large model ( > 800 classifiers). I've been playing
around with persistence via teneo this last week. I've just now got the
basic thing working and need to look at some of the bottlenecks I've
hit. Most I think revolve around eager fetching.
The DB schema that teneo / hibernate have generated has > 500 tables
which is pretty large model in my book.
This is currently hurting on start up as hibernate evaluates what needs
to change. I'm assuming hibernate has a way to turn that off but haven't
pursued that yet.
On the performance front I'm about to look into tweaking a few things
such as:
- selectively turning off eager fetching of containment on queries. In
general I want it to do this as each object we will pull out of the db
is a pretty large (and diverse) tree. If I needed to specify what to
fetch in each query I would get a very large amount of querying code.
- However, on some operations (particularly when I persist new
versions of these objects) I need to navigate some one to many
containment relationships to add this one to the list. I think it is
eager fetching all the data on every version at the moment which is
causing it to grind. I think hibernate allows you to set fetch
strategies on individual criteria.
- large inheritance trees. I have one class in particular - Product -
that has many subtypes. Hibernate is doing a query on product that joins
on to every subtype. Since its only ever one of these this is wasteful.
I set the MAX_FETCH_DEPTH to 0 to try to stop this but it still does it.
I suspect I may need custom mappings on Product somehow to turn this off.
- Denormalising some classes. I believe there is an Embeddable concept
in EMF that may help this. Also I may choose different inheritance
strategies for different hierarchies.
Hopefully next week I will have more info on my experiences working with
a large model
cheers
Andrew
Martin Taal wrote:
> Hi Tristan,
> As Ed pointed out, it always depends. You should use one of the later
> versions of Teneo as the 1.0.0 release had a performance issue with
> large sets of data (10000s of objects in mem).
>
> If you have performance issues then I would definitely enable hibernate
> sql logging so you can see what queries are fired by hibernate at which
> point. This has helped me a lot to find issues here and there.
>
> EMF uses EMF proxying to prevent eager loading. However Teneo/Hibernate
> does not use that at the moment, Teneo/hibernate uses the standard lazy
> loading mechanism of hibernate (cglib and lazy loaded lists). So both
> native EMF and EMF/Teneo/Hibernate support incremental loading.
>
> gr. Martin
>
> Tristan FAURE wrote:
>> tahnk you ed for you remarks and maybe sorry for my question a little
>> bit too evasive
>>
>> I'll check your links and if anybody has some experience return he is
>> welcome.
>>
>> And about model limit size it doens't depends on the physical size ?
>> Does EMF loads models incrementally ?
>>
>> Ed Merks a écrit :
>>> Tristan,
>>>
>>> Comments below.
>>>
>>> Tristan FAURE wrote:
>>>> Hi
>>>>
>>>> where can i found some information about EMF benchmarks ?
>>> We used to run micro benchmark performance tests, but we've not been
>>> doing that for a while. Loading performance depends a lot on
>>> options, the complexity of the model, not just the size, i.e,. lots
>>> of cross references will make things slower.
>>>> Do you know if there is a model limit size (using XMI) ?
>>> The size of the heap?
>>>> For a teneo use is there some speed benchmarks ?
>>>>
>>>> Is somebody try to use EMF and Teneo for very huge models
>>> Searching the newsgroups will turn up previous discussions (the
>>> second search both lower down):
>>>
>>> http://www.eclipse.org/newsportal/thread.php?group=eclipse.t ools.emf
>>>
>>> http://www.eclipse.org/newsportal/thread.php?group=eclipse.t echnology.emft
>>>
>>>
>>>
>>>> (Actually i can't quantify what is the limit between models and huge
>>>> models ;)) ?
>>> Exactly. It's relative.
>>>> what kind of problem do you have ?
>>> Eventually you run out of memory.
>>>> Do you need to apply a process to avoid performance problem ?
>>>>
>>>>
>>>> /EMF Performance Tips/
>>>>
>>>> < http://www.google.ca/url?sa=t&source=web&ct=res& cd=1&url=http%3A%2F%2Fwww.eclipse.org%2Fmodeling%2Femf%2 Fdocs%2Fperformance%2FEMFPerformanceTips.html&ei=-jbmSMT PJIqO8wTn5735Cw&usg=AFQjCNH8hcsWZ-j-8MfFwYjl_reru_wU_w&a mp;sig2=7pxIPBkobe-P3D0nz3jK4Q>
>>>>
>>>>
>
>
|
|
|
| Re: [EMF] [Teneo] : benchmarks ? And Return of Experience ? [message #423649 is a reply to message #423647] |
Sat, 04 October 2008 09:17 |
Martin Taal
Messages: 5468
Registered: July 2009 |
Senior Member |
|
|
Hi Andrew,
See my comments inline.
gr. Martin
andrew987650000@yahoo.com.au wrote:
> We have a pretty large model ( > 800 classifiers). I've been playing
> around with persistence via teneo this last week. I've just now got the
> basic thing working and need to look at some of the bottlenecks I've
> hit. Most I think revolve around eager fetching.
>
> The DB schema that teneo / hibernate have generated has > 500 tables
> which is pretty large model in my book.
>
> This is currently hurting on start up as hibernate evaluates what needs
> to change. I'm assuming hibernate has a way to turn that off but haven't
> pursued that yet.
MT>> As a default if you don't set the Environment.HBM2DDL_AUTO hibernate property then Teneo will
set it to update, so if you set it to none (undocumented feature of hibernate afaik) it should
prevent teneo from setting it.
I am working with a 425 table model and hibernate is initialized in about 10 seconds.
In my case I don't update the database schema so probably this is were the time is spend by hibernate.
>
> On the performance front I'm about to look into tweaking a few things
> such as:
>
> - selectively turning off eager fetching of containment on queries. In
> general I want it to do this as each object we will pull out of the db
> is a pretty large (and diverse) tree. If I needed to specify what to
> fetch in each query I would get a very large amount of querying code.
MT>> depends on how much time you want to spend on this but an idea I got when reading this is that
you can write your own query generator using your model which is available at runtime. Then you can
parameter control your query generator to use eager fetching or not in different locations.
> - However, on some operations (particularly when I persist new
> versions of these objects) I need to navigate some one to many
> containment relationships to add this one to the list. I think it is
> eager fetching all the data on every version at the moment which is
> causing it to grind. I think hibernate allows you to set fetch
> strategies on individual criteria.
MT>> I am not sure if you use bags or lists but you should maybe try hibernates idbag or persist
lists as bags (so non-ordered). Lists are slower to persist/update than bags and much slower than
idbags.
Another thing to watch for is that EMF sends notifications on the basis of changes in objects,
sometimes these notifications result in loading of data at unexpected moments in time. Although it
can't be prevented always (EMF sends the notifications for a reason) it can be good to be aware of it.
>
> - large inheritance trees. I have one class in particular - Product -
> that has many subtypes. Hibernate is doing a query on product that joins
> on to every subtype. Since its only ever one of these this is wasteful.
> I set the MAX_FETCH_DEPTH to 0 to try to stop this but it still does it.
> I suspect I may need custom mappings on Product somehow to turn this off.
MT>> Afaik the fetch_depth is related to associations and not so much depth of the inheritance tree.
Another option would be to use single table inheritance for this inheritance tree. Hibernate has a
limitation though that a whole inheritance tree should use one inheritance strategy for the whole
tree. So you can do join_table for the top of the tree and single table for branches. But if your
product inheritance tree is an independent tree then it can be set to single table using the
inheritance annotation.
>
> - Denormalising some classes. I believe there is an Embeddable concept
> in EMF that may help this. Also I may choose different inheritance
> strategies for different hierarchies.
MT>> Yes, that was also my idea.
>
> Hopefully next week I will have more info on my experiences working with
> a large model
>
> cheers
>
> Andrew
>
>
> Martin Taal wrote:
>> Hi Tristan,
>> As Ed pointed out, it always depends. You should use one of the later
>> versions of Teneo as the 1.0.0 release had a performance issue with
>> large sets of data (10000s of objects in mem).
>>
>> If you have performance issues then I would definitely enable
>> hibernate sql logging so you can see what queries are fired by
>> hibernate at which point. This has helped me a lot to find issues here
>> and there.
>>
>> EMF uses EMF proxying to prevent eager loading. However
>> Teneo/Hibernate does not use that at the moment, Teneo/hibernate uses
>> the standard lazy loading mechanism of hibernate (cglib and lazy
>> loaded lists). So both native EMF and EMF/Teneo/Hibernate support
>> incremental loading.
>>
>> gr. Martin
>>
>> Tristan FAURE wrote:
>>> tahnk you ed for you remarks and maybe sorry for my question a little
>>> bit too evasive
>>>
>>> I'll check your links and if anybody has some experience return he is
>>> welcome.
>>>
>>> And about model limit size it doens't depends on the physical size ?
>>> Does EMF loads models incrementally ?
>>>
>>> Ed Merks a écrit :
>>>> Tristan,
>>>>
>>>> Comments below.
>>>>
>>>> Tristan FAURE wrote:
>>>>> Hi
>>>>>
>>>>> where can i found some information about EMF benchmarks ?
>>>> We used to run micro benchmark performance tests, but we've not been
>>>> doing that for a while. Loading performance depends a lot on
>>>> options, the complexity of the model, not just the size, i.e,. lots
>>>> of cross references will make things slower.
>>>>> Do you know if there is a model limit size (using XMI) ?
>>>> The size of the heap?
>>>>> For a teneo use is there some speed benchmarks ?
>>>>>
>>>>> Is somebody try to use EMF and Teneo for very huge models
>>>> Searching the newsgroups will turn up previous discussions (the
>>>> second search both lower down):
>>>>
>>>>
>>>> http://www.eclipse.org/newsportal/thread.php?group=eclipse.t ools.emf
>>>>
>>>> http://www.eclipse.org/newsportal/thread.php?group=eclipse.t echnology.emft
>>>>
>>>>
>>>>
>>>>> (Actually i can't quantify what is the limit between models and
>>>>> huge models ;)) ?
>>>> Exactly. It's relative.
>>>>> what kind of problem do you have ?
>>>> Eventually you run out of memory.
>>>>> Do you need to apply a process to avoid performance problem ?
>>>>>
>>>>>
>>>>> /EMF Performance Tips/
>>>>>
>>>>> < http://www.google.ca/url?sa=t&source=web&ct=res& cd=1&url=http%3A%2F%2Fwww.eclipse.org%2Fmodeling%2Femf%2 Fdocs%2Fperformance%2FEMFPerformanceTips.html&ei=-jbmSMT PJIqO8wTn5735Cw&usg=AFQjCNH8hcsWZ-j-8MfFwYjl_reru_wU_w&a mp;sig2=7pxIPBkobe-P3D0nz3jK4Q>
>>>>>
>>>>>
>>
>>
--
With Regards, Martin Taal
Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Cell: +31 (0)6 288 48 943
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
|
|
|
| Re: [EMF] [Teneo] : benchmarks ? And Return of Experience ? [message #423660 is a reply to message #423649] |
Mon, 06 October 2008 08:13 |
Tristan Faure
Messages: 460
Registered: July 2009 |
Senior Member |
|
|
Hi thank you all for your comments
all of them will be very useful for my study
Martin Taal a écrit :
> Hi Andrew,
> See my comments inline.
>
> gr. Martin
>
> andrew987650000@yahoo.com.au wrote:
>> We have a pretty large model ( > 800 classifiers). I've been playing
>> around with persistence via teneo this last week. I've just now got
>> the basic thing working and need to look at some of the bottlenecks
>> I've hit. Most I think revolve around eager fetching.
>>
>> The DB schema that teneo / hibernate have generated has > 500 tables
>> which is pretty large model in my book.
>>
>> This is currently hurting on start up as hibernate evaluates what
>> needs to change. I'm assuming hibernate has a way to turn that off but
>> haven't pursued that yet.
> MT>> As a default if you don't set the Environment.HBM2DDL_AUTO
> hibernate property then Teneo will set it to update, so if you set it to
> none (undocumented feature of hibernate afaik) it should prevent teneo
> from setting it.
> I am working with a 425 table model and hibernate is initialized in
> about 10 seconds.
> In my case I don't update the database schema so probably this is were
> the time is spend by hibernate.
>
>>
>> On the performance front I'm about to look into tweaking a few things
>> such as:
>>
>> - selectively turning off eager fetching of containment on queries. In
>> general I want it to do this as each object we will pull out of the db
>> is a pretty large (and diverse) tree. If I needed to specify what to
>> fetch in each query I would get a very large amount of querying code.
> MT>> depends on how much time you want to spend on this but an idea I
> got when reading this is that you can write your own query generator
> using your model which is available at runtime. Then you can parameter
> control your query generator to use eager fetching or not in different
> locations.
>
>> - However, on some operations (particularly when I persist new
>> versions of these objects) I need to navigate some one to many
>> containment relationships to add this one to the list. I think it is
>> eager fetching all the data on every version at the moment which is
>> causing it to grind. I think hibernate allows you to set fetch
>> strategies on individual criteria.
> MT>> I am not sure if you use bags or lists but you should maybe try
> hibernates idbag or persist lists as bags (so non-ordered). Lists are
> slower to persist/update than bags and much slower than idbags.
> Another thing to watch for is that EMF sends notifications on the basis
> of changes in objects, sometimes these notifications result in loading
> of data at unexpected moments in time. Although it can't be prevented
> always (EMF sends the notifications for a reason) it can be good to be
> aware of it.
>
>>
>> - large inheritance trees. I have one class in particular - Product -
>> that has many subtypes. Hibernate is doing a query on product that
>> joins on to every subtype. Since its only ever one of these this is
>> wasteful. I set the MAX_FETCH_DEPTH to 0 to try to stop this but it
>> still does it. I suspect I may need custom mappings on Product somehow
>> to turn this off.
> MT>> Afaik the fetch_depth is related to associations and not so much
> depth of the inheritance tree.
> Another option would be to use single table inheritance for this
> inheritance tree. Hibernate has a limitation though that a whole
> inheritance tree should use one inheritance strategy for the whole tree.
> So you can do join_table for the top of the tree and single table for
> branches. But if your product inheritance tree is an independent tree
> then it can be set to single table using the inheritance annotation.
>
>>
>> - Denormalising some classes. I believe there is an Embeddable concept
>> in EMF that may help this. Also I may choose different inheritance
>> strategies for different hierarchies.
> MT>> Yes, that was also my idea.
>
>>
>> Hopefully next week I will have more info on my experiences working
>> with a large model
>>
>> cheers
>>
>> Andrew
>>
>>
>> Martin Taal wrote:
>>> Hi Tristan,
>>> As Ed pointed out, it always depends. You should use one of the later
>>> versions of Teneo as the 1.0.0 release had a performance issue with
>>> large sets of data (10000s of objects in mem).
>>>
>>> If you have performance issues then I would definitely enable
>>> hibernate sql logging so you can see what queries are fired by
>>> hibernate at which point. This has helped me a lot to find issues
>>> here and there.
>>>
>>> EMF uses EMF proxying to prevent eager loading. However
>>> Teneo/Hibernate does not use that at the moment, Teneo/hibernate uses
>>> the standard lazy loading mechanism of hibernate (cglib and lazy
>>> loaded lists). So both native EMF and EMF/Teneo/Hibernate support
>>> incremental loading.
>>>
>>> gr. Martin
>>>
>>> Tristan FAURE wrote:
>>>> tahnk you ed for you remarks and maybe sorry for my question a
>>>> little bit too evasive
>>>>
>>>> I'll check your links and if anybody has some experience return he
>>>> is welcome.
>>>>
>>>> And about model limit size it doens't depends on the physical size ?
>>>> Does EMF loads models incrementally ?
>>>>
>>>> Ed Merks a écrit :
>>>>> Tristan,
>>>>>
>>>>> Comments below.
>>>>>
>>>>> Tristan FAURE wrote:
>>>>>> Hi
>>>>>>
>>>>>> where can i found some information about EMF benchmarks ?
>>>>> We used to run micro benchmark performance tests, but we've not
>>>>> been doing that for a while. Loading performance depends a lot on
>>>>> options, the complexity of the model, not just the size, i.e,. lots
>>>>> of cross references will make things slower.
>>>>>> Do you know if there is a model limit size (using XMI) ?
>>>>> The size of the heap?
>>>>>> For a teneo use is there some speed benchmarks ?
>>>>>>
>>>>>> Is somebody try to use EMF and Teneo for very huge models
>>>>> Searching the newsgroups will turn up previous discussions (the
>>>>> second search both lower down):
>>>>>
>>>>>
>>>>> http://www.eclipse.org/newsportal/thread.php?group=eclipse.t ools.emf
>>>>>
>>>>> http://www.eclipse.org/newsportal/thread.php?group=eclipse.t echnology.emft
>>>>>
>>>>>
>>>>>
>>>>>> (Actually i can't quantify what is the limit between models and
>>>>>> huge models ;)) ?
>>>>> Exactly. It's relative.
>>>>>> what kind of problem do you have ?
>>>>> Eventually you run out of memory.
>>>>>> Do you need to apply a process to avoid performance problem ?
>>>>>>
>>>>>>
>>>>>> /EMF Performance Tips/
>>>>>>
>>>>>> < http://www.google.ca/url?sa=t&source=web&ct=res& cd=1&url=http%3A%2F%2Fwww.eclipse.org%2Fmodeling%2Femf%2 Fdocs%2Fperformance%2FEMFPerformanceTips.html&ei=-jbmSMT PJIqO8wTn5735Cw&usg=AFQjCNH8hcsWZ-j-8MfFwYjl_reru_wU_w&a mp;sig2=7pxIPBkobe-P3D0nz3jK4Q>
>>>>>>
>>>>>>
>>>
>>>
>
>
|
|
|
| Re: [EMF] [Teneo] : benchmarks ? And Return of Experience ? [message #423755 is a reply to message #423649] |
Tue, 07 October 2008 05:51 |
Andrew H
Messages: 117
Registered: July 2009 |
Senior Member |
|
|
Thanks for the help Martin.
See inline
Martin Taal wrote:
> Hi Andrew,
> See my comments inline.
> gr. Martin
> andrew987650000@yahoo.com.au wrote:
>> We have a pretty large model ( > 800 classifiers). I've been playing
>> around with persistence via teneo this last week. I've just now got the
>> basic thing working and need to look at some of the bottlenecks I've
>> hit. Most I think revolve around eager fetching.
>>
>> The DB schema that teneo / hibernate have generated has > 500 tables
>> which is pretty large model in my book.
>>
>> This is currently hurting on start up as hibernate evaluates what needs
>> to change. I'm assuming hibernate has a way to turn that off but haven't
>> pursued that yet.
> MT>> As a default if you don't set the Environment.HBM2DDL_AUTO hibernate
property then Teneo will
> set it to update, so if you set it to none (undocumented feature of
hibernate afaik) it should
> prevent teneo from setting it.
> I am working with a 425 table model and hibernate is initialized in about 10
seconds.
> In my case I don't update the database schema so probably this is were the
time is spend by hibernate.
That did the trick. Starts much quicker now.
>>
>> On the performance front I'm about to look into tweaking a few things
>> such as:
>>
>> - selectively turning off eager fetching of containment on queries. In
>> general I want it to do this as each object we will pull out of the db
>> is a pretty large (and diverse) tree. If I needed to specify what to
>> fetch in each query I would get a very large amount of querying code.
> MT>> depends on how much time you want to spend on this but an idea I got
when reading this is that
> you can write your own query generator using your model which is available
at runtime. Then you can
> parameter control your query generator to use eager fetching or not in
different locations.
Interesting. How would I go about doing that?
>> - However, on some operations (particularly when I persist new
>> versions of these objects) I need to navigate some one to many
>> containment relationships to add this one to the list. I think it is
>> eager fetching all the data on every version at the moment which is
>> causing it to grind. I think hibernate allows you to set fetch
>> strategies on individual criteria.
> MT>> I am not sure if you use bags or lists but you should maybe try
hibernates idbag or persist
> lists as bags (so non-ordered). Lists are slower to persist/update than bags
and much slower than
> idbags.
> Another thing to watch for is that EMF sends notifications on the basis of
changes in objects,
> sometimes these notifications result in loading of data at unexpected
moments in time. Although it
> can't be prevented always (EMF sends the notifications for a reason) it can
be good to be aware of it.
Good to know. I'll keep these options in mind when tuning collections.
>>
>> - large inheritance trees. I have one class in particular - Product -
>> that has many subtypes. Hibernate is doing a query on product that joins
>> on to every subtype. Since its only ever one of these this is wasteful.
>> I set the MAX_FETCH_DEPTH to 0 to try to stop this but it still does it.
>> I suspect I may need custom mappings on Product somehow to turn this off.
> MT>> Afaik the fetch_depth is related to associations and not so much depth
of the inheritance tree.
> Another option would be to use single table inheritance for this inheritance
tree. Hibernate has a
> limitation though that a whole inheritance tree should use one inheritance
strategy for the whole
> tree. So you can do join_table for the top of the tree and single table for
branches. But if your
> product inheritance tree is an independent tree then it can be set to single
table using the
> inheritance annotation.
I actually started with single table but changed it to join when I saw
what was generated. I had naively hoped that hibernate would change
approach in this case and not do the extra joins, buy was wrong. Having
all the product heirarchy flattened in this case is pretty ugly so I'm
going to leave it as it is now. Annoying that there doesn't seem to be a
way to stop hibernate doing the queries like this. But I think we can
probably live with it.
>>
>> - Denormalising some classes. I believe there is an Embeddable concept
>> in EMF that may help this. Also I may choose different inheritance
>> strategies for different hierarchies.
> MT>> Yes, that was also my idea.
>>
>> Hopefully next week I will have more info on my experiences working with
>> a large model
>>
>> cheers
>>
>> Andrew
>>
>>
>> Martin Taal wrote:
>>> Hi Tristan,
>>> As Ed pointed out, it always depends. You should use one of the later
>>> versions of Teneo as the 1.0.0 release had a performance issue with
>>> large sets of data (10000s of objects in mem).
>>>
>>> If you have performance issues then I would definitely enable
>>> hibernate sql logging so you can see what queries are fired by
>>> hibernate at which point. This has helped me a lot to find issues here
>>> and there.
>>>
>>> EMF uses EMF proxying to prevent eager loading. However
>>> Teneo/Hibernate does not use that at the moment, Teneo/hibernate uses
>>> the standard lazy loading mechanism of hibernate (cglib and lazy
>>> loaded lists). So both native EMF and EMF/Teneo/Hibernate support
>>> incremental loading.
>>>
>>> gr. Martin
>>>
>>> Tristan FAURE wrote:
>>>> tahnk you ed for you remarks and maybe sorry for my question a little
>>>> bit too evasive
>>>>
>>>> I'll check your links and if anybody has some experience return he is
>>>> welcome.
>>>>
>>>> And about model limit size it doens't depends on the physical size ?
>>>> Does EMF loads models incrementally ?
>>>>
>>>> Ed Merks a écrit :
>>>>> Tristan,
>>>>>
>>>>> Comments below.
>>>>>
>>>>> Tristan FAURE wrote:
>>>>>> Hi
>>>>>>
>>>>>> where can i found some information about EMF benchmarks ?
>>>>> We used to run micro benchmark performance tests, but we've not been
>>>>> doing that for a while. Loading performance depends a lot on
>>>>> options, the complexity of the model, not just the size, i.e,. lots
>>>>> of cross references will make things slower.
>>>>>> Do you know if there is a model limit size (using XMI) ?
>>>>> The size of the heap?
>>>>>> For a teneo use is there some speed benchmarks ?
>>>>>>
>>>>>> Is somebody try to use EMF and Teneo for very huge models
>>>>> Searching the newsgroups will turn up previous discussions (the
>>>>> second search both lower down):
>>>>>
>>>>>
>>>>> http://www.eclipse.org/newsportal/thread.php?group=eclipse.t ools.emf
>>>>>
>>>>>
http://www.eclipse.org/newsportal/thread.php?group=eclipse.t echnology.emft
>>>>>
>>>>>
>>>>>
>>>>>> (Actually i can't quantify what is the limit between models and
>>>>>> huge models ;)) ?
>>>>> Exactly. It's relative.
>>>>>> what kind of problem do you have ?
>>>>> Eventually you run out of memory.
>>>>>> Do you need to apply a process to avoid performance problem ?
>>>>>>
>>>>>>
>>>>>> /EMF Performance Tips/
>>>>>>
>>>>>>
< http://www.google.ca/url?sa=t&source=web&ct=res& cd=1&url=http%3A%2F%2Fwww.eclipse.org%2Fmodeling%2Femf%2 Fdocs%2Fperformance%2FEMFPerformanceTips.html&ei=-jbmSMT PJIqO8wTn5735Cw&usg=AFQjCNH8hcsWZ-j-8MfFwYjl_reru_wU_w&a mp;sig2=7pxIPBkobe-P3D0nz3jK4Q>
>>>>>>
>>>>>>
>>>
>>>
|
|
|
| Re: [EMF] [Teneo] : benchmarks ? And Return of Experience ? [message #423767 is a reply to message #423755] |
Tue, 07 October 2008 08:38 |
Martin Taal
Messages: 5468
Registered: July 2009 |
Senior Member |
|
|
Hi Andrew,
To write your own query generator you have to traverse the in-memory model and create a query on the
basis of the in-memory model.
For example this piece of code (not necessarily syntacticallly correct) will do a query for each
eclass in the system:
Session s = dataStore.getSessionFactory().getSession();
for (EClass eClass : ePackage.getEClasses()) {
final Query qry = s.createQuery("from " + eClass.getName());
System.err.println("Querying for " + eClass.getName());
for (Object o : qry.list()) {
System.err.println(o);
}
}
and the following code will generate an eager join with each many-to-one of an eclass. So for your
query generator you have to find some parameters to control what gets joined. Note that you can for
example add an eannotation in your model to classify the ereferences and use that classification
(the annotation) in the query generator to generate the correct eager joins.
public List<Object> queryEagerly(EClass eClass, Session session) {
final StringBuilder sb = new StringBuilder();
sb.append("select e from " + eClass.getName() + "
System.err.println("Querying for " + eClass.getName());
for (EReference eref : eClass.getEAllReferences()) {
if (!eref.isMany()) {
sb.append(" left outer join fetch e." + eref.getName());
}
}
// add the where clause here
final Query qry = session.createQuery(sb.toString());
return qry.list();
}
Hope this helps.
gr. Martin
Andrew H wrote:
> Thanks for the help Martin.
>
> See inline
>
> Martin Taal wrote:
>
>> Hi Andrew,
>> See my comments inline.
>
>> gr. Martin
>
>> andrew987650000@yahoo.com.au wrote:
>>> We have a pretty large model ( > 800 classifiers). I've been playing
>>> around with persistence via teneo this last week. I've just now got
>>> the basic thing working and need to look at some of the bottlenecks
>>> I've hit. Most I think revolve around eager fetching.
>>>
>>> The DB schema that teneo / hibernate have generated has > 500 tables
>>> which is pretty large model in my book.
>>>
>>> This is currently hurting on start up as hibernate evaluates what
>>> needs to change. I'm assuming hibernate has a way to turn that off
>>> but haven't pursued that yet.
>> MT>> As a default if you don't set the Environment.HBM2DDL_AUTO hibernate
> property then Teneo will
>> set it to update, so if you set it to none (undocumented feature of
> hibernate afaik) it should
>> prevent teneo from setting it.
>> I am working with a 425 table model and hibernate is initialized in
>> about 10
> seconds.
>> In my case I don't update the database schema so probably this is were
>> the
> time is spend by hibernate.
>
> That did the trick. Starts much quicker now.
>
>
>>>
>>> On the performance front I'm about to look into tweaking a few things
>>> such as:
>>>
>>> - selectively turning off eager fetching of containment on queries.
>>> In general I want it to do this as each object we will pull out of
>>> the db is a pretty large (and diverse) tree. If I needed to specify
>>> what to fetch in each query I would get a very large amount of
>>> querying code.
>> MT>> depends on how much time you want to spend on this but an idea I got
> when reading this is that
>> you can write your own query generator using your model which is
>> available
> at runtime. Then you can
>> parameter control your query generator to use eager fetching or not in
> different locations.
>
> Interesting. How would I go about doing that?
>
>>> - However, on some operations (particularly when I persist new
>>> versions of these objects) I need to navigate some one to many
>>> containment relationships to add this one to the list. I think it is
>>> eager fetching all the data on every version at the moment which is
>>> causing it to grind. I think hibernate allows you to set fetch
>>> strategies on individual criteria.
>> MT>> I am not sure if you use bags or lists but you should maybe try
> hibernates idbag or persist
>> lists as bags (so non-ordered). Lists are slower to persist/update
>> than bags
> and much slower than
>> idbags.
>> Another thing to watch for is that EMF sends notifications on the
>> basis of
> changes in objects,
>> sometimes these notifications result in loading of data at unexpected
> moments in time. Although it
>> can't be prevented always (EMF sends the notifications for a reason)
>> it can
> be good to be aware of it.
>
> Good to know. I'll keep these options in mind when tuning collections.
>
>>>
>>> - large inheritance trees. I have one class in particular - Product -
>>> that has many subtypes. Hibernate is doing a query on product that
>>> joins on to every subtype. Since its only ever one of these this is
>>> wasteful. I set the MAX_FETCH_DEPTH to 0 to try to stop this but it
>>> still does it. I suspect I may need custom mappings on Product
>>> somehow to turn this off.
>> MT>> Afaik the fetch_depth is related to associations and not so much
>> depth
> of the inheritance tree.
>> Another option would be to use single table inheritance for this
>> inheritance
> tree. Hibernate has a
>> limitation though that a whole inheritance tree should use one
>> inheritance
> strategy for the whole
>> tree. So you can do join_table for the top of the tree and single
>> table for
> branches. But if your
>> product inheritance tree is an independent tree then it can be set to
>> single
> table using the
>> inheritance annotation.
>
> I actually started with single table but changed it to join when I saw
> what was generated. I had naively hoped that hibernate would change
> approach in this case and not do the extra joins, buy was wrong. Having
> all the product heirarchy flattened in this case is pretty ugly so I'm
> going to leave it as it is now. Annoying that there doesn't seem to be a
> way to stop hibernate doing the queries like this. But I think we can
> probably live with it.
>
>>>
>>> - Denormalising some classes. I believe there is an Embeddable
>>> concept in EMF that may help this. Also I may choose different
>>> inheritance strategies for different hierarchies.
>> MT>> Yes, that was also my idea.
>
>>>
>>> Hopefully next week I will have more info on my experiences working
>>> with a large model
>>>
>>> cheers
>>>
>>> Andrew
>>>
>>>
>>> Martin Taal wrote:
>>>> Hi Tristan,
>>>> As Ed pointed out, it always depends. You should use one of the
>>>> later versions of Teneo as the 1.0.0 release had a performance issue
>>>> with large sets of data (10000s of objects in mem).
>>>>
>>>> If you have performance issues then I would definitely enable
>>>> hibernate sql logging so you can see what queries are fired by
>>>> hibernate at which point. This has helped me a lot to find issues
>>>> here and there.
>>>>
>>>> EMF uses EMF proxying to prevent eager loading. However
>>>> Teneo/Hibernate does not use that at the moment, Teneo/hibernate
>>>> uses the standard lazy loading mechanism of hibernate (cglib and
>>>> lazy loaded lists). So both native EMF and EMF/Teneo/Hibernate
>>>> support incremental loading.
>>>>
>>>> gr. Martin
>>>>
>>>> Tristan FAURE wrote:
>>>>> tahnk you ed for you remarks and maybe sorry for my question a
>>>>> little bit too evasive
>>>>>
>>>>> I'll check your links and if anybody has some experience return he
>>>>> is welcome.
>>>>>
>>>>> And about model limit size it doens't depends on the physical size
>>>>> ? Does EMF loads models incrementally ?
>>>>>
>>>>> Ed Merks a écrit :
>>>>>> Tristan,
>>>>>>
>>>>>> Comments below.
>>>>>>
>>>>>> Tristan FAURE wrote:
>>>>>>> Hi
>>>>>>>
>>>>>>> where can i found some information about EMF benchmarks ?
>>>>>> We used to run micro benchmark performance tests, but we've not
>>>>>> been doing that for a while. Loading performance depends a lot on
>>>>>> options, the complexity of the model, not just the size, i.e,.
>>>>>> lots of cross references will make things slower.
>>>>>>> Do you know if there is a model limit size (using XMI) ?
>>>>>> The size of the heap?
>>>>>>> For a teneo use is there some speed benchmarks ?
>>>>>>>
>>>>>>> Is somebody try to use EMF and Teneo for very huge models
>>>>>> Searching the newsgroups will turn up previous discussions (the
>>>>>> second search both lower down):
>>>>>>
>>>>>>
>>>>>> http://www.eclipse.org/newsportal/thread.php?group=eclipse.t ools.emf
>>>>>>
> http://www.eclipse.org/newsportal/thread.php?group=eclipse.t echnology.emft
>>>>>>
>>>>>>
>>>>>>
>>>>>>> (Actually i can't quantify what is the limit between models and
>>>>>>> huge models ;)) ?
>>>>>> Exactly. It's relative.
>>>>>>> what kind of problem do you have ?
>>>>>> Eventually you run out of memory.
>>>>>>> Do you need to apply a process to avoid performance problem ?
>>>>>>>
>>>>>>>
>>>>>>> /EMF Performance Tips/
>>>>>>>
> < http://www.google.ca/url?sa=t&source=web&ct=res& cd=1&url=http%3A%2F%2Fwww.eclipse.org%2Fmodeling%2Femf%2 Fdocs%2Fperformance%2FEMFPerformanceTips.html&ei=-jbmSMT PJIqO8wTn5735Cw&usg=AFQjCNH8hcsWZ-j-8MfFwYjl_reru_wU_w&a mp;sig2=7pxIPBkobe-P3D0nz3jK4Q>
>
>>>>>>>
>>>>>>>
>>>>
>>>>
>
>
>
>
--
With Regards, Martin Taal
Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Cell: +31 (0)6 288 48 943
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org
|
|
|
| Re: [EMF] [Teneo] : benchmarks ? And Return of Experience ? [message #423846 is a reply to message #423767] |
Tue, 07 October 2008 23:19 |
Andrew H
Messages: 117
Registered: July 2009 |
Senior Member |
|
|
Thanks Martin
That's a very interesting idea. I think that will be very helpful for
implementing some of the searches I need to do
cheers
Andrew
Martin Taal wrote:
> Hi Andrew,
> To write your own query generator you have to traverse the in-memory model
and create a query on the
> basis of the in-memory model.
> For example this piece of code (not necessarily syntacticallly correct) will
do a query for each
> eclass in the system:
> Session s = dataStore.getSessionFactory().getSession();
> for (EClass eClass : ePackage.getEClasses()) {
> final Query qry = s.createQuery("from " + eClass.getName());
> System.err.println("Querying for " + eClass.getName());
> for (Object o : qry.list()) {
> System.err.println(o);
> }
> }
> and the following code will generate an eager join with each many-to-one of
an eclass. So for your
> query generator you have to find some parameters to control what gets
joined. Note that you can for
> example add an eannotation in your model to classify the ereferences and use
that classification
> (the annotation) in the query generator to generate the correct eager joins.
> public List<Object> queryEagerly(EClass eClass, Session session) {
> final StringBuilder sb = new StringBuilder();
> sb.append("select e from " + eClass.getName() + "
> System.err.println("Querying for " + eClass.getName());
> for (EReference eref : eClass.getEAllReferences()) {
> if (!eref.isMany()) {
> sb.append(" left outer join fetch e." + eref.getName());
> }
> }
> // add the where clause here
> final Query qry = session.createQuery(sb.toString());
> return qry.list();
> }
> Hope this helps.
> gr. Martin
> Andrew H wrote:
>> Thanks for the help Martin.
>>
>> See inline
>>
>> Martin Taal wrote:
>>
>>> Hi Andrew,
>>> See my comments inline.
>>
>>> gr. Martin
>>
>>> andrew987650000@yahoo.com.au wrote:
>>>> We have a pretty large model ( > 800 classifiers). I've been playing
>>>> around with persistence via teneo this last week. I've just now got
>>>> the basic thing working and need to look at some of the bottlenecks
>>>> I've hit. Most I think revolve around eager fetching.
>>>>
>>>> The DB schema that teneo / hibernate have generated has > 500 tables
>>>> which is pretty large model in my book.
>>>>
>>>> This is currently hurting on start up as hibernate evaluates what
>>>> needs to change. I'm assuming hibernate has a way to turn that off
>>>> but haven't pursued that yet.
>>> MT>> As a default if you don't set the Environment.HBM2DDL_AUTO hibernate
>> property then Teneo will
>>> set it to update, so if you set it to none (undocumented feature of
>> hibernate afaik) it should
>>> prevent teneo from setting it.
>>> I am working with a 425 table model and hibernate is initialized in
>>> about 10
>> seconds.
>>> In my case I don't update the database schema so probably this is were
>>> the
>> time is spend by hibernate.
>>
>> That did the trick. Starts much quicker now.
>>
>>
>>>>
>>>> On the performance front I'm about to look into tweaking a few things
>>>> such as:
>>>>
>>>> - selectively turning off eager fetching of containment on queries.
>>>> In general I want it to do this as each object we will pull out of
>>>> the db is a pretty large (and diverse) tree. If I needed to specify
>>>> what to fetch in each query I would get a very large amount of
>>>> querying code.
>>> MT>> depends on how much time you want to spend on this but an idea I got
>> when reading this is that
>>> you can write your own query generator using your model which is
>>> available
>> at runtime. Then you can
>>> parameter control your query generator to use eager fetching or not in
>> different locations.
>>
>> Interesting. How would I go about doing that?
>>
>>>> - However, on some operations (particularly when I persist new
>>>> versions of these objects) I need to navigate some one to many
>>>> containment relationships to add this one to the list. I think it is
>>>> eager fetching all the data on every version at the moment which is
>>>> causing it to grind. I think hibernate allows you to set fetch
>>>> strategies on individual criteria.
>>> MT>> I am not sure if you use bags or lists but you should maybe try
>> hibernates idbag or persist
>>> lists as bags (so non-ordered). Lists are slower to persist/update
>>> than bags
>> and much slower than
>>> idbags.
>>> Another thing to watch for is that EMF sends notifications on the
>>> basis of
>> changes in objects,
>>> sometimes these notifications result in loading of data at unexpected
>> moments in time. Although it
>>> can't be prevented always (EMF sends the notifications for a reason)
>>> it can
>> be good to be aware of it.
>>
>> Good to know. I'll keep these options in mind when tuning collections.
>>
>>>>
>>>> - large inheritance trees. I have one class in particular - Product -
>>>> that has many subtypes. Hibernate is doing a query on product that
>>>> joins on to every subtype. Since its only ever one of these this is
>>>> wasteful. I set the MAX_FETCH_DEPTH to 0 to try to stop this but it
>>>> still does it. I suspect I may need custom mappings on Product
>>>> somehow to turn this off.
>>> MT>> Afaik the fetch_depth is related to associations and not so much
>>> depth
>> of the inheritance tree.
>>> Another option would be to use single table inheritance for this
>>> inheritance
>> tree. Hibernate has a
>>> limitation though that a whole inheritance tree should use one
>>> inheritance
>> strategy for the whole
>>> tree. So you can do join_table for the top of the tree and single
>>> table for
>> branches. But if your
>>> product inheritance tree is an independent tree then it can be set to
>>> single
>> table using the
>>> inheritance annotation.
>>
>> I actually started with single table but changed it to join when I saw
>> what was generated. I had naively hoped that hibernate would change
>> approach in this case and not do the extra joins, buy was wrong. Having
>> all the product heirarchy flattened in this case is pretty ugly so I'm
>> going to leave it as it is now. Annoying that there doesn't seem to be a
>> way to stop hibernate doing the queries like this. But I think we can
>> probably live with it.
>>
>>>>
>>>> - Denormalising some classes. I believe there is an Embeddable
>>>> concept in EMF that may help this. Also I may choose different
>>>> inheritance strategies for different hierarchies.
>>> MT>> Yes, that was also my idea.
>>
>>>>
>>>> Hopefully next week I will have more info on my experiences working
>>>> with a large model
>>>>
>>>> cheers
>>>>
>>>> Andrew
>>>>
>>>>
>>>> Martin Taal wrote:
>>>>> Hi Tristan,
>>>>> As Ed pointed out, it always depends. You should use one of the
>>>>> later versions of Teneo as the 1.0.0 release had a performance issue
>>>>> with large sets of data (10000s of objects in mem).
>>>>>
>>>>> If you have performance issues then I would definitely enable
>>>>> hibernate sql logging so you can see what queries are fired by
>>>>> hibernate at which point. This has helped me a lot to find issues
>>>>> here and there.
>>>>>
>>>>> EMF uses EMF proxying to prevent eager loading. However
>>>>> Teneo/Hibernate does not use that at the moment, Teneo/hibernate
>>>>> uses the standard lazy loading mechanism of hibernate (cglib and
>>>>> lazy loaded lists). So both native EMF and EMF/Teneo/Hibernate
>>>>> support incremental loading.
>>>>>
>>>>> gr. Martin
>>>>>
>>>>> Tristan FAURE wrote:
>>>>>> tahnk you ed for you remarks and maybe sorry for my question a
>>>>>> little bit too evasive
>>>>>>
>>>>>> I'll check your links and if anybody has some experience return he
>>>>>> is welcome.
>>>>>>
>>>>>> And about model limit size it doens't depends on the physical size
>>>>>> ? Does EMF loads models incrementally ?
>>>>>>
>>>>>> Ed Merks a écrit :
>>>>>>> Tristan,
>>>>>>>
>>>>>>> Comments below.
>>>>>>>
>>>>>>> Tristan FAURE wrote:
>>>>>>>> Hi
>>>>>>>>
>>>>>>>> where can i found some information about EMF benchmarks ?
>>>>>>> We used to run micro benchmark performance tests, but we've not
>>>>>>> been doing that for a while. Loading performance depends a lot on
>>>>>>> options, the complexity of the model, not just the size, i.e,.
>>>>>>> lots of cross references will make things slower.
>>>>>>>> Do you know if there is a model limit size (using XMI) ?
>>>>>>> The size of the heap?
>>>>>>>> For a teneo use is there some speed benchmarks ?
>>>>>>>>
>>>>>>>> Is somebody try to use EMF and Teneo for very huge models
>>>>>>> Searching the newsgroups will turn up previous discussions (the
>>>>>>> second search both lower down):
>>>>>>>
>>>>>>>
>>>>>>> http://www.eclipse.org/newsportal/thread.php?group=eclipse.t ools.emf
>>>>>>>
>> http://www.eclipse.org/newsportal/thread.php?group=eclipse.t echnology.emft
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> (Actually i can't quantify what is the limit between models and
>>>>>>>> huge models ;)) ?
>>>>>>> Exactly. It's relative.
>>>>>>>> what kind of problem do you have ?
>>>>>>> Eventually you run out of memory.
>>>>>>>> Do you need to apply a process to avoid performance problem ?
>>>>>>>>
>>>>>>>>
>>>>>>>> /EMF Performance Tips/
>>>>>>>>
>>
< http://www.google.ca/url?sa=t&source=web&ct=res& cd=1&url=http%3A%2F%2Fwww.eclipse.org%2Fmodeling%2Femf%2 Fdocs%2Fperformance%2FEMFPerformanceTips.html&ei=-jbmSMT PJIqO8wTn5735Cw&usg=AFQjCNH8hcsWZ-j-8MfFwYjl_reru_wU_w&a mp;sig2=7pxIPBkobe-P3D0nz3jK4Q>
>>
>>>>>>>>
>>>>>>>>
>>>>>
>>>>>
>>
>>
>>
>>
|
|
|
Goto Forum:
Current Time: Fri Apr 19 09:39:40 GMT 2024
Powered by FUDForum. Page generated in 0.08420 seconds |
] 
Search
Help
Register
Login
Home
