Eclipse Community Forums: EMF "Technology" (Ecore Tools, EMFatic, etc) » General architecture question on performance

Help

Home

Home » Modeling » EMF "Technology" (Ecore Tools, EMFatic, etc) » General architecture question on performance

Show: Today's Messages :: Show Polls :: Message Navigator

General architecture question on performance [message #64008]

Thu, 07 December 2006 20:29

Eclipse User

Originally posted by: kalin.nakov.gmail.com

Hi,

My application needs to handle very large models, mainly tens of
thousands objects, a number high enough so that loading and saving into
plain files and loading everything into memory is not an option. My
investigation leads me to using Teneo and JPOX resources. So far so
good, I guess I would be able to open, edit and validate my model stored
in a database and now I can do code generation with Jet based on a
Resource object (loaded EMF model). But I have the following
architecture questions before I start implementing my application to
avoid running into a dead end when scalability issues start to occur:

1) When I open a very huge model in the EMF editor, is all the model
loaded into the memory, or only the currently visible items in the EMF
editor? Is it loaded in a top-down manner by first loading the top
hierarchical levels and when I expand the tree it loads the child items
from the database?

2) When I save my document from the EMF editor, are only the changes
saved, or the whole file is written in the repository from scratch?

3) When I validate my document from the EMF editor, are all the objects
iterated and validated, or only the ones that I have changed?

4) When I use JET for code generation and use xpath queries on a EMF
model, are those queries optimized for relational database access, or
issuing a query like //MyClass will iterate all the objects in the database?

Thanks in advance,
Kalin

Report message to a moderator

Re: General architecture question on performance [message #64036 is a reply to message #64008]

Thu, 07 December 2006 21:07

Eclipse User

Originally posted by: merks.ca.ibm.com

Kalin,

Comments below.

Kalin Nakov wrote:
> Hi,
>
> My application needs to handle very large models, mainly tens of
> thousands objects, a number high enough so that loading and saving
> into plain files and loading everything into memory is not an option.
> My investigation leads me to using Teneo and JPOX resources. So far so
> good, I guess I would be able to open, edit and validate my model
> stored in a database and now I can do code generation with Jet based
> on a Resource object (loaded EMF model). But I have the following
> architecture questions before I start implementing my application to
> avoid running into a dead end when scalability issues start to occur:
>
> 1) When I open a very huge model in the EMF editor, is all the model
> loaded into the memory, or only the currently visible items in the EMF
> editor?
Some operations will want to visit the whole model. For example, when
you execute a delete, the editor will want to find all incoming
references to clean those up. Or when you use the properties view, the
editor will want to visit all objects to offer as choices the ones that
will fit into the feature.
> Is it loaded in a top-down manner by first loading the top
> hierarchical levels and when I expand the tree it loads the child
> items from the database?
Generally the root note is displayed and things are loaded on demand as
you visit them.
>
> 2) When I save my document from the EMF editor, are only the changes
> saved, or the whole file is written in the repository from scratch?
The default resource implementation loads the whole resource (but does
not follow links/proxies to other resources) and saves the whole resource.
>
> 3) When I validate my document from the EMF editor, are all the
> objects iterated and validated, or only the ones that I have changed?
All objects are visited.
>
> 4) When I use JET for code generation and use xpath queries on a EMF
> model, are those queries optimized for relational database access, or
> issuing a query like //MyClass will iterate all the objects in the
> database?
No, they walk they in memory model.
>
> Thanks in advance,
> Kalin

Report message to a moderator

Re: General architecture question on performance [message #64131 is a reply to message #64036]

Fri, 08 December 2006 04:36

Martin Taal

Messages: 5468
Registered: July 2009

Senior Member

Kalin,
Just to add some jpox specific info. Jpox will normally lazy load references. This means that lists
are only loaded at the moment that the first entry in the list is required (or with operations as
size()). So initially when you load the resource from the database only the top level objects should
be loaded, their children are still in the db (and not loaded).

As Ed pointed out there are several operations which force EMF to iterate over all objects. To
prevent this you can try to override EMF's iterate behavior. The lists used by Teneo
(org.eclipse.emf.teneo.jpox.elist.EListWrapper) has an isLoaded method. When you have your own
iterate logic you can for example skip lists which not have been loaded yet. This only applies if
the logic iterating through the model does not mind empty lists ofcourse.

Regarding filling the listbox choice you can try to replace this with jdo-queries to the database.

The JpoxDataStore offers methods to get all crossreferences to a certain object
(JpoxDataStore.getCrossReferencers).

gr. Martin

Ed Merks wrote:
> Kalin,
>
> Comments below.
>
> Kalin Nakov wrote:
>> Hi,
>>
>> My application needs to handle very large models, mainly tens of
>> thousands objects, a number high enough so that loading and saving
>> into plain files and loading everything into memory is not an option.
>> My investigation leads me to using Teneo and JPOX resources. So far so
>> good, I guess I would be able to open, edit and validate my model
>> stored in a database and now I can do code generation with Jet based
>> on a Resource object (loaded EMF model). But I have the following
>> architecture questions before I start implementing my application to
>> avoid running into a dead end when scalability issues start to occur:
>>
>> 1) When I open a very huge model in the EMF editor, is all the model
>> loaded into the memory, or only the currently visible items in the EMF
>> editor?
> Some operations will want to visit the whole model. For example, when
> you execute a delete, the editor will want to find all incoming
> references to clean those up. Or when you use the properties view, the
> editor will want to visit all objects to offer as choices the ones that
> will fit into the feature.
>> Is it loaded in a top-down manner by first loading the top
>> hierarchical levels and when I expand the tree it loads the child
>> items from the database?
> Generally the root note is displayed and things are loaded on demand as
> you visit them.
>>
>> 2) When I save my document from the EMF editor, are only the changes
>> saved, or the whole file is written in the repository from scratch?
> The default resource implementation loads the whole resource (but does
> not follow links/proxies to other resources) and saves the whole resource.
>>
>> 3) When I validate my document from the EMF editor, are all the
>> objects iterated and validated, or only the ones that I have changed?
> All objects are visited.
>>
>> 4) When I use JET for code generation and use xpath queries on a EMF
>> model, are those queries optimized for relational database access, or
>> issuing a query like //MyClass will iterate all the objects in the
>> database?
> No, they walk they in memory model.
>>
>> Thanks in advance,
>> Kalin

--

With Regards, Martin Taal

Springsite/Elver.org
Office: Hardwareweg 4, 3821 BV Amersfoort
Postal: Nassaulaan 7, 3941 EC Doorn
The Netherlands
Tel: +31 (0)84 420 2397
Fax: +31 (0)84 225 9307
Mail: mtaal@springsite.com - mtaal@elver.org
Web: www.springsite.com - www.elver.org

Report message to a moderator

Re: General architecture question on performance [message #596664 is a reply to message #64008]

Thu, 07 December 2006 21:07

Ed Merks

Messages: 33140
Registered: July 2009

Senior Member

Kalin,

Comments below.

Kalin Nakov wrote:
> Hi,
>
> My application needs to handle very large models, mainly tens of
> thousands objects, a number high enough so that loading and saving
> into plain files and loading everything into memory is not an option.
> My investigation leads me to using Teneo and JPOX resources. So far so
> good, I guess I would be able to open, edit and validate my model
> stored in a database and now I can do code generation with Jet based
> on a Resource object (loaded EMF model). But I have the following
> architecture questions before I start implementing my application to
> avoid running into a dead end when scalability issues start to occur:
>
> 1) When I open a very huge model in the EMF editor, is all the model
> loaded into the memory, or only the currently visible items in the EMF
> editor?
Some operations will want to visit the whole model. For example, when
you execute a delete, the editor will want to find all incoming
references to clean those up. Or when you use the properties view, the
editor will want to visit all objects to offer as choices the ones that
will fit into the feature.
> Is it loaded in a top-down manner by first loading the top
> hierarchical levels and when I expand the tree it loads the child
> items from the database?
Generally the root note is displayed and things are loaded on demand as
you visit them.
>
> 2) When I save my document from the EMF editor, are only the changes
> saved, or the whole file is written in the repository from scratch?
The default resource implementation loads the whole resource (but does
not follow links/proxies to other resources) and saves the whole resource.
>
> 3) When I validate my document from the EMF editor, are all the
> objects iterated and validated, or only the ones that I have changed?
All objects are visited.
>
> 4) When I use JET for code generation and use xpath queries on a EMF
> model, are those queries optimized for relational database access, or
> issuing a query like //MyClass will iterate all the objects in the
> database?
No, they walk they in memory model.
>
> Thanks in advance,
> Kalin

Ed Merks
Professional Support: https://www.macromodeling.com/

Report message to a moderator