Eclipse Community Forums: EMF » My Model is getting to big...

Help

Home

Home » Modeling » EMF » My Model is getting to big...(My Model is getting to big...)

Show: Today's Messages :: Show Polls :: Message Navigator

My Model is getting to big... [message #894735]

Tue, 10 July 2012 10:17

Markus Jo

Messages: 83
Registered: January 2012

Member

Hi 2 all,
I have the Problem that I have a business model which needs more than 1GB main memory.

I have an import function which I use to load data from a server. The incoming data is mapped so new EObjects that are added to the model...the model in memory grows and grows and when I reach the one GB my system is breaking.

How can I master this ? Is CDO an option ? Is it much work to use CDO (only for minimizing the model memory) or is it simply already done by doing this:

http://wiki.eclipse.org/CDO/Preparing_EMF_Models

Thanks for help

Report message to a moderator

Re: My Model is getting to big... [message #894741 is a reply to message #894735]

Tue, 10 July 2012 10:29

Ed Merks

Messages: 33140
Registered: July 2009

Senior Member

Markus,

Comments below.

On 10/07/2012 12:17 PM, Markus Jo wrote:
> Hi 2 all,
> I have the Problem that I have a business model which needs more than
> 1GB main memory.
That's a big business. :-P
>
> I have an import function which I use to load data from a server. The
> incoming data is mapped so new EObjects that are added to the
> model...the model in memory grows and grows and when I reach the one
> GB my system is breaking.
You can increase the heap. A 64 bit VM can have a very large heap.
>
> How can I master this ? Is CDO an option ?
Yes, I think CDO can help with that.
> Is it much work to use CDO (only for minimizing the model memory) or
> is it simply already done by doing this:
> http://wiki.eclipse.org/CDO/Preparing_EMF_Models
That just helps generate a nicely CDO enable model. You'll still need to
learn about CDO transactions to manage resources that are stored in a
repository.
>
>
> Thanks for help

Ed Merks
Professional Support: https://www.macromodeling.com/

Report message to a moderator

Re: My Model is getting to big... [message #894742 is a reply to message #894741]

Tue, 10 July 2012 10:45

Markus Jo

Messages: 83
Registered: January 2012

Member

Thx for the fast answer.

The 64VM is no choice.... our customers expect a version for 32bit.

The Project is already in the second half and we are only allowed to make use of 1GB main memory. Would be no Problem with the inital demand....there we have 600mb data. But now we should be able to display two years (instead of one) of data and that´s to much.

The CDO approach seems to need to much time. Is there a solution just for solving the "big model in main memory" problem that we can 'easily' do by hand? Splitting to many model model files ? There is the "Control.." Action in the Editors TreeView Context Menu where you can set an own model file for some parts of the model. But this seems to be done by the client-user. Is there a way to do this automatically without reimplementing EMF ? Something like:

- Outsorcing every Object of type XY to an own ressource
- Unload it when possible
- Load it when needed
- Deleting the resource file when deleting the object in the client.

Its very hard to find the entry points for this in the EMF code.

Report message to a moderator

Re: My Model is getting to big... [message #894795 is a reply to message #894742]

Tue, 10 July 2012 14:36

Ed Merks

Messages: 33140
Registered: July 2009

Senior Member

Comments below.

On 10/07/2012 12:45 PM, Markus Jo wrote:
> Thx for the fast answer.
> The 64VM is no choice.... our customers expect a version for 32bit.
>
> The Project is already in the second half and we are only allowed to
> make use of 1GB main memory. Would be no Problem with the inital
> demand....there we have 600mb data. But now we should be able to
> display two years (instead of one) of data and that´s to much.
>
> The CDO approach seems to need to much time.
You mean it performs poorly?
> Is there a solution just for solving the "big model in main memory"
> problem that we can 'easily' do by hand? Splitting to many model model
> files ?
Yes, you can split objects across resources.
> There is the "Control.." Action in the Editors TreeView Context Menu
> where you can set an own model file for some parts of the model.
All that does it create a resource and put the object in that resource;
of course you can do that programmatically (resourceSet.createResource
and resource.getContents().add(eObject).
> But this seems to be done by the client-user. Is there a way to do
> this automatically without reimplementing EMF ? Something like:
>
> - Outsorcing every Object of type XY to an own ressource
> - Unload it when possible
Unloading doesn't free up the memory if there are other references to
the objects in the resource.
> - Load it when needed
> - Deleting the resource file when deleting the object in the client.
>
> Its very hard to find the entry points for this in the EMF code.
ResourceSet and Resource are your friends. They're not such big APIs.
>
>

Ed Merks
Professional Support: https://www.macromodeling.com/

Report message to a moderator

Re: My Model is getting to big... [message #894807 is a reply to message #894795]

Tue, 10 July 2012 15:15

Thomas Schindl

Messages: 6651
Registered: July 2009

Senior Member

Maybe a dumb question you already using MinimalEObject? It will at least
save some bits ;-)

Tom

Am 10.07.12 16:36, schrieb Ed Merks:
> Comments below.
>
> On 10/07/2012 12:45 PM, Markus Jo wrote:
>> Thx for the fast answer.
>> The 64VM is no choice.... our customers expect a version for 32bit.
>>
>> The Project is already in the second half and we are only allowed to
>> make use of 1GB main memory. Would be no Problem with the inital
>> demand....there we have 600mb data. But now we should be able to
>> display two years (instead of one) of data and that´s to much.
>>
>> The CDO approach seems to need to much time.
> You mean it performs poorly?
>> Is there a solution just for solving the "big model in main memory"
>> problem that we can 'easily' do by hand? Splitting to many model model
>> files ?
> Yes, you can split objects across resources.
>> There is the "Control.." Action in the Editors TreeView Context Menu
>> where you can set an own model file for some parts of the model.
> All that does it create a resource and put the object in that resource;
> of course you can do that programmatically (resourceSet.createResource
> and resource.getContents().add(eObject).
>> But this seems to be done by the client-user. Is there a way to do
>> this automatically without reimplementing EMF ? Something like:
>>
>> - Outsorcing every Object of type XY to an own ressource
>> - Unload it when possible
> Unloading doesn't free up the memory if there are other references to
> the objects in the resource.
>> - Load it when needed
>> - Deleting the resource file when deleting the object in the client.
>>
>> Its very hard to find the entry points for this in the EMF code.
> ResourceSet and Resource are your friends. They're not such big APIs.
>>
>>
>
>

Report message to a moderator

Re: My Model is getting to big... [message #894927 is a reply to message #894807]

Wed, 11 July 2012 06:29

Markus Jo

Messages: 83
Registered: January 2012

Member

@Ed
No, I didnt mean JDO performs poorly.... I brain performs that poorly that I think I would need to much project time to be able to use CDO properly. But thanks for your hints. I am currenty trying to swap some Elements to other resources when the doSave(..) method of the MetamodelEditor is triggered.
One question to that....If I move an Object to antoher ressource and save this resource before saving the "main"-resource, do I have to remove the moved object from the resource or does EMF do this for me ?

@Thomas
Hi, I found a presentation of that but there were just one or two lines about that and I did not found an implementation of MinimalEObject yet. Do you have the class ? How do I activate it ? By specifying its path in the genmodel under "Root Extends Class/Interface" ?

Report message to a moderator

Re: My Model is getting to big... [message #894935 is a reply to message #894927]

Wed, 11 July 2012 07:46

Markus Jo

Messages: 83
Registered: January 2012

Member

@Ed
My question is answered....it removes the swapped elements automatically.

@All
But....the little memory window at the lower right of my Application still shows the same high value (after clicking some times on the garbage collector icon). I swapped nearly half size of the main model file to little fragments whichs contents are not displayed/expanded in the clients tree view. So I thought they would not be loaded....not before I expand the model tree in the client to theirs levels. Do I have to activate some kind of lazy loading ? Or is it propable that there is some kind of initilaization in the background that causes loading all fragement model files to the main memory ?

Report message to a moderator

Re: My Model is getting to big... [message #894940 is a reply to message #894935]

Wed, 11 July 2012 07:52

Eike Stepper

Messages: 6682
Registered: July 2009

Senior Member

Am 11.07.2012 09:46, schrieb Markus Jo:
> @Ed
> My question is answered....it removes the swapped elements automatically.
>
> @All
> But....the little memory window at the lower right of my Application still shows the same high value (after clicking
> some times on the garbage collector icon). I swapped nearly half size of the main model file to little fragments
> whichs contents are not displayed/expanded in the clients tree view. So I thought they would not be loaded....not
> before I expand the model tree in the client to theirs levels. Do I have to activate some kind of lazy loading ? Or is
> it propable that there is some kind of initilaization in the background that causes loading all fragement model files
> to the main memory ?
You may find my article http://thegordian.blogspot.de/2008/11/how-scalable-are-my-models.html interesting as it kind of
explains what's going on on your heap ;-)

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper

Report message to a moderator

Re: My Model is getting to big... [message #894942 is a reply to message #894927]

Wed, 11 July 2012 07:58

Eike Stepper

Messages: 6682
Registered: July 2009

Senior Member

Am 11.07.2012 08:29, schrieb Markus Jo:
> @Ed
> No, I didnt mean JDO performs poorly.... I brain performs that poorly that I think I would need to much project time
> to be able to use CDO properly.
I guess you mean CDO. From my experiences with CDO adopters I have the impression that many, if not most, of them have
managed the common scenarios in less than a day. Free and prompt support is given here. Please note that we may also be
able to support you in a more exclusive, commercial fashion: http://www.eclipse.org/cdo/support

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper

Report message to a moderator

Re: My Model is getting to big... [message #894945 is a reply to message #894935]

Wed, 11 July 2012 08:21

Ed Merks

Messages: 33140
Registered: July 2009

Senior Member

Markus,

Proxies are resolved on demand. You can monitor the contents of the
resource set, perhaps setting a breakproint in
ResourceSetImpl.demandLoad to see what in your application is causing
proxies to be resolved.

On 11/07/2012 9:46 AM, Markus Jo wrote:
> @Ed
> My question is answered....it removes the swapped elements automatically.
>
> @All
> But....the little memory window at the lower right of my Application
> still shows the same high value (after clicking some times on the
> garbage collector icon). I swapped nearly half size of the main model
> file to little fragments whichs contents are not displayed/expanded in
> the clients tree view. So I thought they would not be loaded....not
> before I expand the model tree in the client to theirs levels. Do I
> have to activate some kind of lazy loading ? Or is it propable that
> there is some kind of initilaization in the background that causes
> loading all fragement model files to the main memory ?

Ed Merks
Professional Support: https://www.macromodeling.com/

Report message to a moderator

Re: My Model is getting to big... [message #895239 is a reply to message #894945]

Thu, 12 July 2012 08:43

Markus Jo

Messages: 83
Registered: January 2012

Member

Hi Ed,
done it.....the resources were loaded when we add some Listener or ContentAdapters to our model...something like this:

EContentAdapter eContentAdapter = new ModelNotificationListener();
eContentAdapter.setTarget(modelRoot);

or like

editingDomain.getResourceSet().eAdapters().add(AccessControl.getInstance());

Then the whole model gets traversed and alle resources were loaded. Any Idea how we can prevent this ? Are there listeners or flag so set which only register on loaded resources and reregister when a resource is loaded later ?

Report message to a moderator

Re: My Model is getting to big... [message #895240 is a reply to message #895239]

Thu, 12 July 2012 08:47

Markus Jo

Messages: 83
Registered: January 2012

Member

Antoher question....unloading after the notifier are registered/added does not help the garbage collector, right ? Because the Notifier and ContentAdapters hold references to the model objects ?

Report message to a moderator

Re: My Model is getting to big... [message #895318 is a reply to message #895239]

Thu, 12 July 2012 12:46

Ed Merks

Messages: 33140
Registered: July 2009

Senior Member

Markus,

You can specialize the EContentAdapter's resolve method to return
false. Then it will start listening to after the proxy is resolved.

On 12/07/2012 10:43 AM, Markus Jo wrote:
> Hi Ed,
> done it.....the resources were loaded when we add some Listener or
> ContentAdapters to our model...something like this:
>
> EContentAdapter eContentAdapter = new ModelNotificationListener();
> eContentAdapter.setTarget(modelRoot);
>
> or like
>
> editingDomain.getResourceSet().eAdapters().add(AccessControl.getInstance());
>
>
>
> Then the whole model gets traversed and alle resources were loaded.
> Any Idea how we can prevent this ? Are there listeners or flag so set
> which only register on loaded resources and reregister when a resource
> is loaded later ?

Ed Merks
Professional Support: https://www.macromodeling.com/

Report message to a moderator

Re: My Model is getting to big... [message #895319 is a reply to message #895240]

Thu, 12 July 2012 12:47

Ed Merks

Messages: 33140
Registered: July 2009

Senior Member

Markus,

Unloading clears the eAdapters list of all the unloaded EObjects.

On 12/07/2012 10:47 AM, Markus Jo wrote:
> Antoher question....unloading after the notifier are registered/added
> does not help the garbage collector, right ? Because the Notifier and
> ContentAdapters hold references to the model objects ?

Ed Merks
Professional Support: https://www.macromodeling.com/

Report message to a moderator

Previous Topic:	Fits CDO to my needs?
Next Topic:	How to implement Tool tip for tree items

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Thu Apr 25 13:39:58 GMT 2024

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter