Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Modeling » UML2 » CacheAdapter.inverseCrossReferencer and Memory Leaks
CacheAdapter.inverseCrossReferencer and Memory Leaks [message #690216] Wed, 29 June 2011 03:45 Go to next message
Kirsten M. Z. is currently offline Kirsten M. Z.
Messages: 131
Registered: July 2010
Senior Member
Hi @all,

I am struggling against some memory issues, and I have been analyzing for several days now. As far as I can see, using MAT, the problem could be caused by CacheAdapter (i.e. its inverseCrossReferencer) which is not removing old resources.

I have coded a (graphical) UML editor which uses EMF/UML2 internally. Whenever I close an old model and reload it, I waste about 30 MBytes (a small model) which is far to much for ignoring this problem.

Learning from other threads on this board and own investigations, I already use the following code in order to unload my resources and resource set (BTW, there is a regular UML model and several UML profiles in my resource set):

while(getResourceSet().getResources().size() > 0) {
	Resource res = getResourceSet().getResources().get(0);
	res.eAdapters().clear();
	res.unload();
				
	getResourceSet().getResources().remove(res);
}		
CacheAdapter.INSTANCE.clear();


I found the following threads or bug entries within this forum about a similar topic. However, bugs seem to be fixed and the issues solved.

http://www.eclipse.org/forums/index.php/m/624261
http://www.eclipse.org/forums/index.php/m/647513
https://bugs.eclipse.org/bugs/show_bug.cgi?id=197079

Now, I need to get some background information. I already debugged into sources, but sorry, I cannot really figure out the mechanism why (or how) the InverseCrossReferencer does not remove old stuff when unloading. I also haven't found a way to reset the state completely or to use another CacheAdapter, which could also solve the problem. Maybe, CacheAdapter$inverseCrossReferencer is not the only problem (I am still wondering if I can trust MAT), so your experience would help. Please tell me scenarios in which clearing the cache or unloading could fail or even tutorials on unloading.

Thanks in adavance!

[Updated on: Wed, 29 June 2011 03:47]

Report message to a moderator

Re: CacheAdapter.inverseCrossReferencer and Memory Leaks [message #690230 is a reply to message #690216] Wed, 29 June 2011 04:23 Go to previous messageGo to next message
Vlad Varnica is currently offline Vlad Varnica
Messages: 546
Registered: July 2009
Location: Milton Keynes - UK
Senior Member
This memory use has always been existing and it is almost impossible to solve it today.
You can only try to deactivate few recursive methods and it would reduce it by over 50%. I don't know what to deactivate because this job has been done many year ago in our tool and the developer who did the job left the company.
I would say that the memory size is going down with the size of the project. I mean that for small project the size is really unacceptable, but for very large modeling project you never exceed 380Mb which is pretty good considering that you have millions of saved information available live because loaded in memory. If you use GMF then this memory leak is seriously rising but GEF is really good alone.

It seems to me that SAP has started this in memory queries with its ERP and I think this could easily be extended to EMF therefore even if a problem exist we are going in the right direction today.

[Updated on: Wed, 29 June 2011 04:30]

Report message to a moderator

Re: CacheAdapter.inverseCrossReferencer and Memory Leaks [message #690508 is a reply to message #690216] Wed, 29 June 2011 11:45 Go to previous messageGo to next message
Ed Merks is currently offline Ed Merks
Messages: 25999
Registered: July 2009
Senior Member
Kirsten,

Kenn is in the best position to answer but if you want to investigate
yourself, ECrossReferenceAdapter.unsetTarget(Resource) and
unset(EObject) should be getting called for all your resources and all
the objects they contain. That should result in things being cleaned
up. It will be useful to use the debugger to inspect what's left in the
map after you've clean up your old resource set...



On 29/06/2011 12:45 AM, Kirsten M. Z. wrote:
> Hi @all,
>
> I am struggling against some memory issues, and I have been analyzing
> for several days now. As far as I can see, using MAT, the problem
> could be caused by CacheAdapter (i.e. its inverseCrossReferencer)
> which is not removing old resources.
> I have coded a (graphical) UML editor which uses EMF/UML2 internally.
> Whenever I close an old model and reload it, I waste about 30 MBytes
> (a small model) which is far to much for ignoring this problem.
>
> Learning from other thread in this forum and own inverstigations, I
> already use the following code in order to unload my resources and
> resource set (BTW, there is a regular UML model and several UML
> profiles in my resource set):
>
>
> while(getResourceSet().getResources().size() > 0) {
> Resource res = getResourceSet().getResources().get(0);
> res.eAdapters().clear();
> res.unload();
>
> getResourceSet().getResources().remove(res);
> }
> CacheAdapter.INSTANCE.clear();
>
>
> I found the following threads or bug entries within this forum about a
> similar topic. However, bugs seem to be fixed and the issues solved.
>
> http://www.eclipse.org/forums/index.php/m/624261
> http://www.eclipse.org/forums/index.php/m/647513
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=197079
>
> Now, I need to get some background information. I already debugged
> into sources, but sorry, I cannot really figure out the mechanism why
> (or how) the InverseCrossReferencer does not remove old stuff when
> unloading. I also haven't found a way to complete reset the state or
> to use another CacheAdapter, which could also solve the problem.
> Maybe, CacheAdapter$inverseCrossReferencer is not the only problem (I
> am still wondering if I can trust MAT), so your experience would help.
> Please tell me scenarios in which clearing the cache or unloading
> could fail or even tutorials on unloading.
>
> Thanks in adavance!
Re: CacheAdapter.inverseCrossReferencer and Memory Leaks [message #690509 is a reply to message #690230] Wed, 29 June 2011 11:53 Go to previous messageGo to next message
Ed Merks is currently offline Ed Merks
Messages: 25999
Registered: July 2009
Senior Member
Vlad,

Comments below.

On 29/06/2011 1:23 AM, Vlad Varnica wrote:
> This memory too important MBytes use has always been existing and it
> is almost impossible to solve it today.
This sentence is completely incoherent. It starts out as a random
jumble of words not connected in a grammatical way.
> You can only try to deactivate few recursive methods.
It's like a computer is generating these words. How exactly does one
deactivate a method and what does recursion have to do with it?
> I would say that the memory size is going down with the size of the
> project.
Project size? He's talking about a UML model, not a project.
> I mean that for small project the size is really unacceptable, but for
> very large modeling project you never exceed 380Mb which is pretty
> good considering that you have millions of saved information available
> live because loaded in memory.
His concern is that there's a 30 megabyte leak every time he opens and
closes the editor. So doing that 13 times will exceed 380 presumably...
>
> It seems to me that SAP has started this in memory queries with its ERP
Is this at all related to this thread?
> and I think this could easily be extended to EMF therefore even if
> problem exist we are going in the right direction.
A leak is bad and it's not the right direction...
Re: CacheAdapter.inverseCrossReferencer and Memory Leaks [message #690600 is a reply to message #690509] Wed, 29 June 2011 15:50 Go to previous messageGo to next message
Vlad Varnica is currently offline Vlad Varnica
Messages: 546
Registered: July 2009
Location: Milton Keynes - UK
Senior Member
Come on Ed you know there are leaks so ........

[Updated on: Wed, 29 June 2011 15:57]

Report message to a moderator

Re: CacheAdapter.inverseCrossReferencer and Memory Leaks [message #690620 is a reply to message #690600] Wed, 29 June 2011 16:48 Go to previous messageGo to next message
Ed Merks is currently offline Ed Merks
Messages: 25999
Registered: July 2009
Senior Member
Vlad,

It's impossible to figure out what you're saying when most of your
sentences are an ungrammatical collection of unrelated words.

I have no idea what makes you think I was arguing there isn't a leak. I
was giving instructions for how to investigate the leak and arguing that
all leaks should be tracked down and eliminated.


On 29/06/2011 12:50 PM, Vlad Varnica wrote:
> Come on Ed you know there are leaks so don't argue with me !!
Re: CacheAdapter.inverseCrossReferencer and Memory Leaks [message #690678 is a reply to message #690216] Wed, 29 June 2011 21:19 Go to previous messageGo to next message
Kenn Hussey is currently offline Kenn Hussey
Messages: 1618
Registered: July 2009
Senior Member
Kirsten,

Unloading all resources in the resource set should be sufficient. The
sample UML editor does the same and I haven't seen these issues:

public void dispose() {
disposeGen();

for (Resource resource : editingDomain.getResourceSet().getResources()) {
resource.unload();
}
}

Are you sure there are no other references to the objects in your model?

Kenn

On 11-06-29 3:45 AM, Kirsten M. Z. wrote:
> Hi @all,
>
> I am struggling against some memory issues, and I have been analyzing
> for several days now. As far as I can see, using MAT, the problem could
> be caused by CacheAdapter (i.e. its inverseCrossReferencer) which is not
> removing old resources.
> I have coded a (graphical) UML editor which uses EMF/UML2 internally.
> Whenever I close an old model and reload it, I waste about 30 MBytes (a
> small model) which is far to much for ignoring this problem.
>
> Learning from other thread in this forum and own inverstigations, I
> already use the following code in order to unload my resources and
> resource set (BTW, there is a regular UML model and several UML profiles
> in my resource set):
>
>
> while(getResourceSet().getResources().size() > 0) {
> Resource res = getResourceSet().getResources().get(0);
> res.eAdapters().clear();
> res.unload();
>
> getResourceSet().getResources().remove(res);
> }
> CacheAdapter.INSTANCE.clear();
>
>
> I found the following threads or bug entries within this forum about a
> similar topic. However, bugs seem to be fixed and the issues solved.
>
> http://www.eclipse.org/forums/index.php/m/624261
> http://www.eclipse.org/forums/index.php/m/647513
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=197079
>
> Now, I need to get some background information. I already debugged into
> sources, but sorry, I cannot really figure out the mechanism why (or
> how) the InverseCrossReferencer does not remove old stuff when
> unloading. I also haven't found a way to complete reset the state or to
> use another CacheAdapter, which could also solve the problem. Maybe,
> CacheAdapter$inverseCrossReferencer is not the only problem (I am still
> wondering if I can trust MAT), so your experience would help. Please
> tell me scenarios in which clearing the cache or unloading could fail or
> even tutorials on unloading.
>
> Thanks in adavance!
Re: CacheAdapter.inverseCrossReferencer and Memory Leaks [message #691320 is a reply to message #690678] Fri, 01 July 2011 04:34 Go to previous messageGo to next message
Kirsten M. Z. is currently offline Kirsten M. Z.
Messages: 131
Registered: July 2010
Senior Member
Hi Kenn,

I cannot say anything for sure, because I have probably a more complex scenario as in simple examples. However, I know one thing: I have (exactly) one resource set which contains several resources, i.e. a regular UML model and several UML profiles. Within my model I apply some stereotypes, of course. I don't have another resource set with another UML model where I perform cross linkage, if you mean something like that. And I shouldn't have some other EMF objects which are not contained in some of these resources.

for (Resource resource : editingDomain.getResourceSet().getResources()) {
resource.unload();
}


Bascially, that's the code I also used.
Is it possible that the order of unload calls is crucial? For example, profiles must be unloaded first?
Re: CacheAdapter.inverseCrossReferencer and Memory Leaks [message #898425 is a reply to message #690216] Thu, 26 July 2012 07:27 Go to previous messageGo to next message
Alexandre Torres is currently offline Alexandre Torres
Messages: 139
Registered: July 2009
Senior Member
Hi Kirsten, I have the same problem. A little less than 30 mb... but still a problem when I perform batch operations.
Did you managed to solve this problem?
The CacheAdapter has a clear method, I will try to find a way to call it.
My scenario is much more complicated because I use GEF and I'm pretty sure there are "lost adapters" around. Neverthless, MAT points the leak to the CacheAdapter and other UML related classes instead of any GEF, Draw2d, or SWT objects, my usual suspects... In fact, they are not in MAT list of suspects, to my surprise.
it must be the Butler! Smile

Hope there is a solution to this...
By the way, just opening the UML, doing some read and disposing does not cause this problem. Without adding adapters, my batch operations works well with over 500 models, just checking if some class exists, and some stereotype is used or not.





Re: CacheAdapter.inverseCrossReferencer and Memory Leaks [message #898465 is a reply to message #898425] Thu, 26 July 2012 08:30 Go to previous messageGo to next message
Ed Willink is currently offline Ed Willink
Messages: 4025
Registered: July 2009
Senior Member
Hi

Ecore and UML can have very highly recursive references, so that a
single parasitic reference to any part of the model causes everything to
be locked in.

Simple operations such as Resource.unload() are not sufficient to clear
the adapters.

In my experience, trying to get OCL leak friendly, you need to add code
to actively null out references that you understand so that you are
eventually left with the references that need resolution.

You might want to use an approach such as Xtext's
org.eclipse.xtext.junit4.GlobalRegistries that ensures that a variety
iof EMF and Xtext globals are nulled. I have found that user entries in
the EValidator.Registry are a particularly problematic sort of
persistent references.

Regards

Ed Willink


On 26/07/2012 12:27, Alexandre Torres wrote:
> Hi Kirsten, I have the same problem. A little less than 30 mb... but
> still a problem when I perform batch operations. Did you managed to
> solve this problem?
> The CacheAdapter has a clear method, I will try to find a way to call it.
> My scenario is much more complicated because I use GEF and I'm pretty
> sure there are "lost adapters" around. Neverthless, MAT points the
> leak to the CacheAdapter and other UML related classes instead of any
> GEF, Draw2d, or SWT objects, my usual suspects... In fact, they are
> not in MAT list of suspects, to my surprise. it must be the Butler! :)
>
> Hope there is a solution to this...
> By the way, just opening the UML, doing some read and disposing does
> not cause this problem. Without adding adapters, my batch operations
> works well with over 500 models, just checking if some class exists,
> and some stereotype is used or not.
>
>
>
>
>
Re: CacheAdapter.inverseCrossReferencer and Memory Leaks [message #898512 is a reply to message #898465] Thu, 26 July 2012 09:56 Go to previous messageGo to next message
Alexandre Torres is currently offline Alexandre Torres
Messages: 139
Registered: July 2009
Senior Member
Thank you Ed!
I just discovered something amazing.
I moved each execution of my batch into Runnable objects. Each model is loaded, opened, processed (image exporting), and closed in a thread separeted from the main eclipse (using Display.getCurrent().asyncExec...).
This gives time to the GEF/SWT whatever release the locks that are holding my UML elements, in some obscure background process. The speed is not great, but it is not a problem for me - and it has a constant speed. The memory leakage has droped to manageble level (few kbs/model).
Another thing I did was to call System.gc after disposing the stuff. Just becouse JVM is lazy on GC.
And the CacheAdapter is really innocent after all. Doing this to ensure clean up:
//for each resource... 
res.eAdapters.clear(); // empty res adapters.
for (TreeIterator<EObject> ti=res.getAllContents();ti.hasNext();) {
      EObject eo = ti.next();
      //remove any lost adapter
      eo.eAdapters().removeAll(eo.eAdapters());				
}
res.unload();
//...and in the end:
resourceSet.eAdapters.clear();


Thanks!
Re: CacheAdapter.inverseCrossReferencer and Memory Leaks [message #898575 is a reply to message #898512] Thu, 26 July 2012 11:56 Go to previous message
Christian W. Damus is currently offline Christian W. Damus
Messages: 768
Registered: July 2009
Senior Member
Hi, Alexandre,

You don't need to iterate the contents of a resource to remove adapters
from every element. The resource will do that for you when it is
unloaded, so you can avoid an unnecessary walk of the contents tree.

Cheers,

Christian

On 2012-07-26 13:56:58 +0000, Alexandre Torres said:

> Thank you Ed!
> I just discovered something amazing.
> I moved each execution of my batch into Runnable objects. Each model is
> loaded, opened, processed (image exporting), and closed in a thread
> separeted from the main eclipse (using
> Display.getCurrent().asyncExec...).
> This gives time to the GEF/SWT whatever release the locks that are
> holding my UML elements, in some obscure background process. The speed
> is not great, but it is not a problem for me - and it has a constant
> speed. The memory leakage has droped to manageble level (few kbs/model).
> Another thing I did was to call System.gc after disposing the stuff.
> Just becouse JVM is lazy on GC.
> And the CacheAdapter is really innocent after all. Doing this to ensure
> clean up:
>
> //for each resource... res.eAdapters.clear(); // empty res adapters.
> for (TreeIterator<EObject> ti=res.getAllContents();ti.hasNext();) {
> EObject eo = ti.next();
> //remove any lost adapter
> eo.eAdapters().removeAll(eo.eAdapters());
> }
> res.unload();
> //...and in the end:
> resourceSet.eAdapters.clear();
>
>
> Thanks!
Previous Topic:Multilingual models and OWL
Next Topic:[SOLVED] Cannot retrieve InstanceSpecification with UML2 library
Goto Forum:
  


Current Time: Wed Aug 20 10:51:29 EDT 2014

Powered by FUDForum. Page generated in 0.02250 seconds