Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Sapphire » Ideas to reduce memory usage(Ideas to reduce memory usage)
Ideas to reduce memory usage [message #1017194] Mon, 11 March 2013 03:00 Go to next message
Luke V is currently offline Luke V
Messages: 3
Registered: February 2013
Junior Member
I'm using Sapphire to facilitate an XML entry for a design description and documentation format. The XML schema is rather complex (has many levels of hierarchy and is composed of multiple files). The schema comes from an standards committee and I cannot modify it for simplification.

I have a sapphire form/model put together and it works well for smaller xml files (< 256K). However as the files get larger, the load time grows and the memory usage starts to balloon. For a larger xml file (1-2MB), the load time opening the first form is a minute or two and eclipse's memory usage creeps into the 1200MB to 1500MB range. This tends to make eclipse less than responsive (even on a recent desktop machine with 12GB of RAM) and decreases the editor's usefulness. Sapphire 0.5x series would crash the JVM with the larger files. Sapphire 0.6 eventually loads, but is very slow. I've tried all sorts of JVM configuration tweaks, 1.6, 1.7, Hotspot, OpenJDK, and both Eclipse Indigo and Juno without any luck.

I've tried analyzing the issue a bit with the various memory profiler tools. From an initial look, it seemed that much of the utilization was due to creating a new service object for each node in the design. One quick idea was to try and figure out how to share a service object (or have a default one) for nodes that didn't customize the service at all. However, this seemed to be a bit of work (especially without a good working knowledge of Sapphire's internals).

Does anyone have an additional insight or ideas of how to reduce memory usage on designs with large numbers of model nodes?
Re: Ideas to reduce memory usage [message #1017451 is a reply to message #1017194] Mon, 11 March 2013 21:19 Go to previous messageGo to next message
Konstantin Komissarchik is currently offline Konstantin Komissarchik
Messages: 942
Registered: July 2009
Senior Member
Performance is definitely an issue that needs work, especially performance when opening an editor. Some work happened in the 0.6 release, but more is necessary. Typically, the issues are retention of objects that supposed to have gone out of scope (aka, memory leak) or unnecessary repetition of work (too many events). The memory weight of legitimate objects is unlikely to be an issue.

If you are poking around with a profiler, take a look at object counts on the heap for some of the key classes (such as MasterDetailsContentNode). Does the count match your expectation?

Any help you can offer in diagnosing performance issues would be of great benefit to the project.

- Konstantin
Re: Ideas to reduce memory usage [message #1018088 is a reply to message #1017451] Wed, 13 March 2013 05:40 Go to previous messageGo to next message
Greg Amerson is currently offline Greg Amerson
Messages: 116
Registered: March 2010
Senior Member
Hey Konstantin,

I took a few hours yesterday to poke around with some sampling. I saw a few hot spots in the EL parsing and ExtensionSystem initialization. I modified to the code to cache EL function literals and also removed synchronization and switched to eager init for the ExtensionSystem to remove need for sychronization. Then the next hotspots I found were equally divided in the ModelElement.read() and that gets into the heart of the framework so I stopped my attempts there. So after my 2 changes, I'm still not seeing any overall speed ups for opening the editor. Before my changes, i was about from 11-13 seconds opening a 82k xml file and after my changes i'm seeing speeds within statistical variation. I can send you a patch for the changes if you wanted to review but at this time, I don't think the changes are quite worth it.

However, this startup speed issue is something that my users have been reporting. Now granted, some of the slowness could be due to my usage of the framework and could be optimized. But I am willing to commit resources to addressing this problem.
Re: Ideas to reduce memory usage [message #1018315 is a reply to message #1018088] Wed, 13 March 2013 15:55 Go to previous messageGo to next message
Konstantin Komissarchik is currently offline Konstantin Komissarchik
Messages: 942
Registered: July 2009
Senior Member
I fixed a number of performance issues for the 0.6 release using a combination of two techniques. I am sure more cases of both are still present.

1. Used a heap analyzer (such as Eclipse MAT) to look for lingering objects or too many objects of particular type. Memory leaks is the most likely culprit of high memory usage.

2. Used the trace facility in the event system to identify cases where too many events were generated and processed. This is the most likely culprit of slow editor startup. See ListenerContext.TRACE and Event.TRACE_SOURCE_STACK_DEPTH. This analysis is best performed with a very small editor and a very small model instance.

- Konstantin
Re: Ideas to reduce memory usage [message #1061558 is a reply to message #1017194] Sun, 02 June 2013 20:34 Go to previous messageGo to next message
Luke V is currently offline Luke V
Messages: 3
Registered: February 2013
Junior Member
I finally got some time to look back into this. Analyzing a smaller file, I was pointed towards the ServiceContexts. I then loaded up a larger (1.5MB XML file and examined the heap). It appears that duplicating the HashSets containing the service lists for each Element or Property in the model is contributing to the memory issue. An example is below:

The source node from the XML file contains the tag of <dim> and the value of "1". The PropertyInstanceServiceContext for this node retains ~12KB of data. Using the Memory Analyzer to look at the dominator tree, the service ArrayList contributes to 10KB of this with 6KB+ being retained by the MasterConversionService object, almost all retained by the conversions HashSet.

I also noticed, that although not exactly the same, most of the MasterConversionService objects contain references to 99% of the same ~65 conversions. Although I haven't investigated, I would think that one of these conversions being different for a particular node would be the exception rather than the rule.

Would it make sense to pool the MasterConversionService objects and reuse a particular object with the same set of conversions? XML files with the same hierarchy/tags modeled as a Sapphire Property should have the same set of conversions/overrides and shouldn't differ in how they need to be converted...

The latest git repo did help the utilization quite a bit (vs the 0.6x series I started with, eclipse's heap memory usage is now in the 700MBish range for the same file).

index.php/fa/15130/0/
Re: Ideas to reduce memory usage [message #1063876 is a reply to message #1061558] Fri, 14 June 2013 16:18 Go to previous messageGo to next message
Konstantin Komissarchik is currently offline Konstantin Komissarchik
Messages: 942
Registered: July 2009
Senior Member
Thanks for the analysis. I have opened a bug to track this issue. I definitely think we can do something to improve the memory footprint of this service. Could you add yourself to this bug?

https://bugs.eclipse.org/bugs/show_bug.cgi?id=410836
Re: Ideas to reduce memory usage [message #1064561 is a reply to message #1063876] Wed, 19 June 2013 22:08 Go to previous messageGo to next message
Konstantin Komissarchik is currently offline Konstantin Komissarchik
Messages: 942
Registered: July 2009
Senior Member
I implemented some changes to MasterConversionService and ServiceContext to reduce memory lookup. See the referenced bug for Git commit link. Please grab the latest and re-evaluate. Your focus on tuning resource consumption with large model instances is very valuable to the project.
Re: Ideas to reduce memory usage [message #1064579 is a reply to message #1064561] Thu, 20 June 2013 05:28 Go to previous messageGo to next message
Luke V is currently offline Luke V
Messages: 3
Registered: February 2013
Junior Member
MasterConversionService has definitely been driven off the radar... Thanks for your expedient patch! Memory usage is down to 685MB (total for eclipse) from 700MB for the 1.5MB XML file. Looking at the memory analyzer again, there looks like there might be some more low hanging fruit... (I guess the term low hanging is relative judging from the 100 files updated for 85a2748... Smile )

Currently on the top of the "Biggest Top-Level Dominator Classes" list is org.eclipse.sapphire.services.internal.PropertyInstanceServiceContext. Digging into the dominator tree for this object, org.eclipse.sapphire.modeling.el.Literal$1 floated to the top.

In the current model/file that I am using, there ends up being 326,153 instances of these Literal objects. A couple of interesting stats that I noticed about these literals:
1. There are 68217 literals with the value of null, each taking up 376 bytes = ~25MB of "nulls". I haven't had the chance to trace the origin or study if any of these "nulls" are used for the Function.origin members instead, but at a first glance, it would seem that these have the potential to be replaced with the same static copy of Literal.NULL

2. For a literal text value of "false" 520 bytes are retained by the literal object. (376 of these being the literal's overhead). 144 bytes of storage for a 5 character string seems bit off (a bit of googling and some math seems to indicate that this should be 56 bytes for a fresh string.). Also, I noticed that the Literal class had true/false static values which might be of service here (although this will probably take some tweaking on the model to get the data types to be translated through the model correctly).

2. There are 2686 copies of the following literals (each 1032 bytes) for a total of ~2.5MB. Appears at first glance to be ab bit redundant...
<html><head><style>body
{
margin: 0;
overflow: auto;
}
table
{
border-width: 0px;
border-spacing: 0px;
border-collapse: collapse;
}
td
{
margin: 0px;
padding-right: 10px;
}
body, td
{
font-family: "Segoe UI", "sans-serif";
font-size: 9pt;
}
</style></head><body>

Will do some more memory analyzer digging later...

[Updated on: Thu, 20 June 2013 05:30]

Report message to a moderator

Re: Ideas to reduce memory usage [message #1064714 is a reply to message #1064579] Thu, 20 June 2013 16:05 Go to previous message
Konstantin Komissarchik is currently offline Konstantin Komissarchik
Messages: 942
Registered: July 2009
Senior Member
Please open a bug with your latest findings. Each literal is a function, so it gets evaluated and a new function result created. Perhaps this can be optimized.
Previous Topic:Drag and Drop support for a list of string items
Next Topic:Generating a config.xml file
Goto Forum:
  


Current Time: Sat Oct 25 13:37:02 GMT 2014

Powered by FUDForum. Page generated in 0.03275 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software