Eclipse Community Forums: EMF-IncQuery » Query Pattern takes up enormous amounts of memory

Help

Home

Home » Archived » EMF-IncQuery » Query Pattern takes up enormous amounts of memory

Show: Today's Messages :: Show Polls :: Message Navigator

Query Pattern takes up enormous amounts of memory [message #1696103]

Thu, 21 May 2015 19:33

ihave question

Messages: 32
Registered: November 2014

Member

Hello!

I'm in the process of using your wonderful framework, so far with great success.

However I found a query pattern that is extremely memory hungry using your query hotspot evaluator!

Short intro: Variables have equations, which reference other variables by name (reference) as a string. Variables directly reference other variables (influences)

This is it's twin, which does not consume so much memory:

/* Connected, but not referenced in equation */
pattern missingReference(Source : Variable, Linked : Variable) = {
find equation(Source, Equation);
find influences(Linked, Source);
neg find reference(Equation, Linked);
}

But the counterpart, a reference without a connection, just goes wild!

/* Missing link to referenced variable */
pattern missingLink(Source : Variable, Referenced : Variable) = {
find equation(Source, Equation);
find reference(Equation, Referenced);
neg find influences(Referenced, Source);
}

UPDATE

I found while testing thatI basically reused the same structure over and over to generate a test model - the every variable referenced every other! I randomised the names with UUIDs and now the memory consumption dropped from 630Mb down to 33Mb with 4800 elements - big win Smile

however, still takes up 79% (24000 elements 430Mb) of the cumulative memory

Any help on refactoring this pattern would be very much appreciated Smile

[Updated on: Thu, 21 May 2015 19:56]

Report message to a moderator

Re: Query Pattern takes up enormous amounts of memory [message #1696104 is a reply to message #1696103]

Thu, 21 May 2015 20:03

Zoltan Ujhelyi

Messages: 392
Registered: July 2015

Senior Member

Hi,

it is hard to say what is causing the issue here without more details about the model and/or queries what is happening, but I try to give some general hints how to find such issues.

In general, the memory consumption of the pattern is high either if there are some model element references or subpatterns that have a high number of matches, or there are join operations that cannot filter the contents well (in worst case, the number of elements in the join results can be the product of the joined ones). For this reason, you have to look what subpatterns can have many matches.

0. Check whether there is a warning for isolated constraints. I guess, there is not, but it is very easy to check, and if there is, we have a candidate for large memory footprint.
1. At first, I would look at the number of matches of the referenced patterns 'equation' (later E), 'reference' (R) and 'influence' (I). As the missingReference pattern is not consuming a large amount of memory, these are not problematic in themselves.
2. The missingReference pattern has at most E*I matches, as the reference pattern can only filter out elements in that case; while the missingLink might contain E*R matches. As these numbers might be very different, that already might account for the differences.
3. If the neg find can effectively filter either the results of the equation or reference patterns, it might sense to introduce a helper pattern that contains only a single find and the neg find in it. This way you can ensure that the result is filtered.
4. Finally, it is possible that either by inlining the (positive) pattern calls, or restructuring them it is possible to reduce the memory consumption, but it is hard to give hints without knowing the finer structure of the patterns and the models you are using.

For checking such patterns we feature a Rete Visualizer components (see https://wiki.eclipse.org/EMFIncQuery/UserDocumentation/RETE_Visualizer) that can display the Rete network, including both the operations and the size of the nodes. This might give you further hints where to look for memory gains. Again, what you are looking for is a Rete node that stores a large number of partial matches (typical candidates are the join nodes).

I hope, this was helpful. If not, please ask for clarification, or share more details about your problem.

Cheers,
Zoltán

Report message to a moderator