Hi Dave,
Sorry to hear those improvements didn't impact the problems you
have. If you could share your heap dump with us, we could take a
look. You can use https://framadrop.org/ to share the file. If
it's more than 200Mo, it will only be valid for 24 hours, but
we'll read our mails by then and pick it up ;-)
Cheers,
Geneviève
On 2019-01-15 5:12 p.m., David Wootton
wrote:
I built code using these instructions and ran
some crude side by side tests using the Eclipse heap storage
monitor in the status line. I was puzzled that I didn't see
any change in storage utilization (around 2GB for 1.5 million
events with about 10 attributes per event) or time (around 1.5
minutes)
It turns out my analysis modules extend
CallStackAnalysis and the changes apply to CallGraphAnalysis
where the type hierarchies don't intersect.
Dave
Matthew Khouzam ---01/09/2019
09:43:01 AM---Hi David, Sorry for the long delay, the easiest
way to make a trace compass baseline is here.
From: Matthew
Khouzam <matthew.khouzam@xxxxxxxxxxxx>
To: tracecompass
developer discussions <tracecompass-dev@xxxxxxxxxxx>
Date: 01/09/2019
09:43 AM
Subject: Re:
[tracecompass-dev] Fw: Heap storage consumption for traces
Sent by: tracecompass-dev-bounces@xxxxxxxxxxx
Hi David,
Sorry for the long delay, the easiest way to
make a trace compass baseline is here.
First check out this page.
https://projects.eclipse.org/projects/tools.tracecompass/developer
Eclipse Trace
Compass | projects.eclipse.org
projects.eclipse.org
You can use the code
from these repositories to experiment, test, build,
create patches, issue pull requests, etc. This project
uses Gerrit Code Review; please see contributing via
Gerrit. |
Then checkout the patch in your git repo like
this.
$ git fetch
ssh://yourname@xxxxxxxxxxxxxxx:29418/tracecompass/org.eclipse.tracecompass
refs/changes/63/134363/4 && git checkout FETCH_HEAD
then run a maven build (mvn clean install
-Pdeploy-update-site) more info is here.
https://git.eclipse.org/c/tracecompass/org.eclipse.tracecompass.git/tree/README.md
You should then have a local update site you
can use.
From:
tracecompass-dev-bounces@xxxxxxxxxxx
<tracecompass-dev-bounces@xxxxxxxxxxx> on behalf of David
Wootton <dwootton@xxxxxxxxxx>
Sent: Monday, January 7, 2019
3:22:55 PM
To: tracecompass developer
discussions
Subject: Re:
[tracecompass-dev] Fw: Heap storage consumption for traces
Matthew
I don't have an easy way to build a workspace with pth Trace
Compass and my plugin code. At the moment, trying this is not
urgent, so I'll wait until this code gets into a repository I
can work with.
Dave
Matthew Khouzam ---01/07/2019
11:21:17 AM---The patch set is not yet fully merged, you can get
a sneaky preview here. https://urldefense.proofpo
From: Matthew Khouzam
<matthew.khouzam@xxxxxxxxxxxx>
To: tracecompass developer
discussions <tracecompass-dev@xxxxxxxxxxx>
Date: 01/07/2019 11:21 AM
Subject: Re:
[tracecompass-dev] Fw: Heap storage consumption for traces
Sent by: tracecompass-dev-bounces@xxxxxxxxxxx
The patch set is not yet fully merged, you can get a sneaky
preview here. https://git.eclipse.org/r/#/c/134363/
From:
tracecompass-dev-bounces@xxxxxxxxxxx
<tracecompass-dev-bounces@xxxxxxxxxxx> on behalf of David
Wootton <dwootton@xxxxxxxxxx>
Sent: Monday, January 7, 2019
10:42:48 AM
To: tracecompass developer
discussions
Subject: Re:
[tracecompass-dev] Fw: Heap storage consumption for traces
Matthew.
That's good news. Will this be in the Simrel 2019-03 release?
Will there be a Trace Compass repository site I can point to at
some point in time to give this a quick look?
Thanks to Genevieve for the optimization work.
Dave
Matthew Khouzam ---01/07/2019
10:20:52 AM---Hey, Dave, Happy Holidays! Hope Santa was good for
you, she was for me, her name is Genevieve and sh
From: Matthew Khouzam
<matthew.khouzam@xxxxxxxxxxxx>
To: tracecompass developer
discussions <tracecompass-dev@xxxxxxxxxxx>
Date: 01/07/2019 10:20 AM
Subject: Re:
[tracecompass-dev] Fw: Heap storage consumption for traces
Sent by: tracecompass-dev-bounces@xxxxxxxxxxx
Hey, Dave,
Happy Holidays! Hope Santa was good for you, she was for me, her
name is Genevieve and she really optimized the callgraph over
the vacations. I think you will be pleasantly surprised with the
results. 😊
BR
Matthew
From:
tracecompass-dev-bounces@xxxxxxxxxxx
<tracecompass-dev-bounces@xxxxxxxxxxx> on behalf of David
Wootton <dwootton@xxxxxxxxxx>
Sent: Thursday, January 3,
2019 7:20:16 AM
To: tracecompass developer
discussions
Subject: [tracecompass-dev]
Fw: Heap storage consumption for traces
Matthew
I hope you had a good holiday. I'm following up on this now that
I am back at work. Do you need anything from me at this point,
and how can I get it to you?
Dave
----- Forwarded by David Wootton/Poughkeepsie/Contr/IBM on
01/03/2019 07:18 AM -----
From: David
Wootton/Poughkeepsie/Contr/IBM
To: tracecompass developer
discussions <tracecompass-dev@xxxxxxxxxxx>
Date: 12/05/2018 02:28 PM
Subject: Re:
[tracecompass-dev] Heap storage consumption for traces
Matthew
I think I can send you my heap dump, but I need to check. It's
about 200MB and I don't have any public site I can upload it to.
How do I get it to you?
I'm not really familiar with VisualVM so I'm not sure how I
check what owns an object.
I am a little surprised that I seem to only have 52
TmfStateValue objects when I have 4000 ITmfEventField objects.
Maybe my placement for the printf call is wrong and most of the
objects have been released, I placed it in my class overriding
the call stack view, in the traceOpened method. But I am using a
hash table to keep only a single instance of a value.
If I understand, in TraceCompass 4.1, at the point where I
create a TmfStateValue to wrap an object (String, Long, Integer,
etc), I can now just use that object itself in the call to the
state builder pushAttribute call. Is that correct?
Happy holidays
Dave
Matthew Khouzam ---12/05/2018
10:30:53 AM---Hi Dave, these are some pretty cool findings. Now
one thing to keep in mind, we have several caches,
From: Matthew Khouzam
<matthew.khouzam@xxxxxxxxxxxx>
To: tracecompass developer
discussions <tracecompass-dev@xxxxxxxxxxx>
Date: 12/05/2018 10:30 AM
Subject: Re:
[tracecompass-dev] Heap storage consumption for traces
Sent by: tracecompass-dev-bounces@xxxxxxxxxxx
Hi Dave,
these are some pretty cool findings. Now one thing to keep in
mind, we have several caches, so they do trigger visualVM and
rightly so. Can you look up who is the owner of the mapEntries?
Maybe we have a leak with a manager or something.
Also, please note, tmfstatevalues are on the road to
deprecation. The state system now handles objects, so you can
save a few megabytes of useless pointers by just using these
objects. 😊
If you are allowed to share your heapdump, we can look at that
and try to optimize a bit here.
There is finally a possibility, but not a large one. Does your
EventField have a backreference to the event, which contains a
map of its fields? That would be a prime candidate for where
things can go wrong.
BR,
Matthew. (Is it too soon to wish a happy holiday season?)
From:
tracecompass-dev-bounces@xxxxxxxxxxx
<tracecompass-dev-bounces@xxxxxxxxxxx> on behalf of David
Wootton <dwootton@xxxxxxxxxx>
Sent: Wednesday, December 5,
2018 8:58:57 AM
To: tracecompass developer
discussions
Subject: [tracecompass-dev]
Heap storage consumption for traces
A while back, Matthew and I discussed heap storage utilization
by traces. My traces have a number of attributes associated with
event, with the number of attributes varying by type of trace. I
create ITmfEventField and TmfStateValue objects for each
attribute.
I have a trace with about 88,000 events. In that trace, I ended
up with close to 800,000 ITmfEventField and 500,000
TmfStateValue objects.
I created a hash table for ITmfEventField objects that contained
a single instance for each unique field name and value pair. In
this trace, I ended up with about 4,000 unique ITmfEventField
objects.
I created a hash table for the TmfStateValue objects and end up
with 52 unique objects.
I ran my code with VisialVM, and I noticed that the top class
for heap usage was
org.eclipse.tracecompass.internal.statesystem.core.backend.historytree.HTInterval.
From one run to the next the total number of objects varies
somewhere between 1.5 million and 2.5 million. If I click the
Perform-GC button in VisualVM, the count drops to around
700,000, fairly consistently.
I also see a large number of instances of
com.google.com.collect.ImmutableMapEntry (355,000),
com.google.com.collect.ImmutableMapEntry[] (178,000) classes and
com.google.com.collect.ImmutableMapEntry$NonTerminalImmutableMapEntry
(355,000)
Looking at the reference pane in VisualVM for a few (< 10) of
each of these classes, it looks like the ImmutableMapEntry and
ImmutableMapEntry$nonTerminalImmutableMap objects might be
related to StateItem objects and the ImmutableMapEntry[] objects
might be related to ITmfEventField objects.
I'm wondering if it is possible to use a hash table to maintain
single copies of unique instances of those objects or whether
the data in them guarantees each instance is unique. If it is
possible, then is the performance tradeoff for ushing a hash
table worth it?
Dave_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/tracecompass-dev
_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/tracecompass-dev
_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/tracecompass-dev
_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/tracecompass-dev
_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/tracecompass-dev
|