Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
[tracecompass-dev] Trace Compass is slow for LTTng-UST CallStack analysis
Dear experts,

I am trying to profile a multi-threaded application (built with -finstrument-functions -g) running on a Linux based embedded system using LTTng for function tracing and Trace compass for analyzing. But Trace Compass is extremely slow in processing the traces that renders it almost unusable. Here are the steps I followed:
  • collected function entry and exit traces from the remote Linux system using LTTng. resulting in a trace file of ~700MB
  • and imported the trace in Trace Compass. There are ~10 Million events in the trace file.
pasted_image003.png

I clicked on Lttng-UST CallStack -> Flame Chart.
The Flame chart is generated but in the Progress tab, it is stuck at 50% for ever. Navigating the Flame chart view is painful: many times, Trace compass become non responsive and then have to wait for several minutes to be able to use it again.
pasted_image005.png

Also, Flame Graph and Function Duration Statistics are not generated.
I have got similar experience with Lttng-UST CallStack (new) as well.

Questions on performance:
1. Isn't trace compass designed to handle large trace files e.g., 10 Million events from multithreaded (~100 threads) programs? When I tried to use a trimmed trace file (with ~100K events), it is working just fine.
2. Any hints to improve the performance of trace compass? My laptop is a powerful one (8 core, 128GB RAM) and Trace compass is not really using much cpu/memory even when it is generating Flame chart. I tried to increase the heap size to 10GB (-Xms1g -Xmx20g in .ini file), but it is not helping.
3. How to debug the issue of trace compass getting stuck - e.g., from the trace compass logs to see where trace compass is really getting stuck?

Question on addresses to function names mapping:
I have provided a text file containing the address to symbol name mapping (generated with nm -C <executable/shared lib>) in the flame chart view . Even after that, I am only getting function addresses in the flame chart and not the symbol names. 
  • How can I debug this issue - e.g., by looking at the actions performed by trace compass after I supply these files?
  • My traced program is composed of an executable and multiple shared libs (.so). If I have to use the "the root location of the binaries of the trace target" option from flame chart, how can I find out the right path to be given in the "custom target root directory"?
image.png

Versions
Trace compass version: trace-compass-10.2.0-20241204-1911-win32.win32.x86_64
JVM version: jdk-17.0.12_windows-x64_bin.msi


Thanks,
K.

Back to the top