Re: [tracecompass-dev] Question about scaling of data driven analysis

[Robbert Jongeling <robbert.jongeling@xxxxxxxx>]

Hi Loïc and Matthew,

Thank you for your responses.

I have indeed noticed that opening the trace without the analysis attached is much faster.

It takes about 50 seconds to index the 2.2 million events, however when the analysis is attached, it takes too long to wait for (it seems to index only hundreds of events per second compared to thousands or even tens of thousands when no analysis is attached).

After applying the patch you both suggested, creating the view based on my analysis still takes too long to wait for.

Indexing of the trace when the analysis is attached is also still a lot slower than when it is not attached.

(By attached I mean the following:

I currently have an analysis file in which I define a stateprovider and a timegraphview.

Indexing the trace without the timegraphview opened is normal, quick, but indexing the trace while the timegraphview is already opened takes very long. 

In both cases, the view is imported in Trace Compass.)

The patch does improve the performance.

I have profiled two runs of TraceCompass, one without the patch and one with the patch.

In these runs I have first opened the trace and after indexing opened my custom timegraphview. 

At that time, I have started the profiling.

This does show a significant improvement of time spent on historytree related functions, 

but not on the HTNode.readNode method, which takes up the majority of the time in both cases.

Which, I think, is caused by the large amount of attributes on a level in the state system.

When I run my analysis on a different version of the trace, in which the top level attribute of the state system contains not thousands of attributes but just one, generation of the view is almost instantanious.

Is it possible to somehow group the attributes in order to make the generation faster?

Kind regards,

Robbert

---

Van: tracecompass-dev-bounces@xxxxxxxxxxx <tracecompass-dev-bounces@xxxxxxxxxxx> namens Matthew Khouzam <matthew.khouzam@xxxxxxxxxxxx>
Verzonden: dinsdag 21 juni 2016 21:15:08
Aan: tracecompass-dev@xxxxxxxxxxx
Onderwerp: Re: [tracecompass-dev] Question about scaling of data driven analysis

 

Hi Robbert!

Quick question, can you load the 200mb trace easily if there is no
analysis? (remove the analysis from tc)

if that is the case, the problem lies in the state system, perhaps this
patch can fix it. https://git.eclipse.org/r/#/c/61062/

Please keep us posted,

Br,

Matthew

On 16-06-21 09:06 AM, Robbert Jongeling wrote:
> Dear Trace Compass developers,
>
> I am trying to apply Trace Compass to custom, text-based traces and to
> this end have created a parser and an xml-analysis.
> My question relates to the scaling of the application or the analysis
> to larger traces.
> In case of a large trace (200MB) my analysis takes about 2 hours to
> complete, whereas for small traces (10MB) it takes seconds.
> What can I do to increase the performance of my xml analysis?
>
> I have noticed that the issue may be caused because of the size of my
> state system structure, which is about 4 levels deep but on each level
> there may be >1k different entries. E.g.:
>
> |- trace
> || - thingid1
> || - subthingid 1
> || - somevalue
> |...
> || - thigid1000
> || - subthingid 1
> |...
> || - subthingid 1000
> || - someOtherValue
>
>
> I have noticed that the getRelevantInterval method in
> org.eclipse.tracecompass.internal.statesystem.core.backend.historytree.HTNode
> takes up quite a lot of time.
> Also, I found this bug
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=492525, which discusses
> that CTF traces can be packetized.
> Is there a similar way to handle custom (text-based) traces?
>
> I would greatly appreciate any pointers you can give me.
>
>
> Kind regards,
>
> Robbert
>
>
>
> _______________________________________________
> tracecompass-dev mailing list
> tracecompass-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/tracecompass-dev

_______________________________________________
tracecompass-dev mailing list
tracecompass-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/tracecompass-dev