Re: [tracecompass-dev] Reindexing trace file and rebuilding the state sy

[Matthew Khouzam <matthew.khouzam@xxxxxxxxxxxx>]

> Thanks, we have found out the problem of slow indexing is caused by our parser.

Great!

>  We’re working on improving it. Since the state system is immutable, I’m thinking of creating a custom data structure that is mutable for our use. Is it possible to create an ITmfTreeXYDataProvider implementation that does not use the state system?

Yes. the state system is just a cache. You can point it to any datastructure you want as long as you follow the API. If you have a cool solution you can share, please do! 🙂

---

From: tracecompass-dev <tracecompass-dev-bounces@xxxxxxxxxxx> on behalf of Alex Doan <vadoan@xxxxxxxxxxxxxx>
Sent: Thursday, March 2, 2023 10:07 AM
To: tracecompass developer discussions <tracecompass-dev@xxxxxxxxxxx>
Subject: Re: [tracecompass-dev] Reindexing trace file and rebuilding the state system during analysis

 

Thanks, we have found out the problem of slow indexing is caused by our parser. We’re working on improving it. Since the state system is immutable, I’m thinking of creating a custom data structure that is mutable for our use. Is it possible to create an ITmfTreeXYDataProvider implementation that does not use the state system?

 

Thank you,

 

Alex

 

From: tracecompass-dev <tracecompass-dev-bounces@xxxxxxxxxxx> On Behalf Of Matthew Khouzam via tracecompass-dev
Sent: Wednesday, March 1, 2023 14:55
To: tracecompass developer discussions <tracecompass-dev@xxxxxxxxxxx>
Cc: Matthew Khouzam <matthew.khouzam@xxxxxxxxxxxx>
Subject: Re: [tracecompass-dev] Reindexing trace file and rebuilding the state system during analysis

 

CAUTION - This email is from an external source. Please be cautious with links and attachments. (go/taginfo)

 

I think my gut reaction would be to say "wow". That is strange. So what I suggest is tracing trace compass.

 

Here is a guide from the README.md.

 

Tracing Trace Compass

---------------------

Trace Compass can be traced by doing the following in the launch configuration:

* (java 8 only) -vmargs

* -Djava.util.logging.config.file=%gitroot%/logging.properties (where %gitroot% is the directory tracecompass is checked out to)

* -Dorg.eclipse.tracecompass.logging=true

Additionally the folder the trace is being written to (default is `home/.tracecompass/logs`) needs to be created in advance. After running Trace Compass, a `trace_n.json` will be created in the tracing folder. It needs to be converted to true json, so use the `jsonify.sh` script in the root directory to convert it. Then it can be loaded into Trace Compass, if the **Trace Event format** is installed from the incubator, or from a web browser such as Chromium.

The performance impact is low enough as long as the log level is greater than "*FINEST*".

NOTE: thread 1 is always the UI thread for Trace Compass. Also, the thread numbers are the JVM threads and do not correspond necessarily to Process IDs.

 

I would love to see any results you see. Please keep us posted!

 

Matthew

---

From: tracecompass-dev <tracecompass-dev-bounces@xxxxxxxxxxx> on behalf of Alex Doan <vadoan@xxxxxxxxxxxxxx>
Sent: Wednesday, March 1, 2023 11:50 AM
To: tracecompass developer discussions <tracecompass-dev@xxxxxxxxxxx>
Subject: Re: [tracecompass-dev] Reindexing trace file and rebuilding the state system during analysis

 

Hi, I have a quick question regarding the indexing. I have been testing performance on big trace file and it seems to me that the State System only took a few seconds to finish building, getTree in AbstractTreeDataProvider also finish a few milliseconds later, which makes me think the data is ready, but fetchXY in ITmfXYDataProvider refuses to run until after the indexing is finished, which takes upward of 5-10 minutes for a 200MB file. Please help me understand how the indexing works and why the data cannot be shown even though State System is built and ready. Thank you!

 

Best regards,

 

Alex

 

From: tracecompass-dev <tracecompass-dev-bounces@xxxxxxxxxxx> On Behalf Of Matthew Khouzam via tracecompass-dev
Sent: Tuesday, February 28, 2023 10:10
To: tracecompass developer discussions <tracecompass-dev@xxxxxxxxxxx>
Cc: Matthew Khouzam <matthew.khouzam@xxxxxxxxxxxx>
Subject: Re: [tracecompass-dev] Reindexing trace file and rebuilding the state system during analysis

 

CAUTION - This email is from an external source. Please be cautious with links and attachments. (go/taginfo)

 

The state history on disk is designed to be immutable. This is an interesting use case, it leads to maybe looking into partial state system, there was some work here:

https://git.eclipse.org/r/c/tracecompass/org.eclipse.tracecompass/+/194833

 

This would give the notion of checkpoints and a cascaded secondary state system. That being said, it would not be ideal as the trace would be read twice. Can we help you making the trace handling faster maybe?

 

Basically, more info, if you're allowed to share, would help.

 

Matthew

---

From: tracecompass-dev <tracecompass-dev-bounces@xxxxxxxxxxx> on behalf of Alex Doan <vadoan@xxxxxxxxxxxxxx>
Sent: Tuesday, February 28, 2023 8:55 AM
To: tracecompass developer discussions <tracecompass-dev@xxxxxxxxxxx>
Subject: [tracecompass-dev] Reindexing trace file and rebuilding the state system during analysis

 

Hi, I am currently working on a feature where the trace would be analyzed at a higher resolution on zoom. Right now I have a workaround that is to open the trace file at very high resolution at the start then lower the resolution after the fact, however this have the downside of making analysis takes significantly longer, and limit the maximum resolution I can produce due to the performance.

 

Is there a way to change state system attribute mid analysis? Any help would be appreciated.

 

Thank you,

 

Alex

---

This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.

---

This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.

---

This transmission (including any attachments) may contain confidential information, privileged material (including material protected by the solicitor-client or other applicable privileges), or constitute non-public information. Any use of this information by anyone other than the intended recipient is prohibited. If you have received this transmission in error, please immediately reply to the sender and delete this information from your system. Use, dissemination, distribution, or reproduction of this transmission by unintended recipients is not authorized and may be unlawful.