[tracecompass-dev] Throwing around some ideas for instant seeking of tra

[Matthew Khouzam <matthew.khouzam@xxxxxxxxxxxx>]

Hello Trace Compass world,

I figured I would dump some ideas here as discussing them behind closed
doors does not a good open source project make. Right now, Trace Compass
is moderately fast, we read traces at over 30mb/s on spinny hard
drives[1]. But throughput is not the only issue, we also want to
minimize latency.

If a trace has an index available from the start[2], we should be able
to seek anywhere in the trace from the start.
This is already the case if you ship the trace with the trace compass index

One method used by LTTngTop is to assume an event is 23 bits long[3] for
progress bars. This is a great heuristic for progress bars but won't
work as well for seeking.

Let's say we have what I would call an imprecise index. One that does
not give us event totals but rather timestamp checkpoints. We could
populate the histogram with fuzzy data until we have better data. We
could seek in the histogram, we can populate the events view. We can
even make in memory state systems for a given area with some unknowns...
then when the main request catches up with a full state, it will replace
the unknowns with proper data. Later, it can be coalesced, but this
might be a micro-optimisation and should be looked at when the time is
right.

Here are some steps I can think of that would be necessary to support
these indexes
* Define an api for the pre-existing index. If it has the right
information, convert it to our b-tree
* Have the histogram size itself to the total size. allowing later seeks
only allowing time range checks in the populated area
* Make the events table support reading the events out of the scrollbar
range.
* Make an in memory state system and statistics request from the data.
* Support range queries in the state system and histogram. Showing the
in memory state system when the main state system doesn't have data
available at a given time range.
* refresh "UNKNOWN" states to the correct ones when the data is available.
* Make the progress bar for the trace show "% done"

Any thoughts?



[1] my machine at home, Q6600 - 2 gb ram, 7200 rpm hdd
[2] CTF supports it unnoficially, and OTF I think supports it partially
too ***please correct me if you know something I don't***
[3] Discussion with J Desfossez Dec 8 2014