Re: [tracecompass-dev] Trace Compass state snapshot exchange format

[Philippe Proulx <pproulx@xxxxxxxxxxxx>]

----- Original Message -----
> From: "Matthew Khouzam" <matthew.khouzam@xxxxxxxxxxxx>
> To: "tracecompass-dev" <tracecompass-dev@xxxxxxxxxxx>
> Sent: Tuesday, 8 November, 2016 14:15:04
> Subject: Re: [tracecompass-dev] Trace Compass state snapshot exchange format

> Hello tracers,
> 
> I have a summary of my thoughts here, I can go in depth if need be.
> 
> To be honest, I find this very complex as a way to dump the full state
> history and segments at a given time of a trace. Are there any other
> places this can be used? I would go with a tree.json alone. Have the
> types in also, we only support 10000 attributes / ss. So that means we
> are saving 10k (more once Loic's patches are merged) values of worst
> case 4 instead of "String", so we are saving 8 chars or 80 K of disk
> space with this operation. In practice we are probably saving more like
> 80-800 bytes since a real average state system is 10-100 deep. What we
> are also doing is creating an opportunity to lose a file and render the
> state system illegible. I would suggest instead making a full tree.json
> and if you find it too large, zip/bz2/gzip/xz it.

You might save only 80 k for a state dump, however for the whole
history, it could be way more than that. We're talking one byte per
interval, remember.

Having one or multiple files depends on your project and on your design
principles. I personally find it cleaner to separate concerns in
different files. CTF does this, and I would be lying if I said that CTF
was not an inspiration for this. Your CTF trace is invalid too if you
lose this `metadata` file. But why and how would you lose it?

`types.json` and `tree.json` could be merged indeed. However, what I
understand from the state history back-end (and concept) is that the
association between keys and paths (tree nodes) is an extension (that
Trace Compass needs, yes), but this would not be required for other
applications. I suppose this is only interesting if this state history
API is split from Trace Compass one day. In this case `tree.json` would
be a TC-specific file (like CTF packet indexes which LTTng adds to
a CTF trace).

> 
> Also, will it save the segments in the segment store? this is a valid
> issue as we have several (especially with XML) segment stores being used
> at this moment. I think we don't want to lose the segments being
> created. It may be out of scope though for this rfc.

I'm not qualified to answer this. Alex?

> 
> Something I would NOT recommend that would save space too would be to
> replace "value" with "v", as it is repeated very often.

Honestly I don't care about disk usage for a state dump. I only stated
this advantage for state histories, which tend to be big.

> 
> Finally, the 'history.bin' is not really detailed. Is it:
> 
> <philippe>
> 
> * A state history and a state snapshot have an intersection
>  (`types.json` and `tree.json`), which means common code to produce and
>  consume both. They should be already produced when taking the snapshot
>  in fact.
> 
> </philippe>?

No. `history.bin` is part 1 of the first list. It follows the current
state history file format, without the part at the end which maps keys
to paths. Also another modification is that the type ID is not attached
to each interval (this is indicated by `types.json`).

> 
> Could you sell me on why these three files are more interesting that say
> a json output of the analysis id, the state, and the layout, then it is
> the responsibility of the analysis to be able to load its snapshot?

Yes, this would be possible also, and I guess we'll do something like this
for phase 1. I just thought that having common file formats between the
state history and a single state dump would be interesting for code reuse
reasons. Instead of producing another custom format when dumping the current
state, just copy the existing `types.json` and `tree.json` files, and
create a file (`state.json`) dedicated to hold the current values of
known keys.

Same logic goes for reading I guess:

1. Read and interpret `types.json`: create an integer-to-type map.

2. Read and interpret `tree.json`: create the tree, and for each node, get
   its type from the type map according to its key.

The next step is to modify the current values of this tree. You can use
a state history query or a state dump for this. This is only where the 
back-end formats differ.

Compare this to a CTF trace. If you need a partial CTF trace, you can
cut the data stream files, and reuse the same `metadata` file as is.

Phil

> 
> This is just my 2c though.
> 
> Cheers
> 
> Matthew
> _______________________________________________
> tracecompass-dev mailing list
> tracecompass-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from
> this list, visit
> https://dev.eclipse.org/mailman/listinfo/tracecompass-dev