Re: [tracecompass-dev] CTF stress tests

[Matthew Khouzam <matthew.khouzam@xxxxxxxxxxxx>]

Another point:

Should we set limits to the parser? Maybe it would be interesting to
fail saying: we only support 65535 callsites or something like that
might be nicer than supporting 65539 and then surprisingly crashing.

PS. The numbers are arbitrary.

On 14-11-18 09:50 AM, Matthew Khouzam wrote:
> Hi,
> I was looking into the CTF stress tests.
> They are good, but I don't know if we want them in our standard test
> cases. They basically check the scalability of the computer it is being
> run on in most cases and in all cases, the test of reading 12g of files
> is rather prohibitive. That being said, maybe having this as a weekly
> build could be an interesting idea. Also, I don't think the idea that
> busting the heap size is a good indication of a test failure. I can
> always allocate more heap at the problem. :)
>
> Now for a breakdown of the tests:
> ├── gen.sh
> ├── metadata
> │   └── pass
> │       ├── large-metadata - interesting test, I think something like
> this should be used to improve robustness
> │       ├── long-identifier - this should be our approach here, I think
> http://stackoverflow.com/questions/6007568/what-is-max-length-for-an-c-c-identifier-on-common-build-systems
> │       ├── many-callsites - works here until we oome but it's slow to
> test and we don't have a real application to validate the data yet. So
> it's good, but we may wish to put efforts elsewhere for now.
> │       ├── many-stream-class - This test is looking at the max size of
> an array...
> │       ├── many-typealias - interesting test, the parser will suffer.
> │       └── many-typedef - ditto
> └── stream
>     └── pass
>         ├── array-large - Testing the max size of ram and arrays.
>         ├── many-events - This can be an interesting tool for profiling.
> But it's not really a test for the reader...
>         ├── many-packets - This will be good to test distributed indexes
>         ├── many-streams - Good, but even better is...
>         ├── many-traces - interesting and a real problem
>         ├── packet-large - this is actually easier than many-events
>         ├── sequence-large - see array-large
>         ├── string-large - see array-large
>         ├── struct-many-fields - see array-large
>         ├── struct-nest-n-deep - I like this one, it highlights one of
> our optimisations
>         └── variant-many-tags - see array-large
>
> Why no deep variants? just curious.
>
> so, tl;dr the tests are good at making a system suffer and qualifying
> its boundaries. I think this is great for profiling, but for actual
> testing, it should not be part of the per-patch system we have set up.
>
> Any thoughts? 
> _______________________________________________
> tracecompass-dev mailing list
> tracecompass-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://dev.eclipse.org/mailman/listinfo/tracecompass-dev