[tracecompass-dev] CTF stress tests

[Matthew Khouzam <matthew.khouzam@xxxxxxxxxxxx>]

Hi,
I was looking into the CTF stress tests.
They are good, but I don't know if we want them in our standard test
cases. They basically check the scalability of the computer it is being
run on in most cases and in all cases, the test of reading 12g of files
is rather prohibitive. That being said, maybe having this as a weekly
build could be an interesting idea. Also, I don't think the idea that
busting the heap size is a good indication of a test failure. I can
always allocate more heap at the problem. :)

Now for a breakdown of the tests:
├── gen.sh
├── metadata
│   └── pass
│       ├── large-metadata - interesting test, I think something like
this should be used to improve robustness
│       ├── long-identifier - this should be our approach here, I think
http://stackoverflow.com/questions/6007568/what-is-max-length-for-an-c-c-identifier-on-common-build-systems
│       ├── many-callsites - works here until we oome but it's slow to
test and we don't have a real application to validate the data yet. So
it's good, but we may wish to put efforts elsewhere for now.
│       ├── many-stream-class - This test is looking at the max size of
an array...
│       ├── many-typealias - interesting test, the parser will suffer.
│       └── many-typedef - ditto
└── stream
    └── pass
        ├── array-large - Testing the max size of ram and arrays.
        ├── many-events - This can be an interesting tool for profiling.
But it's not really a test for the reader...
        ├── many-packets - This will be good to test distributed indexes
        ├── many-streams - Good, but even better is...
        ├── many-traces - interesting and a real problem
        ├── packet-large - this is actually easier than many-events
        ├── sequence-large - see array-large
        ├── string-large - see array-large
        ├── struct-many-fields - see array-large
        ├── struct-nest-n-deep - I like this one, it highlights one of
our optimisations
        └── variant-many-tags - see array-large

Why no deep variants? just curious.

so, tl;dr the tests are good at making a system suffer and qualifying
its boundaries. I think this is great for profiling, but for actual
testing, it should not be part of the per-patch system we have set up.

Any thoughts?