| Re: [tracecompass-dev] Question about scaling of data driven analysis |
|
Hi Robert,
As Loic said, large number of attributes in the state system cause scaling issues. The number of attributes per tree level does not matter, it's the total number. Some questions/paths to try: * Limiting the number of attributes in the tree would be preferable. Do you need all of them for your analysis? If not, you should stick with the bare minimum needed. * Maybe you can split your data in multiple analyses if there are distinct subsets of data for different views * As for grouping them, do many attribute values start and end at the same time? If so, they could be grouped in a custom state value for one attribute. That is a new state value type that was added recently, but it is not used yet and not supported in data-driven analyses. But it should be "fairly easy" to add that support. But that doesn't help you today. But knowing someone has an actual need for it may make it happen faster ;-) * The thingidx, do they follow a pattern of start, something's happening, then end? If so, that is a killer of state system performance. You could consider using a segment store. See [1] for instructions on writing an XML pattern to create segments and [2] for the path on the source of TraceCompass of an example pattern using segments. If you do so, I have to warn you you may reach the limit of the segment stores (the available memory). Some prototypes of scalable segment stores are on gerrit right now, we just need to review them and accept one of them. * If the attributes have a high frequency of change, using a type that takes less space may help also. Integers take less space than strings. If you have string attributes that could be an enum, then you should use an enum and save ints instead (see the <definedValue> tag of the XML) * You can see the size on disk of the state system by selecting the analysis in the project explorer and looking in the Properties view under "Analysis module properties". The smaller, the better, so when you try something, it may give you an indication on the effect it had on the state system. I hope some of this can help you. Also, Loic, who replied earlier, is doing is master's degree in improving the state system performances. If possible, you could provide him a trace, the parser and the analysis so he can see if the solutions he proposes improve a variety of cases or if there is more that should be done. Regards, Geneviève [1] http://archive.eclipse.org/tracecompass/doc/org.eclipse.tracecompass.doc.user/Data-driven-analysis.html#Defining_an_XML_pattern_provider [2] tmf/org.eclipse.tracecompass.analysis.xml.core.tests/test_xml_files/test_valid/test_valid_pattern.xml On 06/22/2016 06:50 AM, Robbert Jongeling wrote:
|