I'd like to know if you already have some ideas of the required number of code samples to have relatively accurate statistics over a specific framework. Do you have ways to compute the standard deviation or something like that?
In other words, I wonder how many developers of a specific framework must share their projects in the knowledge base for the data mining results to be relevant. So do you have some ideas of how I can estimate that?
This of course, get us back to the more general question of bad framework usage. What happens if the knowledge base is mostly populated with bad examples of framework usage? Is it possible to estimate at what time we achieved a number of analysis high enough to trust the results?
That's a real tough question to which I don't have a good answer yet. We didn't conduct experiments on how many examples we needed to create "good" results.
Just one example for illustration:
For (mid-size API) SWT Button we found approx. 160 different ways how developers used (i.e., invoked methods on) instances of SWT Button. These observations have been observed in thousands of different method contexts (e.g. PreferencePage#performOk, DialogPage#createContents etc.)
By now, you need at least one example for each method context - assuming that there is only one common way of how to use this type in this method. If you have different ways (like using a Button as check/radio/push button) more examples are needed to identify these patterns.
To be honest: I can't give you a good number yet. We should define some experiments and evaluation statistics on that. Do you (anyone?) have any concrete ideas how to evaluate this? I may find a student to work on this interesting question if we have a concrete vision of what to measure/achieve