Home » Archived » Test and Performance Tools Platform (TPTP) » Different variations of a rule without killing perf?
| Different variations of a rule without killing perf? [message #61144] |
Tue, 28 March 2006 15:31  |
Eclipse User |
|
|
|
Originally posted by: stuart.a.ballard.gmail.com
I'm writing static analysis rules which, if all goes according to plan,
will be run against a very large codebase, so performance is important. To
make matters worse, my rules pretty much require looking at the body of
every method (at least, I haven't yet found a filter that will allow me to
efficiently trim the set of methods to be looked at).
Furthermore, each of the overall rules that I want to write has about
three different variations which should be able to be enabled or disabled
independently. I can, of course, write entirely separate rules for each
variation, but that would involve running a chunk of common code once for
each variation - which triples the performance problem mentioned in the
first paragraph.
What I'd ideally like to be able to do is write code like:
foreach (MethodDeclaration meth in methods) {
if (methodMeetsFirstCriterion(meth)) {
if (variation1Enabled() && variation1(meth)) {
reportResult(meth, "First criterion, variation 1");
} else if (variation2Enabled() && variation2(meth)) {
reportResult(meth, "First criterion, variation 2");
} else if (variation3Enabled() && variation3(meth)) {
reportResult(meth, "First criterion, variation 3");
}
}
if (methodMeetsSecondCriterion(meth)) {
...
}
}
Is there a way to do something in a performant way that will produce
results equivalent to this, assuming that the methodMeetsNthCriterion()
and variationN() are nontrivial amounts of code?
(They're not *tremendously* complicated, but I also imagine that looping
over every method in a large codebase has a pretty significant overhead of
its own, in which case I'd rather only pay that once than three times per
criterion. Feel free to correct me if my assumptions are faulty and it's
really cheap to do that)
|
|
|
| Re: Different variations of a rule without killing perf? [message #61525 is a reply to message #61144] |
Tue, 28 March 2006 18:52  |
 Steve Gutz
Messages: 70
Registered: July 2009 |
Member |
|
|
What you want to do is something we routinely do in the rules. You will
find that most of time taken to analysis a set of resources is actually
associated with building the parse tree. Even very complex rules will
generally accumulate only a few seconds of execution time over several
hundred classes. On a garden-variety PC you will find that analyzing
several thousand files will take only a couple minutes - less if you are
only running your handful of rules.
Depending on the meaning of "methodMeetsFirstCriterion" you may be able to
use some of the RuleFiltering support to quickly reduce the size of the data
you need to manage. The data is stored in linked lists which are fairly
responsive by nature, but reducing the size
If you want to get more specific I can help you out. If your rule is
intellectually sensitive we can take the discussion off-line.
Will the variation flags be managed through RuleParameters or do you have
some sort of external property to control these? You would want the former
if you want to let the user change the values at runtime.
Steve
"Stuart Ballard" <stuart.a.ballard@gmail.com> wrote in message
news:c4004a3f9d52a62d3d901c802356b676$1@www.eclipse.org...
> I'm writing static analysis rules which, if all goes according to plan,
> will be run against a very large codebase, so performance is important. To
> make matters worse, my rules pretty much require looking at the body of
> every method (at least, I haven't yet found a filter that will allow me to
> efficiently trim the set of methods to be looked at).
>
> Furthermore, each of the overall rules that I want to write has about
> three different variations which should be able to be enabled or disabled
> independently. I can, of course, write entirely separate rules for each
> variation, but that would involve running a chunk of common code once for
> each variation - which triples the performance problem mentioned in the
> first paragraph.
>
> What I'd ideally like to be able to do is write code like:
>
> foreach (MethodDeclaration meth in methods) {
> if (methodMeetsFirstCriterion(meth)) {
> if (variation1Enabled() && variation1(meth)) {
> reportResult(meth, "First criterion, variation 1");
> } else if (variation2Enabled() && variation2(meth)) {
> reportResult(meth, "First criterion, variation 2");
> } else if (variation3Enabled() && variation3(meth)) {
> reportResult(meth, "First criterion, variation 3");
> }
> }
> if (methodMeetsSecondCriterion(meth)) {
> ...
> }
> }
>
> Is there a way to do something in a performant way that will produce
> results equivalent to this, assuming that the methodMeetsNthCriterion()
> and variationN() are nontrivial amounts of code?
>
> (They're not *tremendously* complicated, but I also imagine that looping
> over every method in a large codebase has a pretty significant overhead of
> its own, in which case I'd rather only pay that once than three times per
> criterion. Feel free to correct me if my assumptions are faulty and it's
> really cheap to do that)
>
|
|
|
| Re: Different variations of a rule without killing perf? [message #61625 is a reply to message #61525] |
Tue, 28 March 2006 21:15 |
| Eclipse User |
|
|
|
Originally posted by: stuart.a.ballard.gmail.com
Steve Gutz wrote:
> What you want to do is something we routinely do in the rules. You will
> find that most of time taken to analysis a set of resources is actually
> associated with building the parse tree. Even very complex rules will
> generally accumulate only a few seconds of execution time over several
> hundred classes. On a garden-variety PC you will find that analyzing
> several thousand files will take only a couple minutes - less if you are
> only running your handful of rules.
So does it only build the parse tree once and then run all the enabled
rules over the same parse tree? That makes sense and would definitely
mitigate my concerns :)
> Depending on the meaning of "methodMeetsFirstCriterion" you may be able to
> use some of the RuleFiltering support to quickly reduce the size of the data
> you need to manage. The data is stored in linked lists which are fairly
> responsive by nature, but reducing the size
So far I'm only in the early stages of experimentation but I'm not sure if
there's a way to filter that will achieve what I want. I basically need to
filter on the type of the first statement in the method body (or on the
fact that the method body has *no* statements).
> If you want to get more specific I can help you out. If your rule is
> intellectually sensitive we can take the discussion off-line.
Not at all, it's for an open source project, so I'll keep the discussion
here. I was just trying not to put in too many irrelevant details to keep
the question simple :) I actually mentioned the nature of the rules in the
jdt newsgroup in the question that got me pointed to tptp in the first
place. The criteria I'm trying to match are:
- Find methods with no statements in the body.
- Find void methods where the first statement in the body is "return;".
- Find methods where the first statement in the body is a throw statement.
- Find methods where the first statement in the body is "return " and the
expression being returned is a literal constant.
- Find void methods where the only statement is a call to the superclass
implementation of the same method with the same arguments.
- Find methods where the first statement is "return " and the expression
is the result of invoking the superclass implementation of the same method
with the same arguments.
- Find methods that throw any exception called "NotImplementedException",
regardless of what package that NotImplementedException is in.
The "variations" I was talking about are that I need to be able to break
the results down further based on the existence, or not, of a particular
comment in the body or whether they throw NotImplementedException *as
well* as meeting one of the other criteria.
> Will the variation flags be managed through RuleParameters or do you have
> some sort of external property to control these? You would want the former
> if you want to let the user change the values at runtime.
Aha! I hadn't discovered RuleParameters but (based on the name, anyway)
they sound like exactly what I was looking for. I was thinking I'd need to
have a separate rule for each variation, which was the main reason I was
worried about perf.
On the other hand, I do want to get different messages in the output
depending on which variation is met. Is that possible?
I'll look into the docs to see if these RuleParameter thingies do what I
want them to :)
Stuart.
|
|
|
| Re: Different variations of a rule without killing perf? [message #62014 is a reply to message #61625] |
Wed, 29 March 2006 13:12 |
Steve Gutz
Messages: 70
Registered: July 2009 |
Member |
|
|
Most of what you need to do is fairly simple (at least compared to some
rules I've written). Here is something to get you started. This rule
should find empty methods:
// Imports removed
public class RuleEmptyMethod extends AbstractAnalysisRule {
public void analyze( AnalysisHistory history) {
// Extract the resource being analyzed
CodeReviewResource resource = (CodeReviewResource)
getProvider().getProperty(history.getHistoryId(),
CodeReviewProvider.RESOURCE_PROPERTY);
List list =
resource.getTypedNodeList(resource.getResourceCompUnit(),
ASTNode.METHOD_DECLARATION);
for( Iterator it = list.iterator(); it.hasNext(); ) {
MethodDeclaration md = (MethodDeclaration)it.next();
if( md.getBody().statements().size() == 0 ) {
resource.generateResultsForASTNode(this,
history.getHistoryId(), md.getName() );
}
}
}
}
You can use this as the foundation for most of your other rules since they
all seems to work with statements.
BTW you are correct that there isn't a RuleFilter to do what you want for
this. However you have inspired me to make some improvements in the API to
let you create your own filters - not sure why I didn't think of this before
:-)
Steve
"Stuart Ballard" <stuart.a.ballard@gmail.com> wrote in message
news:f33f3f00d987107d79b325c5cd790c7b$1@www.eclipse.org...
> Steve Gutz wrote:
> > What you want to do is something we routinely do in the rules. You will
> > find that most of time taken to analysis a set of resources is actually
> > associated with building the parse tree. Even very complex rules will
> > generally accumulate only a few seconds of execution time over several
> > hundred classes. On a garden-variety PC you will find that analyzing
> > several thousand files will take only a couple minutes - less if you are
> > only running your handful of rules.
>
> So does it only build the parse tree once and then run all the enabled
> rules over the same parse tree? That makes sense and would definitely
> mitigate my concerns :)
>
> > Depending on the meaning of "methodMeetsFirstCriterion" you may be able
to
> > use some of the RuleFiltering support to quickly reduce the size of the
data
> > you need to manage. The data is stored in linked lists which are fairly
> > responsive by nature, but reducing the size
>
> So far I'm only in the early stages of experimentation but I'm not sure if
> there's a way to filter that will achieve what I want. I basically need to
> filter on the type of the first statement in the method body (or on the
> fact that the method body has *no* statements).
>
> > If you want to get more specific I can help you out. If your rule is
> > intellectually sensitive we can take the discussion off-line.
>
> Not at all, it's for an open source project, so I'll keep the discussion
> here. I was just trying not to put in too many irrelevant details to keep
> the question simple :) I actually mentioned the nature of the rules in the
> jdt newsgroup in the question that got me pointed to tptp in the first
> place. The criteria I'm trying to match are:
>
> - Find methods with no statements in the body.
> - Find void methods where the first statement in the body is "return;".
> - Find methods where the first statement in the body is a throw statement.
> - Find methods where the first statement in the body is "return " and the
> expression being returned is a literal constant.
> - Find void methods where the only statement is a call to the superclass
> implementation of the same method with the same arguments.
> - Find methods where the first statement is "return " and the expression
> is the result of invoking the superclass implementation of the same method
> with the same arguments.
> - Find methods that throw any exception called "NotImplementedException",
> regardless of what package that NotImplementedException is in.
>
> The "variations" I was talking about are that I need to be able to break
> the results down further based on the existence, or not, of a particular
> comment in the body or whether they throw NotImplementedException *as
> well* as meeting one of the other criteria.
>
> > Will the variation flags be managed through RuleParameters or do you
have
> > some sort of external property to control these? You would want the
former
> > if you want to let the user change the values at runtime.
>
> Aha! I hadn't discovered RuleParameters but (based on the name, anyway)
> they sound like exactly what I was looking for. I was thinking I'd need to
> have a separate rule for each variation, which was the main reason I was
> worried about perf.
>
> On the other hand, I do want to get different messages in the output
> depending on which variation is met. Is that possible?
>
> I'll look into the docs to see if these RuleParameter thingies do what I
> want them to :)
>
> Stuart.
>
|
|
|
| Re: Different variations of a rule without killing perf? [message #62103 is a reply to message #62014] |
Wed, 29 March 2006 14:08 |
| Eclipse User |
|
|
|
Originally posted by: stuart.a.ballard.gmail.com
Steve Gutz wrote:
> Most of what you need to do is fairly simple (at least compared to some
> rules I've written). Here is something to get you started. This rule
> should find empty methods:
Yeah, I've got the simpler rules working at this point, although I'm still
having difficulty coming up with an architecture that avoids doing work
several times over. I'll try to find the option in Eclipse to export what
I've got into a zip so I can post it somewhere to better demonstrate my
problem.
One suggestion that would have saved me some pain: the tutorial at
http://www.eclipse.org/tptp/home/documents/process/developme nt/static_analysis/TPTP_static_analysis_tutorial_part2.html
(which is *incredibly* useful, everything I've accomplished so far came
pretty much straight from there) is just a little bit *too* terse when it
comes to quickFixes. It mentions that the quickfix() method receives the
analysis result being fixed, but it doesn't mention that you need to cast
it to CodeReviewResult to get access to the getMyNode() method to find
what you're actually fixing. I spent quite a while hunting through all the
methods declared on IAnalysisResult and AbstractAnalysisQuickFix and
AbstractAnalysisRule trying to get at this information before I eventually
found the solution by using the debugger to find out what the actual class
of the object was.
BTW, what would be the right newsgroup to ask questions about general AST
manipulation as required for implementing a quickfix?
Thanks very much for all your help!
Stuart.
|
|
|
| Re: Different variations of a rule without killing perf? [message #62150 is a reply to message #62103] |
Wed, 29 March 2006 14:30 |
| Eclipse User |
|
|
|
Originally posted by: stuart.a.ballard.gmail.com
Stuart Ballard wrote:
> I'll try to find the option in Eclipse to export what
> I've got into a zip so I can post it somewhere to better demonstrate my
> problem.
Ok, that was apparently easy enough. I *think* that
http://sab39.dev.netreach.com/org.nongnu.japitools.stubfinde r_1.0.0.jar
should be a viable plugin including the source.
Basically, as you'll see if you go into the "Japitools stub methods"
analysis category, I have five rules. "Empty method" and "Unconditional
throw" are the two out of my original set that I've implemented so far.
"Marked non-stub" is methods that match one of these two criteria but
contain a comment indicating that this is correct and expected - "Nothing
to do here" or "Always throw <type>" (these are skipped by the "empty
method stub" and "unconditional throw" rules). "Flagged Stub" is anything
that throws NotImplementedException (these are also excluded from all the
other rules). "Uncharacteristic Flagged Stub" is anything that throws
NotImplementedException that *doesn't* match "Empty Method" or
"Unconditional Throw".
So basically the problem is that in order to implement these five
different categories of result, I need to do the work of determining "is
it an empty method", say, at least three times (for "Empty method", for
"Marked non-stub" and for "Uncharacteristic flagged stub"). I need to
determine if it throws NotImplementedException *five* times because every
single rule relies on that knowledge. And so on.
What I'm looking for is whether there's some way I could provide these
five (and more, when I finish my list of criteria) different categories of
result without doing so much redundant work on each method.
Thanks again!
Stuart.
PS I'm testing on the following class (with a trivial declaration of
NotImplementedException to make it compile):
public class TestClass {
public void emptyMethod() {
}
public void emptyWithComment() {
// This is a comment.
}
public void emptySkip() {
// Nothing to do here.
}
public void throwMethod() {
throw new RuntimeException();
}
public void throwWithSkip() {
// Always throw RuntimeException.
throw new RuntimeException();
}
public void flaggedEmptyMethod() throws NotImplementedException {
}
public void flaggedEmptyWithComment() throws NotImplementedException {
// This is a comment.
}
public void flaggedEmptySkip() throws NotImplementedException {
// Nothing to do here.
}
public void flaggedThrowMethod() throws NotImplementedException {
throw new RuntimeException();
}
public void flaggedThrowWithSkip() throws NotImplementedException {
// Always throw RuntimeException.
throw new RuntimeException();
}
public void flaggedUncharacteristic() throws NotImplementedException {
System.out.println("This does something!");
}
}
|
|
|
| Re: Different variations of a rule without killing perf? [message #62629 is a reply to message #62150] |
Thu, 30 March 2006 12:03 |
Steve Gutz
Messages: 70
Registered: July 2009 |
Member |
|
|
Well if it's the same method you're comparing for each rule you could use a
provider property to cache it, though you need to be carefule with this
since you can leave object hanging around if you're not careful.
for your rule you can do:
rule.getProvider().setProperty( "myproperty", value) and,
value = rule.getProvider().getProperty( "myproperty" )
These properties span the scope of a rule so you can exploit this to share
information during your analysis session. The first time you can look to
see if the property has a value, if not you can do your empty mthod check
and set the property. In subsequent checks you'll be able to just read your
true/false value from the property.
Steve
"Stuart Ballard" <stuart.a.ballard@gmail.com> wrote in message
news:d202eae3ce80a3d0b75cd3ef315ce4ba$1@www.eclipse.org...
> Stuart Ballard wrote:
> > I'll try to find the option in Eclipse to export what
> > I've got into a zip so I can post it somewhere to better demonstrate my
> > problem.
>
> Ok, that was apparently easy enough. I *think* that
> http://sab39.dev.netreach.com/org.nongnu.japitools.stubfinde r_1.0.0.jar
> should be a viable plugin including the source.
>
> Basically, as you'll see if you go into the "Japitools stub methods"
> analysis category, I have five rules. "Empty method" and "Unconditional
> throw" are the two out of my original set that I've implemented so far.
> "Marked non-stub" is methods that match one of these two criteria but
> contain a comment indicating that this is correct and expected - "Nothing
> to do here" or "Always throw <type>" (these are skipped by the "empty
> method stub" and "unconditional throw" rules). "Flagged Stub" is anything
> that throws NotImplementedException (these are also excluded from all the
> other rules). "Uncharacteristic Flagged Stub" is anything that throws
> NotImplementedException that *doesn't* match "Empty Method" or
> "Unconditional Throw".
>
> So basically the problem is that in order to implement these five
> different categories of result, I need to do the work of determining "is
> it an empty method", say, at least three times (for "Empty method", for
> "Marked non-stub" and for "Uncharacteristic flagged stub"). I need to
> determine if it throws NotImplementedException *five* times because every
> single rule relies on that knowledge. And so on.
>
> What I'm looking for is whether there's some way I could provide these
> five (and more, when I finish my list of criteria) different categories of
> result without doing so much redundant work on each method.
>
> Thanks again!
> Stuart.
>
> PS I'm testing on the following class (with a trivial declaration of
> NotImplementedException to make it compile):
>
> public class TestClass {
> public void emptyMethod() {
> }
> public void emptyWithComment() {
> // This is a comment.
> }
> public void emptySkip() {
> // Nothing to do here.
> }
> public void throwMethod() {
> throw new RuntimeException();
> }
> public void throwWithSkip() {
> // Always throw RuntimeException.
> throw new RuntimeException();
> }
> public void flaggedEmptyMethod() throws NotImplementedException {
> }
> public void flaggedEmptyWithComment() throws NotImplementedException {
> // This is a comment.
> }
> public void flaggedEmptySkip() throws NotImplementedException {
> // Nothing to do here.
> }
> public void flaggedThrowMethod() throws NotImplementedException {
> throw new RuntimeException();
> }
> public void flaggedThrowWithSkip() throws NotImplementedException {
> // Always throw RuntimeException.
> throw new RuntimeException();
> }
> public void flaggedUncharacteristic() throws NotImplementedException {
> System.out.println("This does something!");
> }
> }
>
>
|
|
|
Goto Forum:
Current Time: Fri Apr 26 14:07:35 GMT 2024
Powered by FUDForum. Page generated in 0.03224 seconds |
] 
Search
Help
Register
Login
Home
