[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
[cdt-dev] Junit testing of CDT components
|
Hey all,
I have been working on the CDT JUnit test suites, and wanted to put a
question out to the community.
My question related to test cases that trigger bugs that have bugzilla
reports against them, but will not be fixed for a while. In this case
with the current setup these failures will show up every time the tests
are run, and make it a bit harder to pull out the new failures. What do
people think we should do about this? Some options that have come to
mind are:
1 - Do nothing, accept that this is how the results look, and take the
time to pull out new failures.
2 - Reverse the failure/assertion condition, and put a bugzilla id into
the output string. This would hide the failure until the bug was
fixed.. Pros of this are that it would not show up in the results, so
any failures or asserts would have to be new issues and make it easy to
catch things as they broke. As well, when the issue was fixed the it
would show up in the logs and be easy to track back to bugzilla, and
close the pr/update the test. The cons here is that we would manually
have to go through and update the test cases when things changed, and it
would be hard to keep track of the open issues that are being tested for
as they would not show up in the test results.
3 - Post process out the failures with an associated bugzilla id. We
could setup the results processing to search for failures where the
output message started with PR: and then assume they are known issues.
We would then have to make sure when problems were fixed, that the PR
number was removed from the output message. Pros: Fairly easy to
maintain, and we can get a bit more information on the number of problem
reports being tested. Cons: If a problem was fixed, and the test case
was not updated to remove the PR number, if the bug resurfaced it would
not show up in the logs. As well if the test was not updated it would be
hard to find it to clean it up after the fact.
4 - Modify junit to have an idea of expected failures. Pros - get
exactly what we want/need Cons - way more work then I think we want to
get into..
5 - Something else?
Any comments/ideas?
Thanks
-Peter