Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
[cdt-dev] Junit testing of CDT components

Hey all,
I have been working on the CDT JUnit test suites, and wanted to put a question out to the community.

My question related to test cases that trigger bugs that have bugzilla reports against them, but will not be fixed for a while. In this case with the current setup these failures will show up every time the tests are run, and make it a bit harder to pull out the new failures. What do people think we should do about this? Some options that have come to mind are:

1 - Do nothing, accept that this is how the results look, and take the time to pull out new failures.

2 - Reverse the failure/assertion condition, and put a bugzilla id into the output string. This would hide the failure until the bug was fixed.. Pros of this are that it would not show up in the results, so any failures or asserts would have to be new issues and make it easy to catch things as they broke. As well, when the issue was fixed the it would show up in the logs and be easy to track back to bugzilla, and close the pr/update the test. The cons here is that we would manually have to go through and update the test cases when things changed, and it would be hard to keep track of the open issues that are being tested for as they would not show up in the test results.

3 - Post process out the failures with an associated bugzilla id. We could setup the results processing to search for failures where the output message started with PR: and then assume they are known issues. We would then have to make sure when problems were fixed, that the PR number was removed from the output message. Pros: Fairly easy to maintain, and we can get a bit more information on the number of problem reports being tested. Cons: If a problem was fixed, and the test case was not updated to remove the PR number, if the bug resurfaced it would not show up in the logs. As well if the test was not updated it would be hard to find it to clean it up after the fact.

4 - Modify junit to have an idea of expected failures. Pros - get exactly what we want/need Cons - way more work then I think we want to get into..

5 - Something else?

Any comments/ideas?


Back to the top