Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
RE: [wtp-dev] Thursday I-build schedule

+1 for "Build will not be declared a success if there are any test errors"

The "easy way out" was intended to be a very temporary condition, just until M4 is complete (which is not that far off!) intended to make it easier for developers to specify proposed API, provide a JUnit test for it in advance so it could be counted by the test counter, and have the proposed-api-junit failure not prevent the build from being declared a success.

But, its not working. I'm not sure if its just a bad idea or if component leads just haven't been communicating well. (And, I say this knowing full well I'm one of the culprits, allowing failing xml validation tests last week to go unmentioned and unexplained -- they are non critical but should have been removed until fixed).

So, fellow component leads .. please make sure tests are clean this week, or communicate clearly if they are failing due to the "provide JUnit tests in advance" policy.
That way, we'll be able to easily decide next week if a change in policy/procedure is needed. (my current impression is we do not have any that are failing due to the "specified in advance" policy)

And, IMHO, I think the main criteria of success of an I-build is if it can be used as a target for further development, so, in many cases, its legitimate to remove failing tests if they are failing due to testing some fringe cases, etc. Might it even be possible to provide the 'proposed-api-tests" and rig things to always have them "fake" success? Long long term, it would be great if the JUnit framework provided a way to specify the "level" of the test, right in the test ... critical, noncritical, stress-only, performance, etc. .... but that's a Release 2 item, if even then.

Thanks for the comments Ted -- it helps to have another perspective.  And three cheers for continuos improvement through continuous iteration :)

Other's opinions welcome.

"Ted Bashor" <tbashor@xxxxxxx>
Sent by: wtp-dev-bounces@xxxxxxxxxxx

04/06/2005 01:32 PM

Please respond to
"General discussion of project-wide or architectural issues."

"General discussion of project-wide or architectural issues." <wtp-dev@xxxxxxxxxxx>
RE: [wtp-dev] Thursday I-build schedule

"Component owners must justify test errors if they want to declare it a pass despite errors."
Is there any way we can change that policy to "Build will not be declared a success if there are any test errors"?

Seems to me that either the error is critical, and the build is unstable, or it's non-critical, and the test should be commented out until there's time to fix it.  The impression one gets is that failing tests are allowed to hang out indefinitely through the milestone.

This "successful with errors" state is really inconvenient for someone trying to determine whether to adopt the build.  Ideally one should be able to look at the download site and see a green check or red X and get a general sense of stability.  Obviously there are going to be bugs and unfinished work in an integration build, but I think we really need a binary thumbs up or thumbs down that doesn't require digging through email.  

If people really don't want to comment out tests, we could have two status flags, one for "no test errors" and one for "test errors but declared sucessful", but to me that indicates the need for separate test suites.

Another recommendation is that test results start going out with the first candidate build (that builds successfully).  I know that's more spam to the mailing list, but I think it will give us a better shot at achieving a clean I-build sometime between Tues and Thurs.

Thanks, Ted

                -----Original Message-----
                From: wtp-dev-bounces@xxxxxxxxxxx on behalf of David M Williams
                Sent: Wed 4/6/2005 9:35 AM
                To: wtp-dev@xxxxxxxxxxx
                Subject: [wtp-dev] Thursday I-build schedule

                Thanks goes to Ozgur for confirming. And thinking generally that if I'm confused others might be (which ... I know ... is seldom the case :) I thought I'd re-post the I-build schedule for Thursday. And, now that most of us have gone through daylight-savings-time change, I will even do the math for EDT:
                        midnight, 11:30 AM, 6:30 PM.
                And, BTW, no need to wait till asked for "Component owners must justify test errors" part. Let us all know status of failed tests here on wtp-dev.
                = = = = =  original post = = = = =
                Builds are started on 4:00 AM GMT,  15:30 GMT, 22:30 GMT
                Failure Policy:
                Rebuild until success.  (Success will be achieved when there are no compile/testing errors).  Component owners must justify test errors if they want to declare it a pass despite errors.  All other unfixed errors will be the source of public humiliation. You must fix your errors.  This becomes the top priority on thursdays.

wtp-dev mailing list

Back to the top