[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [jakarta.ee-spec.committee] alternate tests
|
I'm fine with adding alternative tests so we can get on with the show and avoid the coverage decrease, which is a major and valid concern.
My personal experience is between excludes and alternatives, the process gets complicated quickly.
We haven't had the benefit of licensees being able to talk to each other or work on tests together over the last 20 years. I am confident over time we will start to see less objection to simply fixing tests and releasing a patched TCK.
-David
> On Jun 21, 2019, at 4:32 PM, Bill Shannon <bill.shannon@xxxxxxxxxx> wrote:
>
> One of the open issues in our TCK Process document is the use of
> alternate tests to resolve test challenges.
>
> Assuming we continue with the approach of not allowing the required
> tests to increase except when a new spec version is released, we
> run the risk of test challenges decreasing the number of tests over
> the lifetime of a spec version as broken or bad tests are excluded.
>
> The way we've dealt with this in the past is to allow for the addition
> of alternate tests. For example, if TestA is passed by 2 implementations
> but fails for the 3rd implementation because the test is buggy and
> includes an implementation dependency that doesn't work with the 3rd
> implementation, what should we do? If the test is a minor test and
> excluding it has little impact on the test coverage, that is often
> the simplest approach.
>
> But if TestA is testing a key API, we might not want to suffer the
> reduced test coverage if we exclude the test. Instead, we would introduce
> TestA', an updated version of TestA that fixes the bug or removes the
> implementation dependency, as an alternate test. Implementations
> could choose whether to pass TestA or TestA'. That allows us to retain
> the same test coverage without invalidating the results of the first
> 2 implementations.
>
> As currently written, our TCK Process does not allow for alternate tests.
>
> Do we want to allow for alternate tests?
> _______________________________________________
> jakarta.ee-spec.committee mailing list
> jakarta.ee-spec.committee@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec.committee
