Re: [jakarta.ee-spec.committee] alternate tests

[Scott Stark <sstark@xxxxxxxxxx>]

I don’t see this as any different from challenge that should result in an exclude and a temporary drop in tests. If we can’t write tests that don’t work across implementations there is a problem with the spec. I would rather not have to deal with alternative tests to simplify the process.

> On Jun 21, 2019, at 4:32 PM, Bill Shannon <bill.shannon@xxxxxxxxxx> wrote:
> 
> One of the open issues in our TCK Process document is the use of
> alternate tests to resolve test challenges.
> 
> Assuming we continue with the approach of not allowing the required
> tests to increase except when a new spec version is released, we
> run the risk of test challenges decreasing the number of tests over
> the lifetime of a spec version as broken or bad tests are excluded.
> 
> The way we've dealt with this in the past is to allow for the addition
> of alternate tests.  For example, if TestA is passed by 2 implementations
> but fails for the 3rd implementation because the test is buggy and
> includes an implementation dependency that doesn't work with the 3rd
> implementation, what should we do?  If the test is a minor test and
> excluding it has little impact on the test coverage, that is often
> the simplest approach.
> 
> But if TestA is testing a key API, we might not want to suffer the
> reduced test coverage if we exclude the test.  Instead, we would introduce
> TestA', an updated version of TestA that fixes the bug or removes the
> implementation dependency, as an alternate test.  Implementations
> could choose whether to pass TestA or TestA'.  That allows us to retain
> the same test coverage without invalidating the results of the first
> 2 implementations.
> 
> As currently written, our TCK Process does not allow for alternate tests.
> 
> Do we want to allow for alternate tests?
> _______________________________________________
> jakarta.ee-spec.committee mailing list
> jakarta.ee-spec.committee@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec.committee