Re: [jakarta.ee-spec.committee] alternate tests

["Kazumura, Kenji" <kzr@xxxxxxxxxxxxxx>]

+1
Either tests or spec must be changed.

-Kenji Kazumura


> -----Original Message-----
> From: jakarta.ee-spec.committee-bounces@xxxxxxxxxxx
> [mailto:jakarta.ee-spec.committee-bounces@xxxxxxxxxxx] On Behalf Of Scott Stark
> Sent: Saturday, June 22, 2019 9:30 AM
> To: Jakarta specification committee <jakarta.ee-spec.committee@xxxxxxxxxxx>
> Subject: Re: [jakarta.ee-spec.committee] alternate tests
> 
> I don’t see this as any different from challenge that should result in an exclude and a temporary drop in tests. If
> we can’t write tests that don’t work across implementations there is a problem with the spec. I would rather not
> have to deal with alternative tests to simplify the process.
> 
> > On Jun 21, 2019, at 4:32 PM, Bill Shannon <bill.shannon@xxxxxxxxxx> wrote:
> >
> > One of the open issues in our TCK Process document is the use of
> > alternate tests to resolve test challenges.
> >
> > Assuming we continue with the approach of not allowing the required
> > tests to increase except when a new spec version is released, we
> > run the risk of test challenges decreasing the number of tests over
> > the lifetime of a spec version as broken or bad tests are excluded.
> >
> > The way we've dealt with this in the past is to allow for the addition
> > of alternate tests.  For example, if TestA is passed by 2 implementations
> > but fails for the 3rd implementation because the test is buggy and
> > includes an implementation dependency that doesn't work with the 3rd
> > implementation, what should we do?  If the test is a minor test and
> > excluding it has little impact on the test coverage, that is often
> > the simplest approach.
> >
> > But if TestA is testing a key API, we might not want to suffer the
> > reduced test coverage if we exclude the test.  Instead, we would introduce
> > TestA', an updated version of TestA that fixes the bug or removes the
> > implementation dependency, as an alternate test.  Implementations
> > could choose whether to pass TestA or TestA'.  That allows us to retain
> > the same test coverage without invalidating the results of the first
> > 2 implementations.
> >
> > As currently written, our TCK Process does not allow for alternate tests.
> >
> > Do we want to allow for alternate tests?
> > _______________________________________________
> > jakarta.ee-spec.committee mailing list
> > jakarta.ee-spec.committee@xxxxxxxxxxx
> > To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> > https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec.committee
> 
> _______________________________________________
> jakarta.ee-spec.committee mailing list
> jakarta.ee-spec.committee@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec.committee