+1 I agree with the desire for a more rapid release cadence and the openness of the TCK during API development I think this can be mitigated.
From: jakarta.ee-spec.committee-bounces@xxxxxxxxxxx <jakarta.ee-spec.committee-bounces@xxxxxxxxxxx>
On Behalf Of Kevin Sutter
Sent: 24 June 2019 20:48
To: Jakarta specification committee <jakarta.ee-spec.committee@xxxxxxxxxxx>
Subject: Re: [jakarta.ee-spec.committee] alternate tests
Agree. Let's leave the idea of "alternate tests" out for now. We can adjust later if the need arises.
---------------------------------------------------
Kevin Sutter
STSM, MicroProfile and Jakarta EE architect
e-mail: sutter@xxxxxxxxxx Twitter: @kwsutter
phone: tl-553-3620 (office), 507-253-3620 (office)
LinkedIn: https://www.linkedin.com/in/kevinwsutter
From: "Kazumura, Kenji" <kzr@xxxxxxxxxxxxxx>
To: Jakarta specification committee <jakarta.ee-spec.committee@xxxxxxxxxxx>
Date: 06/23/2019 01:46 AM
Subject: [EXTERNAL] Re: [jakarta.ee-spec.committee] alternate tests
Sent by: jakarta.ee-spec.committee-bounces@xxxxxxxxxxx
+1
Either tests or spec must be changed.
-Kenji Kazumura
> -----Original Message-----
> From: jakarta.ee-spec.committee-bounces@xxxxxxxxxxx
> [mailto:jakarta.ee-spec.committee-bounces@xxxxxxxxxxx] On Behalf Of Scott Stark
> Sent: Saturday, June 22, 2019 9:30 AM
> To: Jakarta specification committee <jakarta.ee-spec.committee@xxxxxxxxxxx>
> Subject: Re: [jakarta.ee-spec.committee] alternate tests
>
> I don’t see this as any different from challenge that should result in an exclude and a temporary drop in tests. If
> we can’t write tests that don’t work across implementations there is a problem with the spec. I would rather not
> have to deal with alternative tests to simplify the process.
>
> > On Jun 21, 2019, at 4:32 PM, Bill Shannon <bill.shannon@xxxxxxxxxx> wrote:
> >
> > One of the open issues in our TCK Process document is the use of
> > alternate tests to resolve test challenges.
> >
> > Assuming we continue with the approach of not allowing the required
> > tests to increase except when a new spec version is released, we
> > run the risk of test challenges decreasing the number of tests over
> > the lifetime of a spec version as broken or bad tests are excluded.
> >
> > The way we've dealt with this in the past is to allow for the addition
> > of alternate tests. For example, if TestA is passed by 2 implementations
> > but fails for the 3rd implementation because the test is buggy and
> > includes an implementation dependency that doesn't work with the 3rd
> > implementation, what should we do? If the test is a minor test and
> > excluding it has little impact on the test coverage, that is often
> > the simplest approach.
> >
> > But if TestA is testing a key API, we might not want to suffer the
> > reduced test coverage if we exclude the test. Instead, we would introduce
> > TestA', an updated version of TestA that fixes the bug or removes the
> > implementation dependency, as an alternate test. Implementations
> > could choose whether to pass TestA or TestA'. That allows us to retain
> > the same test coverage without invalidating the results of the first
> > 2 implementations.
> >
> > As currently written, our TCK Process does not allow for alternate tests.
> >
> > Do we want to allow for alternate tests?
> > _______________________________________________
> > jakarta.ee-spec.committee mailing list
> > jakarta.ee-spec.committee@xxxxxxxxxxx
> > To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> > https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec.committee
>
> _______________________________________________
> jakarta.ee-spec.committee mailing list
> jakarta.ee-spec.committee@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec.committee
_______________________________________________
jakarta.ee-spec.committee mailing list
jakarta.ee-spec.committee@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec.committee
