Re: [jakarta.ee-spec.committee] alternate tests

["Kevin Sutter" <sutter@xxxxxxxxxx>]

Agree.  Let's
leave the idea of "alternate tests" out for now.  We can
adjust later if the need arises.


---------------------------------------------------
Kevin Sutter 
STSM, MicroProfile and Jakarta EE architect
e-mail:  sutter@xxxxxxxxxx     Twitter:  @kwsutter
phone: tl-553-3620 (office), 507-253-3620 (office)    
LinkedIn: https://www.linkedin.com/in/kevinwsutter



From:
       "Kazumura,
Kenji" <kzr@xxxxxxxxxxxxxx>
To:
       Jakarta
specification committee <jakarta.ee-spec.committee@xxxxxxxxxxx>
Date:
       06/23/2019
01:46 AM
Subject:
       [EXTERNAL]
Re: [jakarta.ee-spec.committee] alternate tests
Sent
by:        jakarta.ee-spec.committee-bounces@xxxxxxxxxxx

---

+1
Either tests or spec must be changed.

-Kenji Kazumura


> -----Original Message-----
> From: jakarta.ee-spec.committee-bounces@xxxxxxxxxxx
> [mailto:jakarta.ee-spec.committee-bounces@xxxxxxxxxxx]
On Behalf Of Scott Stark
> Sent: Saturday, June 22, 2019 9:30 AM
> To: Jakarta specification committee <jakarta.ee-spec.committee@xxxxxxxxxxx>
> Subject: Re: [jakarta.ee-spec.committee] alternate tests
> 
> I don’t see this as any different from challenge that should result
in an exclude and a temporary drop in tests. If
> we can’t write tests that don’t work across implementations there
is a problem with the spec. I would rather not
> have to deal with alternative tests to simplify the process.
> 
> > On Jun 21, 2019, at 4:32 PM, Bill Shannon <bill.shannon@xxxxxxxxxx>
wrote:
> >
> > One of the open issues in our TCK Process document is the use
of
> > alternate tests to resolve test challenges.
> >
> > Assuming we continue with the approach of not allowing the required
> > tests to increase except when a new spec version is released,
we
> > run the risk of test challenges decreasing the number of tests
over
> > the lifetime of a spec version as broken or bad tests are excluded.
> >
> > The way we've dealt with this in the past is to allow for the
addition
> > of alternate tests.  For example, if TestA is passed by
2 implementations
> > but fails for the 3rd implementation because the test is buggy
and
> > includes an implementation dependency that doesn't work with
the 3rd
> > implementation, what should we do?  If the test is a minor
test and
> > excluding it has little impact on the test coverage, that is
often
> > the simplest approach.
> >
> > But if TestA is testing a key API, we might not want to suffer
the
> > reduced test coverage if we exclude the test.  Instead,
we would introduce
> > TestA', an updated version of TestA that fixes the bug or removes
the
> > implementation dependency, as an alternate test.  Implementations
> > could choose whether to pass TestA or TestA'.  That allows
us to retain
> > the same test coverage without invalidating the results of the
first
> > 2 implementations.
> >
> > As currently written, our TCK Process does not allow for alternate
tests.
> >
> > Do we want to allow for alternate tests?
> > _______________________________________________
> > jakarta.ee-spec.committee mailing list
> > jakarta.ee-spec.committee@xxxxxxxxxxx
> > To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
> > https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec.committee
> 
> _______________________________________________
> jakarta.ee-spec.committee mailing list
> jakarta.ee-spec.committee@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
> https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec.committee
_______________________________________________
jakarta.ee-spec.committee mailing list
jakarta.ee-spec.committee@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec.committee