Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jakarta.ee-spec.committee] alternate tests

One more thing, for completeness -- In the case where alternative tests are provided, it is up to a vendor to decide which alternative they wish to use.

The idea is to prevent creation of a moving target.

-- Ed

On 6/25/2019 9:33 AM, Ed Bratt wrote:

For what it's worth, the current TCK User Guides contain details about test alternatives. Effectively, if an alternative is provided it serves the same function as the original test. A vendor implementation must pass one, or the other. In that text, the decision to create an alternate test is at the discretion of the Maintenance Lead -- for Jakarta, that would most likely translate to the Specification Committee Committer team.

This text will need to be revised in all the user guides, regardless which direction is settled on.

I personally don't mind keeping the option open to providing alternative tests. I am not aware that this has presented problems in the past.

-- Ed

On 6/25/2019 2:23 AM, Steve Millidge (Payara) wrote:

+1 I agree with the desire for a more rapid release cadence and the openness of the TCK during API development I think this can be mitigated.

 

From: jakarta.ee-spec.committee-bounces@xxxxxxxxxxx <jakarta.ee-spec.committee-bounces@xxxxxxxxxxx> On Behalf Of Kevin Sutter
Sent: 24 June 2019 20:48
To: Jakarta specification committee <jakarta.ee-spec.committee@xxxxxxxxxxx>
Subject: Re: [jakarta.ee-spec.committee] alternate tests

 

Agree.  Let's leave the idea of "alternate tests" out for now.  We can adjust later if the need arises.


---------------------------------------------------
Kevin Sutter
STSM, MicroProfile and Jakarta EE architect
e-mail:  sutter@xxxxxxxxxx     Twitter:  @kwsutter
phone: tl-553-3620 (office), 507-253-3620 (office)    
LinkedIn: https://www.linkedin.com/in/kevinwsutter



From:        "Kazumura, Kenji" <kzr@xxxxxxxxxxxxxx>
To:        Jakarta specification committee <jakarta.ee-spec.committee@xxxxxxxxxxx>
Date:        06/23/2019 01:46 AM
Subject:        [EXTERNAL] Re: [jakarta.ee-spec.committee] alternate tests
Sent by:        jakarta.ee-spec.committee-bounces@xxxxxxxxxxx




+1
Either tests or spec must be changed.

-Kenji Kazumura


> -----Original Message-----
> From: jakarta.ee-spec.committee-bounces@xxxxxxxxxxx
> [mailto:jakarta.ee-spec.committee-bounces@xxxxxxxxxxx] On Behalf Of Scott Stark
> Sent: Saturday, June 22, 2019 9:30 AM
> To: Jakarta specification committee <jakarta.ee-spec.committee@xxxxxxxxxxx>
> Subject: Re: [jakarta.ee-spec.committee] alternate tests
>
> I don’t see this as any different from challenge that should result in an exclude and a temporary drop in tests. If
> we can’t write tests that don’t work across implementations there is a problem with the spec. I would rather not
> have to deal with alternative tests to simplify the process.
>
> > On Jun 21, 2019, at 4:32 PM, Bill Shannon <bill.shannon@xxxxxxxxxx> wrote:
> >
> > One of the open issues in our TCK Process document is the use of
> > alternate tests to resolve test challenges.
> >
> > Assuming we continue with the approach of not allowing the required
> > tests to increase except when a new spec version is released, we
> > run the risk of test challenges decreasing the number of tests over
> > the lifetime of a spec version as broken or bad tests are excluded.
> >
> > The way we've dealt with this in the past is to allow for the addition
> > of alternate tests.  For example, if TestA is passed by 2 implementations
> > but fails for the 3rd implementation because the test is buggy and
> > includes an implementation dependency that doesn't work with the 3rd
> > implementation, what should we do?  If the test is a minor test and
> > excluding it has little impact on the test coverage, that is often
> > the simplest approach.
> >
> > But if TestA is testing a key API, we might not want to suffer the
> > reduced test coverage if we exclude the test.  Instead, we would introduce
> > TestA', an updated version of TestA that fixes the bug or removes the
> > implementation dependency, as an alternate test.  Implementations
> > could choose whether to pass TestA or TestA'.  That allows us to retain
> > the same test coverage without invalidating the results of the first
> > 2 implementations.
> >
> > As currently written, our TCK Process does not allow for alternate tests.
> >
> > Do we want to allow for alternate tests?
> > _______________________________________________
> > jakarta.ee-spec.committee mailing list
> > jakarta.ee-spec.committee@xxxxxxxxxxx
> > To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> > https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec.committee
>
> _______________________________________________
> jakarta.ee-spec.committee mailing list
> jakarta.ee-spec.committee@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe from this list, visit
> https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec.committee
_______________________________________________
jakarta.ee-spec.committee mailing list
jakarta.ee-spec.committee@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec.committee



_______________________________________________
jakarta.ee-spec.committee mailing list
jakarta.ee-spec.committee@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec.committee


_______________________________________________
jakarta.ee-spec.committee mailing list
jakarta.ee-spec.committee@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec.committee

Back to the top