Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jakarta.ee-spec.committee] TCK Issues to Resolve
Richard Monson-Haefel wrote on 08/22/18 05:24 AM:
The Draft of the TCK document (see link below) has been available and commented on.  More comments are welcome.

Note that people who haven't changed their notification preferences for this document to "notify me of all updates" are missing many important points and discussions.


While some issues are easy to resolve in comments on the document others require a more focused resolution. These other issues are enumerated below.  Please take the time to read and respond.
Thanks for moving these issues into email!  :-)

  1. TCK Challenges: When a vendor challenges a TCK test who should determine if the challenge is valid?   The following options have been suggested:
    • The TCK Project (assuming its separate from the Specification Project)
    • The Specification Project with Escalation to the Specification Committee allowed
    • The Specification Committee only.
I believe we decided previously that TCK projects need to be separate from spec projects for IP reasons, but it would be good to confirm that decision.

With separate TCK and spec projects, you can imagine that some people will only care about the spec and only join the spec project, and other people will be focussed on creating the tests, not defining the spec, and so will only join the TCK project.

But who's best qualified to judge whether a test challenge is valid?  I think that requires both detailed knowledge of the test and how it tests the requirement, as well as detailed knowledge of the requirement and its intent.  Ideally, someone(s) with both kinds of knowledge would be involved in the decision.

The way we did it for Java EE is that the challenge would come to the TCK engineer, who would consult with the spec engineer if needed.  I think it would be fine to continue that approach here.

The test challenge would be filed against the TCK project.  The TCK project members can consult with spec project members as desired.

If the vendor disagrees with the decision, the issue can be escalated to the spec project, and if necessary to the spec committee.

  1. Vendor Releases and TCK patches:  If a vendor releases a new version minor of their product, which continues to support the same specifications claimed by the original major version, can the new minor version be tested against the original TCK used for the original version, or does it need to be tested against the latest patch/revision of that TCK?
To some extent, it shouldn't matter, depending on how updates to the TCK are managed.

For Java EE we always tried to manage updates to a TCK so that the updates would not cause a failure when the updated TCK was run against an existing project that passed the previous micro level of the TCK.  That means only excluding tests, or providing alternate tests where the vendor can choose whether to pass the original test or the alternate test.

If we choose to allow TCK updates to introduce new required tests, I would argue for allowing the vendor to use the original TCK version.  Note that this runs the risk of vendors claiming they're "more compatible" because they pass the newer TCK.  Our goal with Java was always that compatibility is binary - you either are compatible or you are not compatible, no relative comparisons are allowed.

  1. Versioning:  Should the TCK versions match the major and minor versions of the Specification?
Yes.

  1. Versioning:  If an errata level error (anything that doesn't change the described behavior of the software or its API usage) is found in a specification which of the following options should we use?
    • An errata is published with the correction but the specification is not updated.
    • The specification is updated and a new Rev version is released. (e.g. X Specification Version 1.2 Rev 2)
    • The specification is updated and a new Maintenance version is released (e.g. X Specification Version 1.2 MR 2)
I'm fine with either of the first two.  (There was alot of confusion in the JCP over MRs, and I'd like to not repeat it here.  See my JCP process writeup.)  If errata are published separately from the specification document, we need good document management procedures that allow users to easily discover the errata that apply to a given spec.  IETF does this pretty well.

  1. Specification Updates and TCK updates:  What kinds of corrections to a specification should require a new version of the TCK?
In general, test coverage is a "quality" choice for the TCK.  No TCK will test every requirement of a spec.  If a new requirement is added to a spec, is it required that the TCK be updated?  I don't think so.

However, I expect every TCK to include a signature test, and any change to the API signatures in a spec would require an update to the signature test in the TCK.

And obviously any change to a spec requirement that invalidates an existing test would require an update to the TCK.

Back to the top