Re: [jakarta.ee-spec.committee] Notes on TCK maintenance

[Bill Shannon <bill.shannon@xxxxxxxxxx>]

You've all read the TCK Users Guide template I sent out a few weeks ago, right?  :-)

David Blevins wrote on 07/18/18 10:20 AM:

Pushing some rough notes on what we need to draft up or I'll forget the details:

- TCK major version is released.  Say Jakarta EE 9

   - All vendors work against this "golden master"

   - Test challenges happen naturally and result in a growing exclude list

   - "Passing" means you complete the TCK, minus the tests in the exclude list

Which exclude list?  The one that was updated 3 minutes before you released your product?  The one that existed when you downloaded the TCK?

- At the moment the Specification project would like to unexclude, update or add new tests, a maintenance version is released, such as Jakarta EE 9.0.1

A maintenance version of the spec?  Or of the TCK?  I assume the latter, e.g., "Jakarta EE TCK 9.0.1".

   - TCK and Specification are versioned/patched in lock-step form

I disagree.  The TCK, just like any SI, needs to be able to evolve without changing the spec.  Yes, there should be rules on how it can evolve without changing the spec, but those rules should allow some evolution.

   - These maintenance allow the exclude list to be reset to zero and new tests to be added

Allow but not require the exclude list to be reset to zero.  In general, removing a test from the exclude list is equivalent to adding a new required test.

   - Maintenance versions are subject to the same Specification Committee approval processes as major versions

I suspect the full specification approval process will be too heavyweight to allow timely updates to the TCK for vendors trying to ship a product on a schedule.

If the specification approval process includes a separate path for maintenance, that might be acceptable, but a multi-week voting and approval process but be appropriate for the spec but seems like too much for TCK bugs.

   - A cadence for the when or how often maintenance versions must be release is not mandated.  In practice it may be quarterly (this is for our notes)

   - A vendor may pass any of the versions, 9.0.x, and claim compatibility

I think we should do something similar to what is currently done with the Oracle TCKs and require you to pass the version that was most recent no more than X days before your release (where "X" is "180" or so), or a newer version.

   - A compatibility page will be maintained to document all passing implementations, sorting the most up-to-date implementations at the top

I don't think that such a sort is appropriate as it implies newer releases are somehow "more compatible" than older releases.

   - Maintenance versions, 9.0.x, may not change the goals or add features to the specification

Are you talking about maintenance versions of the TCK?  No version of the TCK may change the goals or add features to the specification.

   - Test challenges happen naturally and result in a growing exclude list to the maintenance version and the process repeats.

Does a change to the exclude list result in effectively a new maintenance version of the TCK?  Or is the exclude list versioned independently of the TCK?

   - "Passing" means you complete the TCK, minus the tests in the exclude list

Plus adhering to any compatibility rules that are part of the TCK (assuming we choose to define such rules, which should include at least "run rules").

- Test challenge process and exclude list management

   - Specification projects are responsible for the handling of test challenges

   - Challenges are handled and discussed openly on the projects xyz list (could be dev or we make a dedicated list)

   - Additions to the exclude list must pass a super majority vote by the specification project

Super majority of those who vote?  (Within what time period?)  Or super majority of all Committers? I fear that the latter will result in projects that can't make progress due to inactive Committers.

   - Tests may not be altered, fixed, patched or added via an exclude list.  This requires a maintenance version as detailed above

As a side note commentary, if we did release a maintenance version it would be one way to show "speed" and that we are faster than before.  Thus, I don't see them as a problem to be avoided, but an opportunity that should be encouraged.  I'd recommend framing things up with quarterly as a goal and our timings and approval processes to ensure that remains realistic.

Are you talking about a maintenance version of the TCK?  Or of the spec?  Obviously there's a big "it depends" for both of them, although I agree with designing our process so that it allows for at least that frequency.  Achieving that frequency is a function of demand and resources.