| Re: [jakarta.ee-spec.committee] Notes on TCK maintenance |
|
And adding my previous replies to the thread (attached). David Blevins wrote on 11/28/18 9:26
AM:
|
--- Begin Message ---
- From: Bill Shannon <bill.shannon@xxxxxxxxxx>
- Date: Wed, 18 Jul 2018 15:09:26 -0700
- User-agent: Mozilla/5.0 (X11; SunOS i86pc; rv:52.0) Gecko/20100101 Thunderbird/52.7.0
You've all read the TCK Users Guide template I sent out a few weeks ago, right? :-)
David Blevins wrote on 07/18/18 10:20 AM:
Which exclude list? The one that was updated 3 minutes before you released your product? The one that existed when you downloaded the TCK?Pushing some rough notes on what we need to draft up or I'll forget the details:
- TCK major version is released. Say Jakarta EE 9- All vendors work against this "golden master"- Test challenges happen naturally and result in a growing exclude list- "Passing" means you complete the TCK, minus the tests in the exclude list
A maintenance version of the spec? Or of the TCK? I assume the latter, e.g., "Jakarta EE TCK 9.0.1".
- At the moment the Specification project would like to unexclude, update or add new tests, a maintenance version is released, such as Jakarta EE 9.0.1
I disagree. The TCK, just like any SI, needs to be able to evolve without changing the spec. Yes, there should be rules on how it can evolve without changing the spec, but those rules should allow some evolution.- TCK and Specification are versioned/patched in lock-step form
Allow but not require the exclude list to be reset to zero. In general, removing a test from the exclude list is equivalent to adding a new required test.- These maintenance allow the exclude list to be reset to zero and new tests to be added
I suspect the full specification approval process will be too heavyweight to allow timely updates to the TCK for vendors trying to ship a product on a schedule.- Maintenance versions are subject to the same Specification Committee approval processes as major versions
If the specification approval process includes a separate path for maintenance, that might be acceptable, but a multi-week voting and approval process but be appropriate for the spec but seems like too much for TCK bugs.
I think we should do something similar to what is currently done with the Oracle TCKs and require you to pass the version that was most recent no more than X days before your release (where "X" is "180" or so), or a newer version.- A cadence for the when or how often maintenance versions must be release is not mandated. In practice it may be quarterly (this is for our notes)- A vendor may pass any of the versions, 9.0.x, and claim compatibility
I don't think that such a sort is appropriate as it implies newer releases are somehow "more compatible" than older releases.- A compatibility page will be maintained to document all passing implementations, sorting the most up-to-date implementations at the top
Are you talking about maintenance versions of the TCK? No version of the TCK may change the goals or add features to the specification.- Maintenance versions, 9.0.x, may not change the goals or add features to the specification
Does a change to the exclude list result in effectively a new maintenance version of the TCK? Or is the exclude list versioned independently of the TCK?- Test challenges happen naturally and result in a growing exclude list to the maintenance version and the process repeats.
Plus adhering to any compatibility rules that are part of the TCK (assuming we choose to define such rules, which should include at least "run rules").- "Passing" means you complete the TCK, minus the tests in the exclude list
Super majority of those who vote? (Within what time period?) Or super majority of all Committers? I fear that the latter will result in projects that can't make progress due to inactive Committers.
- Test challenge process and exclude list management- Specification projects are responsible for the handling of test challenges- Challenges are handled and discussed openly on the projects xyz list (could be dev or we make a dedicated list)- Additions to the exclude list must pass a super majority vote by the specification project
Are you talking about a maintenance version of the TCK? Or of the spec? Obviously there's a big "it depends" for both of them, although I agree with designing our process so that it allows for at least that frequency. Achieving that frequency is a function of demand and resources.- Tests may not be altered, fixed, patched or added via an exclude list. This requires a maintenance version as detailed above
As a side note commentary, if we did release a maintenance version it would be one way to show "speed" and that we are faster than before. Thus, I don't see them as a problem to be avoided, but an opportunity that should be encouraged. I'd recommend framing things up with quarterly as a goal and our timings and approval processes to ensure that remains realistic.
--- End Message ---
--- Begin Message ---
- From: Bill Shannon <bill.shannon@xxxxxxxxxx>
- Date: Wed, 18 Jul 2018 15:10:46 -0700
- User-agent: Mozilla/5.0 (X11; SunOS i86pc; rv:52.0) Gecko/20100101 Thunderbird/52.7.0
Again, I believe maintenance of the TCK has to be allowed to occur without changing the spec, which results in them not being in lock step.
Scott Stark wrote on 07/18/18 01:04 PM:
At one point we were discussing a scenario where the TCK version was advancing without a corresponding update in the spec, one example of which could possibly be fixing a faulty TCK test that had no corresponding spec issue as raised by Bill S. In that same conversation I thought we were also saying that an update of the spec/TCK would reset the minimum version of the TCK that one needed to pass. This was to address someone sitting on the TCK version that had less stringent test requirements.
I don't recall a point at which we unanimously switched to a view that the spec and TCK would always be released in lock step. This needs to be hammered out in the draft discussion.
On Wed, Jul 18, 2018 at 10:20 AM David Blevins <dblevins@xxxxxxxxxxxxx> wrote:
_______________________________________________Pushing some rough notes on what we need to draft up or I'll forget the details:
- TCK major version is released. Say Jakarta EE 9- All vendors work against this "golden master"- Test challenges happen naturally and result in a growing exclude list- "Passing" means you complete the TCK, minus the tests in the exclude list
- At the moment the Specification project would like to unexclude, update or add new tests, a maintenance version is released, such as Jakarta EE 9.0.1- TCK and Specification are versioned/patched in lock-step form- These maintenance allow the exclude list to be reset to zero and new tests to be added- Maintenance versions are subject to the same Specification Committee approval processes as major versions- A cadence for the when or how often maintenance versions must be release is not mandated. In practice it may be quarterly (this is for our notes)- A vendor may pass any of the versions, 9.0.x, and claim compatibility- A compatibility page will be maintained to document all passing implementations, sorting the most up-to-date implementations at the top- Maintenance versions, 9.0.x, may not change the goals or add features to the specification- Test challenges happen naturally and result in a growing exclude list to the maintenance version and the process repeats.- "Passing" means you complete the TCK, minus the tests in the exclude list
- Test challenge process and exclude list management- Specification projects are responsible for the handling of test challenges- Challenges are handled and discussed openly on the projects xyz list (could be dev or we make a dedicated list)- Additions to the exclude list must pass a super majority vote by the specification project- Tests may not be altered, fixed, patched or added via an exclude list. This requires a maintenance version as detailed above
As a side note commentary, if we did release a maintenance version it would be one way to show "speed" and that we are faster than before. Thus, I don't see them as a problem to be avoided, but an opportunity that should be encouraged. I'd recommend framing things up with quarterly as a goal and our timings and approval processes to ensure that remains realistic.
jakarta.ee-spec.committee mailing list
jakarta.ee-spec.committee@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jakarta.ee-spec.committee
_______________________________________________ jakarta.ee-spec.committee mailing list jakarta.ee-spec.committee@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://dev.eclipse.org/mailman/listinfo/jakarta.ee-spec.committee
--- End Message ---