| Re: [jakarta.ee-spec] [External] : Re: Process for TCK service releases that include TCK updates for running signature tests on newer JDK versions... |
I would have expected, when we hold a ballot, for any Spec., those components are expected to be ready for all required compatibility configurations -- both with a stand-alone compatible implementation and with a Platform implementation. If no suitable platform is available at the time the component is finished it is plausible that the platform will simply have to conform to the component TCK. If a Platform implementation is required and none is available, that component won't be ready for ballot. In a case where the Platform is, for some reason or other, going to imply changes on the component -- that seems like new requirements and a new release to me.
OR put another way: if the tests can't be frozen, then the Spec. won't be ready for a ballot.
We should always operate on the principle -- the Spec encompasses
all of the Spec. text, the Spec. binary artifacts, and the TCK. If
any of these change -- we are effectively changing the Spec. (I
need to keep reminding myself, Compatibility tests are not the
same as product tests.) Under this principle, the tests are not
subject to change after the release ballot is started. We did a
bit of a sleight of hand to allow for adding additional JDK
support between 9 and 9.1 but that was expressly not supposed to
cause any test changes. There might have been other TCK test
changes, but those all should have been due to challenges or other
errata type fixes.
-- Ed
On 8/31/21 12:14 PM, Ed Bratt wrote:
Hmm. I'm probably diverting this away from the original subject -- I suspect this might be more useful in a general 'TCK release process requirements' thread.
I think that makes sense. In general, whomever is taking responsibility to handle any update (via Challenge, JDK update, or just ongoing evolution) ought to have the facilities to validate, whatever their change might be -- before making any new release. Adding that as an explicit requirement for all TCK bug-fix updates, seems appropriate.
This will/could become a bit more problematic as we start to expand the number of TCKs that are completely independent of the Platform TCK. I suspect that most component APIs do not consider testing with a Platform CI as part of their requirement set. With the centralized Platform TCK project, most of this testing was built into their modify-build-validate process. If a change were made to RESTFul Web Services, in EE 9 or earlier, that change would be validated in both the stand-alone and platform TCKs as part of their standard build/release process. When RESTful web services provides an update to their TCK (in EE 10 and beyond), should we expect that team to have validated their TCK change on both their own component compatible implementation AND a platform implementation?
RESTFul Web Services is a `Wave 4` SPEC, meaning that the RESTFul Web Services SPEC ballot will be well ahead of the Platform SPEC ballot. In previous releases, the RESTFul Web Services Platform TCK tests were not frozen until the Platform TCK was completed but for EE10 we will have some Platform tests in the RESTFul Web Services SPEC TCK. Can we validate SPEC level tests at the time of Ballot (e.g. only non-platform RESTFul Web Services TCK tests are validated at ballot time), and allow SPEC teams to update their TCKs after their initial ballot completes?
From a development perspective, I think that SPEC teams will need to be able to update their (Platform level) TCK tests after their ballot completes as the platform requirements could change or more simply test bugs may need to be fixed, perhaps in the form of a RESTFul Web Services TCK service release that only releases the TCK artifacts from the RESTFul Web Services 3.1 branch.
Perhaps the Platform TCK project will need to include some tooling that allows for the test running of component TCK changes. I realize this is a bit problematic. An alternative would be to simply put that onus onto the component Spec. dev. teams. If we provided enough boiler-plate to make it easy for them to run these tests, perhaps this isn't too much of an additional burden.
-- Ed
On 8/31/2021 7:59 AM, Scott Marlow wrote:
From your point, I am thinking that the TCK producing team needs to block releasing of the TCK until verification has been provided that the TCK can be passed (to be defined as to how). I also want to say that all compatible implementations need to be treated the same and IMO our process shouldn't dictate that https://ci.eclipse.org/jakartaee-tck must be used to verify the produced TCK.
Scott
On 8/27/21 5:13 PM, Ed Bratt wrote:
Currently, it is possible to update the parameters that runs tests using GlassFish so that it can pull down a preliminary TCK and runs the tests. GlassFish has the benefit that it can be used this way for all TCK tests. Other implementations may not be as flexible, but there must be a way for the TCK producing team to test the TCK product it's going to produce. While this tooling may be unique across the implementation landscape -- I am imagining that some kind of test set up will be implemented, regardless which vendor provides the initial ballot compatible implementation. There is simply no way we can produce a TCK and not have a process in place that provides for modify, built, test, review-results, modify, build, test ... cycles. This will have to be part of the process for any candidate compatible implementation that we intend to use for any release ballot.
Probably this needs to be discussed in more detail. These requirements probably need to be made more explicit and the component teams will probably need to become more aware of these obligations. (i.e. they won't be able to just assume that GlassFish will be primed and ready when they think they're ready to deliver their Spec/API/TCK)
We can't release TCKs without confirming that they run in the anticipated environment and they produce the anticipated compatibility verification. I don't see this as much of a problem for just excluding tests but if we add a feature (even if it's just a new Java SE version), we have to be able to test and iterate this prospective update with some compatible version. We will simply have to include this obligation on at least one of the initial ballot compatible implementations. It's got to be a two way partnership -- not a unilateral relationship.
-- Ed
On 8/27/2021 1:01 PM, Scott Marlow wrote:
On 8/26/21 8:24 PM, Ed Bratt wrote:
I would recommend this be brought to the Specification Committee for discussion and once everyone is satisfied, that a resolution be proposed to approve this as a new TCK process guide.
It would be nice to see a change-bar version of the document (I guess I can get that from the current source diff)
Under the section 'Process for Releasing a point revision' (the last section) -- My preference would be to include documentation to reference a compatible implementation that successfully passed the revised TCK. For exclude only updates, this should be easy if you can get one or more of the original compatible implementations to rerun their tests. For updates, that add new Java versions, this could be more difficult but, presumably we'd be releasing the update for the purposes of qualifying a particular implementation so, probably that version could be included (though I guess that might not be an open-source compatible implementation). In my opinion, we always want evidence that the TCK was run and an implementation successfully passed it. Referencing the certification request associated with that implementation would be the easiest way to capture this.
The particular implementation that created the TCK challenge hasn't yet created their certification request yet as they are blocked on waiting for the new TCK release to be published. They may also be waiting for other TCK challenges to be processed before creating their compatibility request. I do like the suggestion but I'm not yet understanding how we can accomplish it. At the very least, I would like the particular implementation to download the (not yet released) TCK to verify it after it has been built and communicate that the (not yet released) TCK is working as expected.
We are also transitioning over to not having a reference implementation to use for verifying not yet released TCKs. We can ask the various compatible implementations to test the new TCK but we cannot expect them to do that in a timely manner.
For reference, the referenced section currently contains:
"
Process for releasing a point revision of a TCK
The process for releasing a point revision of a TCK entails filing an issue in the jakartaee/specifications repository with the following details:
- Link to the TCK release to be published.
- Updated TCK links in the specification's _index.md file.
- A high-level description of what tests were excluded from the TCK and why.
"
Scott
On 8/18/2021 9:16 AM, Scott Marlow wrote:
On 8/5/21 11:01 AM, Kevin Sutter wrote:
Hi Scott,
I think we should pursue an update to the TCK process to allow service releases to fix Signature tests related to newer versions of Java. Not sure if we have to be that specific, but we do need to allow for these type of updates. The alternative of ignoring certain tests gets tricky and nebulous since ignored tests may actually have an issue, but we wouldn't know as casual observers. It would be much better to be clearer and service releases would allow us to be clear. Thanks!
I just updated https://github.com/jakartaee/jakarta.ee/pull/1018 to be less specific about service releases to fix tests for newer versions of Java (could be signature test changes or dealing with removal of Java security manager or something else).
Does anyone disagree with updating the TCK Process version from `1.0` to `1.1`? For what reason/condition should we update the version to `2.0`?
Does anyone else need to review https://github.com/jakartaee/jakarta.ee/pull/1018 before it gets merged?
---------------------------------------------------
Kevin Sutter
STSM, Jakarta EE and MicroProfile architect @ IBM
e-mail: sutter@xxxxxxxxxx Twitter: @kwsutter
phone: tl-553-3620 (office), 507-253-3620 (office)
LinkedIn: https://www.linkedin.com/in/kevinwsutter
Part-time schedule: Tue, Wed, Thu (off on Mon and Fri)
From: "Scott Marlow" <smarlow@xxxxxxxxxx>
To: jakarta.ee-spec@xxxxxxxxxxx
Date: 08/05/2021 08:25
Subject: [EXTERNAL] [jakarta.ee-spec] Process for TCK service releases that include TCK updates for running signature tests on newer JDK versions...
Sent by: "jakarta.ee-spec" <jakarta.ee-spec-bounces@xxxxxxxxxxx>
For Jakarta EE Platform 9.1+ supports implementations running TCK compatibility certification tests on JDK versions Java SE 8, Java SE 11+. In support of running TCK tests on JDK versions greater than SE 11, we expect that some tests will need to be revised (e.g. see jaxb-tck/issues/57 [1] for updating signature tests related to need new signature tooling library and signature map files).
Last December, we started making changes to the `TCK Process 1.0` that includes the following text [2] which introduces an alternative to excluding (challenged) TCK tests:
`The specification project may approve (user) workarounds for an `accepted` TCK challenge (as alternative to excluding TCK tests).`
My question today is whether the above quoted text is enough to cover Jakarta EE 9.1 compatibility certification requests against Java SE 17 (which will include signature test failures due to jaxb-tck/issues/57 [1])? If the answer/vote is yes, certain signature test failures can be ignored on newer JDK versions, if and only if the signature test failure is caused by inadequate TCK signature support for the relevant Java SE (e.g. JDK 17) version. If the answer/vote is no, we will need an additional TCK process change to specifically allow a SPEC TCK service release that updates signature tests to resolve the signature test failure (e.g. allow jakarta-xml-binding-tck-3.0.2.zip [4] to be officially released by Spec team so that implementations can submit certification requests against jakarta-xml-binding-tck-3.0.2.zip).
Scott
[1] https://github.com/eclipse-ee4j/jaxb-tck/issues/57
[2] https://github.com/jakartaee/jakarta.ee/pull/1018/files#diff-1fe254a18287c0db31fd9cb0a6bca11b1efda926095c3a65b73ef2ae0c89360dR223
[3] https://jakarta.ee/committees/specification/tckprocess/
[4] https://download.eclipse.org/ee4j/jakartaee-tck/jakartaee9-eftl/promoted/jakarta-xml-binding-tck-3.0.2.zip_______________________________________________
jakarta.ee-spec mailing list
jakarta.ee-spec@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec
_______________________________________________ jakarta.ee-spec mailing list jakarta.ee-spec@xxxxxxxxxxx To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec
_______________________________________________ jakarta.ee-spec mailing list jakarta.ee-spec@xxxxxxxxxxx To unsubscribe from this list, visit https://urldefense.com/v3/__https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec__;!!ACWV5N9M2RV99hQ!awIHuBnS0vFrSdaLJF1CkeydaJ6HBDuZO-HU31C_-GlXemvLd-yxK7aypDQTQqQ$