| Re: [jakarta.ee-spec] Compatibility Certification Requirements for EE 9.1 |
First off, Ed, thank you for clearly stating your desire and need for the extra CCR work. Although I still think this is unnecessary overhead for 9.1, I much better understand where you were coming from on our Spec call this week. Thanks.
Scott,
I think you stated the key argument against this extra CCR work...
"If the specifications were actually being updated to use Java 11 language features then there would be a more straightforward need to do full API/spec/TCK updates and validate that with an implementation that had a CCR."
We are making updates to allow for execution of the Platform TCK with Java 11, but the TCK will continue to work with Java 8 as well. So, the existing CCRs for the standalone TCKs are still valid with Java SE 8.
If we are verifying that Glassfish is running the Platform and Web Profile TCKs with Java 11, and we are verifying the components TCKs still execute with Java SE 11 (which we are), then I think we only need CCRs for the Platform and Web Profile TCKs. The "paperwork" to document all of the component TCKs run with Java SE 11 is just busy work in my mind.
If we want or need to still generate and promote all of the corresponding service releases for the standalone TCKs, that's another question. As Scott pointed out, the TCKs for DI, CDI, and BV were not updated for Java 11. How many of the standalone TCKs in [1] actually have some changes that would show a difference between the service release and the 9.0 release version? Maybe that's a question for Scott Marlow.
Good question, we just ran the Standalone TCKs [1] generated from the Jakarta EE 9 Platform TCK and do see 3060 failures with JDK11 + GF 6.1. Result Test results link.
From the linked failures, we see failures in most of the TCKs:
Common Annotations
Authentication
Authorization
Concurrency
Connector
_expression_ Language (only signature tests)
JSONB (only signature tests)
JSONP (only signature tests)
Messaging (only signature
tests)
Persistence (only signature
tests)
Server Pages (only
signature tests)
Standard Tag Library
RESTful Web Services (only signature tests)
Security API
Servlet
SOAP with Attachments
Enterprise Web Services
Transactions
Scott
---------------------------------------------------
Kevin Sutter
STSM, Jakarta EE and MicroProfile architect @ IBM
e-mail: sutter@xxxxxxxxxx Twitter: @kwsutter
phone: tl-553-3620 (office), 507-253-3620 (office)
LinkedIn: https://www.linkedin.com/in/kevinwsutter
Part-time schedule: Tue, Wed, Thu (off on Mon and Fri)
From: Scott Stark <starksm64@xxxxxxxxx>
To: Jakarta specification discussions <jakarta.ee-spec@xxxxxxxxxxx>
Cc: jakartaee-platform developer discussions <jakartaee-platform-dev@xxxxxxxxxxx>
Date: 03/25/2021 15:13
Subject: [EXTERNAL] Re: [jakarta.ee-spec] Compatibility Certification Requirements for EE 9.1
Sent by: "jakarta.ee-spec" <jakarta.ee-spec-bounces@xxxxxxxxxxx>
Now I understand my confusion around the use of 'standalone' in the contexts of the TCKs. The list of TCKs in [1] as you say have their source in the jakartaee-tck project intermingled with the CTS source. Just looking at the
https://github.com/eclipse-ee4j/jakartaee-tck/tree/master/src/com/sun/ts/tests/servlettree there is nothing that distinguishes this as standalone. It apparently is generated from the
https://github.com/eclipse-ee4j/jakartaee-tck/tree/master/install/servletsubtree.
This is a confusing relationship and I would hardly call these standalone. The reality is that the CTS/platform TCK project includes the TCK tests for several specifications and produces a TCK binary for several specifications. A question is whether the released standalone TCKs for 9.0 under https://download.eclipse.org/jakartaee/could be used as is or whether there are actual changes needed to be able to run under Java 11.
The distinction matters in my mind because if the jakarta-servlet-tck-5.0.0.zip TCK can be used as is, the 9.1 platform release is just an update at the platform level and there is no need for a CCR at the servlet spec level. It is just another compatible implementation that is free to create a CCR, but it is not required for validation of the servlet spec or TCK.
I don't see how the TCKs in [2] are being run. Presumably this does have to use the binaries from the 9.0 release as we have not produced any new versions as far as I know.
If the specifications were actually being updated to use Java 11 language features then there would be a more straightforward need to do full API/spec/TCK updates and validate that with an implementation that had a CCR.
I'm of the opinion that we need to start reducing the work specification projects need to do. The platform CCR has to validate that it can run against all the TCKs in the CTS and externally. That CCR should be sufficient as proof that the specification TCKs run under Java 11 and are suitable for the 9.1 release. I don't see a need for a CCR in every specification project.
On Wed, Mar 24, 2021 at 8:37 PM Ed Bratt <ed.bratt@xxxxxxxxxx> wrote:
In the Jakarta EE Specification committee, we had a lengthy discussion about compatibility certification requirements for Jakarta EE 9.1. This e-mail is my attempt to describe the problem and also describe a solution.
Jakarta EE 9.1 has explicitly set a requirement that it will support JDK 11. Jakarta EE 9, explicitly required JDK 8. The 9.1 platform specification now implies that all component specifications from 9.0 are also valid for JDK 11. The thought experiment I offered in the Specification Team meeting today was: "If a user were interested in running the stand-alone Servlet TCK, should they expect that TCK would work correctly on JDK 11?" I think the answer to this question is "yes." If there is agreement on this, expanding on that a bit -- one might then ask -- what evidence is provided to demonstrate that the JDK 11 requirement for each and every TCK is verified?
It is my position that each specification must have a compatibility certification request (CCR) that verifies their TCK runs correctly on JDK 11. (The good news, is we already have almost everything we need.)
During this project, we have been tracking the TCK updates and result status using the Jakarta EE Platform TCK (AKA CTS).
Stand-alone TCKs that are created by the Jakarta EE Platform TCK project have also been rebuilt -- changed or not. These are planned to be released as service releases. However, the SHA Sum values we referenced for their previous Jakarta EE 9 release CCR and the component specification pages will need to be updated and we cannot just replace the EE 9 entry since that was, and still would be an accurate TCK for Jakarta EE 9.
When Compatibility Certification Requests for Jakarta EE 9 were created, published, and approved, these were all verified against JDK 8.
If we proceed only with Platform compatible certification requests, we will be providing evidence that an EE Platform (likely multiple platforms, yay!) pass the Jakarta EE Platform TCK and that the platform implementation is compatible with the component specifications that the Platform Specification indicates it can be used to certify as compatible. However this does not demonstrate that the component specifications have verified their stand-alone TCKs.
Some background about the Platform TCK project: The stand-alone TCKs from the Jakarta EE Platform project are produced from common TCK test code that is used to create the Platform TCK and all the respective stand-alone TCKs. As part of the production process, to verify the stand-alone TCKs, GlassFish is used as a "known archive" of compatible implementations -- This is true only for the specific specifications that have stand-alone TCKs and are produced from this project. That list of specifications is identified in [1] below. There are 19 specifications in this list.
The stand-alone TCK verification uses compatible implementation from the component project as a quality measure before release. It has traditionally been up to the project teams to decide if they want to use the test results from the Platform TCK project, or if they want to provide separate implementation results. For example, Enterprise Security and the Soteria implementation have generally opted to take the results produced by the Platform TCK project. Eclipse Jersey and Jakarta REST generally opt to use their own test results even though the Jakarta EE Platform TCK project has these data available. In all these cases, the TCK can be verified and documented -- what was tested -- in what operating environment, and with what JDK in a CCR.
The component specifications are not changing. However, the TCKs produced by the Platform TCK project have been rebuilt. These new TCKs are being tested compatible against compatible implementations contained in Eclipse GlassFish, with JDK 11.
Without a new CCR, even the specifications with TCKs that are not changing will have produced no evidence that their TCK runs correctly with JDK 11. In cases where the previously used compatible implementation will pass on both JDKs -- great -- all that is needed is a verification re-run, and an update that denotes that fact.
If the previous statements are found to be accurate and agreed upon, the following can be documented with the tests that are already being run, today:
All specifications in lists [1] and [4] below can verify that their TCKs pass with JDK 11, using the results we are currently generating out of the Platform TCK project (25 specifications). Specification stand-alone TCKs from list [2] are explicitly included with each platform CCR (that brings us to 29 specifications). These results are also being produced in the Jenkins workspace for the Jakarta EE TCK project. The remainder, projects from list [3], would need to have their current stand-alone TCK and implementation status validated. I'm pretty confident they are in good shape and I don't anticipate much delay getting these data.
A recent TCK test run Blue Ocean "graph" is available here(recent results with Eclipse GF 6.1, JDK 11, Full profile). A recent stand-alone TCK test run, Blue Ocean "graph" is available here(recent results of all stand-alone TCKs that are produced from the Jakarta EE TCK Project, run on JDK 11).
I am willing to do most of the documentation work for all of this (see [5] below for a breakdown) and I think there is sufficient time to accomplish this before the Platform and Web Profile specifications are taken to ballot, presuming that the TCK results hold at their current level -- if any component projects want to produce their own CCR for JDK 11, they are welcome to do that. If we can get Spec. teams to proactively approve the CCRs, we can probably have them all completed before the platform ballot is started (but that's not certain). I am confident most, if not all, of the CCRs can be submitted before the platform ballot starts.
As I said in the Spec. meeting today -- I thought this was the plan all along but -- even if that wasn't well understood, I believe this is an evidence requirement that we must adhere to. I think this provides the evidence trail that we want to maintain for Jakarta EE and with luck, meeting this requirement won't impact any project schedule targets. In most, if not all cases, we're already doing the work. It's just a matter of doing the paper work to document that fact. I'm willing to do nearly all that "paperwork."
-- Ed
[1] -- Stand-alone specification TCKs that are produced by Jakarta EE Platform TCK project:[2] Specifications that are explicitly called out by Platform specification and must be included in a platform compatibility certification request:
- Annotations
- Concurrency
- Connector
- _expression_ Language
- Authorization
- Authentication
- Rest
- Messaging
- Persistence
- Faces
- JSONB
- JSONP
- Server Pages
- Tag Library
- Transactions
- SOAP with Attachments
- Enterprise Security
- Servlet
- WebSocket
[3] Specifications with stand-alone TCKs that will validate component compatibility for a given platform, but are not produced by the Platform TCK project:
- DI
- CDI
- Bean Validation
- XML Binding (required for JDK 11 and higher)
[4] Specifications that are certified compatible only with the Platform TCK (i.e. do not have stand-alone TCKs or are the platform):
- Activation
- Batch
- Compatibility
[5] Work needed to validate:
- Platform
- Web Profile
- Enterprise Beans
- Enterprise Web Services
- Managed Beans
- Web Services Metadata
_______________________________________________
- Collect the current TCK result data (using the data generated in Jakarta EE TCK Project)
- Generate TCK result pages (under the Eclipse GlassFish web-site)
- File the necessary CCRs for each specification
- File PR(s) to add these TCK details to the Specification pages (Add the link to the 9.1 TCK; as needed, add the compatible implementation details.)
- File the component certification e-mail with the EMO.
jakarta.ee-spec mailing list
jakarta.ee-spec@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec_______________________________________________
jakarta.ee-spec mailing list
jakarta.ee-spec@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec
_______________________________________________ jakarta.ee-spec mailing list jakarta.ee-spec@xxxxxxxxxxx To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec