Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jakartaee-platform-dev] Individual specification TCK requirements for specs getting major/minor updates in EE 11


On 2/21/24 17:16, Emily Jiang wrote:
My comments are inline.

On Wed, Feb 21, 2024 at 5:54 PM Brian Stansberry via jakartaee-platform-dev <jakartaee-platform-dev@xxxxxxxxxxx> wrote:


On Wed, Feb 21, 2024 at 11:05 AM Scott Marlow <smarlow@xxxxxxxxxx> wrote:


On 2/20/24 17:36, Brian Stansberry via jakartaee-platform-dev wrote:

There's been some discussion in the last week on a PR against the Jakarta EE TCK process guide[1] and on today's platform call[2] that needs discussion on this list.

The background is that last year there was a 'Requirements on standalone TCKs'
specification committee issue[3] that resulted in an update to the TCK process guide[4]. I think I can fairly correctly characterize the key motivation there as a (valid IMHO) desire to ensure that as new 'standalone' TCKs are created we don't lose the ability to test that EE component impls work correctly in an actual EE environment. 

We are coming up on release deadlines for various component specs, but I don't think the precise implications of the TCK process change are understood (for example, see the discussion at [5]). So my goal here is 1) to get more clarity, 2) to increase awareness of these requirements and 3) to work through any implications for completing the EE 11 release.

I'll frame all this in terms of questions. Some I'm pretty sure I know the answer, but maybe I'm wrong, or maybe others are less sure. Note that I'm going to refer a fair bit to things discussed in the links below, particularly the change included in [4].

1) What is a 'standalone TCK'?

This is a term for an individual specification's TCK that can be executed independent of the larger platform or profile TCK. This does not just mean a TCK for a 'standalone specification', i.e. a Jakarta spec like Jakarta MVC that is not part of any platform or profile.

I like Emily Jiang's use of 'individual specification TCK' in the [4] change, as it avoids the ambiguous meaning of 'standalone'.

2) Is this relevant to TCK service releases?

It's been clearly agreed that this is only relevant for TCKs that are producing updates for major or minor spec releases. Service releases are unaffected; service releases are not allowed to make these kinds of changes.

No.  I agree with the answers that TCK service releases do not need to be concerned with the TCK Process changes.

Do we need to update the new section of the TCK Process to state that TCK service releases are not impacted by the new section?  FYI, there are other sections that do apply to service releases.


3) What are Type 1 tests?

I believe this refers to tests of things like 'CDI in Java SE'[6] -- that is, cases where specs have requirements that are only relevant when not running in an EE container. What matters is the type of spec requirement, not how a test has historically been executed.

This is a relatively small subset of all EE TCK tests.

4) How should Type 3 tests work?

My expectation is most tests are Type 3 tests.
 
The requirements in [4] state that an individual spec must provide a mechanism to run Type 3 tests within a platform/profile, but also must make it possible to run them outside of one. (I assume the latter is to avoid a circular dependency problem where a spec can't release because there is no platform/profile to test against, because the spec hasn't been released...)

In the past we have made exceptions to allow Components Specification to be released as final without testing on a platform/profile as type 3 tests create a chicken + egg problem of needing both a standalone implementation and platform/profile at the same time.  

If the Specification Committee needs to make an exception during the Component Specification review to ignore type 3 TCK tests since they cannot deploy on an EE 11 implementation yet, does that mean the Component Specification review cannot really happen until the last EE 11 wave?  The other question to answer is how to handle restarting the early EE 11 waves if any of the type 3 TCK tests are found to need code changes (note that we don't do service releases during an EE release for minor/major Component Specifications as per Specification Committee discussions in the past). 


Type 3 are executed both inside or outside the platform. e.g. CDI tcks can be executed by Weld, which is enough for TCK ratification. By the time of releasing EE 11 platform, Glassfish will integrate cdi impl in glassfish and then execute the Type 3 CDI Tcks to prove Glassfish is indeed CDI 4.1 compatible.

Is the described behavior of the "Type 3" meant to say that all EE features are portable to (non-EE) Java SE implementations?  I don't believe so but I am trying to understand if type 3 is specifically talking about the subset of TCK tests that have no EE dependencies?

Scott


If a component spec needs to perform another release, e.g. CDI 4.2, to fix some errors, CDI 4.2 would be released and then all the specs depending on CDI. EE 11 will then include CDI 4,2 and the updated other specs. CDI 4.1 would not be included by any platform releases.

Probably we should clarify this, but my interpretation of [4] tells me that there is no requirement that a spec is tested with a platform/profile in order to release. The text is about TCK production requirements, not about TCK execution requirements. How and when the tests are executed is not covered in this chunk of text, which is conceptually ok, but it may be more understandable if there's some language about this.  And how the requirements in the guide are enforced is another question.

L56 on [4] says "The individual specification TCK must provide a mechanism for running the required TCK tests (types 1 and 3) outside the context of a platform or profile. Type 2 tests must be excluded and not run." My _assumption_ is the point of this sentence is to require a TCK mechanism that can be executed to ratify the individual spec, and that impls not tied to a platform/profile impl can use that mechanism to certify. 

It's the language on L54 that defines what I see as the requirement to produce something that platforms/profiles can run. So platforms/profile impls would use that mechanism when certifying. I _assume_ it is not required to run the tests using this mechanism in order to release a spec.

correct.
A question though is how the L54 requirement is enforced. The L56 requirement is enforced by requiring an impl to run and pass the TCK using the L56 mechanism before the spec is released. But, as you say, it's not feasible to have a platform/profile impl run and pass the L54 mechanism before the individual spec can release.

 
L54 will be enforced when releasing the platform/web profile/core profile specification.

Is this "how to enforce" thing a new problem though? Apologies if I'm wrong, but AIUI from our chat yesterday the language in [4] largely codifies the way many individual spec TCKs have worked for many years. The platform TCK source included many tests, organized in packages many of which could be mapped to various individual specs. And then, using a 'vehicle.properties' file with "/name/of/package = standalone" entries an individual spec TCK (aka a 'standalone tck') was generated, and that's what was used to certify individual spec impls. But by using a different vehicle.properties file with, say "/name/of/package = appclient ejb servlet jsp" the same tests could be executed as part of testing a platform or web profile impl. How was this "dual use at different times" thing handled?  Between when an individual spec was released and when the platform was released were changes in /name/of/package not allowed (unless individual spec did a service release)? And then 'enforcement' of these requirements was basically just the platform TCK contributors knowing how it was meant to work and taking appropriate care?

If any platform TCK error occurs, the teams have to work together to identify the root cause and then fix that particular specification, which might involve releasing a new version of that spec to be included in EE 11.
 
If so a lot of what we're discussing here is 1) the old platform TCK is being refactored so we can't just rely on the old tech to produce TCKs that meet the longstanding requirements and 2) responsibility for meeting the requirements is devolving from the platform TCK contributors to the individual spec working groups.  Both are non-trivial for sure!

The projects such as CDI, REST, etc are fine. All of the TCKs using arquillian are Type 3 tests and noop. The Jakarta Data TCKs are fine too. The process is pretty much trying to promote the tck style exhibited by CDI, Data, etc.


FYI, https://www.eclipse.org/lists/el-dev/msg00190.html already indicates that it is easy to produce a type 3 _expression_ Language TCK since EL can just work in either just Java SE (no EE impl) or with an EE impl.  Note that as per https://jakarta.ee/specifications/platform/10/jakarta-platform-spec-10.0#a2161 _expression_ Language type 3 tests should be run from web (e.g. Servlet/Server Pages).  

Also note that https://jakarta.ee/specifications/_expression_-language/5.0/ mentions the implementation used as the initial _expression_ Language implementation (Eclipse Expressly 5.0.0-M2).  For _expression_ Language 6.0 what do people expect to appear under the "Compatible Implementations" section of the https://jakarta.ee/specifications/_expression_-language/6.0 page?  Just the _expression_ Language implementation?  Or details on the EE 11 Compatible Implementation as well?



Do we have any advice/best practices for how to do this? Anything we can extract from CDI, which handles this?

Use Arquillian test framework. Look at how Jakarta Data TCKs are doing and they are good examples. For the specs that only have unit tests, they have to fix their test framework if they do a minor or major release.
5) What about existing individual specification TCKs?

We've long had individual specification TCKs, and I'm sure some of what they include is Type 2 or 3 tests. Are we requiring that specs with those kinds of TCKs that are doing major/minor updates for EE 11 update those existing tests so they can run in a platform/profile impl?
I think one example is the Batch TCK if I recall correctly (I think the Batch TCK includes use of EJB).  I think Batch already meets the requirement.  But there are others that don't like the Persistence Spec (type 1) TCK which is currently in the Platform TCK.

From an individual specification point of view that doesn't have an EE implementation to test with, I'm not really sure of what the Specification Committee was thinking when they made the TCK Process change with regard to actual specification review.  Perhaps the individual specification team will hack together a milestone release of some slightly modified open source EE implementation that has some EE 11 specifications but I think this is a dangerous path to suddenly require individual specification teams to adapt to for the EE 11 release as the hacked together implementation will not really prove that the component specification can be used on EE 11 platform/profile implementations.


Spec releases will happen as before. Nothing changed regarding the integration tests. If platform tcks have issues when perform platform/web profile/core profile, the identified component spec e.g. CDI might have to perform another release, e.g. CDI 4.2. Then CDI 4.2 will be included in EE 11 not CDI 4.1. This pattern could happen in the past Java/Jakarta EE releases. Again, nothing new.

6) What new individual specification TCKs are we expecting?

I know we are refactoring the platform tck to try and move toward individual specification TCKs, but I haven't followed the details on where this is going.

From the TCK Refactor branch, we can look at the install folder and see the following individual TCKs:

1. https://github.com/jakartaee/platform-tck/tree/tckrefactor/install/caj Common Annotations
2. https://github.com/jakartaee/platform-tck/tree/tckrefactor/install/connector Connector/JCA
3. https://github.com/jakartaee/platform-tck/tree/tckrefactor/install/el _expression_ Language
4. https://github.com/jakartaee/platform-tck/tree/tckrefactor/install/jacc Authorization
5. https://github.com/jakartaee/platform-tck/tree/tckrefactor/install/jaxws to be removed since jaxws is optional?
6. https://github.com/jakartaee/platform-tck/tree/tckrefactor/install/jms Messaging
7. https://github.com/jakartaee/platform-tck/tree/tckrefactor/install/jpa Persistence
8. https://github.com/jakartaee/platform-tck/tree/tckrefactor/install/jsp Server Pages
9. https://github.com/jakartaee/platform-tck/tree/tckrefactor/install/jstl Standard Tag Library
10. https://github.com/jakartaee/platform-tck/tree/tckrefactor/install/jta Transactions
11. https://github.com/jakartaee/platform-tck/tree/tckrefactor/install/saaj Soap with Attachments
12. https://github.com/jakartaee/platform-tck/tree/tckrefactor/install/websocket WebSocket

Note that https://github.com/jakartaee/platform-tck/tree/tckrefactor/install/servlet is already migrated to the Servlet Spec.


7) Are the spec groups producing major/minor releases aware of these requirements? Where do things stand?

No for _expression_ Language as per https://www.eclipse.org/lists/el-dev/msg00190.html.  I'm not sure of why that is.



My guess is in most cases the answer to the first question is 'No'.

+1 



_______________________________________________
jakartaee-platform-dev mailing list
jakartaee-platform-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev


--
Brian Stansberry
Principal Architect, Red Hat JBoss EAP
WildFly Project Lead
He/Him/His
_______________________________________________
jakartaee-platform-dev mailing list
jakartaee-platform-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev


--
Thanks
Emily

Back to the top