We discussed this in the platform call, but we believe the TCK process document needs more clarity.
Scott, can you please state precisely what clarification you need?
Thanks,
Ed
From:
jakartaee-platform-dev <jakartaee-platform-dev-bounces@xxxxxxxxxxx> on behalf of Scott Marlow via jakartaee-platform-dev <jakartaee-platform-dev@xxxxxxxxxxx>
Date: Monday, February 26, 2024 at 17:24
To: emijiang6 <emijiang6@xxxxxxxxxxxxxx>, jakartaee-platform developer discussions <jakartaee-platform-dev@xxxxxxxxxxx>
Cc: Scott Marlow <smarlow@xxxxxxxxxx>
Subject: [EXTERNAL] Re: [jakartaee-platform-dev] Individual specification TCK requirements for specs getting major/minor updates in EE 11
On 2/21/24 17:16, Emily Jiang wrote:
On 2/20/24 17:36, Brian Stansberry via jakartaee-platform-dev wrote:
There's been some discussion in the last week on a PR against the Jakarta EE TCK process guide[1] and on today's platform call[2] that needs discussion on this list.
The background is that last year there was a 'Requirements on standalone TCKs'
specification committee issue[3] that resulted in an update to the TCK process guide[4]. I think I can fairly correctly characterize the key motivation there as a (valid IMHO) desire to
ensure that as new 'standalone' TCKs are created we don't lose the ability to test that EE component impls work correctly in an actual EE environment.
We are coming up on release deadlines for various component specs, but I don't think the precise implications of the TCK process change are understood (for example, see the discussion
at [5]). So my goal here is 1) to get more clarity, 2) to increase awareness of these requirements and 3) to work through any implications for completing the EE 11 release.
I'll frame all this in terms of questions. Some I'm pretty sure I know the answer, but maybe I'm wrong, or maybe others are less sure. Note that I'm going to refer a fair bit to things
discussed in the links below, particularly the change included in [4].
1) What is a 'standalone TCK'?
This is a term for an individual specification's TCK that can be executed independent of the larger platform or profile TCK. This does not just mean a TCK for a 'standalone specification',
i.e. a Jakarta spec like Jakarta MVC that is not part of any platform or profile.
I like Emily Jiang's use of 'individual specification TCK' in the [4] change, as it avoids the ambiguous meaning of 'standalone'.
2) Is this relevant to TCK service releases?
It's been clearly agreed that this is only relevant for TCKs that are producing updates for major or minor spec releases. Service releases are unaffected; service releases are not allowed
to make these kinds of changes.
No. I agree with the answers that TCK service releases do not need to be concerned with the TCK Process changes.
Do we need to update the new section of the TCK Process to state that TCK service releases are not impacted by the new section? FYI, there are other sections that do apply to service releases.
3) What are Type 1 tests?
I believe this refers to tests of things like 'CDI in Java SE'[6] -- that is, cases where specs have requirements that are only relevant when not running in an EE container. What matters
is the type of spec requirement, not how a test has historically been executed.
This is a relatively small subset of all EE TCK tests.
4) How should Type 3 tests work?
My expectation is most tests are Type 3 tests.
The requirements in [4] state that an individual spec must provide a mechanism to run Type 3 tests within a platform/profile, but also must make it possible to run them outside of one.
(I assume the latter is to avoid a circular dependency problem where a spec can't release because there is no platform/profile to test against, because the spec hasn't been released...)
In the past we have made exceptions to allow Components Specification to be released as final without testing on a platform/profile as type 3 tests create a chicken + egg problem of needing both a standalone implementation and platform/profile at the same
time.
If the Specification Committee needs to make an exception during the Component Specification review to ignore type 3 TCK tests since they cannot deploy on an EE 11 implementation yet, does that mean the Component Specification review cannot really happen
until the last EE 11 wave? The other question to answer is how to handle restarting the early EE 11 waves if any of the type 3 TCK tests are found to need code changes (note that we don't do service releases during an EE release for minor/major Component
Specifications as per Specification Committee discussions in the past).
Type 3 are executed both inside or outside the platform. e.g. CDI tcks can be executed by Weld, which is enough for TCK ratification. By the time of releasing EE 11 platform, Glassfish will integrate cdi impl in glassfish and then execute
the Type 3 CDI Tcks to prove Glassfish is indeed CDI 4.1 compatible.
Is the described behavior of the "Type 3" meant to say that all EE features are portable to (non-EE) Java SE implementations? I don't believe so but I am trying to understand if type 3 is specifically talking about the subset
of TCK tests that have no EE dependencies?
Scott
If a component spec needs to perform another release, e.g. CDI 4.2, to fix some errors, CDI 4.2 would be released and then all the specs depending on CDI. EE 11 will then include CDI 4,2 and the updated other specs. CDI 4.1 would not be
included by any platform releases.
Probably we should clarify this, but my interpretation of [4] tells me that there is no requirement that a spec is tested with a platform/profile in order to
release. The text is about TCK production requirements, not about TCK execution requirements. How and when the tests are executed is not covered in this chunk of text, which is conceptually ok, but it may be more understandable if there's some language about
this. And how the requirements in the guide are enforced is another question.
L56 on [4] says "The individual specification TCK must provide a mechanism for running the required TCK tests (types 1 and 3) outside the context
of a platform or profile. Type 2 tests must be excluded and not run." My _assumption_ is the point of this sentence is to require a TCK mechanism that can be executed to ratify
the individual spec, and that impls not tied to a platform/profile impl can use that mechanism to certify.
It's the language on L54 that defines what I see as the requirement to produce something that platforms/profiles can run. So platforms/profile impls would use that mechanism when certifying.
I _assume_ it is not required to run the tests using this mechanism in order to release a spec.
A question though is how the L54 requirement is enforced. The L56 requirement is enforced by requiring an impl to run and pass the TCK using the L56 mechanism before the spec is released.
But, as you say, it's not feasible to have a platform/profile impl run and pass the L54 mechanism before the individual spec can release.
L54 will be enforced when releasing the platform/web profile/core profile specification.
Is this "how to enforce" thing a new problem though? Apologies if I'm wrong, but AIUI from our chat yesterday the language in [4] largely codifies the way many individual spec TCKs have
worked for many years. The platform TCK source included many tests, organized in packages many of which could be mapped to various individual specs. And then, using a 'vehicle.properties' file with "/name/of/package = standalone" entries an individual spec
TCK (aka a 'standalone tck') was generated, and that's what was used to certify individual spec impls. But by using a different vehicle.properties file with, say "/name/of/package = appclient ejb servlet jsp" the same tests could be executed as part of testing
a platform or web profile impl. How was this "dual use at different times" thing handled? Between when an individual spec was released and when the platform was released were changes in /name/of/package not allowed (unless individual spec did a service release)?
And then 'enforcement' of these requirements was basically just the platform TCK contributors knowing how it was meant to work and taking appropriate care?
If any platform TCK error occurs, the teams have to work together to identify the root cause and then fix that particular specification, which might involve releasing a new version of that spec to be included in EE 11.
If so a lot of what we're discussing here is 1) the old platform TCK is being refactored so we can't just rely on the old tech to produce TCKs that meet the longstanding requirements and
2) responsibility for meeting the requirements is devolving from the platform TCK contributors to the individual spec working groups. Both are non-trivial for sure!
The projects such as CDI, REST, etc are fine. All of the TCKs using arquillian are Type 3 tests and noop. The Jakarta Data TCKs are fine too. The process is pretty much trying to promote the tck style exhibited by CDI, Data, etc.
Use Arquillian test framework. Look at how Jakarta Data TCKs are doing and they are good examples. For the specs that only have unit tests, they have to fix their test framework if they do a minor or major release.
5) What about existing individual specification TCKs?
We've long had individual specification TCKs, and I'm sure some of what they include is Type 2 or 3 tests. Are we requiring that specs with those kinds of TCKs that are doing major/minor
updates for EE 11 update those existing tests so they can run in a platform/profile impl?
I think one example is the Batch TCK if I recall correctly (I think the Batch TCK includes use of EJB). I think Batch already meets the requirement. But there are others that don't like the Persistence Spec (type 1) TCK which is currently
in the Platform TCK.
From an individual specification point of view that doesn't have an EE implementation to test with, I'm not really sure of what the Specification Committee was thinking when they made the TCK Process change with regard to actual specification review. Perhaps
the individual specification team will hack together a milestone release of some slightly modified open source EE implementation that has some EE 11 specifications but I think this is a dangerous path to suddenly require individual specification teams to adapt
to for the EE 11 release as the hacked together implementation will not really prove that the component specification can be used on EE 11 platform/profile implementations.
Spec releases will happen as before. Nothing changed regarding the integration tests. If platform tcks have issues when perform platform/web profile/core profile, the identified component spec e.g. CDI might have to perform another release,
e.g. CDI 4.2. Then CDI 4.2 will be included in EE 11 not CDI 4.1. This pattern could happen in the past Java/Jakarta EE releases. Again, nothing new.
--
Brian Stansberry
Principal Architect, Red Hat JBoss EAP
_______________________________________________
jakartaee-platform-dev mailing list
jakartaee-platform-dev@xxxxxxxxxxx
To unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev
--
|
