| Re: [jakarta.ee-spec] [External] : TCK release process requirements (continuation from `Process for TCK service releases...`) |
Just to be clear -- my desire to have the new TCK(s) tested by platforms as well as their independent compatible implementations -- should be considered a temporary desire. I would really like to avoid us releasing these new, separated TCKs, only to find that we have, somehow forgotten something crucial out of the platform requirements. I wasn't very involved when CDI first wrote it's TCK but I suspect that there were folks assigned to GlassFish, writing the porting kit elements -- as they were finalizing the CDI TCK -- so -- testing would have been going on, between both CDI with Weld and CDI with GlassFish.
You asked about CI resources - Currently, there are fixed CI resource allocations, at the Eclipse Project level. So -- everything in the GlassFish project is allocated one collection of CI resources -- everything in the Jakarta EE TCK project is allocated to another collection of CI resources. We can make alterations to the allocations but only manually (it's not a shared resource pool). Each Eclipse Project is allocated one 'resource pack' for CI. Each Resource Pack provides two vCPUs.
If memory serves me, GlassFish currently has 12 'resource packs.' The TCK project has 50 resource packs. The TCK project peaked at about 70 vCPUs (in Q2, when we were finalizing the 9.1 release). GlassFish peaked at about 8 vCPUs. The TCK CI systems have been "optimized" to take advantage of parallel test operations. I'm not sure that similar optimizations have been made for GlassFish. It isn't clear to me why these numbers don't peak at the maximum available.
This is a bit off topic to drill into but the TCK
memory/cpu settings could be better tuned. For example, we
are requiring 10GB of memory per VM instance which could be
expressed as a range instead (e.g. changing to allocate at least
4GM VM but try to get as much as 10GB per VM would more likely use
closer to the max available memory). The same is true for our
Java VM heap sizes (could be better tuned). The same for CPUs, we
shouldn't really need two CPUs for (single user) TCK testing, when
a single CPU per VM should be fine. The 10GB memory per VM number
should really be reduced to a much smaller number in a separate
tuning attempt as well.
Even if we could run GlassFish TCK testing with 4GB VMs to fit
better in the 12 (or 50) resource pack environment, IMO, even
better would be to just test TCKs in an environment where compute
resources can be shared (e.g. externally seems to be the only
option at the moment).
My guess is, we'd start seeing resource strain in projects that don't have additional resource packs -- especially if they attempt to run the Platform test jobs with any frequency. We'll probably have to monitor this and see if we need to make adjustments as more component projects pick up their TCK test load.
We may want to continue using the resources in the Platform TCK project since that project has a large pool of resource packs -- but we can decide that if we actually encounter problems. These loads are quite transient -- weeks will go by with little to no use -- then, when projects head toward the finish-line the loads go way, way up.
-- Ed
On 9/3/2021 10:49 AM, Scott Marlow wrote:
On 9/2/21 4:41 PM, Ed Bratt wrote:
If there are no assertion changes (i.e. the tests don't change) then re-release can probably be avoided. These updates can use the micro-version ID bumps.
I agree with no assertion changes == what you stated at the bottom "If the Platform specification (or any other spec that hasn't finished) imposes new or changed requirements (adds or changes an assertion) on RESTFul Web Services, after 3.1 is released -- that forces a new release."
RESTFul Web Services will need to verify itself with a compatible implementation that passes both the newly refactored TCK and a TCK that comes from the Platform TCK project.
Agreed since the RESTFul Web Service TCK currently only contains new tests and one Platform level TCK test. I think that this will change when the RESTFul Web Service TCK can run all of the existing Platform TCK tests (including Jakarta EE Platform + Standalone Java SE tests).
Personally, I would like to see both Jersey and GlassFish (with the new RESTful Web Services implementation) both passing these TCKs prior to the 3.1 ballot.
I have no preference as to which implementation is used but would like to see as many implementations (at least one) pass as possible.
My hope is RESTFul web services project, or through an arrangement between it and the Platform TCK project, will run these TCKs against GlassFish (or, if another CI is willing to provide this service, great) and Jersey periodically -- and hopefully before either of the TCK releases is updated -- even if just a micro update.
Just to bring awareness, what do we lose if https://ci.eclipse.org/glassfish/view/TCK is used instead of https://ci.eclipse.org/jakartaee-tck for testing GlassFish?
So -- if we find a change breaks the previously compatible implementation, we can investigate and correct if necessary.
Good point, should this be coordinated with all of the compatible implementation teams via one of our processes? I have thought about reaching out (nagging) to the compatible implementations when they create compatibility requests, that they should join https://accounts.eclipse.org/mailing-list/jakartaee-tck-dev so we could interact with them directly but I am not sure if mailing list discussions is enough to cover this need.
If the Platform specification (or any other spec that hasn't finished) imposes new or changed requirements (adds or changes an assertion) on RESTFul Web Services, after 3.1 is released -- that forces a new release.
I think that makes sense and does create a wall against changing requirements for a SPEC API after that SPEC API has completed its ballot. The wall doesn't prevent further changes but makes it obvious that there is a cost to change requirements late in the process.
Yes, very much so, thank you!Is this helpful?
-- Ed
On 9/2/2021 8:28 AM, Scott Marlow wrote:
Hi,
[1][2][3][4][5][6][7] are from the "Process for TCK service releases that include TCK updates for running signature tests on newer JDK versions..." discussion thread. I am starting a new thread with the subject that Ed Bratt suggested as I agree that we need a new thread for this sub-discussion.
I am responding here to the last message from Ed [7] for which I will paste his message. I hope this is clear:
> I would have expected, when we hold a ballot, for any Spec., those components are expected to be ready for all required compatibility configurations -- both with a stand-alone compatible implementation and with a Platform implementation. If no suitable platform is
> available at the time the component is finished it is plausible that the platform will simply have to conform to the component TCK. If a Platform implementation is required and none is available, that component won't be ready for ballot. In a case where the Platform is,
> for some reason or other, going to imply changes on the component -- that seems like new requirements and a new release to me.The concrete situation is for how we will deal with the EE 10 Platform TCK tests that *could* get added on top of the jaxrs-api/pull/1002 [8]. IMO, in order to release EE 10 in the first quarter of 2022, we need the overall development process to be as streamlined as possible for this brand new situation where we are adding new features to many EE specifications and also starting the TCK refactoring to improve how easy it is to add new TCK tests (as well as maintain the TCK tests for the future).
I assert that the RESTFul Web Services (3.1) Spec API ballot should only be run once for the EE 10 release and the same for all other Specs as that is what is required and that can include the Java SE TCK tests. But what are the options for releasing [8] Platform level tests that do not require any Spec Ballot to be repeated?
> OR put another way: if the tests can't be frozen, then the Spec. won't be ready for a ballot.
So the Platform level tests cannot be frozen until the Platform spec is frozen but if some (new for EE 10) Platform level tests will be in the jaxrs-api repository [9], those platform level tests shouldn't be validated until after all of the Spec Ballots have completed. IMO, the team maintaining the Platform level tests in the jaxrs-api repository [9] will need a way to fix any test defects identified prior to the Platform spec being released. Also IMO, the Platform TCK tests in the jaxrs-api repository [9] need a way to be released along with any TCKs produced by the Platform TCK.
> We should always operate on the principle -- the Spec encompasses all of the Spec. text, the Spec. binary artifacts, and the TCK. If any of these change -- we are effectively changing the Spec. (I need to keep reminding myself, Compatibility tests are not the same as
> product tests.) Under this principle, the tests are not subject to change after the release ballot is started. We did a bit of a sleight of hand to allow for adding additional JDK support between 9 and 9.1 but that was expressly not supposed to cause any test changes. There
> might have been other TCK test changes, but those all should have been due to challenges or other errata type fixes.The change now is that we need coordination of releasing Platform TCK tests that are maintained by the Standalone SPEC API teams. IMO, we should identify at ballot time if any Platform level tests are maintained by the relevant Spec so we can coordinate linking that Spec API with the Platform Spec final release process.
Scott
[1] https://www.eclipse.org/lists/jakarta.ee-spec/msg01950.html
[2] https://www.eclipse.org/lists/jakarta.ee-spec/msg01951.html
[3] https://www.eclipse.org/lists/jakarta.ee-spec/msg01952.html
[4] https://www.eclipse.org/lists/jakarta.ee-spec/msg01953.html
[5] https://www.eclipse.org/lists/jakarta.ee-spec/msg01954.html
[6] https://www.eclipse.org/lists/jakarta.ee-spec/msg01955.html
[7] https://www.eclipse.org/lists/jakarta.ee-spec/msg01960.html
[8] https://github.com/eclipse-ee4j/jaxrs-api/pull/1002
[9] https://github.com/eclipse-ee4j/jaxrs-api
_______________________________________________ jakarta.ee-spec mailing list jakarta.ee-spec@xxxxxxxxxxx To unsubscribe from this list, visit https://urldefense.com/v3/__https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec__;!!ACWV5N9M2RV99hQ!dnY_NfEDt6cEJMT0wxcHVd-eTD52inS2PxLAdkEJ1gloc334sJr9-2bstJ2QI_0$