Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jakarta.ee-spec] Adding or fixing tests in a TCK's service release


On 3/24/21 2:26 PM, David Blevins wrote:
Another aspect to capturing from today's spec committee call.

We are fixing TCK tests to ensure they run on Java 11.  Strictly speaking this is not allowed per the Jakarta EE TCK Process 1.0 [1]:

    How Tests May be Added to a TCK

    The only time tests may be added to a TCK are in a major or minor
    release. A service release which updates the exclude list MUST not
    have test additions or changes.

What this means to communicate is the only action that can be taken is to remove/exclude tests.  I argued for being able to fix TCKs, but we decided against it and I was ultimately ok with that as I was confident it would only be a matter of time before we'd want to do it and see the value.  Since we are doing it now at large scale for Java 11, we may want to revisit the discussion and policy.

Since we are producing most of the Standalone TCKs from the Jakarta EE Platform TCK project, it is likely that we will trip over this on major/minor releases in the future.  More likely for a minor than a major but it could happen on either to any EE Specifications that do not get updated for the major/minor release.  We will reduce the tripping over this as soon as we do extract the Standalone TCKs from the Platform TCK, for whichever TCKs we do that for. 

To reflect what we are doing for EE 9.1, we could update the TCK Process wording to be specific to the Platform level:

"The only time tests may be added to a TCK are in a major or minor EE release. An EE service release which updates the exclude list MUST not have test additions or changes."

Whatever we change the ^ wording to, I suggest we update https://github.com/jakartaee/jakarta.ee/pull/1018 to reflect it. 

I hope that this helps the discussion!


I can see the clear argument that if a spec team wishes to add new tests that would require at least a minor version, but fixing them is ok.  If we require a CCR for service releases of TCKs that feels like a less risky thing to do in practice.  Our practice now is that we are doing it and we were not planning to create CCRs.

If we look at the TCK-Challenge-processing-status, we can see some of the steps involved in processing accepted TCK challenges:

  1. Build the 9.0.x branch Platform TCK.
  2. Verify excluded tests.
  3. Verify (one-off) that excluded tests in previous service release were properly excluded.
  4. Verify Full Platform tests are passing.
  5. Verify Web Profile tests are passing.
  6. Verify updated Standalone TCK tests are passing.
  7. Promote the EFTL TCKS.
  8. Promote EPL TCKs.
  9. Release the TCKs and update jakarta.ee/specifications/ index.md pages.

If we require a CCR for service releases of each TCK, that will complicate instead of simplify our process (for the EE 9.1 release and even more so for service releases).  The result being that more effort/coordination will be required that might involve the separate Spec teams.  I'm guessing that a new step could be added after step 6 or 8 for creating separate Spec team issues to generate the CCR that blocks the TCK challenge process by however long it takes the generate the CCR.  We need to discuss what the (separate) Spec team would do exactly to generate the CCR and process whatever else they need to process so that someone can do steps 7-9 and run the script to copy the TCKs to the right place.

We are tracking TCK Challenges here which could be updated to also track CCRs but only the Jakarta EE Platform TCK committers can update this project so we might need each separate Spec project to track their CCRs later after we extract Standalone TCKs out of the Platform TCK.  Or perhaps the separate SPEC teams should do the verification of the separate produced TCKs and they can own producing the CCRs for service/minor/major releases as well. 

Scott


Back to the top