Re: [jakarta.ee-spec] [External] : RE: Proposed text for TCK Process Cla

[Ed Bratt <ed.bratt@xxxxxxxxxx>]

Kenji,

Thank you for your feedback. I've paraphrased your questions, followed by my answers. Readers should read Kenji's original text in case I've overly condensed his concerns.

Should we routinely accept exception text after a challenge is accepted?

For my part, I would prefer to see CCRs as clean as they can be. Platform certifications are already becoming quite verbose. My preference is that our process strives for as little extra as possible. I think the process goal ought to be: "TCK x.y.z, M tests passed, 0 skips, 0 failures." The less need need for ancillary text, the better. After a successful challenge, testing with the revised TCK, this type of result is what we should get. In exceptional cases the Spec. Committer team may allow exceptions, but I'd recommend these not be illustrated in the process. In my opinion, the fact a TCK can be revised without traceable confirmation from any vendor that it's been verified is a bit of a process gap. 

I would be curious what others think.

Is this incompatible with existing process requirements documentation?

I did look at the process documentation before I sent the text, though I was focusing heavily on TCK Challenges. 

I don't think the process documentation is explicit about what specifics must be provided as evidence for a successful CCR. The process only requires that the TCK Committer team approve the CCR issue. If there is process text you are concerned about, could you please provide citations so we can more carefully resolve this concern?

If we want to take up the approval process for CCRs we could consider that as well. 

As it stands, a vendor may submit a CCR. So long as it is not egregiously deficient, it will likely be automatically approved. In my opinion, the check-box that asserts the filer claims "... all TCK requirements have been met, ..." is sufficient if we were to retroactively find some serious problem.

I think most CCRs are approved under automatic acceptance (Lazy Consensus for a CCR is defined as automatic approval if there's no negative feedback within 14 days). Sure I'd love to see that all CCRs were reviewed as carefully as the initial CCRs for Release Review ballots, but that simply isn't the case and the process doesn't require it. My opinion is, this is adequate as it is.

Kenji, In any case, If I'm misunderstood your question, please let me know -- also if you or others have different opinions feel free to inject your thoughts. I'm happy to make adjustments.

Cheers.

-- Ed Bratt

On 10/3/2024 9:59 PM, Kenji Kazumura (Fujitsu) wrote:

Ed,

 

I have two comments for the following sentence:

 

If a challenge is automatically accepted, the vendor may submit a CCR with an exception statement, referencing the TCK Challenge Issue ID.

 

 

Even if a challenge is explicitly (but not automatically) accepted, is it a good way to submit a CCR with an exception statement

only when the specification project permits to do so until the service release is ready ?

 

Is it also necessary to change ‘Jakarta EE TCK Process’ in order to allow CCR with an exception statement ?

I am not clear the current TCK process allows to do so.

 

-Kenji Kazumura

 

 

From: jakarta.ee-spec <jakarta.ee-spec-bounces@xxxxxxxxxxx> On Behalf Of Ed Bratt via jakarta.ee-spec
Sent: Friday, October 4, 2024 9:09 AM
To: Jakarta specification disccusions <jakarta.ee-spec@xxxxxxxxxxx>
Cc: Ed Bratt <ed.bratt@xxxxxxxxxx>
Subject: [jakarta.ee-spec] Proposed text for TCK Process Clarification (Issue 74)

 

Specification Committee Members:

Below, please find some starter text to further our effort to resolve the ambiguity in TCK process when a committer team is not responsive to a TCK Challenge Issue. I believe that the stated resolution goal of the committee is to ensure that each Specification project explicitly states its intentions with respect to any automatic actions that might be associated with a TCK Challenge.

This is in intended to address Spec. Committee Issue 74.

Please review and let's try to finalize this at our next meeting. Ideally, we can make progress via e-mail. If there is lots of feedback, it may become necessary to move this text into a Google document (or equivalent). I'm hoping this isn't that controversial.

One question I have is: do we want to establish a date for completing this task? I've suggested Dec. 2. I am presuming that we will monitor and nag groups that don't take action but I didn't feel compelled to assert this, initially.

I look forward to your feedback.

Thank you,

-- Ed Bratt

----

To: jakartaee-spec-project-leads@xxxxxxxxxxx

Subject: TCK Challenge Clarification Action Needed

Hello Specification Committer Team members

It has come to the Specification Committee's attention that there is potential confusion about how TCK Challenges are to be handled. Particularly in the event that a committer team is not immediately responsive to the challenge issue. This message requests your committer team take specific actions to remove this ambiguity from the test challenge process.

The following text (or something close to it) appears under the heading of "TCK Test Appeals Steps" in most TCK User Guides:

Challenges can be resolved by a specification project lead, or a project challenge triage team, after a consensus of the specification project committers is reached or attempts to gain consensus fails. Specification projects may exercise lazy consensus, voting or any practice that follows the principles of Eclipse Foundation Development Process. The expected timeframe for a response is two weeks or less. If consensus cannot be reached by the specification project for a prolonged period of time, the default recommendation is to exclude the tests and address the dispute in a future revision of the specification.

(Emphasis added) This has caused confusion by some parties when filing TCK Challenges.

To resolve this potential confusion, the specification committee is requesting that each Specification Committer team clarify their policy regarding any automatic decision making policies for TCK Challenges. Specifically, we request that teams clarify if any type of automatic acceptance, or Lazy Consensus, or default actions will take place.

Please take the following actions:

1. Update your project's top-level read me and describe any automatic policy that the team will follow for any specification TCK challenge issue filed

2. Update your TCK User Guide text to match the choices made in (1).

Since accepting a challenge obligates the TCK committer team to make a new release of the TCK, we recommend that committer teams decide this issue carefully.

Here is some suggested text for your top-level readme:

TCK Test Challenge Process

This specification committer team [does|does not] allow automatic acceptance (Lazy Consensus) of TCK Challenges. More details about TCK Challenges are included in the TCK User Guide under 'TCK Test Appeals Steps'. You may also read about Challenges in Jakarta EE TCK Process. After filing a TCK Challenge Issue, a vendor is allowed to pursue escalation following the Eclipse Development Process Grievance Handling procedure.

If your specification project allows automatic acceptance, please include a description of how this is intended to work. For example:

For [project-name] TCK, a TCK Challenge will be deemed 'Accepted' if the issue is not resolved and there has been no update to the TCK Challenge issue for a period of [XX] days. If a challenge is automatically accepted, the vendor may submit a CCR with an exception statement, referencing the TCK Challenge Issue ID. Vendors are not permitted to alter the TCK except where specifically allowed in the TCK User Guide. 

Edit this to suit your team's process. If your specification does not offer automatic acceptance, leave the second paragraph out. Of course, change any of the text if it not compatible with your User Guide. Be sure to fill in the duration your team chooses if you will allow automatic acceptance. We recommend this be sufficient to allow for potential member absence or vacation time periods.

The specification committee encourages your project to follow the guidance offered in Jakarta EE TCK Process and that you strive for active and swift resolution for all TCK Challenges. If your team wants to include text about response time goals (e.g. This team strives to respond to all TCK Challenges within [NN] days) that would be encouraged. You might also consider adding something like: If you wish to escalate to the developer team, please send e-mail to <your-project-dev e-mail list>.

The committee recommends that all decisions be documented in the TCK Challenge Issue; that Accepted issues include reference to the specific change implemented (generally to the PR), followed by a reference to the released TCK update (maybe a repository tag, or download link).

The Specification Committee team is available to assist you or your team if you have any questions.

The Specification Team has established a target of Monday Dec. 2, to complete these updates.

Thank you

-- Ed Bratt, for the Jakarta EE Specification Committee