[jakarta.ee-spec.committee] alternate tests

[Bill Shannon <bill.shannon@xxxxxxxxxx>]

One of the open issues in our TCK Process document is the use of
alternate tests to resolve test challenges.

Assuming we continue with the approach of not allowing the required
tests to increase except when a new spec version is released, we
run the risk of test challenges decreasing the number of tests over
the lifetime of a spec version as broken or bad tests are excluded.

The way we've dealt with this in the past is to allow for the addition
of alternate tests.  For example, if TestA is passed by 2 implementations
but fails for the 3rd implementation because the test is buggy and
includes an implementation dependency that doesn't work with the 3rd
implementation, what should we do?  If the test is a minor test and
excluding it has little impact on the test coverage, that is often
the simplest approach.

But if TestA is testing a key API, we might not want to suffer the
reduced test coverage if we exclude the test.  Instead, we would introduce
TestA', an updated version of TestA that fixes the bug or removes the
implementation dependency, as an alternate test.  Implementations
could choose whether to pass TestA or TestA'.  That allows us to retain
the same test coverage without invalidating the results of the first
2 implementations.

As currently written, our TCK Process does not allow for alternate tests.

Do we want to allow for alternate tests?