| Re: [jakartaee-platform-dev] [External] : Re: Interceptors TCK -- what version of Platform TCK? |
During the Platform TCK call today, we talked about a few different options for refactoring what remains in the Platform TCK with different costs. Keeping in mind that the Platform TCK still has around 2.2 million lines of test code (based on `find -type f| xargs cat | wc -l` in src folder). Refactoring all of the Platform TCK tests has the highest risk that it could cause the EE 11 release to take longer than 1-2 years (IMO since we cannot validate any SPEC API Ballots until completing refactoring). If we refactor a small subset of Platform TCK tests, we would see a lower risk to the overall EE 11 release duration depending on how we do the refactoring.
Some options to discuss/consider:
- Completely refactor all remaining tests in the Platform TCK and take as much time as that takes. Cost is very high from an EE 11 release scheduling perspective as the Platform TCK project wouldn't be able to produce any standalone TCKs until the refactoring is complete, so the schedule impact is the first SPEC wave is blocked on starting the ballot until the Platform TCK refactoring is complete. Impact on EE11 release is that we have some initial overlap period where EE 11 SPECs are developed + TCK tests are refactored but the TCK refactoring does become a blocker for the EE 11 release starting the first SPEC ballot wave. Further cost is that we don't know how long the complete refactoring would take.
- Refactor a subset of each Platform TCK test bucket (e.g. specification being tested) but do not refactor all tests. The cost on the EE 11 implementations is that they still have to run both the old (Ant based Platform TCK tests) + new (refactored Platform TCK tests). To minimize the cost on EE 11 implementations, the Ant based Platform TCK tests would continue to be configured/run the same way as run with Platform TCK in EE 10. There is also a EE 11 implementors complexity cost of having some tests run via Ant and some tests run via Maven/Arquillian.
- Refactor a subset of each Platform TCK test bucket (e.g. specification being tested) but do not refactor all tests. The tests that are not refactored would be removed from the EE 11 release but could be refactored and added in future EE releases. The cost of reduced coverage varies depending on how much of each test bucket is refactored. The cost of risk to the EE 11 release also varies based on how much of each test bucket is refactored. The reduced test coverage could impact the quality of EE implementations.
4. Optimization of #2, Refactor a subset of each Platform TCK
test bucket (e.g. specification being tested) but do not refactor
all tests. The cost on the EE 11 implementations is that they
still have to run both the old (Ant based Platform TCK tests) +
new (refactored Platform TCK tests). To minimize the cost on EE
11 implementations, the tests not refactored would continue to be
run via the Platform TCK.
I think we all get the idea of how the first SPEC wave (and other SPEC API waves) are blocked on being able to run the relevant TCK. Perhaps we could be clever and come up with a solution for removing the SPEC API ballot bottleneck on the TCK with a contract that guarantees that any subsequent refactored TCK test changes must pass implementations that have submitted a compatibility certification request for the respective specification (e.g. think of a cooperative communications process that coordinates with the spec implementations that communicate their subsequent TCK test results back on a tracking Platform TCK issue). There could be other ways to improve our processes to allow SPEC API development to proceed forward in parallel while the TCK refactoring might still be happening throughout the EE 11 development process.
Scott