During the Platform TCK call today, we talked about a few
different options for refactoring what remains in the Platform
TCK with different costs. Keeping in mind that the Platform TCK
still has around 2.2 million lines of test code (based on `find
-type f| xargs cat | wc -l` in src folder). Refactoring all of
the Platform TCK tests has the highest risk that it could cause
the EE 11 release to take longer than 1-2 years (IMO since we
cannot validate any SPEC API Ballots until completing
refactoring). If we refactor a small subset of Platform TCK
tests, we would see a lower risk to the overall EE 11 release
duration depending on how we do the refactoring.
Some options to discuss/consider:
- Completely refactor all remaining tests in the Platform TCK
and take as much time as that takes. Cost is very high from
an EE 11 release scheduling perspective as the Platform TCK
project wouldn't be able to produce any standalone TCKs until
the refactoring is complete, so the schedule impact is the
first SPEC wave is blocked on starting the ballot until the
Platform TCK refactoring is complete. Impact on EE11 release
is that we have some initial overlap period where EE 11 SPECs
are developed + TCK tests are refactored but the TCK
refactoring does become a blocker for the EE 11 release
starting the first SPEC ballot wave. Further cost is that we
don't know how long the complete refactoring would take.
- Refactor a subset of each Platform TCK test bucket (e.g.
specification being tested) but do not refactor all tests.
The cost on the EE 11 implementations is that they still have
to run both the old (Ant based Platform TCK tests) + new
(refactored Platform TCK tests). To minimize the cost on EE
11 implementations, the Ant based Platform TCK tests would
continue to be configured/run the same way as run with
Platform TCK in EE 10. There is also a EE 11 implementors
complexity cost of having some tests run via Ant and some
tests run via Maven/Arquillian.
- Refactor a subset of each Platform TCK test bucket (e.g.
specification being tested) but do not refactor all tests.
The tests that are not refactored would be removed from the EE
11 release but could be refactored and added in future EE
releases. The cost of reduced coverage varies depending on
how much of each test bucket is refactored. The cost of risk
to the EE 11 release also varies based on how much of each
test bucket is refactored. The reduced test coverage could
impact the quality of EE implementations.
4. Optimization of #2, Refactor a subset of each Platform TCK
test bucket (e.g. specification being tested) but do not refactor
all tests. The cost on the EE 11 implementations is that they
still have to run both the old (Ant based Platform TCK tests) +
new (refactored Platform TCK tests). To minimize the cost on EE
11 implementations, the tests not refactored would continue to be
run via the Platform TCK.
I'm unclear on what #2 is then. My previous understanding was that it was to make the *Platform* TCK work like the Security, Authentication and Faces standalone TCKs do in EE 10. Basically two test executions, one using Arquillian-based tests, the other using the old Ant approach. That's not too bad for impls who already understand how to run the Ant platform TCK, as we'd just need to master the new way. Which presumably will be easy. ;)
If instead it's about creating more individual spec TCKs that work like Security, Authentication and Faces, well, yuck. :) Getting porting kits set up for those 3 for EE 10 has been real challenge for the WildFly developers, who have a lot of expertise in this area. It would be really painful to have to do more like that, and I imagine it would be far worse for other impls with less expertise.
I think we all get the idea of how the first SPEC wave (and
other SPEC API waves) are blocked on being able to run the
relevant TCK. Perhaps we could be clever and come up with a
solution for removing the SPEC API ballot bottleneck on the TCK
with a contract that guarantees that any subsequent refactored
TCK test changes must pass implementations that have submitted a
compatibility certification request for the respective
specification (e.g. think of a cooperative communications
process that coordinates with the spec implementations that
communicate their subsequent TCK test results back on a tracking
Platform TCK issue). There could be other ways to improve our
processes to allow SPEC API development to proceed forward in
parallel while the TCK refactoring might still be happening
throughout the EE 11 development process.
Scott
_______________________________________________
jakartaee-platform-dev mailing list
jakartaee-platform-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev