Andy Guibert wrote on 2/20/20 11:52 AM:
Now we seem to be going in circles... To recap the
relevant points:
1) How do we create overall platform TCK and how does it
fit together?
> In the same way that CDI and BeanValidation did it
for JavaEE 8 / JakartaEE 8
They didn't. That's the point. That's what we want to avoid.
It's true that CDI/BV do not roll up into the same spot as everywhere else, but based on
my proposed PR to jakartaee-tck I thought we didn't want that for externalized TCKs?
Speaking from experience here, in IBM/OpenLiberty we have an entire team dedicated to setting up and running the JEE CTS. They are the only ones who know how to set up and run the arcane ball of JakartaEE TCKs.
When I was working on the BeanValidation 2.0 implementation for OpenLiberty for JavaEE 8 certification I was able to figure out how to set up and run the BeanValidation 2.0 TCKs against OpenLiberty in under an hour because I was already familiar with Arquillian (as many JEE developers already are). On the other hand only the dedicated CTS teams know how to set up and run the JavaTest stuff.
2) What test framework? What technologies?
> Arquillian+JUnit. These technologies are well-tested
and widely understood in the community, and are perfectly
capable of doing more complex testing such as pulling in
DBs, mail servers, or other external resources. CDI and
BeanVal are reasonably complex specs/TCKs and are already
using Arquillian.
How do we teach Arquillian to run the
existing test cases? Or how do we convert the existing test cases
so that Arquillian can run them? And either way, how do we assure
ourselves that the result has the same pass/fail characteristics
as the original?
The changes were pretty systematic and I did some spot negative testing to ensure validity.
Ultimately, we assure ourselves the results are the same by relying on our developers and our code reviewers. This is a one-time change per-spec and coming up with a more elaborate process would be over-engineering a solution IMO.
3) [From
the meeting minutes] Requirement – allow the use of the
existing TCK tests themselves. If all of the TCK tests
need to be modified just to become part of this new
infrastructure, the process will die. We need the ability
to incorporate existing TCK tests.
>
The conversion can be done on a spec-by-spec case. For
example, everything would work fine if just JSON-B
migrated to the "external TCK" way and all other specs
remained as-is. Also, the conversion could be all-in-one
or partial. For JSON-B I did an all-in-one conversion,
meaning I took _all_ existing TCK tests and converted them
to Arquillian+JUnit so now JSON-B TCK tests don't use
JavaTest at all. Personally I think per-spec all-in-one
conversion is best so there is only ever 1 framework per
spec, but each spec could have the flexibility to do what
they want.
This is the process that the notes claim will die. Depending on
each spec project to convert all of their tests to a new framework
just won't happen. You might end up with 10 specs fully converted
and 27 specs still using the old framework. If old and new can
coexist, that might be fine. Even if we had to run two completely
separate TCK frameworks, that might be acceptable. I'd like to hear
what others think about this entire process of converting tests to a
new framework, whether it needs to be done automatically, whether it
can be done manually and incrementally, etc.
Yes, 10 specs could convert to the new way and 27 specs could stay with the old framework and that would be perfectly fine. IMO That's a win-win because the 10 specs that took the time to convert get the quality-of-life boost that comes with the decoupled/modernized and the 27 specs that stayed with the old way continue on with business as usual.
