[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [jakartaee-platform-dev] TCK tests in the same repo as API andSpec
|
Andy Guibert wrote on 2/24/20 10:37 AM:
Andy Guibert wrote on 2/20/20 11:52 AM:
Now we seem to be going in circles... To recap
the relevant points:
1) How do we create overall platform TCK and
how does it fit together?
> In the same way that CDI and
BeanValidation did it for JavaEE 8 / JakartaEE 8
They didn't. That's the point. That's what we want to
avoid.
It's true that CDI/BV do not roll up into the same spot
as everywhere else, but based on
my proposed PR to jakartaee-tck
I thought we didn't want that for externalized TCKs?
We don't want a distinction between integrated and externalized
TCKs. We want a framework that allows all the TCKs to be
integrated, independent of where they're developed or hosted.
Speaking from experience here, in IBM/OpenLiberty we have
an entire team dedicated to setting up and running the JEE
CTS. They are the only ones who know how to set up and run
the arcane ball of JakartaEE TCKs.
When I was working on the BeanValidation 2.0
implementation for OpenLiberty for JavaEE 8 certification I
was able to figure out how to set up and run the
BeanValidation 2.0 TCKs against OpenLiberty in under an hour
because I was already familiar with Arquillian (as many JEE
developers already are). On the other hand only the
dedicated CTS teams know how to set up and run the JavaTest
stuff.
What's this "JEE" thing you're talking about?
Perhaps you meant
"Java EE"?
Yes, setting up and running the entire CTS is a big job. At Oracle
we also had an entire team dedicated to that work. If we can make
this all easier to do, that would be great, but to a certain extent
the nature of the task is difficult so it's not clear how far we can
go.
But as you say, dealing with an individual spec is quite feasible.
It seems quite surprising to some that we can have 37 things that
are easy to do but when we combine them all together we end up with
one thing that is difficult to do.
2) What test framework? What technologies?
> Arquillian+JUnit. These technologies are
well-tested and widely understood in the
community, and are perfectly capable of doing more
complex testing such as pulling in DBs, mail
servers, or other external resources. CDI and
BeanVal are reasonably complex specs/TCKs and are
already using Arquillian.
How do we teach Arquillian
to run the existing test cases? Or how do we convert
the existing test cases so that Arquillian can run
them? And either way, how do we assure ourselves that
the result has the same pass/fail characteristics as the
original?
The changes were pretty systematic and I did some spot
negative testing to ensure validity.
Ultimately, we assure ourselves the results are the same
by relying on our developers and our code reviewers. This is
a one-time change per-spec and coming up with a more
elaborate process would be over-engineering a solution IMO.
It's a one time change that is critical to the long term health of
Jakarta EE.
There's tens of thousands of tests, many of which are run in
multiple configurations, resulting in many tens of thousands of test
cases. I'd like to believe the community will do the same careful
conversion and review that you've done, but I'm skeptical. It's a
huge task that doesn't scale well.
3)
[From the meeting minutes] Requirement – allow
the use of the existing TCK tests themselves. If
all of the TCK tests need to be modified just to
become part of this new infrastructure, the
process will die. We need the ability to
incorporate existing TCK tests.
>
The conversion can be done on a spec-by-spec
case. For example, everything would work fine if
just JSON-B migrated to the "external TCK" way
and all other specs remained as-is. Also, the
conversion could be all-in-one or partial. For
JSON-B I did an all-in-one conversion, meaning I
took _all_ existing TCK tests and converted them
to Arquillian+JUnit so now JSON-B TCK tests
don't use JavaTest at all. Personally I think
per-spec all-in-one conversion is best so there
is only ever 1 framework per spec, but each spec
could have the flexibility to do what they
want.
This is the process that the notes claim will die.
Depending on each spec project to convert all of their
tests to a new framework just won't happen. You might end
up with 10 specs fully converted and 27 specs still using
the old framework. If old and new can coexist, that might
be fine. Even if we had to run two completely separate
TCK frameworks, that might be acceptable. I'd like to
hear what others think about this entire process of
converting tests to a new framework, whether it needs to
be done automatically, whether it can be done manually and
incrementally, etc.
Yes, 10 specs could convert to the new way and 27 specs
could stay with the old framework and that would be
perfectly fine. IMO That's a win-win because the 10 specs
that took the time to convert get the quality-of-life boost
that comes with the decoupled/modernized and the 27 specs
that stayed with the old way continue on with business as
usual.
I think that's a perfectly valid discussion to have with the
community to see if they would accept that outcome.
I'd love to see something approaching a "design spec" that describes
how you think this would work. Not just hand waving and "go look at
my code", but an actual document with English text that describes
all the components, how they fit together, how a user would use
them, how a developer would add a test suite, etc. Not at low level
detail but maybe 10 - 20 pages of medium level description.
Is that something you, or someone else, could write based on what
you've done so far? Or do you need to do more experimentation to
learn enough to write such a document?
