[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [jakartaee-platform-dev] [EXTERNAL] Re: TCK tests in the same repo as API andSpec
|
Andy Guibert wrote on 3/9/20 8:42 AM:
Andy Guibert wrote on
3/4/20 2:13 PM:
> Bill, I had a sit down meeting with the CTS lead at IBM
and we discussed the
> pros and cons of the current way and the "externalized"
way that I have
> proposed. Ultimately, he said that we already have both
ways working and
> automated and it didn't make a big difference which way
each spec goes.
I don't understand. How do we have both ways working? Do
you mean the old TCK
and the new work you've done?
In this context the two ways are:
1) Most TCK tests with the JavaTest harness
2) The "externailzed" way that CDI, BeanVal, and now
JSON-B are doing. In IBM we already have setups for CDI and
BeanVal and adding the setup for JSON-B is trivial. I expect
this is also true for other organizations that have already
certified on Jakarta EE 8.
I don't think it's fair to compare the incremental cost of adding
another TCK once you've developed all the automation. What matters
is the cost to someone starting from scratch, with no existing
automation. In that case, adding more of the current style of
externalized tests can increase their cost linearly.
> I see two types of stakeholders with the TCK tests:
> 1) People who work on moving specs forward and writing
new TCK tests,
> including getting tests passing for new versions of
specs
> 2) People who run/maintain the TCK tests long term to
ensure things don't regress
Aren't you forgetting the people who run the tests to check
conformance of a
product?
Isn't that just group (2)? I'm not familiar with the
entire certification process, but my understanding is that
people at the implementing company/entity (e.g. IBM) run CTS
and publish the results of the tests passing. Is there also
an additional step I'm not aware of? (e.g. people at Oracle
or Eclipse Foundation need to re-run the tests with the IBM
impl to verify all the tests actually pass)
The people who
maintain the tests are usually the test
developers, who are not generally the people running the tests.
Yes, the people who maintain the tests also run them, but there are
people who run the tests without ever maintaining them. You need to
consider them as distinct groups.
> Based on the opinions I've gathered from people in role
(2) in this thread and
> from direct discussions at IBM, they don't care one way
or another.
> Based on the opinions I've gathered from people in role
(1), they prefer the
> "externalized" way of testing. Furthermore, this
decision of what type of
> testing to go with can be made at a per-spec level so
it doesn't need to be a
> unanimous platform decision. For people in role (1) for
JSON-B, it is
> unanimous that we want to migrate to the "externalized"
TCK way, and we have
> already put in the work to do so.
>
> If anyone from other companies have concrete concerns
with this approach
> (Oracle or others) I think it would be best to set up a
Webex with the
> concerned parties so we can talk it over and report
back to this thread.
My impression is that everyone wants "externalized". Did
you actually find
people who don't want that? Or maybe I don't understand
what you mean by
"externalized"?
By "externalized" I mean the way the CDI, BeanVal, and
soon-to-be JSON-B TCKs are set up which are based on
Maven+Arquillian instead of JavaTest.
So far, everyone I've spoken to (except you) is either in
favor of or neutral on migrating to the externalized TCK
architecture.
How many of the Asian vendors of Java EE products have you talked
to?
I haven't talked to anyone who's in charge of running (but not
developing or maintaining) TCKs who's said "yes, give me more
completely independent TCKs to configure and run!"
The people who say anything like that are the people who are
developing one or more of the TCKs and don't want to be burdened by
the cost of integrating with JavaTest or the CTS.
The people who only run the TCK want one single framework that runs
all the tests.
I think we're just all waiting for more documentation that
tells us how to get
from here to there, and what it will look like once we get
there, especially for
the people who have to run the tests to test conformance of
their product.
I would call this a "project plan", but maybe that's not a
term you're familiar
with.
If I was inventing this process from scratch and CDI and
BeanVal weren't already doing it this way, I would agree
with you that a project plan is needed. However, that is not
the case and BeanValidation has already thoroughly
documented the approach:
So is that exactly the approach you're proposing that we all do? I
assume that doesn't involve converting anything from JavaTest to
something else, right?
For JSON-B it will be the same documentation except
/s/BeanValidation/JSON-B and /s/TestNG/JUnit -- I will raise
a PR shortly on jsonb-api with these instructions.
Wait, so there's a different plan for JSON-B? It doesn't sound like
we have
a plan, but rather multiple similar plans.
If we want to make sure every spec isn't doing their own
thing and have 10 different ways to set up TCKs, we can
simply promote the BeanVal documentation to the
jakartaee-tck repo, generalize the terminology a bit, and
have all specs that want to use this approach (CDI, BeanVal,
JSON-B, JSON-P, etc) refer to the platform-level
documentation.
Which platform level documentation?
I think we
do want to make sure every spec isn't doing there
own thing. If you think you've got a plan for that, let's review
it!
