Andy Guibert wrote on 3/4/20 2:13 PM:
> Bill, I had a sit down meeting with the CTS lead at IBM and we discussed the
> pros and cons of the current way and the "externalized" way that I have
> proposed. Ultimately, he said that we already have both ways working and
> automated and it didn't make a big difference which way each spec goes.
I don't understand. How do we have both ways working? Do you mean the old TCK
and the new work you've done?
In this context the two ways are:
1) Most TCK tests with the JavaTest harness
2) The "externailzed" way that CDI, BeanVal, and now JSON-B are doing. In IBM we already have setups for CDI and BeanVal and adding the setup for JSON-B is trivial. I expect this is also true for other organizations that have already certified on Jakarta EE 8.
> I see two types of stakeholders with the TCK tests:
> 1) People who work on moving specs forward and writing new TCK tests,
> including getting tests passing for new versions of specs
> 2) People who run/maintain the TCK tests long term to ensure things don't regress
Aren't you forgetting the people who run the tests to check conformance of a
product?
Isn't that just group (2)? I'm not familiar with the entire certification process, but my understanding is that people at the implementing company/entity (e.g. IBM) run CTS and publish the results of the tests passing. Is there also an additional step I'm not aware of? (e.g. people at Oracle or Eclipse Foundation need to re-run the tests with the IBM impl to verify all the tests actually pass)
> Based on the opinions I've gathered from people in role (2) in this thread and
> from direct discussions at IBM, they don't care one way or another.
> Based on the opinions I've gathered from people in role (1), they prefer the
> "externalized" way of testing. Furthermore, this decision of what type of
> testing to go with can be made at a per-spec level so it doesn't need to be a
> unanimous platform decision. For people in role (1) for JSON-B, it is
> unanimous that we want to migrate to the "externalized" TCK way, and we have
> already put in the work to do so.
>
> If anyone from other companies have concrete concerns with this approach
> (Oracle or others) I think it would be best to set up a Webex with the
> concerned parties so we can talk it over and report back to this thread.
My impression is that everyone wants "externalized". Did you actually find
people who don't want that? Or maybe I don't understand what you mean by
"externalized"?
By "externalized" I mean the way the CDI, BeanVal, and soon-to-be JSON-B TCKs are set up which are based on Maven+Arquillian instead of JavaTest.
So far, everyone I've spoken to (except you) is either in favor of or neutral on migrating to the externalized TCK architecture.
I think we're just all waiting for more documentation that tells us how to get
from here to there, and what it will look like once we get there, especially for
the people who have to run the tests to test conformance of their product.
I would call this a "project plan", but maybe that's not a term you're familiar
with.
If I was inventing this process from scratch and CDI and BeanVal weren't already doing it this way, I would agree with you that a project plan is needed. However, that is not the case and BeanValidation has already thoroughly documented the approach:
For JSON-B it will be the same documentation except /s/BeanValidation/JSON-B and /s/TestNG/JUnit -- I will raise a PR shortly on jsonb-api with these instructions.
If we want to make sure every spec isn't doing their own thing and have 10 different ways to set up TCKs, we can simply promote the BeanVal documentation to the jakartaee-tck repo, generalize the terminology a bit, and have all specs that want to use this approach (CDI, BeanVal, JSON-B, JSON-P, etc) refer to the platform-level documentation.
