I see that this discussion went too deep into details. Here is my vision and summary of how this process should look like.
Disadvantages of the current approach:
1. It's complicated to run standalone TCK tests. CTS is a complex system requiring special knowledge of how to use it. Most of Jakarta EE developers don't have this experience. JavaTest is not what developers are using in their projects.
2. CTS is not Maven based. It's not possible to run TCK tests as part of Maven builds.
3. CTS is tied with GlassFish. GlassFish is used as a container with all the disadvantages such as complications in CI/CD scripting which should check out, build and produce glassfish.jar with tested component integrated.
4. Some specs may not care much about all Jakarta EE specifications. But they have to now because CTS technically belongs to the Platform project.
Splitting TCK tests out of CTS and making them stand alone Maven artifacts (similar to what MicroProfile has) eliminates the disadvantages above.
In Helidon we don't care about Jakarta Platform tests. We run tests only for components we use as part of our Maven build. I guess that other MicroProfile implementations do it the same way. It also can work the same way for Jakarta EE Platform implementations. They just run TCK tests for all components as part of their build (in a special profile indeed). Cross-spec tests must be a part of one of crossing spec TCKs. Interoperability tests can be released as a separate artifact.
At some point some specs will have TCKs split and some are not. In this case full testing is achieved by running 2 suites: one is as I described above and CTS with the rest of TCKs.
It's important that all TCK tests use the same testing frameworks. I think that we all agreed that it should be JUnit 5 and Arquillian. I think that we should use one parent pom where all dependencies are defined for consistency.
Questions about new tests coverage and quality should be asked and solved during the spec progress review where new TCKs are introduced. It doesn't make sense to me to create a detailed plan now. It can be created later (after Jakarta EE 9 release) when this effort is supported by other specs.
-- Dmitry
From: jakartaee-platform-dev-bounces@xxxxxxxxxxx <jakartaee-platform-dev-bounces@xxxxxxxxxxx> On Behalf Of Bill Shannon
Sent: Tuesday, March 10, 2020 1:11 AM
To: Andy Guibert <andy.guibert@xxxxxxxxx>
Cc: jakartaee-platform developer discussions <jakartaee-platform-dev@xxxxxxxxxxx>
Subject: Re: [jakartaee-platform-dev] [EXTERNAL] Re: TCK tests in the same repo as API andSpec
Andy Guibert wrote on 3/9/20 8:42 AM:
Andy Guibert wrote on 3/4/20 2:13 PM:
> Bill, I had a sit down meeting with the CTS lead at IBM and we discussed the
> pros and cons of the current way and the "externalized" way that I have
> proposed. Ultimately, he said that we already have both ways working and
> automated and it didn't make a big difference which way each spec goes.
I don't understand. How do we have both ways working? Do you mean the old TCK
and the new work you've done?
In this context the two ways are:
1) Most TCK tests with the JavaTest harness
2) The "externailzed" way that CDI, BeanVal, and now JSON-B are doing. In IBM we already have setups for CDI and BeanVal and adding the setup for JSON-B is trivial. I expect this is also true for other organizations that have already certified on Jakarta EE 8.
I don't think it's fair to compare the incremental cost of adding another TCK once you've developed all the automation. What matters is the cost to someone starting from scratch, with no existing automation. In that case, adding more of the current style of externalized tests can increase their cost linearly.
> I see two types of stakeholders with the TCK tests:
> 1) People who work on moving specs forward and writing new TCK tests,
> including getting tests passing for new versions of specs
> 2) People who run/maintain the TCK tests long term to ensure things don't regress
Aren't you forgetting the people who run the tests to check conformance of a
product?
Isn't that just group (2)? I'm not familiar with the entire certification process, but my understanding is that people at the implementing company/entity (e.g. IBM) run CTS and publish the results of the tests passing. Is there also an additional step I'm not aware of? (e.g. people at Oracle or Eclipse Foundation need to re-run the tests with the IBM impl to verify all the tests actually pass)
The people who maintain the tests are usually the test developers, who are not generally the people running the tests. Yes, the people who maintain the tests also run them, but there are people who run the tests without ever maintaining them. You need to consider them as distinct groups.
> Based on the opinions I've gathered from people in role (2) in this thread and
> from direct discussions at IBM, they don't care one way or another.
> Based on the opinions I've gathered from people in role (1), they prefer the
> "externalized" way of testing. Furthermore, this decision of what type of
> testing to go with can be made at a per-spec level so it doesn't need to be a
> unanimous platform decision. For people in role (1) for JSON-B, it is
> unanimous that we want to migrate to the "externalized" TCK way, and we have
> already put in the work to do so.
>
> If anyone from other companies have concrete concerns with this approach
> (Oracle or others) I think it would be best to set up a Webex with the
> concerned parties so we can talk it over and report back to this thread.
My impression is that everyone wants "externalized". Did you actually find
people who don't want that? Or maybe I don't understand what you mean by
"externalized"?
By "externalized" I mean the way the CDI, BeanVal, and soon-to-be JSON-B TCKs are set up which are based on Maven+Arquillian instead of JavaTest.
So far, everyone I've spoken to (except you) is either in favor of or neutral on migrating to the externalized TCK architecture.
How many of the Asian vendors of Java EE products have you talked to?
I haven't talked to anyone who's in charge of running (but not developing or maintaining) TCKs who's said "yes, give me more completely independent TCKs to configure and run!"
The people who say anything like that are the people who are developing one or more of the TCKs and don't want to be burdened by the cost of integrating with JavaTest or the CTS.
The people who only run the TCK want one single framework that runs all the tests.
I think we're just all waiting for more documentation that tells us how to get
from here to there, and what it will look like once we get there, especially for
the people who have to run the tests to test conformance of their product.
I would call this a "project plan", but maybe that's not a term you're familiar
with.
If I was inventing this process from scratch and CDI and BeanVal weren't already doing it this way, I would agree with you that a project plan is needed. However, that is not the case and BeanValidation has already thoroughly documented the approach:
So is that exactly the approach you're proposing that we all do? I assume that doesn't involve converting anything from JavaTest to something else, right?
For JSON-B it will be the same documentation except /s/BeanValidation/JSON-B and /s/TestNG/JUnit -- I will raise a PR shortly on jsonb-api with these instructions.
Wait, so there's a different plan for JSON-B? It doesn't sound like we have a plan, but rather multiple similar plans.
If we want to make sure every spec isn't doing their own thing and have 10 different ways to set up TCKs, we can simply promote the BeanVal documentation to the jakartaee-tck repo, generalize the terminology a bit, and have all specs that want to use this approach (CDI, BeanVal, JSON-B, JSON-P, etc) refer to the platform-level documentation.
Which platform level documentation?
I think we do want to make sure every spec isn't doing there own thing. If you think you've got a plan for that, let's review it!
