I'd like to point out that we already have 2 specs that have their TCKs built in the way I am suggesting: CDI and Bean Validation. Both of these specs are sufficiently complex, and they produce their TCK tests as consumable artifacts. Do spec authors and runtime implementors think this is preferable? I do.
Yes for use as part of the Java EE compliance testing these TCKS were not integrated into CTS and were required to be run separately. The use of different frameworks was a common complaint by many of the Java EE licensees
I am just trying to provide some history for consideration when determining the final direction.
I'm sure all of the vendors already have vendor-specific scripting to pull down the maven artifact and plug it into the runtime they care about. I could start contributing this stuff to the jakartaee-tck repo, but I'd like to pause and ask if this is really necessary? What we're going to end up with is essentially a shell Maven project with a little bit of common pom.xml but mostly vendor-specific gorp in the <profile> for each respective implementation. So I could run TCK tests like this:
mvn verify -Pglassfish
mvn verify -Pliberty
The versions of the CDI and BV TCKs along with their exclude lists that were used for Java EE compliance testing were provided as part of CTS. Licensees were not directed to download these separately.
But is there much value in having this all combined into a single repo? We don't have this level of flexibility in the jakartaee-tck repo today (it's just setup for Glassfish out of the box).
Andy’s effort doesn’t affect Jakarta EE 9 release. As I was saying
before, JSONB TCKs in CTS repository will be used for certification.
TCKs in JSONB repository can be used for preliminary compatibility
testing. It’s much easier to use, it works much faster and it’s easier
to create Jenkins jobs with these tests.
I worry that the TCKs will be out of sync. For an instance, someone adds tests easily to the local version of TCKs. He/she needs to remember to add TCKs to CTS version. I think this duplicate efforts. I think the splitting JSONB tcks needs to be done by the time of releasing JSON-B as part of Jakarta EE9.
Having had experience with this when we started moving the TCK support to their own teams vs one big team, we did not encounter any issues with this ourselves. If you are using the same framework it is easy to pull these into the platform TCK. There was no duplication of effort.
How the concept of the deployment would work with
Andy's approach is not clear. Should the deployment be tested in the
future? Should it be left on the Spec to define the deployment
requirements/API or should it more common to every Spec? Currently, the
TCK framework does that and vendors implement the porting package that
takes care of the deployment. Every TCK uses the same mechanism. But
with having a different test framework in each TCK, the deployment needs
to be thought through.
I don't think the goal of breaking TCKs from the big bucket is to use a different test framework. I envisage the same test framework should be retained at least for the first release. I don
't think using different test framework is something we should ban either. We should allow different specs to use a suitable test framework if there is strong benefit associated with the new test ramework.
Using a consistent framework provides benefits for the platform integration and for the overall user experience. We had this feedback from the Java EE licensees when we first added some of the 3rd party technologies.
Long term I would suggest standardizing on a test framework across the Jakarta EE ecosystem
If the tests should still
be part of the platform TCK, and the tests are in the Spec repository,
how to make sure the tests are run for full platform and not for the
There is a good use case for breaking up the TCKs into standalone and platform as some runtime might only be interested in some standalone specs instead of the full platform. This will help to push individually specs to be certified. For instance, the test group in Arquillian can be adopted to dvide the tests into different buckets.
Not sure I understand the point here. Some of the existing technologies already have a life outside of the platform TCK. There are different requirements for these technologies depending on the container/environment.
You want to make sure that any standalone tests can plug in effortlessly so that can be tested in the various container configurations. CTS handled this today.
Moving away from JavaTest is fine IMHO, I would just make this an orderly change and come to agreement on the framework of choice.
Even the Microprofile has a unified Arquillian based test environment.
Using JUnit in one TCK, Arquillian in other, and JavaTest in the next
brings a challenge for the team that ought to run the TCKs. Are we cool
with having it different in every TCK?
In MicroProfile, you were right that Arquillian is a widely adopted technology. I think at the moment, Jakarta EE adopts different test frameworks and we can live with it. However, for any new specs, I think we should strongly recommend to adopt the mainstream test framework, such as Arquillian etc. However, I think this is orthorgno to the discussion of rehoming TCKs.
If Arquillian is the future, that is fine, but I would still suggest a strategy before doing too much work
We should consider job Andy’s done as a PoC. It should be merged to JSONB repository and Yasson team should adopt it without giving up TCK testing using CTS. TCK team should review it and recommend it (or not recommend) it to other project as a template for Jakarta EE 10 work. I don’t want a situation that splitting is artificially blocked because no-one from TCK team wants to do it.
Andy’s effort doesn’t affect Jakarta EE 9 release. As I was saying before, JSONB TCKs in CTS repository will be used for certification. TCKs in JSONB repository can be used for preliminary compatibility testing. It’s much easier to use, it works much faster and it’s easier to create Jenkins jobs with these tests.
I think it's fine for people to do experiments, ideally in their own
personal forks so there's no confusion as to its relationship with
the official version.
What I don't want is for (e.g.) Andy to get it to the point that
it's good enough for him, he decides he's done and merges it in to
the official repos for his projects, and the rest is left for the
rest of us to figure out.
I'm fine for not everything to be done before some things are
merged, but I want to see a plan that the jakartaee-tck
project committers agree to for how we're going to get to the
desired end point.
Among other things, we need to know that the plan will get us to a
working platform tck in time for a corresponding platform release.
And it would be best if the plan didn't have us running both the
"old" and "new" TCKs in parallel for releases for some non-trivial
This is a big project, so we're going to have to make some
compromises, but that doesn't mean giving up on any sense of
planning. An agile approach is good, but we need confidence that
we'll be able to see the endpoint.
P.S. I'm also fine with adding lots more committers to the
jakartaee-tck project if that helps us make progress more quickly in
the short term. There should be few impediments to spec projects
updating their TCK tests within the existing framework. They can
start with PRs against the jakartaee-tck repo and become committers
Kevin Sutter wrote on 2/3/20 5:40 AM:
Jan, You bring up
excellent points. Having a well thought out plan before jumping
with both feet would be nice. But, on the other hand, we don't
to stifle the excited effort that Andy is demonstrating. It's a
line. Maybe we need to allow one or two teams to experiment
the processes and see what it takes. And, then we can take
experiences and develop a plan that works across all the
The discussion on the various
lists is quite extensive, and I
admit I do not follow every thread that is discussed. But I am
for the plan from any of the committee we have since the
Eclipse Foundation. And I am not aware about any clear
what the final goal is. For start:
- Is the splitting really what is recommended?
- Should the Platform Jakarta EE TCK still be available in the
after the split?
- Should the interoperability between Specs tests (the tests
require multiple Specifications to cooperate as it is
described in one
or the other specification, the tests that used to be part of
CTS only) still be part of the combined (?) Jakarta EE TCK?
- Is there a recommended framework to be used in the future
TCKs (such as the existing JavaTest + perhaps Arquillian
Microprofile + ?)?
Those should be goals presented by the Jakarta representatives
the TCK repo is started to be ripped apart. Not on email, but
the main Jakarta informational page (whatever that is now).
_______________________________________________ jakartaee-platform-dev mailing list jakartaee-platform-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev
Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen@xxxxxxxxxx