Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jakarta.ee-wg] Verifying TCK compatibility claims
On Thu, 19 Jul 2018 at 16:29, Bill Shannon <bill.shannon@xxxxxxxxxx> wrote:
Obviously we're not interested in either extreme.  We're looking for the right cost/benefit tradeoff.

If we think that users are going to be running the TCK against vendor's products because they don't believe the product actually passed the TCK, we have a much bigger problem.

The TCK is a very complex test suite (which doesn't use Arquillian, by the way).  Setting up and running the TCK against someone else's product is a difficult task, even given all the adapters that are needed.  It's far more likely that people will do it wrong and make incorrect claims that the product doesn't pass the TCK.

"The TCK" is actually not what we're talking about here. Today, there may be only one TCK for Java/Jakarta EE, but in future, and this document is specifying what should happen in future, not the status quo that happens today (in fact the intention is that this is an Eclipse wide specification policy, not just Jakarta EE), the TCK should be split up, and made to be a very simple test suite, to lower the barrier of entry and encourage a more open platform that anyone can contribute to. Now no one's suggesting that the TCK be split up and rewritten tomorrow, as you said, it's very complex, but I would hope over time that it heads in this direction, that new iterations on specs would be accompanied with pulling the TCKs for those specs out of the main TCK, etc.

And it's not correct to say that the TCK doesn't use Arquillian - the CDI TCK, which is part of the TCK, does use Arquillian. So we've already got the base DI mechanism for Java EE that is heading in that direction.

The TCK adapter for a proprietary product may make use of proprietary APIs that the vendor does not want exposed.

All products must have a way for end users to deploy things to a container that the vendor does expose, otherwise the product is unusable. I don't see how a vendor can claim that they don't want this exposed.
 
Or the proprietary product may come with a license that prohibits independent testing of the product.

This is the sort of lack of transparency that the new Jakarta EE should seek to prevent. I don't think it helps Jakarta EE or the community.

Or the product may only be usable on hardware or in environments that most users won't have access to.

That doesn't make sense - by definition, a user of the software must have access to the hardware or environments that it is usable on, otherwise they are not a user of it.

I can imagine many reasons a vendor might not want to make their TCK adapter available to the public or available under an open source license.

I can imagine many reasons a vendor might not want the TCKs or specs themselves to be open source, but this is not Java EE, this is Jakarta EE, it is an open project owned by the Eclipse Foundation, a foundation that believes in open collaboration and transparency.
 
All that said, I'm sure many vendors will make their TCK adapters available, and will make their TCK results available, so that others can see exactly how they've tested their products.  But I think there's good reasons that some vendors may not want to do that, and I don't see the benefit of requiring them to do it to be great enough.  The bar is already high enough to become a Java EE / Jakarta EE vendor, we don't need to make it unnecessarily higher.

Of course, if experience makes us suspect that vendors are cheating, we can change the rules and require this.  But in 20 years of Java EE that hasn't been a problem.

We're already making the bar orders of magnitude lower by making the TCKs open, allowing vendors to run them themselves without onerous legal argeements such as NDAs. TCKs can (and in fact for new specifications do this already) be run as part of a vendors CI, they get immediate feedback the moment they break it. The bar is far, far lower than it was before. But in lowering this bar, we've lost some of the assurances that existed before, putting us in new territory that Suns/Oracles 20 years of experience can't speak for. Making the adaptors open is a simple way to get that back.

Now perhaps there are existing Java EE vendors that this is a problem for, and perhaps the existing Java EE TCK is not practical for just any user to run. But given that the scope of this document is not just Jakarta EE, but all of Eclipse, I would rather that this policy get it right, and then when applied to Jakarta EE, the existing vendors be grandfathered in with an exception that they don't have to implement this requirement, than to have the Eclipse wide policy for TCKs not have this simple mechanism for ensuring openness and transparency.



James Roper wrote on 07/18/2018 08:39 PM:
On Thu, 19 Jul 2018 at 12:37, Bill Shannon <bill.shannon@xxxxxxxxxx> wrote:

If you push this to the extreme, you'll need to see the source code for the vendor's product so you can make sure they haven't implemented some special "TCK mode".  Obviously that's not feasible.

Well, if we're talking about extremes, why not go the other way, and have no TCKs, and just trust each vendor at their word that they've been careful and done their own testing to ensure they implement the spec? But we're not talking about extremes here, so there's no point in bringing it up. Publishing an Arquillian adapter as open source is a low cost, high value thing that can be done, extremes are irrelevant when we're talking about such a cheap win. It also provides a very simple starting point for the frank discussions and compromises that you're saying should be done when necessary, since it means everyone can start from the same meaningful place, "we've run the TCK against product X and found Y". Plus most, if not all Arquillian adapters for Java EE containers out there are already open source, so in most cases, adding such a requirement primarily reinforces what is already done, it doesn't add anything new.

In the end, cheating is dealt with in the courts.  But cheating is rare, and it rarely gets to that point.  Before then, there are plenty of opportunities for frank discussion and compromises when necessary.  Most vendors are trying to do the right thing, and when the issues are explained to them they'll do their best to follow the rules.  That's part of the reason that we have rules that go with the TCK, not just tests, and when you sign the license for the TCK you agree to follow the rules.


Mike Milinkovich wrote on 07/18/2018 06:16 PM:
On 2018-07-18 9:05 PM, James Roper wrote:
A discussion about verifying TCKs was cut short on the Google docs, and that's probably because it's hard to follow discussions in the tiny comments boxes, so I wanted to start a thread here.

Firstly, I'm not suggesting that Eclipse run the TCKs themselves, that's obviously not feasible.

The problem is that there is an assumption that just because the TCKs are open, anyone can run them, and so therefore anyone can verify whether a vendor is falsifying test reports. This is not true. For example, most MicroProfile TCKs are built on Arquillian, which is a testing tool that abstracts away an application server, allowing specs to produce vendor agnostic TCKs. Arquillian itself is open, however, the integration with a particular app server is not part of Arquillian, and may or may not be open. If the Arquillian integration for a particular app server is not open and not freely available, then only people that have access to the Arquillian integration for that app server will be able to run the TCK against it. This means, in such a case, it won't be possible for anyone to verify whether the TCK passed against that app server. Even if the app server vendor provides the binaries for Arquillian integration - we need more than that, we need the source code of the Arquillian integration to ensure that they haven't implemented any smoke and mirrors in their integration. For example, what's to stop them from providing Arquillian integration that actually integrates to a different vendors app server instead of theirs, in that case, the TCK might pass, but only because it was testing a different app server.

So I think any implementation that wishes to claim compatibility must, in addition to publishing test results, also publish any integration code necessary to run the TCK as open source. I would go as far as to say that they should also provide scripts and detailed instructions saying how to run the TCK, since for some vendors app servers this might not be at all trivial.

James,

Thanks for this. I now understand what you're trying to say, and agree that it is a valid point.

I wonder if anyone from Oracle could chime in here and comment on how this is handled currently with Java EE certification?

--
Mike Milinkovich
mike.milinkovich@xxxxxxxxxxxxxxxxxxxxxx
(m) +1.613.220.3223



Avast logo

This email has been checked for viruses by Avast antivirus software.
www.avast.com 




_______________________________________________
jakarta.ee-wg mailing list
jakarta.ee-wg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jakarta.ee-wg

_______________________________________________
jakarta.ee-wg mailing list
jakarta.ee-wg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jakarta.ee-wg


--
James Roper
Senior Developer, Office of the CTO

Lightbend – Build reactive apps!
Twitter: @jroper


_______________________________________________
jakarta.ee-wg mailing list
jakarta.ee-wg@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://dev.eclipse.org/mailman/listinfo/jakarta.ee-wg



--
James Roper
Senior Developer, Office of the CTO

Lightbend – Build reactive apps!
Twitter: @jroper

Back to the top