Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jakarta.ee-wg] Verifying TCK compatibility claims

Since the TCK are now open source, why not look into a creating a common test result logging framework that produces a tamper resistant result that can be the basis of the compliance submission. This way there is no extra effort in trying to rerun and verify the results.

As Bill says, if I'm bent on cheating, I can easily provide an Arquillian adapter that simply sets a single flag on a deployment indicating a TCK test, and all the hacks are inside of the server framework. The TCK rules can still say something like your results are subject to an independent audit by a representative of the Eclipse Foundation. The rules for the Java EE TCKs Bill mentioned have this requirement.

On Wed, Jul 18, 2018 at 8:40 PM James Roper <james@xxxxxxxxxxxxx> wrote:
On Thu, 19 Jul 2018 at 12:37, Bill Shannon <bill.shannon@xxxxxxxxxx> wrote:

If you push this to the extreme, you'll need to see the source code for the vendor's product so you can make sure they haven't implemented some special "TCK mode".  Obviously that's not feasible.

Well, if we're talking about extremes, why not go the other way, and have no TCKs, and just trust each vendor at their word that they've been careful and done their own testing to ensure they implement the spec? But we're not talking about extremes here, so there's no point in bringing it up. Publishing an Arquillian adapter as open source is a low cost, high value thing that can be done, extremes are irrelevant when we're talking about such a cheap win. It also provides a very simple starting point for the frank discussions and compromises that you're saying should be done when necessary, since it means everyone can start from the same meaningful place, "we've run the TCK against product X and found Y". Plus most, if not all Arquillian adapters for Java EE containers out there are already open source, so in most cases, adding such a requirement primarily reinforces what is already done, it doesn't add anything new.

Back to the top