Hi all,
A discussion about verifying TCKs was cut short on the Google docs, and that's probably because it's hard to follow discussions in the tiny comments boxes, so I wanted to start a thread here.
Firstly, I'm not suggesting that Eclipse run the TCKs themselves, that's obviously not feasible.
The problem is that there is an assumption that just because the TCKs are open, anyone can run them, and so therefore anyone can verify whether a vendor is falsifying test reports. This is not true. For example, most MicroProfile TCKs are built on Arquillian, which is a testing tool that abstracts away an application server, allowing specs to produce vendor agnostic TCKs. Arquillian itself is open, however, the integration with a particular app server is not part of Arquillian, and may or may not be open. If the Arquillian integration for a particular app server is not open and not freely available, then only people that have access to the Arquillian integration for that app server will be able to run the TCK against it. This means, in such a case, it won't be possible for anyone to verify whether the TCK passed against that app server. Even if the app server vendor provides the binaries for Arquillian integration - we need more than that, we need the source code of the Arquillian integration to ensure that they haven't implemented any smoke and mirrors in their integration. For example, what's to stop them from providing Arquillian integration that actually integrates to a different vendors app server instead of theirs, in that case, the TCK might pass, but only because it was testing a different app server.
So I think any implementation that wishes to claim compatibility must, in addition to publishing test results, also publish any integration code necessary to run the TCK as open source. I would go as far as to say that they should also provide scripts and detailed instructions saying how to run the TCK, since for some vendors app servers this might not be at all trivial.
Regards,
James
--
James Roper
Senior Developer, Office of the CTO
