| Re: [jakarta.ee-spec.committee] [External] : Re: Requiring visibility on optional features |
I'm very much in favor of improving the summary output of the
TCKs. It would make reporting much easier. It would certainly be
nice if a TCK report included whatever optional elements it did or
did not test -- but that does erode the mantra that the TCK result
is a binary -- it passes or it doesn't. I'd want to be a bit
circumspect about this idea.
The Platform has long allowed implementations to include
additional technologies. I think the goal is -- if the
implementation includes a component that the Platform Spec.
doesn't make a requirement, it must pass that component's TCK --
it is then incumbent on the implementer to ensure that the TCK
supports and verifies that component. In some cases that may
require coordination with the TCK and Implementation team.
The Specification (proper) is the complete collection of the
written Spec. document, the TCK (docs. and tests), The Java Docs.,
and the API files. Tomitribe's experience is probably showing us
that some of the details about the platform have become tribal
like knowledge. We should try to improve things where we can.
Of course we want all elements of the Specifications to be more complete, consistent, and easy to use. We will all benefit from that.
Specifically, I suspect the javaee_optional property may not be
documented completely, but there is a table in the Platform TCK
User's Guide (7.1 Keywords to Technology Mappings for Full Profile
Optional) that attempts to describe what is tested under Optional
(e.g. javaee_optional). I can see this
table is quite out of date. There is an issue (jakartaee-tck#509)
already asking that this be updated-- and expanded (for example to
include the web-profile optional property (javaee_web_profile_optional)
definition). The property file that defines these is here.
It's clear there is some updating to do with the documentation.
The final responsibility for these properties (as with the test
themselves) is with the platform committer groups. I certainly
hope that politics plays only a minimal role -- if even that.
The compatibility certification summary for any CCR should always
include evidence that any Jakarta EE component that is included in
the vendor product, is verified compatible. What is a requirement,
is spelled out in a few places -- one might start by reading
sections 1.1
and 1.2
of the Web Profile Spec. Platforms have long benefited by having a
single TCK that validates most of the necessary compatibility. As
we move forward the Platform TCK will become less and less
complete (as components refactor their tests out).
Making it clear and hopefully easy for a vendor to report seems
like a worthwhile enhancement to the TCKs. This is probably just
part of this evolution of Jakarta EE. I hope that the new TCK
platforms we adopt in EE 10 will simplify this.
In an ideal world, the TCK report would clearly log all the relevant property, environmental details (i.e. JDK, OS, Derby, Mail, etc.) that are being used, along with the test result summary. Having a clear output that could be included in each TCK Summary (for the CCRs) would be quite helpful -- but this is all an enhancement that would likely need to happen over time.
We don't require this detail today. We use an honor system. We
rely on vendors following the rules. At the end of the process --
all that matters is that the component TCK has completed
successfully. Whatever sausage making goes on, inside of the TCK
is really up to the Specification team. To the extend we can
improve the TCK reporting, I think we can reasonably ratchet up
the details that are included in the TCK result summaries.
I guess the experience we are gaining from Tomitribe's work is
showing that it's really difficult to know what to implement and
that reversing out requirements from TCKs is less than ideal.
While I don't think this is a new reality, it is something we
could decide to focus on.
-- Ed
Oh brother, so that optional components section is basically a politician's truth or some proof of Godel's incompleteness theorem. There are no optional components except for the disjoint set of technologies between the web and full profiles. Why even have tests for this? If you want to have tech from the full profile in the web profile, run the full profile tck tests.
On Jul 15, 2021 at 9:35:30 AM, Scott Marlow <smarlow@xxxxxxxxxx> wrote:
On 7/15/21 8:58 AM, Kevin Sutter wrote:
I like the approach of having the TCK results highlight the optional vs required aspects of the Specification. But, that will be easier said than done. I know the Platform TCK is very convoluted with the parameters required to run or not run the optional tests. And, the actual results do not highlight this aspect in the least. So, as stated below... We would have a ways to go to get this right, but I like the desire.
FYI just for more background, the Platform TCK keyword.properties currently maps the keywords to certain Platform TCK test folders. Note that the `javaee_optional` + `javaee_web_profile_optional` are (too) big test buckets.
Please also note that while the Web Profile specification does not define any optional components, Web Profile implementations can include Full Platform technologies/components (e.g. appclient container tests are included for Web Profile testing only if `javaee_web_profile_optional` keyword is specified).
At a high level, with the EE 10 Platform TCK, I think we will have more flexible capabilities via JUnit5 LauncherDiscoveryRequest + DiscoveryFilter as well as TestNG test groups, however the actual tests for each desired optional capability also need to be isolated from the other optional capabilities in order to allow for more specific test categories. We will be discussing these options on the Platform TCK mailing list in more detail.
_______________________________________________ jakarta.ee-spec.committee mailing list jakarta.ee-spec.committee@xxxxxxxxxxx To unsubscribe from this list, visit https://urldefense.com/v3/__https://www.eclipse.org/mailman/listinfo/jakarta.ee-spec.committee__;!!ACWV5N9M2RV99hQ!YbYki6yyTioLDlZbu4bn156WNEQtg7lrXIKYDICjm3ZtXcjpa_wIQ3Dbsbc7u5s$