A little more history.
Websockets is one of the few specs where the assertions were included as part of the spec(it was doable as it was a newer technology). We had considered trying to do this for all specs but we were so far down the road with some specs (EJB 1.0 spec was released in 1998 as an example), it would have been too costly from a time perspective to do this retroactively to the specs themselves.
Hope this helps. Please let me know if you have any more questions and I will be happy to try and answer.
Happy Thanksgiving!
Best Lance
The tools are able to walk through the Javadoc, generate the assertion list from the Javadoc, grab the previous generated assertion list from the previous version of the Javadoc, compare the two documents, and add the new assertion IDs to the top of the previous assertion list. That kept the assertion ids enlisted on the tests valid. The API assertions were then quite complete (As far as I remember the tools made the assertions only for methods, one for a method, one for the exception it threw. It did not made the assertions for the class descriptions, so for annotations and empty interfaces such as Serializable no assertions were generated). This saves time a lot. For Spec documents, the assertions were listed manually. There used to be some prototype of a tool grabbing the assertions from the WebSocket spec document based on the assertion ids marked there, but I do not know how well tested it was and whether it made it to the tool repository. As far as I remember, the WebSocket spec assertion list is partially generated by the tool and partially the assertions were added manually. The only flaw was that it asked online for a dtd file for parsing the Javadoc, e.g. https://github.com/eclipse-ee4j/jakartaee-tck-tools/blob/01a68ca8e8fb476bac11123a8427124e61943d08/docs/xsl/assertions/javadoc2assertions-with-ctor-exceptions.xsl#L21-- Jan On 27.11.2019 23:21, Mark Thomas wrote: On 27/11/2019 20:16, Jan Supol wrote:
I'd like to add that the tools were contributed to the Jakarta, too: https://github.com/eclipse-ee4j/jakartaee-tck-tools/tree/master/tools
Thanks. That was going to be my next question :)
For each TCK, there are two assertion lists, one for the Specification document, one for the API. Those can be found here in the internal/docs folder:
https://github.com/eclipse-ee4j/jakartaee-tck/tree/master/internal/docs/websocket
Great. Thanks for the reference.
Websocket Spec document has the assertions marked in the Spec Document directly by the Websocket spec lead.
We'll continue that approach for Jakarta WebSocket 2.0 and keep it in mind if we change anything that might impact those assertions.
Thanks again,
Mark
Thanks,
Jan
On 27.11.2019 20:27, Lance Andersen wrote:
On Nov 27, 2019, at 12:29 PM, Mark Thomas <markt@xxxxxxxxxx <mailto:markt@xxxxxxxxxx>> wrote:
On 27/11/2019 17:15, Lance Andersen wrote:
There are assertion docs generated for the spec and the java doc API, they should be part of the doc bundle, or at least were for the Java EE versions of the respective TCKs. The ids below would be found it those docs with a brief description.
The test in question validates the assertions that are listed in your example (or should anyways ;-) )
HTH Lance
It does, thanks. So there is nothing automated checking that all the assertions in the spec and Javadoc are listed as being tested in the TCK?
Well getting 100% coverage of all assertions for every spec and TCK(s) has always been a goal but not there. Compatibility rules did not allow for additional tests to be added once the spec had gone final as you wanted everyone to be testing against the same set of tests for a given spec version. Test coverage continued to be improved from release to release.
That being said, there were tools which were used to help calculate assertion coverage leveraging the assertion documents/tests but they were separate from CTS/TCKs.
Mark
On Nov 27, 2019, at 11:49 AM, Mark Thomas <markt@xxxxxxxxxx <mailto:markt@xxxxxxxxxx> <mailto:markt@xxxxxxxxxx>> wrote:
Hi,
The WebSocket specification document has references like this in it:
WSC-6.5-2
These appear to cross-reference to the TCK where there are comments in the source like this:
/* * @testName: constructortest * * @assertion_ids: WebSocket:JAVADOC:8; WebSocket:JAVADOC:10; * WebSocket:JAVADOC:6; WebSocket:JAVADOC:7; WebSocket:JAVADOC:70; * WebSocket:JAVADOC:71; WebSocket:SPEC:WSC-3.2.1-1; * WebSocket:SPEC:WSC-3.2.2-1; * * @test_Strategy: Test constructor */
Together that makes me think there might be some sort of tooling somewhere doing something with these references. If that is the case it would be helpful to know what that tooling was and what it was doing before we start making too many changes to the spec. Any pointers?
Note: at this point we are just doing clean-up but we a number of open issues to clarify the spec. Adding that clarification may impact some of these references and/or lead us to want to add new ones.
Thanks,
Mark _______________________________________________ jakartaee-tck-dev mailing list jakartaee-tck-dev@xxxxxxxxxxx <mailto:jakartaee-tck-dev@xxxxxxxxxxx> <mailto:jakartaee-tck-dev@xxxxxxxxxxx> To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev
<http://oracle.com/us/design/oracle-email-sig-198324.gif> <http://oracle.com/us/design/oracle-email-sig-198324.gif><http://oracle.com/us/design/oracle-email-sig-198324.gif>
<http://oracle.com/us/design/oracle-email-sig-198324.gif>Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen@xxxxxxxxxx <mailto:Lance.Andersen@xxxxxxxxxx> <mailto:Lance.Andersen@xxxxxxxxxx>
_______________________________________________ jakartaee-tck-dev mailing list jakartaee-tck-dev@xxxxxxxxxxx <mailto:jakartaee-tck-dev@xxxxxxxxxxx> To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev
_______________________________________________ jakartaee-tck-dev mailing list jakartaee-tck-dev@xxxxxxxxxxx <mailto:jakartaee-tck-dev@xxxxxxxxxxx> To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev
<http://oracle.com/us/design/oracle-email-sig-198324.gif> <http://oracle.com/us/design/oracle-email-sig-198324.gif><http://oracle.com/us/design/oracle-email-sig-198324.gif>
<http://oracle.com/us/design/oracle-email-sig-198324.gif>Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037 Oracle Java Engineering 1 Network Drive Burlington, MA 01803 Lance.Andersen@xxxxxxxxxx <mailto:Lance.Andersen@xxxxxxxxxx>
_______________________________________________ jakartaee-tck-dev mailing list jakartaee-tck-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev
_______________________________________________ jakartaee-tck-dev mailing list jakartaee-tck-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev
_______________________________________________ jakartaee-tck-dev mailing list jakartaee-tck-dev@xxxxxxxxxxx To change your delivery options, retrieve your password, or unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev
_______________________________________________ jakartaee-tck-dev mailing list jakartaee-tck-dev@xxxxxxxxxxxTo change your delivery options, retrieve your password, or unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev
 Lance Andersen| Principal Member of Technical Staff | +1.781.442.2037Oracle Java Engineering 1 Network Drive Burlington, MA 01803Lance.Andersen@xxxxxxxxxx
|