Re: [jakartaee-tck-dev] [External] : Re: standard recipe to follow for r

[Lenny Primak <lenny@xxxxxxxxxxxxx>]



Also latest  TestNG doesn’t work with the latest Arquillian: https://github.com/arquillian/arquillian-core/issues/427

On Oct 19, 2022, at 3:25 PM, Scott Marlow <smarlow@xxxxxxxxxx> wrote:



On 10/11/22 1:18 PM, Scott Stark wrote:

I would not dismiss a complete refactoring at this point as the next full EE release might not be until the end of 2024.

I would leave TestNG out of the picture because it really no longer brings anything to the table that Junit5 does not support. The main benefit TestNG had when CDI/BV were created was the ability to have an external test suite definition. A common external format is something we need to create for Junit5 as it only has an API for this as far as I know.

Common reporting is another thing to look at.

A bigger issue to try to address is the lack of assertions linked between tests and specifications. CDI/BV had used annotation processing to try to do this, but the links to the specification are a manual step that easily becomes obsolete. Ideally the assertions are in the spec asciidoc and these can be transformed into an assertion library that the tests can reference.

In general we need a roadmap for the platform TCK and start creating epics around features we want to target.

https://github.com/eclipse-ee4j/jakartaee-tck/issues/1126 is a rough first pass.

On Oct 10, 2022 at 11:20:14 AM, Gurunandan Rao <gurunandan.rao@xxxxxxxxxx> wrote:

IMO, Platform TCK refactoring should be targeted on SPEC basis for a particular Jakarta EE release, refactoring all of platform TCK is a huge effort and not feasible for a single Jakarta EE release.

Common TCK framework needs to be implemented for the refactored TCK's, some of the common functionality are as follows:

1) Signature test Driver and recording of new Signature.

2) Logging framework for TCK.

3) Test grouping to implement common grouping like SE/EE.

4) Test exclusion.

5) Framework for seamless run of refactored standalone tests in a container for Platform runs.

6) Test execution report.

We should also explore on feasible of a common framework to encapsulate from vendors the test tech Junit/TestNG/Arquillian.

---

From: jakartaee-tck-dev <jakartaee-tck-dev-bounces@xxxxxxxxxxx> on behalf of arjan tijms <arjan.tijms@xxxxxxxxx>
Sent: 08 October 2022 16:00
To: jakartaee-tck developer discussions <jakartaee-tck-dev@xxxxxxxxxxx>
Subject: [External] : Re: [jakartaee-tck-dev] standard recipe to follow for refactoring the Platform TCK tests...

 

Hi,

On Saturday, October 8, 2022, Emily Jiang via jakartaee-tck-dev <jakartaee-tck-dev@xxxxxxxxxxx> wrote:

One of the requirements is that the refactored tests must be runnable by runtimes such as Open Liberty so that the whole integration can be tested.

Yes, that specifically applies to the json and _expression_ language tcks, which run outside runtimes such as open liberty now.

Kind regards,

Arjan Tijms

 

Thanks 

Emily 

Sent from my iPhone

On 7 Oct 2022, at 17:50, arjan tijms <arjan.tijms@xxxxxxxxx> wrote:



Hi,

Not yet jumping towards a definite answer, but there are a couple of options to keep in mind.

There are now two main variants in use to base the tests on: Junit and TestNG. As far as I can tell, this choice is completely arbitrary and neither has any real advantage over the other (please correct me if I'm wrong).

There are two phases to use for the tests with maven. The unit test phase (using surefire) and the integration test phase (using failsafe). Sometimes the choice is arbitrary, sometimes there's an actual reason; integration tests allow one to use the full output produced by a maven module, e.g. a .war.

There are also two main ways how tests are practically conducted using Arquillian. There is the so called "testable" mode (also called "magic mode") where a test class that looks like a client test is magically (hence the name) transported to the server and run from there. Arquillian automagically using a protocol transfers test results to the client (junit or testng). The other mode is "non-testable" (also called "run-on-client"), where the unit or IT test explicitly runs on the client, and where explicit HTTP requests are submitted to the server under test. There's a few sub-variants here again. Some TCKs have e.g. a Servlet where they return some result and that result is then inspected by the client test. E.g. assert response contains "user logged-in". Others have a Servlet that manually inspects some results server-side, and then only return "succeeded" or "failed" to the client test. In some cases there's even a kind of junit/testng emulation like layer implemented and manually invoked at the server side.

Finally, but it largely follows from the choices above, there are TCKs that produce a "single test jar" that contain all the tests, which therefore requires a "runner pom" that imports this test jar via a special directive. Other TCKs are a "multi-module maven project", where each piece of test code is its own maven module and can be used and deployed as a single standalone application (and therefore also debugged as such).

We can probably create a table to categorize the existing TCKs into the above options.

Kind regards,

Arjan Tijms

On Fri, Oct 7, 2022 at 5:50 PM Scott Marlow <smarlow@xxxxxxxxxx> wrote:

Hi,

It would be good to have a standard recipe to follow for refactoring the Platform TCK tests as we discussed on the TCK call [1] this week.

What are the initial steps that we followed for EE 10 to refactor Platform TCK tests to junit?  For the Faces TCK refactoring I recall we discussed a lot on Faces issues (e.g. like how to deal with vendor deployment descriptors).  Does anyone feel ready to respond with what the steps might be? 

Or should we explore those further together (pick a test bucket and start refactoring at least one test in the Platform TCK)? 

Scott

[1] https://docs.google.com/document/d/1V1dDLJkd14EDRMPeuI0VzPtU4Lbli8FFBd1pLDLlOrY/edit#

_______________________________________________
jakartaee-tck-dev mailing list
jakartaee-tck-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev

_______________________________________________
jakartaee-tck-dev mailing list
jakartaee-tck-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev

 

_______________________________________________
jakartaee-tck-dev mailing list
jakartaee-tck-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev

_______________________________________________
jakartaee-tck-dev mailing list
jakartaee-tck-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev

_______________________________________________
jakartaee-tck-dev mailing list
jakartaee-tck-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev