Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jakartaee-tck-dev] standard recipe to follow for refactoring the Platform TCK tests...

There is something I missed to mention here and I faced while working on migrating the servlet TCK to using arquillian junit.
And I would like to discuss it.

I will take this class as example com.sun.ts.tests.servlet.api.jakarta_servlet.servletrequest.URLClient.
The current TCK is using some javadoc annotations such:
  /*
   * @testName: getAttributeNamesTest
   *
   * @assertion_ids: Servlet:JAVADOC:174
   *
   * @test_Strategy: Servlet sets some attributes and verifies they can be
   * retrieved.
   *
   */

The class doesn't have this method but it's inherited.

The current framework will run only the methods found with the javadoc annotations @testName (the method could be in the current class or inherited)
Per default, junit runs only tests with the annotation @Test and will run all annotated methods in inherited classes which we don't want here.
I have marked all test methods with @Test in parent classes but by default all inherited classes will run all @Test methods which is wrong. 
But @Test annotation is very convenient for running a test in any IDE.
What I did currently is to write a maven plugin [1] which parses sources and finds javadoc annotation @testName to generate a text file containing all the className#methodName to run.
Then I wrote a dedicated org.junit.platform.engine.TestEngine [2] which read the generated files from parsing sources to run only those tests.

It works but I find this extra work and a bit more complicated to configure in Maven (and more maintenance in case junit changes something).

My idea was to simply add the missing methods to the classes such in com.sun.ts.tests.servlet.api.jakarta_servlet.servletrequest.URLClient
(this can be done by the same plugin which is currently parsing the classes to discover which tests to run)

public getAttributeNamesTest() {
  super.getAttributeNamesTest();
}

With such changes, no more need to maintain a dedicated test engine, easy to run test in any IDE in debug mode, easier Maven configuration.

wdyt?





On Tue, Oct 11, 2022 at 6:07 PM Olivier Lamy <olamy@xxxxxxxxxxx> wrote:
Hi,
It sounds a bit complicated to have a common recipe.
But for servlet, I choose a the testable jar war so I can simply test it with Jetty using a pom here https://github.com/jetty-project/servlet-tck-run/
Note in the branch, I created I have added a module to run the same way with Glassfish in a profile see https://github.com/olamy/jakartaee-tck/blob/981fa5e53bd4f8a1188e044bfb0acd3d951e6d3c/tck-run/pom.xml#L88
I wanted to avoid any Glassfish specific (I think we should avoid if possible any container impl specific and move to the arquillian container impl part)
but yup this means I still have a specific test I cannot get working with Glassfish (https with cert) :) 

ETA on this WIP I have a huge rebase to get this in sync with current master 



On Sat, Oct 8, 2022 at 8:30 PM arjan tijms <arjan.tijms@xxxxxxxxx> wrote:
Hi,

On Saturday, October 8, 2022, Emily Jiang via jakartaee-tck-dev <jakartaee-tck-dev@xxxxxxxxxxx> wrote:
One of the requirements is that the refactored tests must be runnable by runtimes such as Open Liberty so that the whole integration can be tested.

Yes, that specifically applies to the json and _expression_ language tcks, which run outside runtimes such as open liberty now.

Kind regards,
Arjan Tijms

 

Thanks 
Emily 

Sent from my iPhone

On 7 Oct 2022, at 17:50, arjan tijms <arjan.tijms@xxxxxxxxx> wrote:


Hi,

Not yet jumping towards a definite answer, but there are a couple of options to keep in mind.

There are now two main variants in use to base the tests on: Junit and TestNG. As far as I can tell, this choice is completely arbitrary and neither has any real advantage over the other (please correct me if I'm wrong).

There are two phases to use for the tests with maven. The unit test phase (using surefire) and the integration test phase (using failsafe). Sometimes the choice is arbitrary, sometimes there's an actual reason; integration tests allow one to use the full output produced by a maven module, e.g. a .war.

There are also two main ways how tests are practically conducted using Arquillian. There is the so called "testable" mode (also called "magic mode") where a test class that looks like a client test is magically (hence the name) transported to the server and run from there. Arquillian automagically using a protocol transfers test results to the client (junit or testng). The other mode is "non-testable" (also called "run-on-client"), where the unit or IT test explicitly runs on the client, and where explicit HTTP requests are submitted to the server under test. There's a few sub-variants here again. Some TCKs have e.g. a Servlet where they return some result and that result is then inspected by the client test. E.g. assert response contains "user logged-in". Others have a Servlet that manually inspects some results server-side, and then only return "succeeded" or "failed" to the client test. In some cases there's even a kind of junit/testng emulation like layer implemented and manually invoked at the server side.

Finally, but it largely follows from the choices above, there are TCKs that produce a "single test jar" that contain all the tests, which therefore requires a "runner pom" that imports this test jar via a special directive. Other TCKs are a "multi-module maven project", where each piece of test code is its own maven module and can be used and deployed as a single standalone application (and therefore also debugged as such).

We can probably create a table to categorize the existing TCKs into the above options.

Kind regards,
Arjan Tijms








On Fri, Oct 7, 2022 at 5:50 PM Scott Marlow <smarlow@xxxxxxxxxx> wrote:

Hi,

It would be good to have a standard recipe to follow for refactoring the Platform TCK tests as we discussed on the TCK call [1] this week.

What are the initial steps that we followed for EE 10 to refactor Platform TCK tests to junit?  For the Faces TCK refactoring I recall we discussed a lot on Faces issues (e.g. like how to deal with vendor deployment descriptors).  Does anyone feel ready to respond with what the steps might be? 

Or should we explore those further together (pick a test bucket and start refactoring at least one test in the Platform TCK)? 

Scott

[1] https://docs.google.com/document/d/1V1dDLJkd14EDRMPeuI0VzPtU4Lbli8FFBd1pLDLlOrY/edit#

_______________________________________________
jakartaee-tck-dev mailing list
jakartaee-tck-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev
_______________________________________________
jakartaee-tck-dev mailing list
jakartaee-tck-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev


 
_______________________________________________
jakartaee-tck-dev mailing list
jakartaee-tck-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-tck-dev


--
Olivier


--
Olivier

Back to the top