Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jakartaee-platform-dev] [EXTERNAL] Re: TCK tests in the same repo as API andSpec

On Wed, Mar 11, 2020 at 1:57 PM Scott Marlow <smarlow@xxxxxxxxxx> wrote:
On Wed, Feb 26, 2020 at 10:45 AM Andy Guibert <andy.guibert@xxxxxxxxx> wrote:
Regarding "platform tests" I'd like to move away from tests that are "owned" by the platform and instead have multi-spec integrations be "owned" by one of the specs involved.

You mentioned that you would like input from those that run the Platform TCK (CTS), so I wanted to share my input, as I have been part of a team doing that for several years now.  

I want to see us maintain the current level of Jakarta EE Platform level testing.  I am fine with achieving that via restructuring of the TCK tests, but I think that we need to know which SPECs exactly, would take on the burden of running additional (platform level) TCK tests, which means they have to chase down more failures for each new EE release and other hassles that they don't currently have to deal with.

I consider "platform level" tests to be any test that incorporates interactions between 2 or more specs in a single test. For these cases, we can divide out the TCK tests to the spec whose features are the main premise of the test case. For example, if we have a test that utilizes CDI+BeanValidation+JPA, the primary subject of the test might be that BeanValidation annotations are properly checked on a JPA entity that uses CDI. So in that case the test could be owned/moved to the BeanValidation TCK. It won't be a precise process, but the basic idea is to assign spec ownership to each test (as we split them out) so we minimize the scenarios left in "no mans land" (i.e. platform level). 

Also, a blocking issue for me, would be if the separate SPECs do not offer a way to run the equivalent of the current Full Platform/Web Profile TCK tests.  The reason being that we need more platform level test coverage, not less.  

Currently, with the Full Platform, we run 1062 JSON-B tests, Web Profile has 532 JSON-B tests.  To me, the standalone JSON-B TCK tests would need to be able to run these same tests against a Full Platform/Web Profile Jakarta EE implementation.  

If we can accomplish platform level testing for JSON-B + other SPECs, that will address my concern about not reducing the current EE test coverage.  

I don't fully understand this comment, but I think you're saying that currently we repeat each test for different "containers", namely: Servlet, EJB, App Client, and JSP. Assuming that is what you meant, we can extend the original test classes and override the deployments in each subclass. For example, we could have:
public class SomeTest_standalone { /* @Test methods .... */ }

public class SomeTest_Servlet extends SomeTest_standalone {
  public WebArchive<?> buildWar() { ... }

public class SomeTest_EJB extends SomeTest_standalone {
  public WebArchive<?> buildEjbWar() { ... }

To control which containers get used when, we can either use a naming pattern like "**/*_EJB.class" or we could define system properties or environment variables for which containers get enabled/disabled via JUnit 5 conditional annotations (see
Another blocking issue, is that that other SPECs, like JPA, are using many other SPECs, so it isn't clear as to which SPEC TCK, the tests should really move to.

Like mentioned above, it wont' be an exact science, but in most cases it's pretty clear which spec owns the guts of the test case. If it really is a toss-up between two or more specs as to who should own it, then we can pick either spec. Probably the spec that could most readily accommodate the test without adding a bunch of extra infra. For example, if it's a toss-up between if CDI or JPA should own a test case and the CDI TCK didn't yet require a DB but JPA did, then it would be better to move the test to the JPA TCK.
Regarding the idea of using Arquillian, +100 on that! :)


For example, if we have a test for a BeanValidation + CDI interaction, the let the test be housed in the BeanValidation TCK.  By default, when you run the BeanValidation TCK _all_ tests would run (including the BV+CDI ones), and if an implementation did not want to certify with CDI then they could explicitly disable the CDI tests by doing something like `mvn test -Dcdi=false`.

I think it will be helpful to not have tests in "no mans land" for the sake of test development and maintenance. 

jakartaee-platform-dev mailing list
To unsubscribe from this list, visit

Back to the top