[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
|
Re: [jakartaee-platform-dev] [EXTERNAL] Re: TCK tests in the same repo as API andSpec
|
Andy Guibert wrote on 3/17/20 8:00 AM:
Regarding "platform tests" I'd like to
move away from tests that are "owned" by the
platform and instead have multi-spec integrations be
"owned" by one of the specs involved.
You mentioned that you would like input from those
that run the Platform TCK (CTS), so I wanted to
share my input, as I have been part of a team doing
that for several years now.
I want to see us maintain the current level of
Jakarta EE Platform level testing. I am fine with
achieving that via restructuring of the TCK tests, but
I think that we need to know which SPECs exactly,
would take on the burden of running additional
(platform level) TCK tests, which means they have to
chase down more failures for each new EE release and
other hassles that they don't currently have to deal
with.
I consider "platform level" tests to be any test that
incorporates interactions between 2 or more specs in a
single test. For these cases, we can divide out the TCK
tests to the spec whose features are the main premise of the
test case. For example, if we have a test that utilizes
CDI+BeanValidation+JPA, the primary subject of the test
might be that BeanValidation annotations are properly
checked on a JPA entity that uses CDI. So in that case the
test could be owned/moved to the BeanValidation TCK. It
won't be a precise process, but the basic idea is to assign
spec ownership to each test (as we split them out) so we
minimize the scenarios left in "no mans land" (i.e. platform
level).
In some cases that might be reasonable, but while I don't want
platform tests to be a no-man's land, I don't see a problem with
having some tests that are maintained by the platform project. More
of these "end-to-end" tests would be useful to ensure that real
application scenarios are working as expected.
Also, a blocking issue for me, would be if the
separate SPECs do not offer a way to run the
equivalent of the current Full Platform/Web Profile
TCK tests. The reason being that we need more
platform level test coverage, not less.
Currently, with the Full Platform, we run 1062
JSON-B tests, Web Profile has 532 JSON-B tests. To
me, the standalone JSON-B TCK tests would need to be
able to run these same tests against a Full
Platform/Web Profile Jakarta EE implementation.
If we can accomplish platform level testing for
JSON-B + other SPECs, that will address my concern
about not reducing the current EE test coverage.
I don't fully understand this comment, but I think you're
saying that currently we repeat each test for different
"containers", namely: Servlet, EJB, App Client, and JSP.
I think they're called "vehicles" in the CTS.
Assuming that is what you meant, we can extend the
original test classes and override the deployments in each
subclass. For example, we could have:
@RunWith(Arquillian.class)
public class SomeTest_standalone { /* @Test methods ....
*/ }
public class SomeTest_Servlet extends SomeTest_standalone
{
@Deployment
public WebArchive<?> buildWar() { ... }
}
public class SomeTest_EJB extends SomeTest_standalone {
@Deployment
public WebArchive<?> buildEjbWar() { ... }
}
Ideally the tests themselves wouldn't need to declare this, but that
some higher level logic could find all the "vehicle-independent"
tests and run them in all the vehicles, packaging them accordingly
for the vehicle being tested.
If the test needs some vehicle-dependent code, it might need to test
which vehicle it's running in.
Possibly the tests could just declare which vehicles to run them in,
but that makes adding a new vehicle more difficult.
Another blocking issue, is that that other SPECs,
like JPA, are using many other SPECs, so it isn't
clear as to which SPEC TCK, the tests should really
move to.
Like mentioned above, it wont' be an exact science, but
in most cases it's pretty clear which spec owns the guts of
the test case. If it really is a toss-up between two or more
specs as to who should own it, then we can pick either spec.
Probably the spec that could most readily accommodate the
test without adding a bunch of extra infra. For example, if
it's a toss-up between if CDI or JPA should own a test case
and the CDI TCK didn't yet require a DB but JPA did, then it
would be better to move the test to the JPA TCK.
Regarding the idea of using Arquillian, +100 on
that! :)
Scott
For example, if we have a test for a
BeanValidation + CDI interaction, the let the test
be housed in the BeanValidation TCK. By default,
when you run the BeanValidation TCK _all_ tests
would run (including the BV+CDI ones), and if an
implementation did not want to certify with CDI
then they could explicitly disable the CDI tests
by doing something like `mvn test -Dcdi=false`.
I think it will be helpful to not have tests in
"no mans land" for the sake of test development
and maintenance.
_______________________________________________
jakartaee-platform-dev mailing list
jakartaee-platform-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev
_______________________________________________
jakartaee-platform-dev mailing list
jakartaee-platform-dev@xxxxxxxxxxx
To unsubscribe from this list, visit https://urldefense.com/v3/__https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev__;!!GqivPVa7Brio!OOY7-R154n0pHsKx2_NBHcrybL_HyKg-nlIWyZyS_oV-0_Uv1289VXSP70mEbsyCBw$
