Regarding "platform tests" I'd like to move away from tests that are "owned" by the platform and instead have multi-spec integrations be "owned" by one of the specs involved.
You mentioned that you would like input from those that run the Platform TCK (CTS), so I wanted to share my input, as I have been part of a team doing that for several years now.
I want to see us maintain the current level of Jakarta EE Platform level testing. I am fine with achieving that via restructuring of the TCK tests, but I think that we need to know which SPECs exactly, would take on the burden of running additional (platform level) TCK tests, which means they have to chase down more failures for each new EE release and other hassles that they don't currently have to deal with.
Also, a blocking issue for me, would be if the separate SPECs do not offer a way to run the equivalent of the current Full Platform/Web Profile TCK tests. The reason being that we need more platform level test coverage, not less.
Currently, with the Full Platform, we run 1062 JSON-B tests, Web Profile has 532 JSON-B tests. To me, the standalone JSON-B TCK tests would need to be able to run these same tests against a Full Platform/Web Profile Jakarta EE implementation.
If we can accomplish platform level testing for JSON-B + other SPECs, that will address my concern about not reducing the current EE test coverage.
Another blocking issue, is that that other SPECs, like JPA, are using many other SPECs, so it isn't clear as to which SPEC TCK, the tests should really move to.
Regarding the idea of using Arquillian, +100 on that! :)
For example, if we have a test for a BeanValidation + CDI interaction, the let the test be housed in the BeanValidation TCK. By default, when you run the BeanValidation TCK _all_ tests would run (including the BV+CDI ones), and if an implementation did not want to certify with CDI then they could explicitly disable the CDI tests by doing something like `mvn test -Dcdi=false`.
I think it will be helpful to not have tests in "no mans land" for the sake of test development and maintenance.