Skip to main content
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jakartaee-platform-dev] [EXTERNAL] Re: TCK tests in the same repo as API andSpec

Could someone familiar with the CTS / JT Harness side of things outline what might be needed in order to "adapt' a JUnit and/or TestNG suite ?

It seems like we could get value starting with simple yes/no compliance... pointing the harness to execute the entire API test suite, getting back pass/fail, and requiring someone to dig through logs to find out what went wrong at a finer level of detail.

Then maybe later we could look at adding filtering and more finer-grained reporting of individual method pass/fails?

------------------------------------------------------
Scott Kurz
WebSphere Batch and Developer Experience
skurz@xxxxxxxxxx
--------------------------------------------------------

Inactive hide details for Bill Shannon ---02/24/2020 03:37:48 PM---Some of the proposed solutions depend on an adapter.  I don'Bill Shannon ---02/24/2020 03:37:48 PM---Some of the proposed solutions depend on an adapter.  I don't know that we want to try for a univers

From: Bill Shannon <bill.shannon@xxxxxxxxxx>
To: jakartaee-platform developer discussions <jakartaee-platform-dev@xxxxxxxxxxx>, Scott Kurz <skurz@xxxxxxxxxx>
Date: 02/24/2020 03:37 PM
Subject: Re: [jakartaee-platform-dev] [EXTERNAL] Re: TCK tests in the same repo as API andSpec




Some of the proposed solutions depend on an adapter.  I don't know that we want to try for a universal adapter, or even a large number of adapters.  If the adapter approach is feasible, I would guess that we would still want to limit it to a very small number of test suite styles for developing the individual tests.

I don't want to boil the ocean, I just want to cook dinner.

Scott Kurz wrote on 2/24/20 12:15 PM:
      I haven't been able to get a handle on the overall thread.


      But let me just start with the Batch TCK, which is already using a TestNG suite.


      If there were a way to "adapt" this suite so that the CTS / JT harness driving it could use whatever
            • executing with filtering (run these tests, not those)
            • reporting/logging


      then it'd seem like we'd achieve a lot of the goals:  allowing TCK developers to develop with their preferred test library/framework, not requiring rewrites.



      Did I miss anything?  Any ideas on where to start with such an adapter?   


      ------------------------------------------------------
      Scott Kurz
      WebSphere Batch and Developer Experience
      skurz@xxxxxxxxxx
      --------------------------------------------------------

      Inactive hide details for "Kevin Sutter" ---02/24/2020 02:57:39 PM---I have tried to post this to the jakartaee-tck-dev mailing"Kevin Sutter" ---02/24/2020 02:57:39 PM---I have tried to post this to the jakartaee-tck-dev mailing list since that was the action from last

      From:       "Kevin Sutter" <sutter@xxxxxxxxxx>
      To:       jakartaee-platform developer discussions <jakartaee-platform-dev@xxxxxxxxxxx>
      Date:       02/24/2020 02:57 PM
      Subject:       Re: [jakartaee-platform-dev] [EXTERNAL] Re: TCK tests in the same repo as API andSpec
      Sent by:       jakartaee-platform-dev-bounces@xxxxxxxxxxx



      I have tried to post this to the jakartaee-tck-dev mailing list since that was the action from last week's Platform call. And, since Andy is referencing some of it below, I thought I would start off my reply with a pointer to the minutes:       https://eclipse-ee4j.github.io/jakartaee-platform/minutes/2020-02-18.html. And, specifically the section of the minutes where we discussed the TCK effort.

      Criteria for replacing the TCKs. [BS]
            • Discussed how to ensure that a “refactored” TCK is a sufficient replacement to the original (previous version)
            • Is each individual Spec project responsible to verify the re-factoring? Is that sufficient? Or, do we need some external checks-and-balances?
            • In the past (J2EE, Java EE), the TCKs were incrementally modified. Easier to monitor the changes going in.
            • Structured reviews might help ensure consistency. Participants from Spec Project team, the Platform TCK team, the Spec Project TCK team are required.
            • Common framework for the TCKs? Or, allow each independent TCK to determine the framework used? A common framework is probably key to the success of this effort.
            • Defining a common framework would allow each Project to plug in their TCK and be executed as part of the overall Platform TCK.
            • Requirement – allow the use of the existing TCK tests themselves. If all of the TCK tests need to be modified just to become part of this new infrastructure, the process will die. We need the ability to incorporate existing TCK tests. Maybe that’s through some build magic or wrappering of the tests or something…
            • Excellent start to this discussion, but needs much more work. Who should drive this effort? Platform TCK? An individual TCK (ie. json-b or json-p)? Group effort? Post to the Platform TCK mailing list and ask for volunteers. (KWS)

      Although the discussion on the call was very good, it also demonstrates that we don't have a complete solution yet. The thought on the call is that we should try to enlist some people with TCK experience to help with defining the solution. Since the discussion seems to want to live in the Platform mailing list instead of the TCK mailing list, I'll post here instead...

      I think the main Requirement (highlighted above) that is getting missed in the discussion is the need to have a framework defined that can easily incorporate existing testcode into a cohesive test suite.

      As stated above, if we define a process that requires updates to existing test code source, then the overall goal of separating the TCKs will die. We'll get to the 10-12 gung-ho projects that do the separation, but if it's not an easy process to move forward then the other projects won't find the cycles to do the work. And, we definitely do not want to have duplicate test source in multiple repos. It's fine for now for json-b and json-p as we continue to experiment, but we don't want that practice to continue. It's great to have the individual tests run with each project, but we also need an easy way to incorporate those same tests into an overall TCK test suite.

      Andy, I'm not sure if you are signing up for this? Or, maybe you can find some existing TCK expertise to help you out? But, the end goal is to make this a well-defined process for incorporating separate TCKs into the overall Platform TCK.

      ---------------------------------------------------
      Kevin Sutter
      STSM, MicroProfile and Jakarta EE architect @ IBM
      e-mail:       sutter@xxxxxxxxxx Twitter: @kwsutter
      phone: tl-553-3620 (office), 507-253-3620 (office)
      LinkedIn:       https://www.linkedin.com/in/kevinwsutter



      From:       Andy Guibert <andy.guibert@xxxxxxxxx>
      To:       Bill Shannon <bill.shannon@xxxxxxxxxx>
      Cc:       jakartaee-platform developer discussions <jakartaee-platform-dev@xxxxxxxxxxx>
      Date:       02/24/2020 12:38
      Subject:       [EXTERNAL] Re: [jakartaee-platform-dev] TCK tests in the same repo as API andSpec
      Sent by:       jakartaee-platform-dev-bounces@xxxxxxxxxxx





      On Sun, Feb 23, 2020 at 12:16 PM Bill Shannon <bill.shannon@xxxxxxxxxx> wrote:
      Andy Guibert wrote on 2/20/20 11:52 AM:
      Now we seem to be going in circles... To recap the relevant points:

      1) How do we create overall platform TCK and how does it fit together?
      > In the same way that CDI and BeanValidation did it for JavaEE 8 / JakartaEE 8
      They didn't. That's the point. That's what we want to avoid.

      It's true that CDI/BV do not roll up into the same spot as everywhere else, but based on my proposed PR to jakartaee-tckI thought we didn't want that for externalized TCKs?

      Speaking from experience here, in IBM/OpenLiberty we have an entire team dedicated to setting up and running the JEE CTS. They are the only ones who know how to set up and run the arcane ball of JakartaEE TCKs.
      When I was working on the BeanValidation 2.0 implementation for OpenLiberty for JavaEE 8 certification I was able to figure out how to set up and run the BeanValidation 2.0 TCKs against OpenLiberty in under an hour because I was already familiar with Arquillian (as many JEE developers already are). On the other hand only the dedicated CTS teams know how to set up and run the JavaTest stuff.

      2) What test framework? What technologies?
      > Arquillian+JUnit. These technologies are well-tested and widely understood in the community, and are perfectly capable of doing more complex testing such as pulling in DBs, mail servers, or other external resources. CDI and BeanVal are reasonably complex specs/TCKs and are already using Arquillian.
      How do we teach Arquillian to run the existing test cases? Or how do we convert the existing test cases so that Arquillian can run them? And either way, how do we assure ourselves that the result has the same pass/fail characteristics as the original?

      I converted the existing JSON-B test cases to Arquillian in this commit here: https://github.com/eclipse-ee4j/jsonb-api/pull/221/commits/6a4867bd18a6b88750066ac43debbedfce1bebb4
      The changes were pretty systematic and I did some spot negative testing to ensure validity.
      Ultimately, we assure ourselves the results are the same by relying on our developers and our code reviewers. This is a one-time change per-spec and coming up with a more elaborate process would be over-engineering a solution IMO.

      3) [From the meeting minutes] Requirement – allow the use of the existing TCK tests themselves. If all of the TCK tests need to be modified just to become part of this new infrastructure, the process will die. We need the ability to incorporate existing TCK tests.
      > The conversion can be done on a spec-by-spec case. For example, everything would work fine if just JSON-B migrated to the "external TCK" way and all other specs remained as-is. Also, the conversion could be all-in-one or partial. For JSON-B I did an all-in-one conversion, meaning I took _all_ existing TCK tests and converted them to Arquillian+JUnit so now JSON-B TCK tests don't use JavaTest at all. Personally I think per-spec all-in-one conversion is best so there is only ever 1 framework per spec, but each spec could have the flexibility to do what they want.

      This is the process that the notes claim will die. Depending on each spec project to convert all of their tests to a new framework just won't happen. You might end up with 10 specs fully converted and 27 specs still using the old framework. If old and new can coexist, that might be fine. Even if we had to run two completely separate TCK frameworks, that might be acceptable. I'd like to hear what others think about this entire process of converting tests to a new framework, whether it needs to be done automatically, whether it can be done manually and incrementally, etc.

      Yes, 10 specs could convert to the new way and 27 specs could stay with the old framework and that would be perfectly fine. IMO That's a win-win because the 10 specs that took the time to convert get the quality-of-life boost that comes with the decoupled/modernized and the 27 specs that stayed with the old way continue on with business as usual.
      _______________________________________________
      jakartaee-platform-dev mailing list
      jakartaee-platform-dev@xxxxxxxxxxx
      To change your delivery options, retrieve your password, or unsubscribe from this list, visit
      https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev


      _______________________________________________
      jakartaee-platform-dev mailing list
      jakartaee-platform-dev@xxxxxxxxxxx
      To change your delivery options, retrieve your password, or unsubscribe from this list, visit
      https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev




      _______________________________________________
      jakartaee-platform-dev mailing list
      jakartaee-platform-dev@xxxxxxxxxxx
      To change your delivery options, retrieve your password, or unsubscribe from this list, visit
      https://urldefense.com/v3/__https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev__;!!GqivPVa7Brio!PMMpIrGKQhzDDh1xagIMc2U8M8GLyiW2-vmNwhxTAX38yOfoV9e2RlE-qXTlZqRTow$ 



Back to the top