[
Date Prev][
Date Next][
Thread Prev][
Thread Next][
Date Index][
Thread Index]
[
List Home]
Re: [jakartaee-platform-dev] [EXTERNAL] Re: TCK tests in the same repo as API andSpec
|
This question might be better posed to the
jtharness
mailing list in the openjdk project. That list looks fairly
active so you might be able to get a response from them.
Scott Kurz wrote on 2/24/20 1:04 PM:
Could someone familiar with the CTS / JT Harness
side of things outline what might be needed in order to
"adapt' a JUnit and/or TestNG suite ?
It seems like we could get value starting with
simple yes/no compliance... pointing the harness to execute
the entire API test suite, getting back pass/fail, and
requiring someone to dig through logs to find out what went
wrong at a finer level of detail.
Then maybe later we could look at adding
filtering and more finer-grained reporting of individual
method pass/fails?
------------------------------------------------------
Scott Kurz
WebSphere Batch and Developer Experience
skurz@xxxxxxxxxx
--------------------------------------------------------
Bill Shannon ---02/24/2020 03:37:48 PM---Some
of the proposed solutions depend on an adapter. I don't know
that we want to try for a univers
From: Bill
Shannon <bill.shannon@xxxxxxxxxx>
To: jakartaee-platform
developer discussions
<jakartaee-platform-dev@xxxxxxxxxxx>, Scott Kurz
<skurz@xxxxxxxxxx>
Date: 02/24/2020
03:37 PM
Subject: Re:
[jakartaee-platform-dev] [EXTERNAL] Re: TCK tests in the same
repo as API andSpec
Some of the proposed solutions depend on an adapter. I don't know
that we want to try for a universal adapter, or even a large
number of adapters. If the adapter approach is feasible, I would
guess that we would still want to limit it to a very small number
of test suite styles for developing the individual tests.
I don't want to boil the ocean, I just want to cook dinner.
Scott Kurz wrote on 2/24/20 12:15 PM:
I haven't been able to get a handle on the overall thread.
But let me just start with the Batch TCK, which is already
using a TestNG suite.
If there were a way to "adapt" this suite so that the CTS / JT
harness driving it could use whatever
- executing with filtering (run these tests, not
those)
- reporting/logging
then it'd seem like we'd achieve a lot of the goals: allowing
TCK developers to develop with their preferred test
library/framework, not requiring rewrites.
Did I miss anything? Any ideas on where to start with such an
adapter?
------------------------------------------------------
Scott Kurz
WebSphere Batch and Developer Experience
skurz@xxxxxxxxxx
--------------------------------------------------------
"Kevin Sutter" ---02/24/2020
02:57:39 PM---I have tried to post this to the
jakartaee-tck-dev mailing list since that was the action
from last
From: "Kevin Sutter" <sutter@xxxxxxxxxx>
To: jakartaee-platform developer
discussions <jakartaee-platform-dev@xxxxxxxxxxx>
Date: 02/24/2020 02:57 PM
Subject: Re: [jakartaee-platform-dev]
[EXTERNAL] Re: TCK tests in the same repo as API andSpec
Sent by: jakartaee-platform-dev-bounces@xxxxxxxxxxx
I have tried to post this to the jakartaee-tck-dev mailing
list since that was the action from last week's Platform
call. And, since Andy is referencing some of it below, I
thought I would start off my reply with a pointer to the
minutes: https://eclipse-ee4j.github.io/jakartaee-platform/minutes/2020-02-18.html. And, specifically the section of the minutes
where we discussed the TCK effort.
Criteria for replacing the TCKs. [BS]
- Discussed how to ensure that a “refactored” TCK is
a sufficient replacement to the original (previous
version)
- Is each individual Spec project responsible to
verify the re-factoring? Is that sufficient? Or, do
we need some external checks-and-balances?
- In the past (J2EE, Java EE), the TCKs were
incrementally modified. Easier to monitor the
changes going in.
- Structured reviews might help ensure consistency.
Participants from Spec Project team, the Platform
TCK team, the Spec Project TCK team are required.
- Common framework for the TCKs? Or, allow each
independent TCK to determine the framework used? A
common framework is probably key to the success of
this effort.
- Defining a common framework would allow each
Project to plug in their TCK and be executed as part
of the overall Platform TCK.
- Requirement – allow the use of the existing
TCK tests themselves. If all of the TCK tests need
to be modified just to become part of this new
infrastructure, the process will die. We need the
ability to incorporate existing TCK tests. Maybe
that’s through some build magic or wrappering of the
tests or something…
- Excellent start to this discussion, but needs much
more work. Who should drive this effort? Platform
TCK? An individual TCK (ie. json-b or json-p)? Group
effort? Post to the Platform TCK mailing list and
ask for volunteers. (KWS)
Although the discussion on the call was very good, it also
demonstrates that we don't have a complete solution yet. The
thought on the call is that we should try to enlist some
people with TCK experience to help with defining the
solution. Since the discussion seems to want to live in the
Platform mailing list instead of the TCK mailing list, I'll
post here instead...
I think the main Requirement (highlighted above) that is
getting missed in the discussion is the need to have a
framework defined that can easily incorporate existing
testcode into a cohesive test suite.
As stated above, if we define a process that requires
updates to existing test code source, then the overall goal
of separating the TCKs will die. We'll get to the 10-12
gung-ho projects that do the separation, but if it's not an
easy process to move forward then the other projects won't
find the cycles to do the work. And, we definitely do not
want to have duplicate test source in multiple repos. It's
fine for now for json-b and json-p as we continue to
experiment, but we don't want that practice to continue.
It's great to have the individual tests run with each
project, but we also need an easy way to incorporate those
same tests into an overall TCK test suite.
Andy, I'm not sure if you are signing up for this? Or, maybe
you can find some existing TCK expertise to help you out?
But, the end goal is to make this a well-defined process for
incorporating separate TCKs into the overall Platform TCK.
---------------------------------------------------
Kevin Sutter
STSM, MicroProfile and Jakarta EE architect @ IBM
e-mail: sutter@xxxxxxxxxx Twitter: @kwsutter
phone: tl-553-3620 (office), 507-253-3620 (office)
LinkedIn: https://www.linkedin.com/in/kevinwsutter
From: Andy Guibert <andy.guibert@xxxxxxxxx>
To: Bill Shannon <bill.shannon@xxxxxxxxxx>
Cc: jakartaee-platform developer
discussions <jakartaee-platform-dev@xxxxxxxxxxx>
Date: 02/24/2020 12:38
Subject: [EXTERNAL] Re:
[jakartaee-platform-dev] TCK tests in the same repo as API
andSpec
Sent by: jakartaee-platform-dev-bounces@xxxxxxxxxxx
On Sun, Feb 23, 2020 at 12:16 PM Bill Shannon <bill.shannon@xxxxxxxxxx>
wrote:
Andy Guibert wrote on 2/20/20 11:52 AM:
Now we seem to be going in circles... To recap the relevant
points:
1) How do we create overall platform TCK and how does it fit
together?
> In the same way that CDI and BeanValidation did it for
JavaEE 8 / JakartaEE 8
They didn't. That's the point. That's what we want to avoid.
It's true that CDI/BV do not roll up into the same spot as
everywhere else, but based on my proposed
PR to jakartaee-tckI thought we didn't
want that for externalized TCKs?
Speaking from experience here, in IBM/OpenLiberty we have an
entire team dedicated to setting up and running the JEE CTS.
They are the only ones who know how to set up and run the
arcane ball of JakartaEE TCKs.
When I was working on the BeanValidation 2.0 implementation
for OpenLiberty for JavaEE 8 certification I was able to
figure out how to set up and run the BeanValidation 2.0 TCKs
against OpenLiberty in under an hour because I was already
familiar with Arquillian (as many JEE developers already are).
On the other hand only the dedicated CTS teams know how to set
up and run the JavaTest stuff.
2) What test framework? What technologies?
> Arquillian+JUnit. These technologies are well-tested and
widely understood in the community, and are perfectly capable
of doing more complex testing such as pulling in DBs, mail
servers, or other external resources. CDI and BeanVal are
reasonably complex specs/TCKs and are already using
Arquillian.
How do we teach Arquillian to run the existing test cases?
Or how do we convert the existing test cases so that
Arquillian can run them? And either way, how do we assure
ourselves that the result has the same pass/fail
characteristics as the original?
I converted the existing JSON-B test cases to Arquillian in
this commit here: https://github.com/eclipse-ee4j/jsonb-api/pull/221/commits/6a4867bd18a6b88750066ac43debbedfce1bebb4
The changes were pretty systematic and I did some spot
negative testing to ensure validity.
Ultimately, we assure ourselves the results are the same by
relying on our developers and our code reviewers. This is a
one-time change per-spec and coming up with a more elaborate
process would be over-engineering a solution IMO.
3) [From the meeting minutes] Requirement – allow the use of
the existing TCK tests themselves. If all of the TCK tests
need to be modified just to become part of this new
infrastructure, the process will die. We need the ability to
incorporate existing TCK tests.
> The conversion can be done on a spec-by-spec case. For
example, everything would work fine if just JSON-B migrated
to the "external TCK" way and all other specs remained
as-is. Also, the conversion could be all-in-one or partial.
For JSON-B I did an all-in-one conversion, meaning I took
_all_ existing TCK tests and converted them to
Arquillian+JUnit so now JSON-B TCK tests don't use JavaTest
at all. Personally I think per-spec all-in-one conversion is
best so there is only ever 1 framework per spec, but each
spec could have the flexibility to do what they want.
This is the process that the notes claim will die. Depending
on each spec project to convert all of their tests to a new
framework just won't happen. You might end up with 10 specs
fully converted and 27 specs still using the old framework. If
old and new can coexist, that might be fine. Even if we had to
run two completely separate TCK frameworks, that might be
acceptable. I'd like to hear what others think about this
entire process of converting tests to a new framework, whether
it needs to be done automatically, whether it can be done
manually and incrementally, etc.
Yes, 10 specs could convert to the new way and 27 specs could
stay with the old framework and that would be perfectly fine.
IMO That's a win-win because the 10 specs that took the time
to convert get the quality-of-life boost that comes with the
decoupled/modernized and the 27 specs that stayed with the old
way continue on with business as usual.
_______________________________________________
jakartaee-platform-dev mailing list
jakartaee-platform-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password,
or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev
_______________________________________________
jakartaee-platform-dev mailing list
jakartaee-platform-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password,
or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev
_______________________________________________
jakartaee-platform-dev mailing list
jakartaee-platform-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
https://urldefense.com/v3/__https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev__;!!GqivPVa7Brio!PMMpIrGKQhzDDh1xagIMc2U8M8GLyiW2-vmNwhxTAX38yOfoV9e2RlE-qXTlZqRTow$