Re: [jakartaee-platform-dev] TCK tests in the same repo as API and Spec

["Kevin Sutter" <sutter@xxxxxxxxxx>]

Dmitry,
The one problem
with this approach is how do we keep the two test buckets in sync?  Is
there any way that we could have a single source repository for the testcases
themselves?  And, they get cloned into the respective test bucket
is attempting to test with them?  I fully understand that the test
harnesses and frameworks are different, but does that preclude the use
of common testcase source?  Or, maybe there's some other way to ensure
that these do not diverge?

---------------------------------------------------
Kevin Sutter 
STSM, MicroProfile and Jakarta EE architect @ IBM
e-mail:  sutter@xxxxxxxxxx     Twitter:  @kwsutter
phone: tl-553-3620 (office), 507-253-3620 (office)    
LinkedIn: https://www.linkedin.com/in/kevinwsutter



From:
       Dmitry
Kornilov <dmitry.kornilov@xxxxxxxxxx>
To:
       jakartaee-platform
developer discussions <jakartaee-platform-dev@xxxxxxxxxxx>
Cc:
       Kevin
Sutter <sutter@xxxxxxxxxx>
Date:
       02/04/2020
04:52
Subject:
       [EXTERNAL]
Re: [jakartaee-platform-dev] TCK tests in the same repo as API and Spec

---

We should consider job Andy’s done as
a PoC. It should be merged to JSONB repository and Yasson team should adopt
it without giving up TCK testing using CTS. TCK team should review it and
recommend it (or not recommend) it to other project as a template for Jakarta
EE 10 work. I don’t want a situation that splitting is artificially blocked
because no-one from TCK team wants to do it.

Andy’s effort doesn’t affect Jakarta
EE 9 release. As I was saying before, JSONB TCKs in CTS repository will
be used for certification. TCKs in JSONB repository can be used for preliminary
compatibility testing. It’s much easier to use, it works much faster and
it’s easier to create Jenkins jobs with these tests.

-- Dmitry


On 4 Feb 2020, at 04:54, Bill Shannon
<bill.shannon@xxxxxxxxxx>
wrote:

I think it's fine for people to do experiments,
ideally in their own personal forks so there's no confusion as to its relationship
with the official version.

What I don't want is for (e.g.) Andy to get it to the point that it's good
enough for him, he decides he's done and merges it in to the official repos
for his projects, and the rest is left for the rest of us to figure out.

I'm fine for not everything to be done before some things are merged, but
I want to see a plan that the jakartaee-tck project committers agree
to for how we're going to get to the desired end point.

Among other things, we need to know that the plan will get us to a working
platform tck in time for a corresponding platform release.

And it would be best if the plan didn't have us running both the "old"
and "new" TCKs in parallel for releases for some non-trivial
time.

This is a big project, so we're going to have to make some compromises,
but that doesn't mean giving up on any sense of planning.  An agile
approach is good, but we need confidence that we'll be able to see the
endpoint.


P.S. I'm also fine with adding lots more committers to the jakartaee-tck
project if that helps us make progress more quickly in the short term.
 There should be few impediments to spec projects updating their TCK
tests within the existing framework.  They can start with PRs against
the jakartaee-tck repo and become committers as needed.


Kevin Sutter wrote on 2/3/20 5:40 AM:
Jan,
You bring up some excellent points.  Having a well thought out plan
before jumping in with both feet would be nice.  But, on the other
hand, we don't want to stifle the excited effort that Andy is demonstrating.
 It's a tough line.  Maybe we need to allow one or two teams
to experiment with the processes and see what it takes.  And, then
we can take those experiences and develop a plan that works across all
the projects.

---------------------------------------------------
Kevin Sutter 
STSM, MicroProfile and Jakarta EE architect @ IBM
e-mail:  sutter@xxxxxxxxxx    Twitter:  @kwsutter
phone: tl-553-3620 (office), 507-253-3620 (office)    
LinkedIn: https://www.linkedin.com/in/kevinwsutter



From:        Jan
Supol <jan.supol@xxxxxxxxxx>
To:        jakartaee-platform-dev@xxxxxxxxxxx
Date:        01/31/2020
15:02
Subject:        [EXTERNAL]
Re: [jakartaee-platform-dev] TCK tests in the same repo as API and Spec
Sent by:        jakartaee-platform-dev-bounces@xxxxxxxxxxx

---

The discussion on the various email lists is quite extensive, and I 
admit I do not follow every thread that is discussed. But I am waiting
for the plan from any of the committee we have since the transition to
Eclipse Foundation. And I am not aware about any clear statement about
what the final goal is. For start:

- Is the splitting really what is recommended?

- Should the Platform Jakarta EE TCK still be available in the future 
after the split?

- Should the interoperability between Specs tests (the tests that 
require multiple Specifications to cooperate as it is described in one
or the other specification, the tests that used to be part of the full
CTS only) still be part of the combined (?) Jakarta EE TCK?

- Is there a recommended framework to be used in the future with the 
TCKs (such as the existing JavaTest + perhaps Arquillian similarly to 
Microprofile + ?)?

Those should be goals presented by the Jakarta representatives before 
the TCK repo is started to be ripped apart. Not on email, but clearly on
the main Jakarta informational page (whatever that is now).

-- Jan

On 31.01.2020 20:34, Lance Andersen wrote:
> I also agree with Bill.  There has to be thought  given
to CTS 
> structure first IMHO so that there is consistency in the test 
> structure to make it easier to pull in the standalone tests  and
>  where the platform specific tests are maintained for the standalone
> technologies.
>
> This will take some time to flush this out.  During the Java
EE days, 
> we did find it easier to work out of one master workspace, but part
of 
> that was due to the fact there was one team primarily responsible
 for 
> the test development for all of the Java EE technologies.
>
>> On Jan 31, 2020, at 2:01 PM, dmitry.kornilov@xxxxxxxxxx
>> <mailto:dmitry.kornilov@xxxxxxxxxx>
wrote:
>>
>> Bill, you are right. On the other hand the process of splitting
has 
>> to be started somehow. It will never start with an assumption
that 
>> some developer with TCK knowledge will take an initiative and
start 
>> working on it. There are not too many developers like this and
no one 
>> wants to take a risk and responsibility.
>> On the other hand I don’t think that it’s a right time to do
it now 
>> as part of Jakarta EE 9 release. I already proposed thatwe will
not 
>> move JSONB TCK tests from CTS yet. We will do it after Jakarta
EE 9 
>> is released. We will keep them in sync until that time. Users
may use 
>> TCK tests in JSONB repo as more convenient way of checking 
>> compatibility. But it cannot be used officially for compliance
>> testing for now.
>> -- Dmitry
>> *From:*jakartaee-platform-dev-bounces@xxxxxxxxxxx
>> <mailto:jakartaee-platform-dev-bounces@xxxxxxxxxxx><jakartaee-platform-dev-bounces@xxxxxxxxxxx
>> <mailto:jakartaee-platform-dev-bounces@xxxxxxxxxxx>>*On
Behalf 
>> Of*Bill Shannon
>> *Sent:*Friday, January 31, 2020 7:06 PM
>> *To:*jakartaee-platform developer discussions 
>> <jakartaee-platform-dev@xxxxxxxxxxx
>> <mailto:jakartaee-platform-dev@xxxxxxxxxxx>>;
Andy Guibert 
>> <andy.guibert@xxxxxxxxx<mailto:andy.guibert@xxxxxxxxx>>
>> *Subject:*Re: [jakartaee-platform-dev] TCK tests in the same repo
as 
>> API and Spec
>>
>> This has been discussed numerous times over the last 3 years.
 No one 
>> disagrees with the goal of splitting up the TCK repo.  Whether
the 
>> individual spec TCKs are in the same repo as the API classes or
in a 
>> different repo is just a detail.  The challenge is in splitting
up 
>> the existing TCK repo such that the resulting TCK for each spec
is 
>> functionally identical to the existing standalone TCKs for the
specs, 
>> and that the platform TCK is somehow created by combining all
the 
>> individual TCKs to produce a new platform TCK that is functionally
>> identical to the existing platform TCK.
>>
>> Needless to say, this is not a small job.
>>
>> No one with deep knowledge of the existing TCKs has come forward
with 
>> a detailed plan for how to achieve the above.  Without that,
we're 
>> all just wishing and hoping.
>>
>> And I strongly encourage you to*not*just start hacking on the
TCK to 
>> create what you want for your spec.  We need to solve the
larger 
>> problem and a bunch of uncoordinated hacks to individual TCKs
will 
>> not get us there.
>>
>> Thanks.
>>
>> P.S.  You should be able to find some of the previous history
of this 
>> discussion in the jakartaee-tck-dev mailing list.
>>
>> Andy Guibert wrote on 1/30/20 10:45 AM:
>>> Hi all,
>>> Currently all of the Jakarta EE TCK tests are housed in one
big repo 
>>> and they use a custom test framework from the Java EE days:
>>> https://github.com/eclipse-ee4j/jakartaee-tck
>>>  It is more convenient to have the TCK tests in the same
repo as the 
>>> API and spec docs because as the technologies change over
time all 3 
>>> parts (tck/api/spec) can be updated in the same PR. Additionally,
>>> implementations can then consume the TCK tests as Maven artifacts
>>> and run/verify that they pass the TCK. This is what MicroProfile
has 
>>> done and it works very well.
>>> As an example, I've started this off with the JSON-B TCK test
here:
>>> jsonb-api pr: https://github.com/eclipse-ee4j/jsonb-api/pull/221
>>> yasson (impl) pr: https://github.com/eclipse-ee4j/yasson/pull/379
>>> Just wanted to share this with the wider dev community and
encourage 
>>> other specs to follow suit as time permits.
>>> Cheers, Andy
>>> -- IBM
>>>
>>>
>>> _______________________________________________
>>> jakartaee-platform-dev mailing list
>>> jakartaee-platform-dev@xxxxxxxxxxx <mailto:jakartaee-platform-dev@xxxxxxxxxxx>
>>> To change your delivery options, retrieve your password, or
unsubscribe from this list, visit
>>> https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev
>> _______________________________________________
>> jakartaee-platform-dev mailing list
>> jakartaee-platform-dev@xxxxxxxxxxx
>> <mailto:jakartaee-platform-dev@xxxxxxxxxxx>
>> To change your delivery options, retrieve your password, or 
>> unsubscribe from this list, visit
>> https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev
>
> <http://oracle.com/us/design/oracle-email-sig-198324.gif>
> <http://oracle.com/us/design/oracle-email-sig-198324.gif><http://oracle.com/us/design/oracle-email-sig-198324.gif>
> <http://oracle.com/us/design/oracle-email-sig-198324.gif>Lance
> Andersen| Principal Member of Technical Staff | +1.781.442.2037
> Oracle Java Engineering
> 1 Network Drive
> Burlington, MA 01803
> Lance.Andersen@xxxxxxxxxx<mailto:Lance.Andersen@xxxxxxxxxx>
>
>
>
>
> _______________________________________________
> jakartaee-platform-dev mailing list
> jakartaee-platform-dev@xxxxxxxxxxx
> To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
> https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev
_______________________________________________
jakartaee-platform-dev mailing list
jakartaee-platform-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev





_______________________________________________
jakartaee-platform-dev mailing list
jakartaee-platform-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev


_______________________________________________
jakartaee-platform-dev mailing list
jakartaee-platform-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe
from this list, visit
https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev