Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [jakartaee-platform-dev] TCK tests in the same repo as API and Spec


+1 Kevin! I think the TCK in Jakarta raises the bar higher for people to run TCK freely. As a contributor, I would like to be able to add TCKs whenever I propose API/Spec changes. Otherwise, we will run out of sync in no time. Therefore, I support what Andy proposed to split TCK and rehome them to be with where it should belong. I speak directly from my own experience of working on MicroProfile. It is great to add tests together with spec or api changes to demonstrate the full piece.

This is exactly what I miss on Microprofile, the tests that are mandatory if and only if two of the specs should work together (i.e. not optional tests). Those were known as the CTS only tests, i.e. were not part of the standalone TCKs (For instance CDI & JAX-RS, CDI & WebSocket, BeanValidation & JAX-RS). Standalone  TCKs are what the Microprofile has, it does not have the full platform test. If the tests should still be part of the platform TCK, and the tests are in the Spec repository, how to make sure the tests are run for full platform and not for the standalone TCK?

Using the approach Andy used for JSON-B works fine. What the approach does not have, and what Microprofile does not have, is the deployment. Jakarta EE TCK has the concept of deployment (not necessarily JSR-88), and the deployment behaviour is described in various Specs, such as Servlet of WebSockets. How the concept of the deployment would work with Andy's approach is not clear. Should the deployment be tested in the future? Should it be left on the Spec to define the deployment requirements/API or should it more common to every Spec? Currently, the TCK framework does that and vendors implement the porting package that takes care of the deployment. Every TCK uses the same mechanism. But with having a different test framework in each TCK, the deployment needs to be thought through.

As for releasing TCKs separately:
1. When releasing a new version of a spec, we release API/Spec/TCK and they have either major or minor version changes. They share the same version. With the same version, we know which version of TCK we should execute for the version of Spec you choose. 2. If we found any bugs in TCK, we can do a patch release for the project, which produces a micro/patch version release of API/Spec/TCK so that runtimes can execute the new TCK while end users are not affected. The above fits in well with the general understanding of major/minor/micro release.

It is great that you think about standalone TCK. However, that is the easy part. The hard part is to think about the full platform. The approach Andy described is to JUnitize the tests. The other approach will use some different framework. It is fairly fine to have separate TCKs like this as long as the Full platform TCK is no longer a requirement. Before the split, it needs to be known whether the full platform TCK is what is required, or whether the set of standalone TCKs, such as in Microprofile, is enough. If the full platform TCK is needed for verifying the compatible implementation, the separation needs to be done with the idea of putting all parts together. Otherwise, each Spec will take their part, convert it to use some fancy test framework and say: "Hey, its not my business to make the full platform TCK possible. Someone else should take care".

Even the Microprofile has a unified Arquillian based test environment. Using JUnit in one TCK, Arquillian in other, and JavaTest in the next brings a challenge for the team that ought to run the TCKs. Are we cool with having it different in every TCK?

On Mon, Feb 3, 2020 at 1:40 PM Kevin Sutter <sutter@xxxxxxxxxx <mailto:sutter@xxxxxxxxxx>> wrote:

    Jan,
    You bring up some excellent points.  Having a well thought out
    plan before jumping in with both feet would be nice.  But, on the
    other hand, we don't want to stifle the excited effort that Andy
    is demonstrating. It's a tough line.  Maybe we need to allow one
    or two teams to experiment with the processes and see what it
    takes.  And, then we can take those experiences and develop a plan
    that works across all the projects.

For the experiment, grabbing JSON-B, or JSON-P is easy. Those are basically standalone only technologies. A Servlet based technology seems more appropriate for experimenting.


    ---------------------------------------------------
    Kevin Sutter
    STSM, MicroProfile and Jakarta EE architect @ IBM
    e-mail: sutter@xxxxxxxxxx <mailto:sutter@xxxxxxxxxx>     Twitter:
     @kwsutter
    phone: tl-553-3620 (office), 507-253-3620 (office)
    LinkedIn: https://www.linkedin.com/in/kevinwsutter



    From: Jan Supol <jan.supol@xxxxxxxxxx <mailto:jan.supol@xxxxxxxxxx>>
    To: jakartaee-platform-dev@xxxxxxxxxxx
    <mailto:jakartaee-platform-dev@xxxxxxxxxxx>
    Date: 01/31/2020 15:02
    Subject: [EXTERNAL] Re: [jakartaee-platform-dev] TCK tests in the
    same repo as API and Spec
    Sent by: jakartaee-platform-dev-bounces@xxxxxxxxxxx
    <mailto:jakartaee-platform-dev-bounces@xxxxxxxxxxx>
    ------------------------------------------------------------------------



    The discussion on the various email lists is quite extensive, and I
    admit I do not follow every thread that is discussed. But I am
    waiting
    for the plan from any of the committee we have since the
    transition to
    Eclipse Foundation. And I am not aware about any clear statement
    about
    what the final goal is. For start:

    - Is the splitting really what is recommended?

    - Should the Platform Jakarta EE TCK still be available in the future
    after the split?

    - Should the interoperability between Specs tests (the tests that
    require multiple Specifications to cooperate as it is described in
    one
    or the other specification, the tests that used to be part of the
    full
    CTS only) still be part of the combined (?) Jakarta EE TCK?

    - Is there a recommended framework to be used in the future with the
    TCKs (such as the existing JavaTest + perhaps Arquillian similarly to
    Microprofile + ?)?

    Those should be goals presented by the Jakarta representatives before
    the TCK repo is started to be ripped apart. Not on email, but
    clearly on
    the main Jakarta informational page (whatever that is now).

    -- Jan

    On 31.01.2020 20:34, Lance Andersen wrote:
    > I also agree with Bill.  There has to be thought  given to CTS
    > structure first IMHO so that there is consistency in the test
    > structure to make it easier to pull in the standalone tests  and
    >  where the platform specific tests are maintained for the
    standalone
    > technologies.
    >
    > This will take some time to flush this out.  During the Java EE
    days,
    > we did find it easier to work out of one master workspace, but
    part of
    > that was due to the fact there was one team primarily
    responsible  for
    > the test development for all of the Java EE technologies.
    >
    >> On Jan 31, 2020, at 2:01 PM, dmitry.kornilov@xxxxxxxxxx
    <mailto:dmitry.kornilov@xxxxxxxxxx>
    >> <mailto:dmitry.kornilov@xxxxxxxxxx> wrote:
    >>
    >> Bill, you are right. On the other hand the process of splitting
    has
    >> to be started somehow. It will never start with an assumption that
    >> some developer with TCK knowledge will take an initiative and
    start
    >> working on it. There are not too many developers like this and
    no one
    >> wants to take a risk and responsibility.
    >> On the other hand I don’t think that it’s a right time to do it
    now
    >> as part of Jakarta EE 9 release. I already proposed thatwe will
    not
    >> move JSONB TCK tests from CTS yet. We will do it after Jakarta
    EE 9
    >> is released. We will keep them in sync until that time. Users
    may use
    >> TCK tests in JSONB repo as more convenient way of checking
    >> compatibility. But it cannot be used officially for compliance
    >> testing for now.
    >> -- Dmitry
    >> *From:*jakartaee-platform-dev-bounces@xxxxxxxxxxx
    <mailto:jakartaee-platform-dev-bounces@xxxxxxxxxxx>
    >>
    <mailto:jakartaee-platform-dev-bounces@xxxxxxxxxxx><jakartaee-platform-dev-bounces@xxxxxxxxxxx
    <mailto:jakartaee-platform-dev-bounces@xxxxxxxxxxx>
    >> <mailto:jakartaee-platform-dev-bounces@xxxxxxxxxxx>>*On Behalf
    >> Of*Bill Shannon
    >> *Sent:*Friday, January 31, 2020 7:06 PM
    >> *To:*jakartaee-platform developer discussions
    >> <jakartaee-platform-dev@xxxxxxxxxxx
    <mailto:jakartaee-platform-dev@xxxxxxxxxxx>
    >> <mailto:jakartaee-platform-dev@xxxxxxxxxxx>>; Andy Guibert
    >> <andy.guibert@xxxxxxxxx <mailto:andy.guibert@xxxxxxxxx>
    <mailto:andy.guibert@xxxxxxxxx>>
    >> *Subject:*Re: [jakartaee-platform-dev] TCK tests in the same
    repo as
    >> API and Spec
    >>
    >> This has been discussed numerous times over the last 3 years.
    No one
    >> disagrees with the goal of splitting up the TCK repo.  Whether the
    >> individual spec TCKs are in the same repo as the API classes or
    in a
    >> different repo is just a detail.  The challenge is in splitting up
    >> the existing TCK repo such that the resulting TCK for each spec is
    >> functionally identical to the existing standalone TCKs for the
    specs,
    >> and that the platform TCK is somehow created by combining all the
    >> individual TCKs to produce a new platform TCK that is functionally
    >> identical to the existing platform TCK.
    >>
    >> Needless to say, this is not a small job.
    >>
    >> No one with deep knowledge of the existing TCKs has come
    forward with
    >> a detailed plan for how to achieve the above. Without that, we're
    >> all just wishing and hoping.
    >>
    >> And I strongly encourage you to*not*just start hacking on the
    TCK to
    >> create what you want for your spec.  We need to solve the larger
    >> problem and a bunch of uncoordinated hacks to individual TCKs will
    >> not get us there.
    >>
    >> Thanks.
    >>
    >> P.S.  You should be able to find some of the previous history
    of this
    >> discussion in the jakartaee-tck-dev mailing list.
    >>
    >> Andy Guibert wrote on 1/30/20 10:45 AM:
    >>> Hi all,
    >>> Currently all of the Jakarta EE TCK tests are housed in one
    big repo
    >>> and they use a custom test framework from the Java EE days:
    >>> https://github.com/eclipse-ee4j/jakartaee-tck
    >>>  It is more convenient to have the TCK tests in the same repo
    as the
    >>> API and spec docs because as the technologies change over time
    all 3
    >>> parts (tck/api/spec) can be updated in the same PR. Additionally,
    >>> implementations can then consume the TCK tests as Maven artifacts
    >>> and run/verify that they pass the TCK. This is what
    MicroProfile has
    >>> done and it works very well.
    >>> As an example, I've started this off with the JSON-B TCK test
    here:
    >>> jsonb-api pr: https://github.com/eclipse-ee4j/jsonb-api/pull/221
    >>> yasson (impl) pr: https://github.com/eclipse-ee4j/yasson/pull/379
    >>> Just wanted to share this with the wider dev community and
    encourage
    >>> other specs to follow suit as time permits.
    >>> Cheers, Andy
    >>> -- IBM
    >>>
    >>>
    >>> _______________________________________________
    >>> jakartaee-platform-dev mailing list
    >>> jakartaee-platform-dev@xxxxxxxxxxx
    <mailto:jakartaee-platform-dev@xxxxxxxxxxx>
     <mailto:jakartaee-platform-dev@xxxxxxxxxxx>
    >>> To change your delivery options, retrieve your password, or
    unsubscribe from this list, visit
    >>> https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev
    >> _______________________________________________
    >> jakartaee-platform-dev mailing list
    >> jakartaee-platform-dev@xxxxxxxxxxx
    <mailto:jakartaee-platform-dev@xxxxxxxxxxx>
    >> <mailto:jakartaee-platform-dev@xxxxxxxxxxx>
    >> To change your delivery options, retrieve your password, or
    >> unsubscribe from this list, visit
    >> https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev
    >
    > <http://oracle.com/us/design/oracle-email-sig-198324.gif>
    >
    <http://oracle.com/us/design/oracle-email-sig-198324.gif><http://oracle.com/us/design/oracle-email-sig-198324.gif>
    > <http://oracle.com/us/design/oracle-email-sig-198324.gif>Lance
    > Andersen| Principal Member of Technical Staff | +1.781.442.2037
    > Oracle Java Engineering
    > 1 Network Drive
    > Burlington, MA 01803
    > Lance.Andersen@xxxxxxxxxx <mailto:Lance.Andersen@xxxxxxxxxx>
    <mailto:Lance.Andersen@xxxxxxxxxx>
    >
    >
    >
    >
    > _______________________________________________
    > jakartaee-platform-dev mailing list
    > jakartaee-platform-dev@xxxxxxxxxxx
    <mailto:jakartaee-platform-dev@xxxxxxxxxxx>
    > To change your delivery options, retrieve your password, or
    unsubscribe from this list, visit
    > https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev
    _______________________________________________
    jakartaee-platform-dev mailing list
    jakartaee-platform-dev@xxxxxxxxxxx
    <mailto:jakartaee-platform-dev@xxxxxxxxxxx>
    To change your delivery options, retrieve your password, or
    unsubscribe from this list, visit
    https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev




    _______________________________________________
    jakartaee-platform-dev mailing list
    jakartaee-platform-dev@xxxxxxxxxxx
    <mailto:jakartaee-platform-dev@xxxxxxxxxxx>
    To change your delivery options, retrieve your password, or
    unsubscribe from this list, visit
    https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev



--
Thanks
Emily


_______________________________________________
jakartaee-platform-dev mailing list
jakartaee-platform-dev@xxxxxxxxxxx
To change your delivery options, retrieve your password, or unsubscribe from this list, visit
https://www.eclipse.org/mailman/listinfo/jakartaee-platform-dev


Back to the top