[wtp-dev] Successful CI requires breaking deps and reorgs, discussion re

[Robert Stryker <stryker@xxxxxxxxxx>]

Hey everyone:

So I've been working on getting that dependency graph stuff going, and I haven't been very successful because of the cycles ;) I have some raw data, which is excluding (for the most part) test plugins, but a few test plugins snuck their way into the report because they're in weird folders like "development" instead of "tests". So just gloss over those where possible.

The purpose of this missive is to try to:

 1)  get a properly ordered project with CI with Jenkins, fewer jobs, fewer repos, etc, wherever possible;

 2) To identify what exact coding changes (other than repo merges) will be necessary;

 3) To get +1's by current component leads to do the above, ie, express a willingness to get things done. Without a willingness to make some changes, we might be stuck with an old and convoluted structure and alienate the new releng guy ;)  (ie Nick Boldt)

Full gist with raw data is here:  https://gist.github.com/robstryker/b584195b5067b0909dc2b57cbcc9ef8f

But, I can provide a summary.

WTP can be thought of as basically 5 tiers:

  1) Common/server at the bottom

  2) jsdt / source-editing

  3) javaee stuff  (javaee, ejb, webservices)

  3a/3b) webservices.axis2, webservices.jaxws, and jsf

  4) Dali at the top

Tier 1:

Common/Server is a bit messy at the moment. Common depends on server currently because it exposes an extension point that uses wst.server's IRuntime. JavaEE uses that extension point.  If we refactor that extension point, it could use the facet IRuntime instead and break the circular dependency.

QUESTION 1)
IS SUCH A CHANGE APPROPRIATE AT THIS TIME? If it's not appropriate, or cannot be approved, then we might have real issues in breaking these circular dependencies between repos until the next major release.

Once that's broken (there are gerrit pushes for that change), there's still one more issue: org.eclipse.jst.common.ui  (in common) depends on org.eclipse.jst.common.frameworks (in javaee).

One of these plugins needs to move, either up or down. Luckily the only things between javaee and common are jsdt/sourceediting and server. The data doesn't show either would be affected by either a move up or a move down.

QUESTION 2: CAN ONE OF THESE PLUGINS BE MOVED AT THIS TIME?

Servertools would look much nicer if it was 1 repository.

QUESTION 3: Will the servertools lead consent to a merge of their repos?

Tier 2:  jsdt / source-editing.  These two have circular dependencies among themselves. It'd be great if these 2 projects could figure out a proper heirarchy, or, alternately, if they'd agree to be merged into one repo ;)

QUESTION 4: Are the jsdt / source-editing repos able to break their circular dependencies? If no, are they willing to be merged into one repo?

Tier 3+:  JavaEE / EJB / Webservices:   These guys need a bit more investigation. We could theoretically merge ALL of it into one repo, which would be very convenient but I'm not sure it's 100% appropriate. Alternately, we could deep-dive to see if or how the dependencies could be broken. I haven't had time to deep-dive on a plugin-by-plugin level yet.

Tier 4: Dali:  Dali is fine. Probably doesn't need any changes.

All of the above is regarding primarily the plugins themselves, NOT any test bundles. Nick and I have been working at a pom structure that would allow builds of your plugins (ie mvn clean install)  to occur against only your required dependencies (and fail if you depend on something that is not on your tier or lower), and a different profile to use for integration tests. 

This is required because,  surprisingly, almost all unit tests are actually integration tests with code much further up or further down the stack as compared to where the tests live.

So basically, the 4 questions are, when can we begin making changes (even to API if necessary or moving plugins between repos) to Tier 1, is servertools willing to merge their repos, and are jsdt/sourceediting capable of breaking deps or do they consent to a repo merge?

Thanks all

- Rob Stryker