Skip to main content

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: [cross-project-issues-dev] [WARNING] SimRel Headed Off the Tracks

Thank you Stephan for your balanced, concise, insightful commentary!

More comments below...

On 29.01.2020 23:54, Stephan Herrmann wrote:
Thanks, Ed, for raising these concerns.

The thread is already at a volume which makes it hard to digest, I can only hope I got the essence of what everybody is thinking.

Yes, the thread has frayed into multiple smaller threads, some of which stray far from the original focus.  E.g., release cadence and quality concerns.

One way I could interpret the situation is: It takes a very small number of qualified people to herd the train - full-time, and then the immediate issue is resolved.

Yes, Fredric Gurr, Ed Merks, and Markus Knauer help to ensure that the large volume of inputs result in the correct and appropriate outputs.  The simrel inputs are a large sets of p2 repositories comprising an even larger set of installable units. Fred's work ensures that these inputs map to a simple p2 repository (the train repo) with a correct, coherent, consistent set of installable units, organized into categories.  Markus Knauer's work ensures that the EPP inputs (the package/product definitions), in combination with the train repo, map to a simple p2 repository (the epp repo) as well as a set of canned packages (zip/tar/dmg).  These two simple repos are composed to give us the overall simrel repo.  Ed Merks ensures that the train repo in combination with the epp repo maps to a set of Product Versions in Oomph's Product Catalog, and delivers a set of installers (*.exe/tar/dmg) that is based on the latest installable units from the train repo.  These installers can be used to install the same "packages" as are delivered as canned packages (zip/tar/dmg).

Fred is funded by the Foundation, which stepped in when David Williams funding was cut by IBM.  Ed Merks was funded by itemis, but that has been cut.  Markus Knauer is funded by Eclipse Source, but his daily work is unrelated and so he has stepped back because essentially his work is unfunded.  In the past, both itemis and Eclipse Source were Strategic Developer Members, but each has stepped back from that long ago.  At this point the Foundation is not able to step in because of its own budget constraints and is also not willing to step in because it's a slippery slope to step in purely using existing budget whenever some part of the community fails.  Of course the failure in this case is cross project and broadly sweeping...

There are member companies for whom it should not be a problem to fund the one appointed train master - perhaps via the foundation. I'd love to see Ed's efforts to be compensated in a way which would allow him to give us a long term perspective. I don't represent a member company so I can't comment on whether and how this simple idea could be put to action.

The proposed solution is a Working Group with membership fees as the funding model for this process.  Of course for that to work, some member companies need to get step up and that boils down to the question: why should I (as a member company) get involved (pay) when I can just wait for and hope that some other member company will pay the cost instead?  It feels like a game of chicken played by multi-billion revenue companies, where the amount of money actually required is as a molecule of water.   I remember all too well the days when the Platform was funded almost exclusively by IBM: that game of chicken lasted for years.  Thank goodness Red Hat joined the party and thank the mighty techo deities for Tycho.

I assume all of us subscribed to cross-projects have a shared interest in providing to our users plenty of options of which software they want to install, in various combinations, and all in the latest and greatest version. Without bad surprises on their way.

Yes and no.  Herein we already see a fundamental split in the community.  There is the "we don't care about the train repo" camp, and also the "we don't care about the packages" camp...

Please, those who want to do away with SimRel: tell me, what efforts are incurred by SimRel that or not essential for this original goal? If SimRel incurs non-essential efforts, produces accidental complexity, shouldn't we just name these, so we can fix things (process? automation?).

In my opinion, aside from the question how exactly the p2 repo/installable-unit inputs are managed/represented, the train repo is and must remain one of the primary goals and outputs of the overall process.  Many of us care deeply about this and some of us only care about this.  Producing this is Fred's role and of course it's already primarily automated. In fact, so automated that we have a daily staging train repo.  Only Fred can comment on how much overall work is involved in this aspect.  I can only comment that we could dramatically improve the testing of quality metrics and streamline the process of problem tracking/reporting. Of course there are arguments in favor of reducing the set of participating projects, but none of that (including representation changes) will change the nature of the work involved.   Moreover, I don't think this is the primary burden that we need to try to reduce or eliminate.

For those who argue to reduce this to just the things needed by EPP package (which is likely those people whose inputs are already in EPP packages), just consider that I personally don't want to participate in a process of producing EPP packages if there is no train repo that contains all the things that I personally want in the train repo.  I expect Stephan has similar concerns about his Object Teams contribution; it's not in any package so is easily caught a broad sweep of irrelevance.  Please don't do that.

The underlying point here is that every point of view is valid and while your personal mileage from using the train repo may vary, please consider carefully the mileage value it has for the community as a whole: if it has value for someone, it has value.

(Disclaimer: I don't care about packages because a simrel repo + various oomph setups are all I need to get the Eclipses I want, or to push the same to colleagues in the company). (Disclaimer 2: I don't hate the cbi aggregator, I prefer to fix it when there's a problem).

The "we don't care about the train" camp are also arguing that we could represent the EPP's inputs differently.   Instead of representing it via the aggregator model (in XML), we could represent it as a target definition model (in XML) and a site definition model (in XML).  Of course this is factually true, but to me it's all just so much XML.  Furthermore,  I find this deeply technical level of discussion about representations pointless, because the fundamental point is that someone must manage a representation and a large set of committers must edit a representation.  And in any case, at this point, still no one has stepped up to do anything.  Changing the representation introduces a new set of problems whereas the existing representation is working well and is fully automated already.  So any argument to change the representation at this point is an argument to create more new work with a new different set of problems, while diverging from the issue at hand, i.e.,  even if someone runs off and prototypes how this could be changed (of course it's possible in principle!), there will remain the problem of who will actually deal with all this for the next 10 years?

For a thought experiment wrt the release cadence: is it possible our cadence is still too slow? Imagine a truly continuous SimRel.
Yes, in my wildest fantasy we would have the full slate of daily staging outputs.  There is no technical reason I can think of why this isn't feasible.
Perhaps, SimRel contributions should happen *always*, i.e., whenever a project has any code changes, or buffered to once per day. Some contributions will break downstream projects, which have the option to react, or drop out after 3 build failures. If your dependency drops out, you will notice immediately. Some version bumps, dependency changes will need coordination, here on cross-projects, but they will just happen at their own timing, not driven by release dates. In a world where so many tasks can be automated, isn't the continues flow of all seeing each others HEAD the simplest model of all?

This is exactly what I imagine.   We can make aggregation fail when licenses are bad or unsigned content is contributed, but we can only do that once we've automated that and are in a valid state just once; we're pretty darned close right now.  We can and should add more automated integration tests.  If we all felt comfortable about the quality of this continue integration's output we could all use it as the basis for our daily development environment.  Each of us could be testing it continuously every day by actually using it every day.  We'd notice problems much faster and that will most definitely improve quality.

<digression>In fact I have a Platform SDK workspace with 26 Git repositories that comprise the Platform SDK.  I regularly run the setup so that it updates the installation and TP to the latest IBuild and then I pull all the Git repos. I see exactly what will be (or soon will be) in the next IBuild. I notice problems before anyone else, before there even is a build, and I can easily help fix said problems, e.g., Bug 558313.  I.e., I literally see HEAD of everything...</digression>

In this model, SimRel (or should we call it SimFlow) would concentrate on ensuring that our components stay on track wrt compatibility with each other. The other concern about more regressions due to shorter cycles and no maintenance branch would still need to be discussed but that's probably a different discussion. That discussion could well suggest a much slower cadence, but perhaps that is even possibly *on top* of a continues SimFlow.
Yes, these other discussions (cadence and quality) are really important as well, but you hit the nail on the head with respect to what I want to see this process become.  Continuous, with appropriate checks and balances in place, along with automated reporting, augmented with a growing set of automated integration tests to eliminate the onerous +1 process of EPP.  And then we each eat our own dog food every day, savoring it as the world's finest delicacy, ready to be served to the masses.

Just brain storming ...

Indeed, collectively we've called everything into question and have headed down different threads.  That is goodness.  As such, while the "we don't care about the train" camp brainstorms down one thread, let me brainstorm the converse on behalf of the "we don't care about the packages" camp.  We could most certainly change the representation of the EPP packages and still produce the same outputs.   We can see how this is factually true just by looking at how Oomph's setup model represents a package.  Each package is essentially just a Product Version and that essentially boils down to a p2 task that installs a set of installable units from some p2 repositories.  We could represent package versions primarily in that form rather than generating this form from the epp repo.  Then we could "just eliminate EPP" while still producing all the same outputs.  With this approach, we could streamline things like JDT's Java beta support so that the moment a new version of Java is released, all the packages (at least those installed via the installer as Product Versions) simply contain the latest Java support.

A great many improvements are possible.  We are only limited by resource, not by good ideas.


cross-project-issues-dev mailing list
To change your delivery options, retrieve your password, or unsubscribe from this list, visit

Back to the top