Re: [ecf-dev] ECF build v2 -- Build easier

[Thomas Hallgren <thomas@xxxxxxx>]

Hi ECF team,
It became a fairly long answer to a short question.

Markus Alexander Kuppe wrote:
> 4) Is Buckminster going to be replaced by p2?
>
> During the con call the question came up if Buckminster's functionality
> of resolving/materializing build dependencies is going to be replaced by
> p2 in which case we might not wanna invest on Buckminster. I will ask
> the Buckminster team to respond to this question.
>
>   
The very short answer is yes, to some extent, but P2 does not address 
source repositories or the build aspect.

The longer answer is that much of this has already happened. Buckminster 
leverages P2 to it's full extent and is capable of reading P2 update 
sites, materialize into a target platform definition of type "Software 
Site", publish to a P2 repository, and also upgrade itself using P2 (the 
headless install command).

So what has been replaced in 3.5?

1. Buckminster Version + VersionType is now P2 Omni Version
We have been working very closely with the P2 team. Henrik Lindberg (co 
lead of the Buckminster project) and I desgined the new Omni Version and 
my implementation of it has replaced the old OSGi Version and 
VersionRange in the P2 code base. This enables the P2 SAT4J resolver to 
go beyond OSGi and to store non OSGi IU's and artifacts in P2 
repositories. The main reason for us to do this was that we didnt' want 
to lock Buckminster to the OSGi versioning scheme while moving over to 
P2 (Henrik also made some major refactorings to the P2 code base to make 
it more consistent with respect to error handling. He is now a P2 
committer. A fact that of course helps us in our continued effort to 
integrate).

2. Target Platform Management
Target platforms can now be setup by importing a PDE Target Platform 
Definition (an XML file with .target suffix). Such a definition may well 
contain a "Software Site" which in essence is a P2 provisioned target 
platform. A number of headless commands was added to Buckminster to 
accommodate this.

Buckminsters "eclipse.import" reader is now based on P2 and a new 
materializer named "p2" has replaced the old "targetPlatform" 
materializer. Used in combination they can set up a target platform from 
a number of repositories. The P2 engine is of course in charge of 
provisioning.

3. Buckminster installer
Is just a very thin layer on top of the P2 director. The reason we 
retain this layer is for backward compatibility. I submitted a new 
director application to P2 and it can perform all the tasks that the 
installer can. Incidentally, we provide a headless packaging of the P2 
director on our download page. It's needed when configuring a headless 
Buckminster.

In addition to these replacements, Buckminster has also started the 
development of a repository aggregator. It is intended as the successor 
of the "Galileo Builder" and is capable of slicing and dicing from 
different sources in order to create a new local repository. It's all 
based on P2 of course and if you want to take a look, go here: 
http://wiki.eclipse.org/Getting_Started_With_Aggregator_(Buckminster)

What cannot be replaced?

P2 is designed to deploy runtimes and binary target platforms. The SAT4J 
resolver is only used when doing the former. When a target platform is 
provisioned, P2 uses a "Permissive Slicer". It performs basically the 
same job as the SAT4J resolver but it is more agile (forgiving) and can 
bypass undesired constraints. For instance, it's very common to use a 
target platform that is set up to handle any combination of os/ws/arch. 
The P2 SAT4J based installer is not capable of installing such a thing 
so P2 provides an extra tool named "repo2runnable" that can convert an 
artifact repository into a target platform.

The problem with both the SAT4J resolver and the Slicer is that both 
must have access to all potential candidates in order to do their job. 
In contrast to this, Buckminsters resolver only needs a top component. 
It performs its resolution transitively in a step wise manner without 
knowing all candidates from start. I expect to see improvements on how 
P2 uses the SAT4J resolver so that its usage can be broadened and cover 
this case too.

What about P2 and source code repositories? We have given this a lot of 
though and concluded that using P2 to "install" source will not be easy 
(if even feasible). Source is moving parts and conceptually, the IU 
(Installable Unit, the foundation for everything in p2) is immutable. 
The source of a feature and/or bundle often has incomplete meta-data. 
You see versions like 0.0.0 in a feature (waiting to be replaced during 
a build) and versions like 1.2.3.qualifier in the bundles (the qualifier 
part is replaced during builds). Even if we could do a half decent job 
of creating meta-data from a set of sources, the task of maintaining it 
would not be trivial.

Buckminsters dependency resolver is designed to cope with sources and 
moving parts. It creates its meta-data from the source, on the fly, as 
needed. So dependency resolution will continue to be performed by 
Buckminster as we move forward.

The Buckminster CSPEC in relation to a P2 Installable Unit:
Are they the same? In essence, the answer is yes. Going forward, I see 
the CSPEC become an extension to the IU. In addition to what's covered 
by the IU (runtime provisioning), the CSPEC also covers build 
orchestration through its concept of actions, groups, and artifacts. The 
PDE build team has expressed a lot of interest in a common build 
infrastructure where everything is modeled and if all goes according to 
plan, we'll work together with them to create a unified build model for 
Eclipse. When that happens, our current CSPEC will eventually become 
obsolete.

Other Buckminster artifacts:
P2 has no concept of materializing from different types of sources 
(Buckminster can use Maven repositories, PDE style releng map files, 
CVS, SVN, or Perforce repositories, even web pages with links scanned 
for patterns, etc.). I don't see P2 moving in a direction where it will 
address any of that. So the concept of a CQUERY that controls that kind 
of advanced resolution in combination with an MSPEC that specifies where 
things should go are also things that will not be replaced by P2.

An extensive effort has gone into writing Buckminster documentation 
(thanks to Henrik Lindberg) so you can probably get a lot of answers 
there. You'll find it here:

http://www.eclipse.org/downloads/download.php?file=/tools/buckminster/doc/BuckyBook.pdf

Kind Regards,
Thomas Hallgren