[dtp-dev] New PDE Based Build System for DTP

[rcernich@xxxxxxxxxx]

Hey all,

I've recently been working on integrating the PDE based builder with DTP.
The basic builder design is derived from the build system used by WTP and
STP (and probably others).  The work is currently checked into CVS under
the following path: org.eclipse.datatools.build/pde_build.  Although this
is a work in progress, the system successfully builds all the plugins, with
the exception of some of the ODA plugins.

The goals behind this effort were:
1.  Use a standardized build system.  The new system is similar to systems
used by other Eclipse projects.
2.  Build consistency.  The new system generates a build similar to that
generated by the IDE.  This means any changes to plugin dependencies are
picked up automatically, without having to hand edit a build file.  (Well,
not entirely.  If the dependency is on a new component (e.g. GMF), that
component must be added to the dependencies.properties file, located with
the map files.)
3.  Use standard PDE build mechanism.  The new system delegates to the PDE
builder supplied by Eclipse.  Build scripts are generated from feature and
plugin manifests.  This helps guarantee that the build environment mimics
the runtime environment.
4.  Automated generation of source code plugins.  This removes the need to
manage customized scripts for collecting source code for our SDK.  Any
plugin added to an existing feature will have its source automatically
added to the source plugin.
5.  Automated generation of download site files.  The new system generates
files that can be uploaded directly to the DTP download site.  This allows
DTP to publish information concerning test results, compiler
errors/warnings, build settings, etc.
6.  Automated test execution.  Although this is not currently implemented,
the hooks exist for executing DTP's unit tests.

I will be adding documentation to the wiki page, but here are the basics
regarding usage.
There are two main projects:
org.eclipse.datatools.releng - This project contains the "map" files used
with the org.eclipse.releng.tools plugin.  These files are used by the PDE
builder to locate the project files within CVS.  While these files support
explicit specification of build tags for each project, currently the tag
specified for all DTP projects is HEAD.

org.eclipse.datatools.releng.builder - This project contains all the build
scripts used to run the build.  The breakdown of the project is as follows:
build.xml - the main build file for the project.  Builds all the
components.

components - contains the build files used for each component build.  The
subdirectories can be set as the buildDirectory used by the PDE builder
(i.e. you should be able to invoke the PDE builder directly on each of
these directories).  If you are adding a component you should create a new
directory by copying the template directory.  See the Eclipse help for more
information on the PDE builder.  (Note, these scripts are slightly
different in that the allElements.xml tasks are incorporated into the
customTargets.xml file.  Furthermore, the
scripts/build/defaultCustomTargets.xml file contains most of the logic
needed for invoking the build such as downloading dependencies (e.g. the
Eclipse platform).)

cruise - contains the main build script and cruise configuration.  This
file downloads the DTP releng plugins and initializes most of the
properties required to execute the main build file.

distribution - contains scripts which direct the build of a particular
distribution and type (e.g. dtp.build).  Each subdirectory contains the
build files for a particular build type (build, site, tests, upload) for a
particular distribution (dtp).  If you are adding a new component, you will
need to modify the dtp.build/build.xml so your component will be built.
You will need to modify the files in the other directories appropriately
(e.g. modify the html templates to make a place for the new component).

scripts - contain utility scripts.  For example, build/build.xml contains
tasks for downloading the PDE builder and invoking it for a particular
component; the dependency directory contains scripts for downloading and
installing dependencies.

The features used to drive the builds are checked into the
org.eclipse.datatools.build/pde_build/features directory.

Still to do:
Update ODA plugins so they can be built with the PDE builder (e.g.
bin.includes).
Integrate generation of doc.isv plugin.
Add user doc plugins to their respective features (and make sure they get
picked up by the builder).
Get unit tests running as part of the build process (look at adding code
coverage statistics as well).
Fix the "upload" target (it doesn't work; not sure what credentials and
file paths it should be using).
Make sure all the relevant build properties are passed when invoking the
PDE builder (this is invoked through Java, not through Ant, so we need to
manually pass in the relevant properties as vm args).
The notify scripts need to be updated with proper email addresses.

To execute a build:
Invoke org.eclipse.datatools.releng.builder/cruise/build.xml from Ant (you
do not need to execute from within Eclipse).  You should specify the
following properties: (via -D arguments on the command line or you may edit
them in the build.properties file that is collocated with the build.xml
file)
basearch - e.g. x86
baseos - e.g. win32
basews - e.g. win32
build.clean - set to false if you don't want to spend time downloading the
dependencies.
build.home - e.g. c:\dtp\build.  This is the working directory for the
build (where the files are checked out, the build output is placed, etc.).
build.trial - set to true for testing.  (I'm not entirely sure what this
does.)
env.J2SE14 - set to the classpath for your build dependencies (e.g.
c:\j2sdk1.4.2\jre\lib\rt.jar;c:\j2sdk1.4.2\jre\lib\jsse.jar;c:\j2sdk1.4.2
\jre\lib\jce.jar).  Note, you need to specify rt.jar, jsse.jar and jce.jar.
env.J2SE15 - same as above, but for Java 5.
lpgJavaRuntimeJarFileLocation - the location of lpgjavaruntime.jar (this is
copied in just prior to compilation and deleted just before packaging).
Required for compilation of sql tools.

Let me know if you have any comments, questions, concerns, etc.

Best regards,
Rob Cernich
DTP Connectivity Project Lead