Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Buckminster dev » Hudson plugin and provisioning
Hudson plugin and provisioning [message #520797] Mon, 15 March 2010 07:26 Go to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
Hi Johannes,
I have a question regarding the Hudson Buckminster plugin. How hard would it be to make it capable of provisioning the
Buckminster runtime before it executes the bucky script? I.e. have it start the director and install/update features
listed in the config?

FYI, http://wiki.hudson-ci.org/display/HUDSON/Buckminster+PlugIn doesn't seem to be available right now.

- thomas
Re: Hudson plugin and provisioning [message #520848 is a reply to message #520797] Mon, 15 March 2010 10:16 Go to previous messageGo to next message
Johannes Utzig is currently offline Johannes UtzigFriend
Messages: 329
Registered: July 2009
Senior Member
Hi,

sorry, I check my inbox before I checked the newsgroup, so I sent my
answer directly to your mail address instead of to the newsgroup...

So here's a copy of the mail I sent you earlier for everyone to read:

Hi Thomas,

this is actually something I was already thinking about myself.
Hudson has a thing called a 'Tool Installation' which is a generic
description of any kind of build tool that a build can depend on.
That includes things like ANT, Maven, JDK,...
For some Tools there is an automatic tool installation available which
usually means download something from somewhere and install it.
Having the same thing for Buckminster would have several advantages:
-Easier to configure for the user
-Easy to update/upgrade buckminster
-And most of all, this would enable master/slave scenarios. If a build
is executed on a slave machine and relies on a tool that has an
automatic installation available, it can check if this tool is already
installed and install it on the fly if necessary. This is a great
feature if you have something like a build cloud where new slaves get
added on demand.

So in my opinion it would be better to participate in the logic provided
by hudson instead of doing it plugin-internal during the build run. What
is your opinion on that?

As for the 'how', does this sound ok to you? :
In 'Configure Hudson' you have 3 options:
1. enter a location to a buckminster installation (that's how it works
currently)
2. pick a version and install a default buckminster automatically
(pretty much all the features on the headless update site I would say)
3. provide a list of update sites and features to be installed

If 2. or 3. gets picked and hudson detects a missing tool installation,
the plugin would download the director application
( http://www.eclipse.org/downloads/download.php?file=/tools/bu ckminster/products/director_latest.zip)
and then provision a buckminster runtime with the defined set of
features. I'm not sure yet how an update would work with such a Tool
Installation but I'm sure I can find out about that...

About the wiki:
yes, the complete hudson wiki seems to be down at the moment. Their
infrastructure (SVN, wiki, ...) is a bit fragile in times... I'm sure it
will be on again in a few hours or so.

Thanks for your input on this, I definetly do agree that this would make
things a lot easier for the user.
As for the question how hard it would be, I don't know yet (the
documentation on automatic tool installation is unfortunately very sparse)
but I can give it a shot in a few days and hopefully tell you more about it.


Best regards,
Johannes
Re: Hudson plugin and provisioning [message #520862 is a reply to message #520848] Mon, 15 March 2010 10:48 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 03/15/2010 04:09 PM, Johannes Utzig wrote:
> Hi,
>
> sorry, I check my inbox before I checked the newsgroup, so I sent my
> answer directly to your mail address instead of to the newsgroup...
>
> So here's a copy of the mail I sent you earlier for everyone to read:
>
> Hi Thomas,
>
> this is actually something I was already thinking about myself.
> Hudson has a thing called a 'Tool Installation' which is a generic
> description of any kind of build tool that a build can depend on.
> That includes things like ANT, Maven, JDK,...
> For some Tools there is an automatic tool installation available which
> usually means download something from somewhere and install it.
> Having the same thing for Buckminster would have several advantages:
> -Easier to configure for the user
> -Easy to update/upgrade buckminster
> -And most of all, this would enable master/slave scenarios. If a build
> is executed on a slave machine and relies on a tool that has an
> automatic installation available, it can check if this tool is already
> installed and install it on the fly if necessary. This is a great
> feature if you have something like a build cloud where new slaves get
> added on demand.
>
> So in my opinion it would be better to participate in the logic provided
> by hudson instead of doing it plugin-internal during the build run. What
> is your opinion on that?
>
> As for the 'how', does this sound ok to you? :
Sounds good. Better to reuse as much as possible and be good Hudson citizens then inventing new stuff.

> How would this sound to you:
> In configure Hudson you have 3 options:
> 1. enter a location to a buckminster installation (that's how it works
> currently)
>
This has it's drawbacks. The client is required to install a Buckminster first. And this installation must be on the
same platform where the build is supposed to run. In case you have one Hudson manager that sets things up, and then
clients that configures their builds, this is good though. Hides the complexity from the end user.

> 2. pick a version and install a default buckminster automatically (pretty
> much all the features on the headless update site I would say)
>
I like this. Question is how the builder would now what versions that are available. Perhaps we could put up a file of
some sort at http://download.eclipse.org/tools/buckminster/, i.e. the parent folder of our update sites where we list
the versions and their respective feature sets. That way, the Hudson plugin wouldn't need to hard wire the sites and
features, and it would give us freedom to move things around.

> 3. provide a list of update sites and features to be installed
>
>
Something like this is needed when a client provides their own extensions to Buckminster. Perhaps this can be additional
info provided for case #2?

- thomas
Re: Hudson plugin and provisioning [message #520883 is a reply to message #520862] Mon, 15 March 2010 16:16 Go to previous messageGo to next message
Johannes Utzig is currently offline Johannes UtzigFriend
Messages: 329
Registered: July 2009
Senior Member
Hi Thomas,

please see comments inline.

>
> > How would this sound to you:
> > In configure Hudson you have 3 options:
> > 1. enter a location to a buckminster installation (that's how it works
> > currently)
> >
> This has it's drawbacks. The client is required to install a Buckminster
> first. And this installation must be on the same platform where the
> build is supposed to run. In case you have one Hudson manager that sets
> things up, and then clients that configures their builds, this is good
> though. Hides the complexity from the end user.
>

It's an option I think we will always need. I would expect many company
environments don't want their build infrastructure to automatically
download things from the internet, they might not even be connected to
the internet, so the 'manual way' should always be an option.

> > 2. pick a version and install a default buckminster automatically
> (pretty
> > much all the features on the headless update site I would say)
> >
> I like this. Question is how the builder would now what versions that
> are available. Perhaps we could put up a file of some sort at
> http://download.eclipse.org/tools/buckminster/, i.e. the parent folder
> of our update sites where we list the versions and their respective
> feature sets. That way, the Hudson plugin wouldn't need to hard wire the
> sites and features, and it would give us freedom to move things around.
>

That sounds really good to me. Much better than hard wire it into the
code. Because if I hard wire this information I would have to release a
new version every time there is a new buckminster version or feature...

> > 3. provide a list of update sites and features to be installed
> >
> >
> Something like this is needed when a client provides their own
> extensions to Buckminster. Perhaps this can be additional info provided
> for case #2?
>

Yeah, that's what I thought too, you need it if you extended buckminster
or if you want to install from a intranet mirror instead of the
buckminster update site.
I guess you are right, 2. and 3. can be merged. In 2. you simply select
the version (3.5 or 3.6 could be default) and that's all you need to do.
Then there could be an 'advanced' button that reveals the exact
installation details (download URL for the director, update sites and
features to be installed).
That way the user would be free to edit this information however (s)he
wants.
But first I need to see if this is even possible with jelly script.
If not, I'll do it with 3 alternatives instead of 2.

Once I figured out how to make this work in hudson, we can discuss the
details on the file format that defines the versions and lists the
features. I'll get back to you once I have an inital working
implementation I would say.

This feature will be very handy for a hudson cluster environment, but I
have only one hudson server available for testing. It would be greatly
appreciated if anybody that runs a hudson cluster is interested in such
a feature and would be willing to test automatic tool installation in a
master/slave scenario with a pre-released version the plugin...

Best regards,
Johannes
Re: Hudson plugin and provisioning [message #520948 is a reply to message #520883] Mon, 15 March 2010 21:41 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 03/15/2010 05:16 PM, Johannes Utzig wrote:
> ... so the 'manual way' should always be an option.
>
Yes, I agree.

> Once I figured out how to make this work in hudson, we can discuss the
> details on the file format that defines the versions and lists the
> features. I'll get back to you once I have an inital working
> implementation I would say.
>
Sounds good.

> This feature will be very handy for a hudson cluster environment, but I
> have only one hudson server available for testing. It would be greatly
> appreciated if anybody that runs a hudson cluster is interested in such
> a feature and would be willing to test automatic tool installation in a
> master/slave scenario with a pre-released version the plugin...
>
We slaves running at build.eclipse.org so that's one place where we could test it. One idea could be to use this for
Buckminsters own nightly builds. Let's discuss when you have things that you want tested.

Regards,
Thomas Hallgren
Re: Hudson plugin and provisioning [message #522177 is a reply to message #520862] Sat, 20 March 2010 19:18 Go to previous messageGo to next message
Johannes Utzig is currently offline Johannes UtzigFriend
Messages: 329
Registered: July 2009
Senior Member
Hi,

today I finally had a chance to have a look at the auto
installation/provisioning topic.
The good news is, there are 2 generic installers that could be used
out-of-the-box as soon as I change the code to rely on Hudson's Tool
infrastructure.
One installer can execute an arbitrary shell script, the other can
extract an archive. This should initially be already enough to support
custom buckminster installations.

Leaves the issue of a default installation that gets provisioned from
eclipse.org:
-download a p2 director
-extract it
-use the director application to create a buckminster product of a given
version
-install all features into the product

> > 2. pick a version and install a default buckminster automatically
> (pretty
> > much all the features on the headless update site I would say)
> >
> I like this. Question is how the builder would now what versions that
> are available. Perhaps we could put up a file of some sort at
> http://download.eclipse.org/tools/buckminster/, i.e. the parent folder
> of our update sites where we list the versions and their respective
> feature sets. That way, the Hudson plugin wouldn't need to hard wire the
> sites and features, and it would give us freedom to move things around.
>

It took me quite a while to figure out how this is currently working
with Ant/Maven/JDK because it's hardly documented and seems to be more
magic than actual coding :)
Turns out that there is an updates folder on the hudson web server that
contains several json files that feed the information into the specific
installers:
https://hudson.dev.java.net/updates/hudson.tasks.Ant.AntInst aller.json

Unfortunately no hudson plugin has implemented such a thing yet (ant,
maven, and JDK Tools are Hudson-Core functionality, not plugins) so I
couldn't really find code examples on how to wire the pieces together so
far.

Looks like it will be better to host the file with the version
information directly on Hudson instead of eclipse.org to not dance out
of line.
I will write again once I have a working implementation but it's
somewhat more complicated than I thought so it might take some days still...

Best regards,
Johannes
Re: Hudson plugin and provisioning [message #522214 is a reply to message #522177] Sun, 21 March 2010 10:59 Go to previous messageGo to next message
Johannes Utzig is currently offline Johannes UtzigFriend
Messages: 329
Registered: July 2009
Senior Member
Hi again,

it actually works like a charm now :)
Here's the current workflow, please let me know what you think about it:

1. The administrator opens the configure hudson page
2. He presses 'Add Buckminster' In the Buckminster section
3. He enters a name for the tool installation (that will be shown in the
job configuration)
4. By default 'Install Automatically' is checked and the default
'Install from Eclipse.org/Cloudsmith.com' installer is checked
5. He selects a version (the versions are fed from a json file on
hudson.org)
6. He can also remove this installer or add other installers (execute
shell script/ download and extract from URL)
7. If install automatically gets unchecked, the installer section
disappears and gets replaced with a textfield to enter the buckminster
home location (this is how it's done in the current buckminster plugin)
8. In a job configuration the build manager can choose between all known
buckminster tool installations
9. If the "Install from Eclipse.org/Cloudsmith.com" installer was
selected and the job is started, the installer checks if buckminster is
already installed on this machine. If not:
-it downloads and extracts the p2 director from the URL in the JSON file
-it invokes the p2 installer with the URL and IU given in the JSON file
to provision a buckminster headless product
-it iterates over all repositories listed in the JSON file and installs
all features associated with them.

In the console output of the job the result looks like this:

Unpacking
http://download.eclipse.org/tools/buckminster/products/direc tor_latest.zip
to /home/joe/workspaceHudson2/buckminster/work/tools/Buckminste r_3.5 on
Hudson
[buckminster] $ sh -e
/home/joe/workspaceHudson2/buckminster/work/tools/Buckminste r_3.5/buckminster/hudson1716709923112513365.sh
Installing org.eclipse.buckminster.cmdline.product 1.1.350.r11146.
Operation completed in 13049 ms.
Installed Feature org.eclipse.buckminster.core.headless.feature from
http://download.eclipse.org/tools/buckminster/headless-3.5/
Installed Feature org.eclipse.buckminster.cvs.headless.feature from
http://download.eclipse.org/tools/buckminster/headless-3.5/
Installed Feature org.eclipse.buckminster.pde.headless.feature from
http://download.eclipse.org/tools/buckminster/headless-3.5/
Installed Feature org.eclipse.buckminster.maven.feature from
http://download.eclipse.org/tools/buckminster/headless-3.5/
Installed Feature org.eclipse.buckminster.subversive.headless.feature
from http://download.cloudsmith.com/buckminster/external/

The JSON File consists of an array of Buckminster Installables with the
following contents (shortened):

"id":"3.5", //the unique ID of the installable
"name":"Buckminster 3.5", //the human readable name that will be shown
in the version selection
"url":" http://download.eclipse.org/tools/buckminster/products/direc tor_latest.zip",
//the URL of the director application
"iu":"org.eclipse.buckminster.cmdline.product"; //which IU to install
"repositoryURL":"http://download.eclipse.org/tools/buckminster/headless-3.5",
//where to find the headless product
"repositories":[ //a list of repositories where to install features from
{
"url":"http://download.eclipse.org/tools/buckminster/headless-3.5/",
//the repository URL
"features":[ //a list of features to install from this repo
{
"id":"org.eclipse.buckminster.core.headless.feature"},
{
"id":"org.eclipse.buckminster.cvs.headless.feature"}...


I'm almost ready to release this but there are a few questions remaining
that I need answered first:

1. Is the approach ok with you or should I change something?
2. Is it ok to use the direct download link from eclipse.org for the p2
director?
3. Is it ok to automatically download additional features from
cloudsmith or should I restrict to eclipse.org?
4. Do I need to have the user accept some kind of licence agreement for
features (like subversive, or so)?
5. I need some help writing the final JSON file with the version/feature
informations. What should be included in which version?

Best regards,
Johannes
Re: Hudson plugin and provisioning [message #522222 is a reply to message #522214] Sun, 21 March 2010 13:26 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
Hi Johannes,
This looks great! I have a few questions/suggestions inline.

On 03/21/2010 11:59 AM, Johannes Utzig wrote:
> Hi again,
>
> it actually works like a charm now :)
> Here's the current workflow, please let me know what you think about it:
>
> 1. The administrator opens the configure hudson page
> 2. He presses 'Add Buckminster' In the Buckminster section
> 3. He enters a name for the tool installation (that will be shown in the
> job configuration)
> 4. By default 'Install Automatically' is checked and the default
> 'Install from Eclipse.org/Cloudsmith.com' installer is checked
> 5. He selects a version (the versions are fed from a json file on
> hudson.org)
> 6. He can also remove this installer or add other installers (execute
> shell script/ download and extract from URL)

I'm uncertain what this means. What other installers can install Buckminster? Can you please elaborate?

> 7. If install automatically gets unchecked, the installer section
> disappears and gets replaced with a textfield to enter the buckminster
> home location (this is how it's done in the current buckminster plugin)

How do we cover the case where the admin wants to install a custom feature from some other repository?

> 8. In a job configuration the build manager can choose between all known
> buckminster tool installations

How is a known tool installation identified? Is this the same as the "versions" in #5 or can custom installations be
created (and named)?

> 9. If the "Install from Eclipse.org/Cloudsmith.com" installer was
> selected and the job is started, the installer checks if buckminster is
> already installed on this machine. If not:
> -it downloads and extracts the p2 director from the URL in the JSON file
> -it invokes the p2 installer with the URL and IU given in the JSON file
> to provision a buckminster headless product
> -it iterates over all repositories listed in the JSON file and installs
> all features associated with them.
>

What happens when new versions of the features arrive in the appointed repositories?

> In the console output of the job the result looks like this:
>
> Unpacking
> http://download.eclipse.org/tools/buckminster/products/direc tor_latest.zip
> to /home/joe/workspaceHudson2/buckminster/work/tools/Buckminste r_3.5 on
> Hudson
> [buckminster] $ sh -e
> /home/joe/workspaceHudson2/buckminster/work/tools/Buckminste r_3.5/buckminster/hudson1716709923112513365.sh
>
> Installing org.eclipse.buckminster.cmdline.product 1.1.350.r11146.
> Operation completed in 13049 ms.
> Installed Feature org.eclipse.buckminster.core.headless.feature from
> http://download.eclipse.org/tools/buckminster/headless-3.5/
> Installed Feature org.eclipse.buckminster.cvs.headless.feature from
> http://download.eclipse.org/tools/buckminster/headless-3.5/
> Installed Feature org.eclipse.buckminster.pde.headless.feature from
> http://download.eclipse.org/tools/buckminster/headless-3.5/
> Installed Feature org.eclipse.buckminster.maven.feature from
> http://download.eclipse.org/tools/buckminster/headless-3.5/
> Installed Feature org.eclipse.buckminster.subversive.headless.feature
> from http://download.cloudsmith.com/buckminster/external/
>
> The JSON File consists of an array of Buckminster Installables with the
> following contents (shortened):
>
> "id":"3.5", //the unique ID of the installable
> "name":"Buckminster 3.5", //the human readable name that will be shown
> in the version selection
> "url":" http://download.eclipse.org/tools/buckminster/products/direc tor_latest.zip",
> //the URL of the director application
> "iu":"org.eclipse.buckminster.cmdline.product"; //which IU to install
> "repositoryURL":"http://download.eclipse.org/tools/buckminster/headless-3.5",
> //where to find the headless product
> "repositories":[ //a list of repositories where to install features from
> { "url":"http://download.eclipse.org/tools/buckminster/headless-3.5/",
> //the repository URL
> "features":[ //a list of features to install from this repo
> { "id":"org.eclipse.buckminster.core.headless.feature"},
> { "id":"org.eclipse.buckminster.cvs.headless.feature"}...
>
>
> I'm almost ready to release this but there are a few questions remaining
> that I need answered first:
>
> 1. Is the approach ok with you or should I change something?

I think it would be great if we could use an alternative json file for specific hudson instances. On build.eclipse.org
for instance, all URL's pointing to http://download.eclipse.org/ can be replaced with
file:/home/data/httpd/download.eclipse.org/, a change that removes a lot of load from the web server. That applies both
to the director download and the actual install.


> 2. Is it ok to use the direct download link from eclipse.org for the p2
> director?

I don't think you need to. An external build can use this link:

http://www.eclipse.org/downloads/download.php?file=/tools/bu ckminster/products/director_latest.zip&r=1

Note the r=1 parameter at the end. It causes a bypass of the selection page so the download is immediate. The closest
geographical mirror is chosen.

> 3. Is it ok to automatically download additional features from
> cloudsmith or should I restrict to eclipse.org?

It must be OK. This is about tooling for our build environment and not about what we publish at Eclipse.org.

> 4. Do I need to have the user accept some kind of licence agreement for
> features (like subversive, or so)?

Quite frankly, I have no idea who agrees to what licenses on what tools in our build environment. Who agreed to use
Linux, Java, Hudson, Ant, Maven, CVS, SVN, Git, the Buckminster plug-in, etc.? I guess this must fall into the same
category. It somehow feels that it's the admins responsibility and that he implicitly accepts the licenses of the stuff
that is put to use. But as I said, I really have no idea.

What happens when you install a new Hudson plug-in? Do you have to accept its license and all licenses that it depends
on at that point?

> 5. I need some help writing the final JSON file with the version/feature
> informations. What should be included in which version?
>
The latest on our headless-3.5 and headless-3.6 update sites + the latest Subclipse at cloudsmith external-3.5 and
external-3.6. We might want to switch to Subversive at some point but at present they have some bugzillas that need
attending first.

- thomas
Re: Hudson plugin and provisioning [message #522232 is a reply to message #522222] Sun, 21 March 2010 10:24 Go to previous messageGo to next message
Johannes Utzig is currently offline Johannes UtzigFriend
Messages: 329
Registered: July 2009
Senior Member
Hi Thomas,

please see answers inside:

Thomas Hallgren schrieb:

>
> I'm uncertain what this means. What other installers can install
> Buckminster? Can you please elaborate?
>

Hudson offers two generic installers:
1. Execute a shell script (pseudocode)
This can perfectly fine install buckminster.
wget director
extract director
director create buckminster product
buckminster install features

2.Download and extract an archive
An admin can prepare a prepackaged custom buckminster, deploy it on some
webserver and have the generic download installer extract that archive
on the build machine as needed.

>> 7. If install automatically gets unchecked, the installer section
>> disappears and gets replaced with a textfield to enter the buckminster
>> home location (this is how it's done in the current buckminster plugin)
>
> How do we cover the case where the admin wants to install a custom
> feature from some other repository?

Currently 4 possibilities:
1. manually specify a path to a custom buckminster installation
2. use the generic script based installer
3. use the generic download installer
4. create a 'configure buckminster' build job that doesn't really build
anything, but executes 'install' commands

It would also be possible I guess to add additional features +
repositories in the configuration of the custom installer, but I wasn't
able to do that just yet with the jelly script.

>
>> 8. In a job configuration the build manager can choose between all known
>> buckminster tool installations
>
> How is a known tool installation identified? Is this the same as the
> "versions" in #5 or can custom installations be created (and named)?
>

If you don't activate 'auto installation' you specify a home directory
of your tool and give the Tool Installation a name. Tool Installations
are always identified by name.

>
> What happens when new versions of the features arrive in the appointed
> repositories?
>

I was thinking about that too. There's two different scenarios:
1. The user activated the auto tool installation
In this case the tool installer would have a chance to perform an update
but that might be too expensive to do for every build run.

2. The tool installation has not been installed by the custom installer
which means the installer logic has no chance to run an update.

So I guess the update logic should be placed in the build step, and not
in the installer to make it work no matter how buckminster has been
installed.
Unfortunately I don't see a way to find out what's currently installed
and therefore I don't know which features need updating.
What about an 'update' command for buckminster? How hard would that be
to implement?

>
> I think it would be great if we could use an alternative json file for
> specific hudson instances. On build.eclipse.org for instance, all URL's
> pointing to http://download.eclipse.org/ can be replaced with
> file:/home/data/httpd/download.eclipse.org/, a change that removes a lot
> of load from the web server. That applies both to the director download
> and the actual install.
>

I'm afraid this is not so easy. This information needs to be available
as soon as the manage hudson site is requested. Waiting for a server
reply would cause the site to hang if the server is not available for
some reason. As far as I understand the mechanics so far, this is
avoided in hudson by downloading the update files up front, store them
locally and update them once a day, or so.
I don't see a way how it would be possible to have the user enter a
different URL and process that at runtime, because I don't think it's
possible to update the web form once it has been created.

That said, a workaround of some sort should be possible of course.
The installer could for example look for a custom.json at a specific
place in the hudson dir, or look for a system property, or something
like that. If present, it overrides the remote file.

>
>> 2. Is it ok to use the direct download link from eclipse.org for the p2
>> director?
>
> I don't think you need to. An external build can use this link:
>
> http://www.eclipse.org/downloads/download.php?file=/tools/bu ckminster/products/director_latest.zip&r=1
>
>
> Note the r=1 parameter at the end. It causes a bypass of the selection
> page so the download is immediate. The closest geographical mirror is
> chosen.
>

I tried that, but it doesn't work. Looks like the redirect is causing
problems for the download logic (which is provided by Hudson)

>> 3. Is it ok to automatically download additional features from
>> cloudsmith or should I restrict to eclipse.org?
>
> It must be OK. This is about tooling for our build environment and not
> about what we publish at Eclipse.org.
>

true, thank you

>> 4. Do I need to have the user accept some kind of licence agreement for
>> features (like subversive, or so)?
>
> Quite frankly, I have no idea who agrees to what licenses on what tools
> in our build environment. Who agreed to use Linux, Java, Hudson, Ant,
> Maven, CVS, SVN, Git, the Buckminster plug-in, etc.? I guess this must
> fall into the same category. It somehow feels that it's the admins
> responsibility and that he implicitly accepts the licenses of the stuff
> that is put to use. But as I said, I really have no idea.
>
> What happens when you install a new Hudson plug-in? Do you have to
> accept its license and all licenses that it depends on at that point?
>

I got a little curious when I noticed that the Hudson JDK Installer
requires the user to accept SUN's licence agreement. You are right, this
is most likely unnecessary, so I don't do anything like that.

>> 5. I need some help writing the final JSON file with the version/feature
>> informations. What should be included in which version?
>>
> The latest on our headless-3.5 and headless-3.6 update sites + the
> latest Subclipse at cloudsmith external-3.5 and external-3.6. We might
> want to switch to Subversive at some point but at present they have some
> bugzillas that need attending first.
>

I can't really list the site features at the moment.

buckminser listsite
http://download.eclipse.org/tools/buckminster/headless-3.6/

gives me this (latest buckminster 3.6 headless):

java.lang.IllegalArgumentException: Cannot find a public member 'id' in
a org.eclipse.equinox.internal.p2.metadata.expression.Everythi ng


So every feature from eclipe.org und subclipse (no subversive) from
cloudsmith, right? What about the emma feature, or is that available
from the eclipse.org site?

Best regards,
Johannes
Re: Hudson plugin and provisioning [message #522236 is a reply to message #522232] Sun, 21 March 2010 16:18 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
Hi Johannes,

On 03/21/2010 04:14 PM, Johannes Utzig wrote:
>> I'm uncertain what this means. What other installers can install
>> Buckminster? Can you please elaborate?
>>
>
> Hudson offers two generic installers:
> 1. Execute a shell script (pseudocode)
> This can perfectly fine install buckminster.
> wget director
> extract director
> director create buckminster product
> buckminster install features
>
> 2.Download and extract an archive
> An admin can prepare a prepackaged custom buckminster, deploy it on some
> webserver and have the generic download installer extract that archive
> on the build machine as needed.
>
OK, clear.

>>> 7. If install automatically gets unchecked, the installer section
>>> disappears and gets replaced with a textfield to enter the buckminster
>>> home location (this is how it's done in the current buckminster plugin)
>>
>> How do we cover the case where the admin wants to install a custom
>> feature from some other repository?
>
> Currently 4 possibilities:
> 1. manually specify a path to a custom buckminster installation
> 2. use the generic script based installer
> 3. use the generic download installer
> 4. create a 'configure buckminster' build job that doesn't really build
> anything, but executes 'install' commands
>
> It would also be possible I guess to add additional features +
> repositories in the configuration of the custom installer, but I wasn't
> able to do that just yet with the jelly script.
>
I think the four possibilities covers this :-) No need to dig deeper unless there's a real demand for it.

>> What happens when new versions of the features arrive in the appointed
>> repositories?
>>
>
> I was thinking about that too. There's two different scenarios:
> 1. The user activated the auto tool installation
> In this case the tool installer would have a chance to perform an update
> but that might be too expensive to do for every build run.
>
I agree. This should only happen when explicitly requested (vouches for better build stability).

> What about an 'update' command for buckminster? How hard would that be
> to implement?
>
What's needed is an 'update' command for the director I think. IIRC, there is a p2 bugzilla for that.

For now, I'm happy if the admin can just clean out what's current and reinstall.

>>
>> I think it would be great if we could use an alternative json file for
>> specific hudson instances. On build.eclipse.org for instance, all
>> URL's pointing to http://download.eclipse.org/ can be replaced with
>> file:/home/data/httpd/download.eclipse.org/, a change that removes a
>> lot of load from the web server. That applies both to the director
>> download and the actual install.
>>
>
> I'm afraid this is not so easy. This information needs to be available
> as soon as the manage hudson site is requested. Waiting for a server
> reply would cause the site to hang if the server is not available for
> some reason. As far as I understand the mechanics so far, this is
> avoided in hudson by downloading the update files up front, store them
> locally and update them once a day, or so.
> I don't see a way how it would be possible to have the user enter a
> different URL and process that at runtime, because I don't think it's
> possible to update the web form once it has been created.
>
> That said, a workaround of some sort should be possible of course.
> The installer could for example look for a custom.json at a specific
> place in the hudson dir, or look for a system property, or something
> like that. If present, it overrides the remote file.
>
Yes, this was what I was thinking too. Something simple that can override. Doesn't need to be exposed in the UI.

>>
>>> 2. Is it ok to use the direct download link from eclipse.org for the p2
>>> director?
>>
>> I don't think you need to. An external build can use this link:
>>
>> http://www.eclipse.org/downloads/download.php?file=/tools/bu ckminster/products/director_latest.zip&r=1
>>
>>
>> Note the r=1 parameter at the end. It causes a bypass of the selection
>> page so the download is immediate. The closest geographical mirror is
>> chosen.
>>
>
> I tried that, but it doesn't work. Looks like the redirect is causing
> problems for the download logic (which is provided by Hudson)
>
OK, perhaps it does something with the parameters of the URL. In any case, I think it must be OK to download using a
direct URL. The director is relatively small and this is not something that will happen on every run.

> I can't really list the site features at the moment.
>
> buckminser listsite
> http://download.eclipse.org/tools/buckminster/headless-3.6/
>
> gives me this (latest buckminster 3.6 headless):
>
> java.lang.IllegalArgumentException: Cannot find a public member 'id' in
> a org.eclipse.equinox.internal.p2.metadata.expression.Everythi ng
>
Ouch! Would appreciate a bugzilla on that.

> So every feature from eclipe.org und subclipse (no subversive) from
> cloudsmith, right? What about the emma feature, or is that available
> from the eclipse.org site?
>
Yes, nowadays emma is also included from Eclipse.

Regards,
Thomas Hallgren
Re: Hudson plugin and provisioning [message #522257 is a reply to message #522236] Sun, 21 March 2010 21:25 Go to previous messageGo to next message
Johannes Utzig is currently offline Johannes UtzigFriend
Messages: 329
Registered: July 2009
Senior Member
Hi Thomas,

comments inline

>>
> Yes, this was what I was thinking too. Something simple that can
> override. Doesn't need to be exposed in the UI.
>

Alright, I do this then. I'll have a look for a good place of the
replacement JSON and document the 'backdoor' in the plugin's wiki page
once the new version is deployed.


> OK, perhaps it does something with the parameters of the URL. In any
> case, I think it must be OK to download using a direct URL. The director
> is relatively small and this is not something that will happen on every
> run.
>

Ok, I'll leave it like that then since it only downloads the director
when the tool gets initially installed. If it turns out to be a problem
I can still try to find a solution for the URL redirect.

>>
> Ouch! Would appreciate a bugzilla on that.
>

https://bugs.eclipse.org/bugs/show_bug.cgi?id=306649

>> So every feature from eclipe.org und subclipse (no subversive) from
>> cloudsmith, right? What about the emma feature, or is that available
>> from the eclipse.org site?
>>
> Yes, nowadays emma is also included from Eclipse.
>

Ok, I think that was all I needed then.
Thank you very much for the support. I will post again once I deployed
the new version.
I'll also post the JSON file once I'm done with it so you can double
check if everything is correct.
If everything works, maybe you can try to run this on a slave on
build.eclipse.org or so to see if it works on slaves as well (since I
have no means to test that properly).

If anybody reading this has a hudson running on a windows machine, I'd
greatly appreciate a feedback if everything is working there too.

Thanks again and best regards,
Johannes
Re: Hudson plugin and provisioning [message #522491 is a reply to message #522257] Mon, 22 March 2010 18:58 Go to previous messageGo to next message
Johannes Utzig is currently offline Johannes UtzigFriend
Messages: 329
Registered: July 2009
Senior Member
Alright, I'm done with the override logic and the json file, but
unfortunatly the psf feature is causing trouble on both 3.5 and 3.6.
Any insight on this?

3.5:
Unpacking
http://download.eclipse.org/tools/buckminster/products/direc tor_latest.zip
to /home/joe/workspaceHudson2/buckminster/work/tools/Buckminste r_3.5 on
Hudson
$ sh -e
/home/joe/workspaceHudson2/buckminster/work/tools/Buckminste r_3.5/buckminster/hudson2796441302788835248.sh
Installing org.eclipse.buckminster.cmdline.product 1.1.350.r11146.
Operation completed in 24784 ms.
Installed Feature org.eclipse.buckminster.core.headless.feature from
http://download.eclipse.org/tools/buckminster/headless-3.5/
Installed Feature org.eclipse.equinox.p2.director.feature from
http://download.eclipse.org/tools/buckminster/headless-3.5/
Installed Feature org.eclipse.buckminster.cvs.headless.feature from
http://download.eclipse.org/tools/buckminster/headless-3.5/
Installed Feature org.eclipse.buckminster.pde.headless.feature from
http://download.eclipse.org/tools/buckminster/headless-3.5/
Installed Feature org.eclipse.buckminster.maven.feature from
http://download.eclipse.org/tools/buckminster/headless-3.5/
Installed Feature org.eclipse.buckminster.aggregator.engine.feature from
http://download.eclipse.org/tools/buckminster/headless-3.5/
Cannot complete the install because one or more required items could not
be found.
Software being installed: Buckminster - Team Project Set (PSF)
support for headless operation 1.0.0.r11251
(org.eclipse.buckminster.psf.feature.feature.group 1.0.0.r11251)
Missing requirement: Edit Support for the PSF extension to the RMAP
model 1.0.0.r11162 (org.eclipse.buckminster.rmap.psf.edit 1.0.0.r11162)
requires 'bundle org.eclipse.emf.edit 0.0.0' but it could not be found
Cannot satisfy dependency:
From: Buckminster - Team Project Set (PSF) support for headless
operation 1.0.0.r11251
(org.eclipse.buckminster.psf.feature.feature.group 1.0.0.r11251)
To: org.eclipse.buckminster.rmap.psf.edit


3.6:
Unpacking
http://download.eclipse.org/tools/buckminster/products/direc tor_latest.zip
to /home/joe/workspaceHudson2/buckminster/work/tools/Buckminste r_3.6 on
Hudson
$ sh -e
/home/joe/workspaceHudson2/buckminster/work/tools/Buckminste r_3.6/buckminster/hudson4554671259141863058.sh
Installing org.eclipse.buckminster.cmdline.product 1.2.0.r11290.
Operation completed in 14533 ms.
Installed Feature org.eclipse.buckminster.core.headless.feature from
http://download.eclipse.org/tools/buckminster/headless-3.6/
Installed Feature org.eclipse.buckminster.cvs.headless.feature from
http://download.eclipse.org/tools/buckminster/headless-3.6/
Installed Feature org.eclipse.buckminster.pde.headless.feature from
http://download.eclipse.org/tools/buckminster/headless-3.6/
Installed Feature org.eclipse.buckminster.maven.headless.feature from
http://download.eclipse.org/tools/buckminster/headless-3.6/
Installed Feature org.eclipse.equinox.p2.director.feature from
http://download.eclipse.org/tools/buckminster/headless-3.6/
Installed Feature org.eclipse.buckminster.emma.headless.feature from
http://download.eclipse.org/tools/buckminster/headless-3.6/
Cannot complete the install because one or more required items could not
be found.
[0]Software being installed: Buckminster - Team Project Set (PSF)
support for headless operation 1.0.0.r11210
(org.eclipse.buckminster.psf.feature.feature.group 1.0.0.r11210)
[0]Missing requirement: Edit Support for the PSF extension to the RMAP
model 1.0.0.r11210 (org.eclipse.buckminster.rmap.psf.edit 1.0.0.r11210)
requires 'bundle org.eclipse.emf.edit 0.0.0' but it could not be found
[0]Cannot satisfy dependency:
[0.2]From: Buckminster - Team Project Set (PSF) support for headless
operation 1.0.0.r11210
(org.eclipse.buckminster.psf.feature.feature.group 1.0.0.r11210)
[0.2]To: org.eclipse.buckminster.rmap.psf.edit [1.0.0,1.1.0)
ERROR: Command returned status 1


Best regards,
Johannes
Re: Hudson plugin and provisioning [message #522518 is a reply to message #522491] Mon, 22 March 2010 16:32 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
Hi Johannes,
Seems we're missing something in the psf headless support. I'll look into that.

- thomas

On 03/22/2010 07:58 PM, Johannes Utzig wrote:
> Alright, I'm done with the override logic and the json file, but
> unfortunatly the psf feature is causing trouble on both 3.5 and 3.6.
> Any insight on this?
>
> 3.5:
> Unpacking
> http://download.eclipse.org/tools/buckminster/products/direc tor_latest.zip
> to /home/joe/workspaceHudson2/buckminster/work/tools/Buckminste r_3.5 on
> Hudson
> $ sh -e
> /home/joe/workspaceHudson2/buckminster/work/tools/Buckminste r_3.5/buckminster/hudson2796441302788835248.sh
>
> Installing org.eclipse.buckminster.cmdline.product 1.1.350.r11146.
> Operation completed in 24784 ms.
> Installed Feature org.eclipse.buckminster.core.headless.feature from
> http://download.eclipse.org/tools/buckminster/headless-3.5/
> Installed Feature org.eclipse.equinox.p2.director.feature from
> http://download.eclipse.org/tools/buckminster/headless-3.5/
> Installed Feature org.eclipse.buckminster.cvs.headless.feature from
> http://download.eclipse.org/tools/buckminster/headless-3.5/
> Installed Feature org.eclipse.buckminster.pde.headless.feature from
> http://download.eclipse.org/tools/buckminster/headless-3.5/
> Installed Feature org.eclipse.buckminster.maven.feature from
> http://download.eclipse.org/tools/buckminster/headless-3.5/
> Installed Feature org.eclipse.buckminster.aggregator.engine.feature from
> http://download.eclipse.org/tools/buckminster/headless-3.5/
> Cannot complete the install because one or more required items could not
> be found.
> Software being installed: Buckminster - Team Project Set (PSF)
> support for headless operation 1.0.0.r11251
> (org.eclipse.buckminster.psf.feature.feature.group 1.0.0.r11251)
> Missing requirement: Edit Support for the PSF extension to the RMAP
> model 1.0.0.r11162 (org.eclipse.buckminster.rmap.psf.edit 1.0.0.r11162)
> requires 'bundle org.eclipse.emf.edit 0.0.0' but it could not be found
> Cannot satisfy dependency:
> From: Buckminster - Team Project Set (PSF) support for headless
> operation 1.0.0.r11251
> (org.eclipse.buckminster.psf.feature.feature.group 1.0.0.r11251)
> To: org.eclipse.buckminster.rmap.psf.edit
>
>
> 3.6:
> Unpacking
> http://download.eclipse.org/tools/buckminster/products/direc tor_latest.zip
> to /home/joe/workspaceHudson2/buckminster/work/tools/Buckminste r_3.6 on
> Hudson
> $ sh -e
> /home/joe/workspaceHudson2/buckminster/work/tools/Buckminste r_3.6/buckminster/hudson4554671259141863058.sh
>
> Installing org.eclipse.buckminster.cmdline.product 1.2.0.r11290.
> Operation completed in 14533 ms.
> Installed Feature org.eclipse.buckminster.core.headless.feature from
> http://download.eclipse.org/tools/buckminster/headless-3.6/
> Installed Feature org.eclipse.buckminster.cvs.headless.feature from
> http://download.eclipse.org/tools/buckminster/headless-3.6/
> Installed Feature org.eclipse.buckminster.pde.headless.feature from
> http://download.eclipse.org/tools/buckminster/headless-3.6/
> Installed Feature org.eclipse.buckminster.maven.headless.feature from
> http://download.eclipse.org/tools/buckminster/headless-3.6/
> Installed Feature org.eclipse.equinox.p2.director.feature from
> http://download.eclipse.org/tools/buckminster/headless-3.6/
> Installed Feature org.eclipse.buckminster.emma.headless.feature from
> http://download.eclipse.org/tools/buckminster/headless-3.6/
> Cannot complete the install because one or more required items could not
> be found.
> [0]Software being installed: Buckminster - Team Project Set (PSF)
> support for headless operation 1.0.0.r11210
> (org.eclipse.buckminster.psf.feature.feature.group 1.0.0.r11210)
> [0]Missing requirement: Edit Support for the PSF extension to the RMAP
> model 1.0.0.r11210 (org.eclipse.buckminster.rmap.psf.edit 1.0.0.r11210)
> requires 'bundle org.eclipse.emf.edit 0.0.0' but it could not be found
> [0]Cannot satisfy dependency:
> [0.2]From: Buckminster - Team Project Set (PSF) support for headless
> operation 1.0.0.r11210
> (org.eclipse.buckminster.psf.feature.feature.group 1.0.0.r11210)
> [0.2]To: org.eclipse.buckminster.rmap.psf.edit [1.0.0,1.1.0)
> ERROR: Command returned status 1
>
>
> Best regards,
> Johannes
Re: Hudson plugin and provisioning [message #522524 is a reply to message #522518] Mon, 22 March 2010 21:32 Go to previous messageGo to next message
Johannes Utzig is currently offline Johannes UtzigFriend
Messages: 329
Registered: July 2009
Senior Member
Thomas Hallgren schrieb:
> Hi Johannes,
> Seems we're missing something in the psf headless support. I'll look
> into that.
>

I pushed the JSON file to the hudson server and excluded the psf support
for now, so please drop me a note when the issue is fixed so I can
update the JSON file.
I released version 0.9.4 just 5 minutes ago so hudson's update center
should make it available in a few hours.
Auto Installation should work now hopefully (at least it does on my
machine :) ).
The backdoor to replace the json file against a custom one is documented
here:
http://wiki.hudson-ci.org/display/HUDSON/Buckminster+PlugIn

Basically you just create a folder 'buckminster' in
'HUDSON_HOME/userContent', create a file 'buckminster.json' and list
your features and repositories in there. That will override whatever
information comes from
http://hudson-ci.org/updates/hudson.plugins.buckminster.Buck minsterInstallation.BuckminsterInstaller.json

The file format is also explained in the wiki.

If you decide to give it a try, please let me know if it worked correctly.

Best regards,
Johannes
Re: Hudson plugin and provisioning [message #522633 is a reply to message #522524] Tue, 23 March 2010 07:42 Go to previous messageGo to next message
Johannes Utzig is currently offline Johannes UtzigFriend
Messages: 329
Registered: July 2009
Senior Member
I tried the new version today on our hudson instance at work. The code
itself seems to work fine, but the PDE feature can no longer be
installed from the 3.6 update site. Did anything change since yesterday,
or did I hit a bad mirror?

[0]Software being installed: Buckminster - PDE support for headless
operation 1.2.0.r11341
(org.eclipse.buckminster.pde.headless.feature.feature.group 1.2.0.r11341)
[0]Missing requirement: Buckminster PDE Plug-in 1.2.0.r11341
(org.eclipse.buckminster.pde 1.2.0.r11341) requires 'bundle
org.eclipse.pde.api.tools 1.0.201' but it could not be found
[0]Cannot satisfy dependency:
[0.2]From: Buckminster - PDE support for headless operation 1.2.0.r11341
(org.eclipse.buckminster.pde.headless.feature.feature.group 1.2.0.r11341)
[0.2]To: org.eclipse.buckminster.pde [1.2.0,1.3.0)
Re: Hudson plugin and provisioning [message #522640 is a reply to message #522633] Tue, 23 March 2010 13:09 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
I managed to produce a bad build but it is fixed now. Still has one problem with the psf feature though. I'm onto that one.

- thomas

On 03/23/2010 01:30 PM, Johannes Utzig wrote:
> I tried the new version today on our hudson instance at work. The code
> itself seems to work fine, but the PDE feature can no longer be
> installed from the 3.6 update site. Did anything change since yesterday,
> or did I hit a bad mirror?
>
> [0]Software being installed: Buckminster - PDE support for headless
> operation 1.2.0.r11341
> (org.eclipse.buckminster.pde.headless.feature.feature.group 1.2.0.r11341)
> [0]Missing requirement: Buckminster PDE Plug-in 1.2.0.r11341
> (org.eclipse.buckminster.pde 1.2.0.r11341) requires 'bundle
> org.eclipse.pde.api.tools 1.0.201' but it could not be found
> [0]Cannot satisfy dependency:
> [0.2]From: Buckminster - PDE support for headless operation 1.2.0.r11341
> (org.eclipse.buckminster.pde.headless.feature.feature.group 1.2.0.r11341)
> [0.2]To: org.eclipse.buckminster.pde [1.2.0,1.3.0)
>
Re: Hudson plugin and provisioning [message #522669 is a reply to message #522524] Tue, 23 March 2010 14:55 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 03/22/2010 10:32 PM, Johannes Utzig wrote:
> ... please drop me a note when the issue is fixed so I can
> update the JSON file.

PSF support is OK now.

- thomas
Re: Hudson plugin and provisioning [message #522677 is a reply to message #522669] Tue, 23 March 2010 14:56 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 03/23/2010 03:55 PM, Thomas Hallgren wrote:
> On 03/22/2010 10:32 PM, Johannes Utzig wrote:
>> ... please drop me a note when the issue is fixed so I can
>> update the JSON file.
>
> PSF support is OK now.
>
For 3.6 I should add. Not sure we need it for 3.5 (it was meant to be a sneak preview, not really production quality).

- thomas
Re: Hudson plugin and provisioning [message #522683 is a reply to message #522677] Tue, 23 March 2010 10:32 Go to previous messageGo to next message
Johannes Utzig is currently offline Johannes UtzigFriend
Messages: 329
Registered: July 2009
Senior Member
Hi Thomas,
>>
>> PSF support is OK now.
>>
> For 3.6 I should add. Not sure we need it for 3.5 (it was meant to be a
> sneak preview, not really production quality).
>
> - thomas

That's great news. I'll update the JSON file as soon as I'm home (the
3.6 version).
PDE is also installable again, thank you very much for the quick
reaction, everything seems to work now.

Best regards,
Johannes
Re: Hudson plugin and provisioning [message #522710 is a reply to message #522683] Tue, 23 March 2010 17:13 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
Hi Johannes,
I'm testing your plug-in and I wonder if there is any way to influence the fact that it uses the Hudson workspace as the
Eclipse workspace. A typical layout in my builds (using Ant as the intemediate) looks like this:

/build
/signing (staging area for the Eclipse signer)
/output (buckminster.output.root)
/temp (buckminster.temp.root)
/workspace (the eclipse workspace)
/targetPlatform

/org.eclipse.xxx.releng (releng project checked out by Hudson)
/xxx (other projects)

In the releng project, I will typically find the cquery, rmap, properties, etc. used by the build.

Using git, all projects will typically be checked out by Hudson and our git reader will then bind them to the workspace
together with other stuff that is checked out.

- thomas


On 03/23/2010 04:25 PM, Johannes Utzig wrote:
> Hi Thomas,
>>>
>>> PSF support is OK now.
>>>
>> For 3.6 I should add. Not sure we need it for 3.5 (it was meant to be a
>> sneak preview, not really production quality).
>>
>> - thomas
>
> That's great news. I'll update the JSON file as soon as I'm home (the
> 3.6 version).
> PDE is also installable again, thank you very much for the quick
> reaction, everything seems to work now.
>
> Best regards,
> Johannes
Re: Hudson plugin and provisioning [message #522716 is a reply to message #522710] Tue, 23 March 2010 17:34 Go to previous messageGo to next message
Johannes Utzig is currently offline Johannes UtzigFriend
Messages: 329
Registered: July 2009
Senior Member
Hi Thomas,

it was a choice made relatively early. It derives from the way I'm using
Buckminster in a CI environment. Developers and Hudson use the same RMAP
and I always want hudson to do the checkout because of the change log,
the update in favor of checkout and possibly automatically triggered
builds on SCM changes:
<searchPath name="mystuff">
<provider readerType="local"
componentTypes="osgi.bundle,eclipse.feature" mutable="true" source="true">
<uri format="file:///{0}/{1}/">
<bc:propertyRef key="workspace.root" />
<bc:propertyRef key="buckminster.component" />
</uri>
</provider>
<provider readerType="cvs"
componentTypes="eclipse.feature,osgi.bundle,buckminster,site.feature "
source="true">
<uri
format=":extssh:{0}@myserver:22/cvsroot/,module/{1}">
<bc:propertyRef key="user.name" />
<bc:propertyRef key="buckminster.component" />
</uri>
</provider>
</searchPath>

That way everything has the same layout like in my eclipse workspace and
it enables buckminster to easily find everything that hudson checked out.

If you prefer an eclipse workspace inside of a hudson workspace
scenario, how about an additional setting in the advanced section of the
build step configuration that lets you configure a nested workspace?
output and temp default to 'buckminster.output' and 'buckminster.temp'
in the workspace root, so that's similar enough I guess.

Best regards,
Johannes
Re: Hudson plugin and provisioning [message #522724 is a reply to message #522716] Tue, 23 March 2010 13:02 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
Hi Johannes,

On 03/23/2010 06:34 PM, Johannes Utzig wrote:
> Hi Thomas,
>
> it was a choice made relatively early. It derives from the way I'm using
> Buckminster in a CI environment. Developers and Hudson use the same RMAP
> and I always want hudson to do the checkout because of the change log,
> the update in favor of checkout and possibly automatically triggered
> builds on SCM changes:
> <searchPath name="mystuff">
> <provider readerType="local"
> componentTypes="osgi.bundle,eclipse.feature" mutable="true" source="true">
> <uri format="file:///{0}/{1}/">
> <bc:propertyRef key="workspace.root" />
> <bc:propertyRef key="buckminster.component" />
> </uri>
> </provider>
> <provider readerType="cvs"
> componentTypes="eclipse.feature,osgi.bundle,buckminster,site.feature "
> source="true">
> <uri
> format=":extssh:{0}@myserver:22/cvsroot/,module/{1}">
> <bc:propertyRef key="user.name" />
> <bc:propertyRef key="buckminster.component" />
> </uri>
> </provider>
> </searchPath>
>
> That way everything has the same layout like in my eclipse workspace and
> it enables buckminster to easily find everything that hudson checked out.
>
I can understand that. It's the same scenario that I use for git. But the physical location of the checked out material
doesn't need to be the same as the physical location of the workspace root and Buckminster needs to bind things to a
workspace anyway so it doesn't really matter where things are as long as you can keep things from colliding.


> If you prefer an eclipse workspace inside of a hudson workspace
> scenario, how about an additional setting in the advanced section of the
> build step configuration that lets you configure a nested workspace?
> output and temp default to 'buckminster.output' and 'buckminster.temp'
> in the workspace root, so that's similar enough I guess.
>
That would be an elegant solution.

- thomas
Re: Hudson plugin and provisioning [message #522793 is a reply to message #522724] Tue, 23 March 2010 17:44 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
I added an issue for this:

http://issues.hudson-ci.org/browse/HUDSON-6040

- thomas
Re: Hudson plugin and provisioning [message #522889 is a reply to message #522257] Wed, 24 March 2010 11:45 Go to previous messageGo to next message
Daniel Weber is currently offline Daniel WeberFriend
Messages: 51
Registered: July 2009
Member
Hi Johannes,

Johannes Utzig wrote:
> [...]
> If anybody reading this has a hudson running on a windows machine,

I have :)

> I'd greatly appreciate a feedback if everything is working there too.

OK. What I just did was this:

* updated the buckminster plugin to version 0.9.4 via hudson's plugin manager
* configured a new auto-install Buckminster instance and requested it to be version 3.6
* created a new job using that new buckminster installation
* scheduled a build of the newly created job

This lead to:

Unpacking http://download.eclipse.org/tools/buckminster/products/direc tor_latest.zip to D:\Documents and Settings\MyUserName\.hudson\tools\Buckminster_3.6 on Hudson
ERROR: Failed to install http://download.eclipse.org/tools/buckminster/products/direc tor_latest.zip to D:\Documents and Settings\MyUserName\.hudson\tools\Buckminster_3.6
Finished: FAILURE

Turned out this was caused by a missing proxy configuration. I already had a proxy set up in the Plugin Manager's Advanced section, installing and updating plugins works like a charm. So I

* restarted hudson using -DproxySet -DproxyHost...

which helped with the director download. I never used the auto installation before, so I don't know if e.g. installing an SDK would work without these options. The plugin should probably make use of the proxy settings as they are configured in the plugin manager, shouldn't it?

However, after having unpacked the director, the next step fails:

Unpacking http://download.eclipse.org/tools/buckminster/products/direc tor_latest.zip to D:\Documents and Settings\MyUserName\.hudson\tools\Buckminster_3.6 on Hudson
[buckminster] $ sh -e "D:\Documents and Settings\MyUserName\.hudson\tools\Buckminster_3.6\buckminste r\hudson7672730827943675296.sh "
D:\Documents and Settings\MyUserName\.hudson\tools\Buckminster_3.6\buckminste r\hudson7672730827943675296.sh: no such file or directory: \D:\Documents%20and%20Settings\MyUserName\.hudson\tools\Buck minster_3.6\director\\director [1]
ERROR: Command returned status 1
Finished: FAILURE

At first glance, I expected the sh call to fail as I'm under windows, but it turned out I actually have sh.exe on my PATH. This will be an issue for windows installations. Unfortunately the hudson7672730827943675296.sh does no longer exist, so I could not check what it actually tried to do. The %20 seems odd... it's always these blanks under windows...

The director directory exists as it was used above (if I replace "%20" by " "). If you replace sh by cmd.exe, the plugin will have to use director.bat (instead of director) to run the installation.

Just let me know if you need more information or another test run ;)

Regards,
Daniel
Re: Hudson plugin and provisioning [message #522905 is a reply to message #522889] Wed, 24 March 2010 13:14 Go to previous messageGo to next message
Johannes Utzig is currently offline Johannes UtzigFriend
Messages: 329
Registered: July 2009
Senior Member
Hi Daniel,

please see comments inline.

Am 24.03.2010 12:45, schrieb Daniel Weber:
> Hi Johannes,
>
> Johannes Utzig wrote:
>> [...]
>> If anybody reading this has a hudson running on a windows machine,
>
> I have :)
>

That's just great, thanks a lot :)

>
> which helped with the director download. I never used the auto
> installation before, so I don't know if e.g. installing an SDK would
> work without these options. The plugin should probably make use of the
> proxy settings as they are configured in the plugin manager, shouldn't it?
>

Well, yes, it should, but this is out of my hands. The installer extends
the DownloadFromURLInstaller which is a generic Hudson Installer. So if
it isn't respecting global proxy settings, it's a bug in Hudson Core.

>
> At first glance, I expected the sh call to fail as I'm under windows,
> but it turned out I actually have sh.exe on my PATH. This will be an
> issue for windows installations. Unfortunately the
> hudson7672730827943675296.sh does no longer exist, so I could not check
> what it actually tried to do. The %20 seems odd... it's always these
> blanks under windows...
>

Yes, Hudson has to ability to execute sh scripts on Windows machines, so
that is as expected.
Ahh, spaces in paths, you got to love those little trouble makers... :)
I'll fix that.

> The director directory exists as it was used above (if I replace "%20"
> by " "). If you replace sh by cmd.exe, the plugin will have to use
> director.bat (instead of director) to run the installation.
>

I'll stick with hudson's sh capabilities because there's a lot of
infrastructure already present for that, so 'director' should work I hope :)

> Just let me know if you need more information or another test run ;)
>

Thank you very much for your effort, I really appreciate it.
Thomas raised a bunch of issues/enhancements. I will try to implement
these as well as fixes for the issues you found tonight and hopefully
release a new version right away.
Once it's out, I'd be happy if you'd try again and let me know how it
worked.

Best regards,
Johannes
Re: Hudson plugin and provisioning [message #523025 is a reply to message #522905] Wed, 24 March 2010 19:05 Go to previous messageGo to next message
Johannes Utzig is currently offline Johannes UtzigFriend
Messages: 329
Registered: July 2009
Senior Member
>
> Thank you very much for your effort, I really appreciate it.
> Thomas raised a bunch of issues/enhancements. I will try to implement
> these as well as fixes for the issues you found tonight and hopefully
> release a new version right away.
> Once it's out, I'd be happy if you'd try again and let me know how it
> worked.
>

All pending issues have been implemented and spaces in the path should
no longer become URI encoded.
So if you want to give it another shot, 0.9.5 should be available in the
update center in a few hours.

Best Regards,
Johannes
Re: Hudson plugin and provisioning [message #523026 is a reply to message #522793] Wed, 24 March 2010 19:08 Go to previous messageGo to next message
Johannes Utzig is currently offline Johannes UtzigFriend
Messages: 329
Registered: July 2009
Senior Member
I implemented the new configuration options and fixed the double slash
issue. 0.9.5 has beed released and should find its way into the update
center soon.
Let me know if the new configuration options meet your requirements.

Thanks for your feedback and best regards,
Johannes
Re: Hudson plugin and provisioning [message #523045 is a reply to message #523026] Wed, 24 March 2010 20:51 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
Johannes,
Thanks for your very quick response to my enhancement requests. I'll try this out first thing tomorrow.

Regards,
Thomas Hallgren

On 03/24/2010 08:08 PM, Johannes Utzig wrote:
> I implemented the new configuration options and fixed the double slash
> issue. 0.9.5 has beed released and should find its way into the update
> center soon.
> Let me know if the new configuration options meet your requirements.
>
> Thanks for your feedback and best regards,
> Johannes
Re: Hudson plugin and provisioning [message #523103 is a reply to message #523025] Thu, 25 March 2010 05:44 Go to previous messageGo to next message
Daniel Weber is currently offline Daniel WeberFriend
Messages: 51
Registered: July 2009
Member
Hi Johannes,

> All pending issues have been implemented and spaces in the path should
> no longer become URI encoded.
> So if you want to give it another shot, 0.9.5 should be available in the
> update center in a few hours.

thanks for addressing this so quickly :)

The new version did not show up yet, I'll give it a try as soon as it is available.
The proxy issue I encountered has already been reported, see http://issues.hudson-ci.org/browse/HUDSON-5271

Cheers,
Daniel
Re: Hudson plugin and provisioning [message #523165 is a reply to message #523026] Thu, 25 March 2010 07:30 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas HallgrenFriend
Messages: 3240
Registered: July 2009
Senior Member
On 03/24/2010 08:08 PM, Johannes Utzig wrote:
> ... 0.9.5 has beed released and should find its way into the update
> center soon.

It hasn't done so yet. How long does this usually take?

Regards,
Thomas Hallgren
Re: Hudson plugin and provisioning [message #523171 is a reply to message #523165] Thu, 25 March 2010 12:24 Go to previous messageGo to next message
Johannes Utzig is currently offline Johannes UtzigFriend
Messages: 329
Registered: July 2009
Senior Member
>
> It hasn't done so yet. How long does this usually take?
>

It takes unusual long at the moment. I checked the Hudson release feed:
https://hudson.dev.java.net/servlets/ProjectRSS?type=news
Looks like the last one that made it through is the Nested View Plugin
which was released on the 22th.
Might be a server issue, I'm not sure. Usually the update center should
pick up the new versions no later than 24 hours after the release.
All plugins released after the 22th are still not available in the
update center (and the wiki information hasn't been updated either).
I'll give it a few more hours and if it doesn't work I ask in the dev
mailing list.
Re: Hudson plugin and provisioning [message #524465 is a reply to message #523103] Thu, 01 April 2010 00:00 Go to previous messageGo to next message
Daniel Weber is currently offline Daniel WeberFriend
Messages: 51
Registered: July 2009
Member
This is a multi-part message in MIME format.
--------------070400000902030306050605
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Hi again,

installation still fails under windows using 0.9.5, some files could not be found.

> Yes, Hudson has to ability to execute sh scripts on Windows machines, so that is as expected.

Hm, I watched hud-son spawning sh.exe from where it was found in PATH, there did not seem to be any special treatment for sh scripts. Is this really expected to work out-ofthe-box?

I grabbed the sources and tried to debug and fix the installation issues under windows.

Find attached a patch that implements the following changes: The installation mechanism no longer generates and runs a shell script but spawns processes for the necessary installation steps (i.e. use the directoy to install buckminster and then call buckminster itself to add additional features). The only OS-specific part is now the selection of the respective start scripts (director vs. director.bat, same for buckminster), the rest is the same for all OSes. Because the plugin now starts the processes directly and checks their return values, the installation will also fail if any of the installation commands fails. This did not work in my first try (where I created and ran a .bat file to perform the installation).

The patch has been created against SVN Revision 29269 and also includes a small fix in EclipseBuckminsterBuilder (ran into an exception on the configuration page).

One more thing I noticed: after updating to 0.9.5, the drop down box on the auto install configuration page is empty unless userContent/buckminster/buckminster.json exists. A default installer configuration seems to be missing? Or maybe something went wrong during my update.

Regards,
Daniel


Daniel Weber wrote:
> Hi Johannes,
>
>> All pending issues have been implemented and spaces in the path should
>> no longer become URI encoded.
>> So if you want to give it another shot, 0.9.5 should be available in
>> the update center in a few hours.
>
> thanks for addressing this so quickly :)
>
> The new version did not show up yet, I'll give it a try as soon as it is
> available.
> The proxy issue I encountered has already been reported, see
> http://issues.hudson-ci.org/browse/HUDSON-5271
>
> Cheers,
> Daniel
>
>
>


--------------070400000902030306050605
Content-Type: text/plain;
name="installation-windows.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="installation-windows.patch"

diff --git src/main/java/hudson/plugins/buckminster/BuckminsterInstalla tion.java src/main/java/hudson/plugins/buckminster/BuckminsterInstalla tion.java
index 24165bf..98ed23f 100644
--- src/main/java/hudson/plugins/buckminster/BuckminsterInstalla tion.java
+++ src/main/java/hudson/plugins/buckminster/BuckminsterInstalla tion.java
@@ -6,6 +6,7 @@ package hudson.plugins.buckminster;
import hudson.EnvVars;
import hudson.Extension;
import hudson.FilePath;
+import hudson.Launcher.ProcStarter;
import hudson.model.EnvironmentSpecific;
import hudson.model.Hudson;
import hudson.model.Node;
@@ -117,37 +118,40 @@ public class BuckminsterInstallation extends ToolInstallation implements Environ
throws IOException, InterruptedException {
return null;
}
-
- @Override
- public FilePath performInstallation(ToolInstallation tool, Node node,
- TaskListener log) throws IOException, InterruptedException {
- FilePath director = super.performInstallation(tool, node, log);
- FilePath buckminsterDir = director.child("buckminster");
- if(buckminsterDir.exists())
- {
- FilePath executableWin = buckminsterDir.child("buckminster.bat");
- FilePath executableUnix = buckminsterDir.child("buckminster");
- if(executableUnix.exists() || executableWin.exists())
- // here we could do an update...
- return buckminsterDir;
- }
-// FilePath expected = preferredLocation(tool, node);
- BuckminsterInstallable inst = (BuckminsterInstallable) getInstallable();
- String command = CommandLineBuilder.createInstallScript(inst, director, node, log);
- FilePath script = buckminsterDir.createTextTempFile("hudson", ".sh", command);
- try {
- String[] cmd = {"sh", "-e", script.getRemote()};
- int r = node.createLauncher(log).launch().cmds(cmd).stdout(log).pwd( buckminsterDir).join();
- if (r != 0) {
- throw new IOException("Command returned status " + r);
- }
- } finally {
- script.delete();
- }
- return buckminsterDir;
-
- }
-
+
+ @Override
+ public FilePath performInstallation(ToolInstallation tool, Node node,
+ TaskListener log) throws IOException, InterruptedException
+ {
+ FilePath director = super.performInstallation(tool, node, log);
+ FilePath buckminsterDir = director.child("buckminster");
+ if(buckminsterDir.exists())
+ {
+ FilePath executableWin = buckminsterDir.child("buckminster.bat");
+ FilePath executableUnix = buckminsterDir.child("buckminster");
+ if(executableUnix.exists() || executableWin.exists())
+ // here we could do an update...
+ return buckminsterDir;
+ }
+ else
+ {
+ // Must exist so that it can be used as PWD
+ buckminsterDir.mkdirs();
+ }
+ // FilePath expected = preferredLocation(tool, node);
+ BuckminsterInstallable inst = (BuckminsterInstallable)getInstallable();
+ for(ProcStarter starter : CommandLineBuilder.createInstallationCommandStarters(
+ inst, director, node, log))
+ {
+ int r = starter.pwd(buckminsterDir).join();
+ if(r != 0)
+ {
+ throw new IOException("Command returned status " + r);
+ }
+ }
+ return buckminsterDir;
+ }
+
@Extension
public static final class DescriptorImpl extends
DownloadFromUrlInstaller.DescriptorImpl<BuckminsterInstaller > {
diff --git src/main/java/hudson/plugins/buckminster/EclipseBuckminsterB uilder.java src/main/java/hudson/plugins/buckminster/EclipseBuckminsterB uilder.java
index 6cdb47b..06be49d 100644
--- src/main/java/hudson/plugins/buckminster/EclipseBuckminsterB uilder.java
+++ src/main/java/hudson/plugins/buckminster/EclipseBuckminsterB uilder.java
@@ -318,7 +318,7 @@ public class EclipseBuckminsterBuilder extends Builder {
FilePath path = project.getSomeWorkspace();
if(path==null)
//we tried, but we couldn't get a workspace
- FormValidation.ok();
+ return FormValidation.ok();
return path.validateRelativePath(value, true, true);
}

diff --git src/main/java/hudson/plugins/buckminster/command/CommandLine Builder.java src/main/java/hudson/plugins/buckminster/command/CommandLine Builder.java
index f2d07ee..28301a5 100644
--- src/main/java/hudson/plugins/buckminster/command/CommandLine Builder.java
+++ src/main/java/hudson/plugins/buckminster/command/CommandLine Builder.java
@@ -1,6 +1,8 @@
package hudson.plugins.buckminster.command;

import hudson.FilePath;
+import hudson.Functions;
+import hudson.Launcher.ProcStarter;
import hudson.model.AbstractBuild;
import hudson.model.BuildListener;
import hudson.model.Computer;
@@ -297,42 +299,64 @@ public class CommandLineBuilder {
}
throw new FileNotFoundException("No equinox launcher jar has been found in "+pluginDir.getCanonicalPath());
}
-
- public static String createInstallScript(BuckminsterInstallable installable, FilePath toolDir, Node node, TaskListener log) throws MalformedURLException, IOException, InterruptedException
- {
- FilePath directorDir = toolDir.child("director");
- String directorDirPath = directorDir.absolutize().toURI().getPath();
- FilePath buckyDir = toolDir.child("buckminster");
- String buckyDirPath = buckyDir.absolutize().toURI().getPath();
- List<JDK> jdks = Hudson.getInstance().getJDKs();
- String vmArgument = "";
- if(jdks!=null && jdks.size()>0)
- {
- JDK jdk = Hudson.getInstance().getJDKs().get(0);
- jdk = jdk.forNode(node, log);
- jdk = jdk.forEnvironment(Computer.currentComputer().getEnvironment ());
- File javaBinDir = jdk.getBinDir();
- File javaExecutable = new File(javaBinDir,"java");
- vmArgument = "-vm "+"\""+javaExecutable.getCanonicalPath()+"\"";
- }

- //TODO: put IU to JSON
- String command = "{0}/director {1} -r \"{2}\" -d \"{3}\" -p Buckminster -i \"{4}\"";
- command = MessageFormat.format(command, directorDirPath,vmArgument ,installable.repositoryURL, buckyDirPath, installable.iu);
- StringBuilder builder = new StringBuilder(command);
- for (Repository repo : installable.repositories) {
-
- for (Feature feature : repo.features) {
- builder.append("\n");
- command = "{0}/buckminster {1} install {2} {3}";
- command = MessageFormat.format(command, buckyDirPath,vmArgument, repo.url,feature.id);
- builder.append(command);
- builder.append("\n");
- command = "echo \"Installed Feature {0} from {1}\"";
- command = MessageFormat.format(command, feature.id, repo.url);
- builder.append(command);
- }
- }
- return builder.toString();
- }
-}
\ No newline at end of file
+ public static List<ProcStarter> createInstallationCommandStarters(
+ BuckminsterInstallable installable, FilePath toolDir, Node node, TaskListener log)
+ throws IOException, InterruptedException
+ {
+ FilePath directorDir = toolDir.child("director");
+ FilePath buckyDir = toolDir.child("buckminster");
+ String buckyDirPath = buckyDir.absolutize().getRemote();
+ List<JDK> jdks = Hudson.getInstance().getJDKs();
+ String vmArgument = "";
+ if(jdks != null && jdks.size() > 0)
+ {
+ JDK jdk = Hudson.getInstance().getJDKs().get(0);
+ jdk = jdk.forNode(node, log);
+ jdk = jdk.forEnvironment(Computer.currentComputer().getEnvironment ());
+ File javaBinDir = jdk.getBinDir();
+ File javaExecutable = new File(javaBinDir, "java");
+ vmArgument = "-vm " + "\"" + javaExecutable.getCanonicalPath() + "\"";
+ }
+
+ List<ProcStarter> cmds = new ArrayList<ProcStarter>();
+ // TODO: put IU to JSON
+ final String director = directorDir.child(directorScriptName()).absolutize()
+ .getRemote();
+ String[] command = { director, vmArgument, "-r",
+ quote(installable.repositoryURL), "-d", quote(buckyDirPath), "-p",
+ "Buckminster", "-i", quote(installable.iu) };
+ cmds.add(node.createLauncher(log).launch().cmds(command).std out(log));
+ final String bucky = buckyDir.child(buckyScriptName()).absolutize().getRemote();
+ for(Repository repo : installable.repositories)
+ {
+ for(Feature feature : repo.features)
+ {
+ String installFeatureCmd[] = { bucky, vmArgument, "install", repo.url,
+ feature.id };
+ cmds.add(node.createLauncher(log).launch().cmds(installFeatu reCmd)
+ .stdout(log));
+ }
+ }
+ return cmds;
+ }
+
+ private static String buckyScriptName()
+ {
+ if(Functions.isWindows())
+ return "buckminster.bat";
+ return "buckminster";
+ }
+
+ private static String directorScriptName()
+ {
+ if(Functions.isWindows())
+ return "director.bat";
+ return "director";
+ }
+
+ private static String quote(final String original)
+ {
+ return "\"" + original + "\"";
+ }
+}

--------------070400000902030306050605--
Re: Hudson plugin and provisioning [message #524496 is a reply to message #524465] Thu, 01 April 2010 07:39 Go to previous messageGo to next message
Johannes Utzig is currently offline Johannes UtzigFriend
Messages: 329
Registered: July 2009
Senior Member
Hi Daniel,

please see comments inline.

Am 01.04.2010 06:45, schrieb Daniel Weber:
> Hi again,
>
> installation still fails under windows using 0.9.5, some files could not
> be found.
>
>> Yes, Hudson has to ability to execute sh scripts on Windows machines,
>> so that is as expected.
>
> Hm, I watched hud-son spawning sh.exe from where it was found in PATH,
> there did not seem to be any special treatment for sh scripts. Is this
> really expected to work out-ofthe-box?
>

I noticed that by now, my mistake, I thought they have some special
treatment because the 'execute shell script' build step allowed the user
to use unix commands even on a windows box, but it seems like it relies
on an sh.exe beeing in the path or configured in Hudson...


> I grabbed the sources and tried to debug and fix the installation issues
> under windows.
> Find attached a patch that implements the following changes: The
> installation mechanism no longer generates and runs a shell script but
> spawns processes for the necessary installation steps (i.e. use the
> directoy to install buckminster and then call buckminster itself to add
> additional features). The only OS-specific part is now the selection of
> the respective start scripts (director vs. director.bat, same for
> buckminster), the rest is the same for all OSes. Because the plugin now
> starts the processes directly and checks their return values, the
> installation will also fail if any of the installation commands fails.
> This did not work in my first try (where I created and ran a .bat file
> to perform the installation).

First of all, thank you very much for your effort on this. As a matter
of fact, I went down the same road just yesterday:
http://www.eclipse.org/forums/index.php?t=msg&th=165216& amp;start=0&
The installation and update is no longer done with an sh script,
everything is executed as a single director invocation.
However, as you can see in the given Thread there's still a slight
problem to make it run on both Windows and Linux.
But I think this line of your patch should do the trick:
+ final String director =
directorDir.child(directorScriptName()).absolutize()
+ .getRemote();

The problem in the thread comes from relativ paths, if I go back to
absolute ones, the command line gets longer, but it should work then (if
I add the .bat extension for windows like you did).
I will make the necessary changes tonight and send Flavio another
version to test on windows.
I'll publish version 1.0.1 as soon it's confirmed that it works on
Windows, Linux and Master/Slave.

>
> The patch has been created against SVN Revision 29269 and also includes
> a small fix in EclipseBuckminsterBuilder (ran into an exception on the
> configuration page).
>

That bug is fixed already in version 1.0.0, but thank you anyway :)

> One more thing I noticed: after updating to 0.9.5, the drop down box on
> the auto install configuration page is empty unless
> userContent/buckminster/buckminster.json exists. A default installer
> configuration seems to be missing? Or maybe something went wrong during
> my update.
>

Is it possible that it's a few days ago that you checked? If so, it's
probably the same as here:
http://issues.hudson-ci.org/browse/HUDSON-6085
I made a change to the json file one of these days and introduced a
syntax error that prevented Hudson from parsing (and downloading) it
properly :-/

Best regards,
Johannes
Re: Hudson plugin and provisioning [message #524516 is a reply to message #524496] Thu, 01 April 2010 08:34 Go to previous messageGo to next message
Daniel Weber is currently offline Daniel WeberFriend
Messages: 51
Registered: July 2009
Member
Hi Johannes,

> First of all, thank you very much for your effort on this.

You're welcome, I'm glad not everything was useless :)
I started setting up a development environment and "hacking around" last week and did not sync since then.

It might well be that the remaining issues have been solved in 1.0.0, I'll check.

Regards,
Daniel

Johannes Utzig wrote:
> Hi Daniel,
>
> please see comments inline.
>
> Am 01.04.2010 06:45, schrieb Daniel Weber:
>> Hi again,
>>
>> installation still fails under windows using 0.9.5, some files could not
>> be found.
>>
>>> Yes, Hudson has to ability to execute sh scripts on Windows machines,
>>> so that is as expected.
>>
>> Hm, I watched hud-son spawning sh.exe from where it was found in PATH,
>> there did not seem to be any special treatment for sh scripts. Is this
>> really expected to work out-ofthe-box?
>>
>
> I noticed that by now, my mistake, I thought they have some special
> treatment because the 'execute shell script' build step allowed the user
> to use unix commands even on a windows box, but it seems like it relies
> on an sh.exe beeing in the path or configured in Hudson...
>
>
>> I grabbed the sources and tried to debug and fix the installation issues
>> under windows.
>> Find attached a patch that implements the following changes: The
>> installation mechanism no longer generates and runs a shell script but
>> spawns processes for the necessary installation steps (i.e. use the
>> directoy to install buckminster and then call buckminster itself to add
>> additional features). The only OS-specific part is now the selection of
>> the respective start scripts (director vs. director.bat, same for
>> buckminster), the rest is the same for all OSes. Because the plugin now
>> starts the processes directly and checks their return values, the
>> installation will also fail if any of the installation commands fails.
>> This did not work in my first try (where I created and ran a .bat file
>> to perform the installation).
>
> First of all, thank you very much for your effort on this. As a matter
> of fact, I went down the same road just yesterday:
> http://www.eclipse.org/forums/index.php?t=msg&th=165216& amp;start=0&
> The installation and update is no longer done with an sh script,
> everything is executed as a single director invocation.
> However, as you can see in the given Thread there's still a slight
> problem to make it run on both Windows and Linux.
> But I think this line of your patch should do the trick:
> + final String director =
> directorDir.child(directorScriptName()).absolutize()
> + .getRemote();
>
> The problem in the thread comes from relativ paths, if I go back to
> absolute ones, the command line gets longer, but it should work then (if
> I add the .bat extension for windows like you did).
> I will make the necessary changes tonight and send Flavio another
> version to test on windows.
> I'll publish version 1.0.1 as soon it's confirmed that it works on
> Windows, Linux and Master/Slave.
>
>>
>> The patch has been created against SVN Revision 29269 and also includes
>> a small fix in EclipseBuckminsterBuilder (ran into an exception on the
>> configuration page).
>>
>
> That bug is fixed already in version 1.0.0, but thank you anyway :)
>
>> One more thing I noticed: after updating to 0.9.5, the drop down box on
>> the auto install configuration page is empty unless
>> userContent/buckminster/buckminster.json exists. A default installer
>> configuration seems to be missing? Or maybe something went wrong during
>> my update.
>>
>
> Is it possible that it's a few days ago that you checked? If so, it's
> probably the same as here:
> http://issues.hudson-ci.org/browse/HUDSON-6085
> I made a change to the json file one of these days and introduced a
> syntax error that prevented Hudson from parsing (and downloading) it
> properly :-/
>
> Best regards,
> Johannes
>
>
Re: Hudson plugin and provisioning [message #524752 is a reply to message #524516] Fri, 02 April 2010 09:56 Go to previous messageGo to next message
Johannes Utzig is currently offline Johannes UtzigFriend
Messages: 329
Registered: July 2009
Senior Member
Hi Flavio,

I installed Windows on a virtual box and configured it as a hudson slave
to finally have my own testing environment for that.
Everything seems to work fine now on both the Linux host and the Windows
slave (no cygwin installed). I've just released version 1.0.1 that
contains the fixes. I hope it works better for you now.
The new version will show up as usual with a few days delay in your
update center.

Thank you for all the help.

Best regards,
Johannes

Hi Daniel,

I just replied to Flavio in the other thread in the buckminster newsgroup:

"I installed Windows on a virtual box and configured it as a hudson
slave to finally have my own testing environment for that.
Everything seems to work fine now on both the Linux host and the Windows
slave (no cygwin installed). I've just released version 1.0.1 that
contains the fixes. I hope it works better for you now.
The new version will show up as usual with a few days delay in your
update center."

Same applies to you of course, thank you very much for the patch you
sent me, the absolute paths finally did the trick.
I had to change a few things though (in case you are interested):
The directorScriptName and buckyScriptName do not work like that in a
distributed build (Master/Slave).
The code is always executed on the master, therefore the methods would
only return the correct script name if Master and Slave run the same
operating system.
The trick was to make this kind of invocation over a VirtualChannel to
have it executed on the current Build machine.
Other than that I took a similar approach like you in the patch you
submitted, feel free to have a look at the code changes.

Best regards,
Johannes
Re: Hudson plugin and provisioning [message #525588 is a reply to message #524752] Wed, 07 April 2010 04:04 Go to previous messageGo to next message
Daniel Weber is currently offline Daniel WeberFriend
Messages: 51
Registered: July 2009
Member
Hi Johannes,

glad I could help :) And thanks for the explanations re Master/Slave issues. This was my first encounter with a piece of hudson code. Up to now I just used it.

Which Windows version does your Virtual box use? The sh problem is solved in 1.0.1, but now I'm having trouble with the director/buckminster.bat files on my WinXP32 machine.

* Parsing the vmargs does not work. The loop in the batch file splits -Dbuckminster.output.root=D:\hudson in two seperate args, omitting the '=':

"...\bin\java.exe" -Dbuckminster.output.root D:\hudson ...

The same happens when I copy the original command line to a "DOS Box". If I add "" around the -D args, it works:

"...\buckminster.bat" ... -vmargs "-Dbuckminster.output.root=D:\hudson"

This might depend on the windows version being used, I don't know.

* The batch files (director/buckminster) have trouble when being installed in a directory with blanks (it's them again... ;)). But there's nothing your plugin can do about that. I'll see if I can find an Eclipse bug for this or file a new one. It's this for statement that does not work if %EXE_DIR% contains blanks:

for /f "delims= tokens=1" %%c in ('dir /B /S /OD %EXE_DIR%\plugins\org.eclipse.equinox.launcher_*.jar') do set EQUINOXJAR=%%c

This works:

for /f "delims= tokens=1" %%c in ('dir /B /S /OD "%EXE_DIR%\plugins\org.eclipse.equinox.launcher_*.jar"') do set EQUINOXJAR=%%c

Regards,
Daniel

Johannes Utzig wrote:
> Hi Flavio,
>
> I installed Windows on a virtual box and configured it as a hudson slave
> to finally have my own testing environment for that.
> Everything seems to work fine now on both the Linux host and the Windows
> slave (no cygwin installed). I've just released version 1.0.1 that
> contains the fixes. I hope it works better for you now.
> The new version will show up as usual with a few days delay in your
> update center.
>
> Thank you for all the help.
>
> Best regards,
> Johannes
>
> Hi Daniel,
>
> I just replied to Flavio in the other thread in the buckminster newsgroup:
>
> "I installed Windows on a virtual box and configured it as a hudson
> slave to finally have my own testing environment for that.
> Everything seems to work fine now on both the Linux host and the Windows
> slave (no cygwin installed). I've just released version 1.0.1 that
> contains the fixes. I hope it works better for you now.
> The new version will show up as usual with a few days delay in your
> update center."
>
> Same applies to you of course, thank you very much for the patch you
> sent me, the absolute paths finally did the trick.
> I had to change a few things though (in case you are interested):
> The directorScriptName and buckyScriptName do not work like that in a
> distributed build (Master/Slave).
> The code is always executed on the master, therefore the methods would
> only return the correct script name if Master and Slave run the same
> operating system.
> The trick was to make this kind of invocation over a VirtualChannel to
> have it executed on the current Build machine.
> Other than that I took a similar approach like you in the patch you
> submitted, feel free to have a look at the code changes.
>
> Best regards,
> Johannes
>
Re: Hudson plugin and provisioning [message #525594 is a reply to message #525588] Wed, 07 April 2010 05:44 Go to previous message
Johannes Utzig is currently offline Johannes UtzigFriend
Messages: 329
Registered: July 2009
Senior Member
Hi Daniel,

please see comments inline.


>
> * Parsing the vmargs does not work. The loop in the batch file splits
> -Dbuckminster.output.root=D:\hudson in two seperate args, omitting the '=':
>
> "...\bin\java.exe" -Dbuckminster.output.root D:\hudson ...
>
> The same happens when I copy the original command line to a "DOS
> Box". If I add "" around the -D args, it works:
> "...\buckminster.bat" ... -vmargs "-Dbuckminster.output.root=D:\hudson"
>
> This might depend on the windows version being used, I don't know.
>

Yes, I've noticed that unfortunately. See here for further references:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=287575
My Windows Slave is a plain 32 bit XP, the reason it worked for me is
something different:
My Windows Slave contained spaces in the paths (because I wanted to test
it well :P). Because of these spaces all arguments got quoted with ""
and that's why the bug didn't occur to me.
Yesterday I committed a fix for that. Instead of the buckminster(.bat)
the plugin uses the equinox launcher again. Version 1.0.2 has been
released yesterday and is hopefully available soon. Sorry for the
trouble and thank you for the feedback.


> * The batch files (director/buckminster) have trouble when being
> installed in a directory with blanks (it's them again... ;)). But
> there's nothing your plugin can do about that. I'll see if I can find an
> Eclipse bug for this or file a new one. It's this for statement that
> does not work if %EXE_DIR% contains blanks:
>
> for /f "delims= tokens=1" %%c in ('dir /B /S /OD
> %EXE_DIR%\plugins\org.eclipse.equinox.launcher_*.jar') do set
> EQUINOXJAR=%%c
>
> This works:
>
> for /f "delims= tokens=1" %%c in ('dir /B /S /OD
> "%EXE_DIR%\plugins\org.eclipse.equinox.launcher_*.jar"') do set
> EQUINOXJAR=%%c
>

I noticed that a while ago and already fixed it in TRUNK:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=303262
https://bugs.eclipse.org/bugs/show_bug.cgi?id=307978

Best regards,
Johannes
Previous Topic:running a buckminster build with Hudson at Eclipse
Next Topic:Can Buckminster build eclipse cdt?
Goto Forum:
  


Current Time: Fri Jun 22 13:32:39 GMT 2018

Powered by FUDForum. Page generated in 0.03994 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top