| headless build - problems with tmp folder [message #48009] |
Wed, 25 February 2009 14:56  |
Eclipse User |
|
|
|
Originally posted by: birt.lewold.at
while executing a headless build the eclipse build process creates several temp files in
${buildDirectory}/tmp. These are used during internal ant tasks like "jarUp, jarIng, ..."
We have like 20 plugins - some relating to each other, some not. They are checked out into
${buildDirectory}/plugins (as expected by the headleass build process).
I have the following problem: Our C.I. server sometimes builds the plugins in parallel
(depending on how many executors are available, and of course the plugins must not relate
to each other).
This leads to the situation, that the first plugin build finishing cleans the tmp
directory, and the other build running in parallel fails.
I think the main problem is to use the same tmp folder for different builds, and to delete
it completely when finishing, thereby killing other builds running in parallel.
I have 2 simple questions (where either of them could solve my problem):
Q1) We are still on Eclipse 3.3 (both for build and runtime environment). Did this
behavior change with Eclipse 3.4 - afaik. I could build 3.3 deliverables with eclipse 3.4.
Q2) Is there a way to specify the "tmp" folder used by the internal ant tasks?
Q3) .... well if !Q1 && !Q2 .... do you have another hint?
Thanks,
Chris
|
|
|
| Re: headless build - problems with tmp folder [message #48100 is a reply to message #48009] |
Wed, 25 February 2009 16:48  |
| Eclipse User |
|
|
|
Originally posted by: birt.lewold.at
Here some more investigations:
the headless build creates and scripts named assemble.PLUGINNAME.win32.win32.x86.xml ....
those files contain an entry
<property name="assemblyTempDir" value="${buildDirectory}/tmp"/>
At the end of the script there is a
<delete dir="${assemblyTempDir}"/>
This is quite deadly, if two such scripts execute in parallel.
Such parallel execution however is quite common when using continous integration systems.
Way better would have been (for example) .....
<property name="assemblyTempDir" value="${buildDirectory}/tmp_PLUGINNAME"/>
Chris
Chris wrote:
> while executing a headless build the eclipse build process creates
> several temp files in ${buildDirectory}/tmp. These are used during
> internal ant tasks like "jarUp, jarIng, ..."
>
> We have like 20 plugins - some relating to each other, some not. They
> are checked out into
> ${buildDirectory}/plugins (as expected by the headleass build process).
>
> I have the following problem: Our C.I. server sometimes builds the
> plugins in parallel (depending on how many executors are available, and
> of course the plugins must not relate to each other).
>
>
> This leads to the situation, that the first plugin build finishing
> cleans the tmp directory, and the other build running in parallel fails.
>
>
> I think the main problem is to use the same tmp folder for different
> builds, and to delete it completely when finishing, thereby killing
> other builds running in parallel.
>
>
> I have 2 simple questions (where either of them could solve my problem):
>
> Q1) We are still on Eclipse 3.3 (both for build and runtime
> environment). Did this behavior change with Eclipse 3.4 - afaik. I could
> build 3.3 deliverables with eclipse 3.4.
>
> Q2) Is there a way to specify the "tmp" folder used by the internal ant
> tasks?
>
> Q3) .... well if !Q1 && !Q2 .... do you have another hint?
>
>
> Thanks,
> Chris
|
|
|
|
| Re: headless build - problems with tmp folder [message #48395 is a reply to message #48246] |
Wed, 25 February 2009 22:49 |
| Eclipse User |
|
|
|
Originally posted by: birt.lewold.at
I am NOT running multiple builds on the same files.
I am running two builds on two totally different features.
We have the following feature/plugin folder structure (we don't use this
structure because we like it that way - we use it as the eclipse build
process forces us to use this structure)....
/buildDirectry/
/plugins
/pluginA
/pluginB
/pluginC
/features
/featureA
/featureB
/featureC
Btw - we have about 25 such plugins right now and we build them using a
continous build process (controlled by Hudson). Some of the features
relate to each other, others don't.
Now assume the build server detects changes on pluginA and pluginB at
the same time. (not too uncommon, as it polls our SCM in the same
intervall for various projects).
So the build server triggers build of featureA and featureB. They DON'T
relate to each other, have no common files or whatever
Now the eclipse build process creates a folder /buildDirectory/tmp - it
places tmp files for both builds in this directory.
Assume featureA build finishes earlier - it wipes tmp .... peng -
featureB build is killed.
.... please don't tell me this is working as intended ....
Btw - the solution would be very simple. The assemblyTempDir property
defined in ALL the generated assemble.feature.xml files just would have
to be unique ... that shouldn't be a difficult thing - it's just hidden
by the eclipse build process, and out of the developers control.
Chris
|
|
|
|
| Re: headless build - problems with tmp folder [message #49045 is a reply to message #48554] |
Wed, 04 March 2009 15:13 |
| Eclipse User |
|
|
|
Originally posted by: birt.lewold.at
Hello Chris,
thanks for your response, sorry to say that I can't agree with this concept.
There are several reasons ....
a) what you are telling me is, that I would have to change the buildDirectory regularly -
depending on plugin dependency. On (not even very) large projects with 40+ plugins /
features this is simply not possible.
b) The build process already takes care of many things needed to build multiple features
within one buildDirectory. e.g.
b1) various temp folders are created right underneath the plugin directory, where they
don't conflicht
b2) the generated assemble and package scripts contain the feature name in order to avoid
conflicts etc.
The only (afaik) situation where there is a collision is at the end, when JARing up the
archives. The solution would be very simple - again use a tmp folder under e.g. the
feature directory, or create a tmp folder containing the feature name (similar to the
assemble script names).
This would allow to run multiple plugin builds of plugins belonging to the same project to
be executed in parallel.
Chris
Chris Williams wrote:
> Chris wrote:
>
>> I am NOT running multiple builds on the same files.
>> I am running two builds on two totally different features.
>
>> We have the following feature/plugin folder structure (we don't use
>> this structure because we like it that way - we use it as the eclipse
>> build process forces us to use this structure)....
>
>> /buildDirectry/
>> /plugins
>> /pluginA
>> /pluginB
>> /pluginC
>> /features
>> /featureA
>> /featureB
>> /featureC
>
> ..
>
>> So the build server triggers build of featureA and featureB. They
>> DON'T relate to each other, have no common files or whatever
>
> The issue here is that you're using a shared buildDirectory for what are
> separate builds. A buildDirectory is assumed by the build process to be
> a modifiable workspace that is devoted just to that single build. If
> featureA and featureB are not related, they should not use the same
> workspace for their builds. If there's no overlap in the contents of the
> features at all, then I suggest breaking out the structure to be more like:
>
> /featureAbuildDirectory
> /features
> /featureA
> /plugins
> /pluginA_1
> /pluginA_2
> /featureBbuildDirectory
> /features
> /featureB
> /plugins
> /pluginB_1
> /pluginB_2
>
>
> If there is some overlap between features (multiple use the same
> plugin), then retain your original structure, but have Hudson use
> separate workspaces for each build that are just copies of your
> buildDirectory. Or make Hudson use a lock for those builds that use the
> same buildDirectory to avoid having multiple builds trying to modify the
> same shared filesystem location.
>
|
|
|
| Re: headless build - problems with tmp folder [message #592348 is a reply to message #48009] |
Wed, 25 February 2009 16:48 |
 Chris
Messages: 97
Registered: July 2009 |
Member |
|
|
Here some more investigations:
the headless build creates and scripts named assemble.PLUGINNAME.win32.win32.x86.xml ....
those files contain an entry
<property name="assemblyTempDir" value="${buildDirectory}/tmp"/>
At the end of the script there is a
<delete dir="${assemblyTempDir}"/>
This is quite deadly, if two such scripts execute in parallel.
Such parallel execution however is quite common when using continous integration systems.
Way better would have been (for example) .....
<property name="assemblyTempDir" value="${buildDirectory}/tmp_PLUGINNAME"/>
Chris
Chris wrote:
> while executing a headless build the eclipse build process creates
> several temp files in ${buildDirectory}/tmp. These are used during
> internal ant tasks like "jarUp, jarIng, ..."
>
> We have like 20 plugins - some relating to each other, some not. They
> are checked out into
> ${buildDirectory}/plugins (as expected by the headleass build process).
>
> I have the following problem: Our C.I. server sometimes builds the
> plugins in parallel (depending on how many executors are available, and
> of course the plugins must not relate to each other).
>
>
> This leads to the situation, that the first plugin build finishing
> cleans the tmp directory, and the other build running in parallel fails.
>
>
> I think the main problem is to use the same tmp folder for different
> builds, and to delete it completely when finishing, thereby killing
> other builds running in parallel.
>
>
> I have 2 simple questions (where either of them could solve my problem):
>
> Q1) We are still on Eclipse 3.3 (both for build and runtime
> environment). Did this behavior change with Eclipse 3.4 - afaik. I could
> build 3.3 deliverables with eclipse 3.4.
>
> Q2) Is there a way to specify the "tmp" folder used by the internal ant
> tasks?
>
> Q3) .... well if !Q1 && !Q2 .... do you have another hint?
>
>
> Thanks,
> Chris
|
|
|
|
|
|
| Re: headless build - problems with tmp folder [message #592679 is a reply to message #48554] |
Wed, 04 March 2009 15:13 |
Chris
Messages: 97
Registered: July 2009 |
Member |
|
|
Hello Chris,
thanks for your response, sorry to say that I can't agree with this concept.
There are several reasons ....
a) what you are telling me is, that I would have to change the buildDirectory regularly -
depending on plugin dependency. On (not even very) large projects with 40+ plugins /
features this is simply not possible.
b) The build process already takes care of many things needed to build multiple features
within one buildDirectory. e.g.
b1) various temp folders are created right underneath the plugin directory, where they
don't conflicht
b2) the generated assemble and package scripts contain the feature name in order to avoid
conflicts etc.
The only (afaik) situation where there is a collision is at the end, when JARing up the
archives. The solution would be very simple - again use a tmp folder under e.g. the
feature directory, or create a tmp folder containing the feature name (similar to the
assemble script names).
This would allow to run multiple plugin builds of plugins belonging to the same project to
be executed in parallel.
Chris
Chris Williams wrote:
> Chris wrote:
>
>> I am NOT running multiple builds on the same files.
>> I am running two builds on two totally different features.
>
>> We have the following feature/plugin folder structure (we don't use
>> this structure because we like it that way - we use it as the eclipse
>> build process forces us to use this structure)....
>
>> /buildDirectry/
>> /plugins
>> /pluginA
>> /pluginB
>> /pluginC
>> /features
>> /featureA
>> /featureB
>> /featureC
>
> ..
>
>> So the build server triggers build of featureA and featureB. They
>> DON'T relate to each other, have no common files or whatever
>
> The issue here is that you're using a shared buildDirectory for what are
> separate builds. A buildDirectory is assumed by the build process to be
> a modifiable workspace that is devoted just to that single build. If
> featureA and featureB are not related, they should not use the same
> workspace for their builds. If there's no overlap in the contents of the
> features at all, then I suggest breaking out the structure to be more like:
>
> /featureAbuildDirectory
> /features
> /featureA
> /plugins
> /pluginA_1
> /pluginA_2
> /featureBbuildDirectory
> /features
> /featureB
> /plugins
> /pluginB_1
> /pluginB_2
>
>
> If there is some overlap between features (multiple use the same
> plugin), then retain your original structure, but have Hudson use
> separate workspaces for each build that are just copies of your
> buildDirectory. Or make Hudson use a lock for those builds that use the
> same buildDirectory to avoid having multiple builds trying to modify the
> same shared filesystem location.
>
|
|
|
Powered by
FUDForum. Page generated in 0.02567 seconds
] 
Search
Help
Register
Login
Home
