Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Plugin Development Environment (PDE) » headless build - problems with tmp folder
headless build - problems with tmp folder [message #48009] Wed, 25 February 2009 14:56 Go to next message
Eclipse UserFriend
Originally posted by: birt.lewold.at

while executing a headless build the eclipse build process creates several temp files in
${buildDirectory}/tmp. These are used during internal ant tasks like "jarUp, jarIng, ..."

We have like 20 plugins - some relating to each other, some not. They are checked out into
${buildDirectory}/plugins (as expected by the headleass build process).

I have the following problem: Our C.I. server sometimes builds the plugins in parallel
(depending on how many executors are available, and of course the plugins must not relate
to each other).


This leads to the situation, that the first plugin build finishing cleans the tmp
directory, and the other build running in parallel fails.


I think the main problem is to use the same tmp folder for different builds, and to delete
it completely when finishing, thereby killing other builds running in parallel.


I have 2 simple questions (where either of them could solve my problem):

Q1) We are still on Eclipse 3.3 (both for build and runtime environment). Did this
behavior change with Eclipse 3.4 - afaik. I could build 3.3 deliverables with eclipse 3.4.

Q2) Is there a way to specify the "tmp" folder used by the internal ant tasks?

Q3) .... well if !Q1 && !Q2 .... do you have another hint?


Thanks,
Chris
Re: headless build - problems with tmp folder [message #48100 is a reply to message #48009] Wed, 25 February 2009 16:48 Go to previous messageGo to next message
Eclipse UserFriend
Originally posted by: birt.lewold.at

Here some more investigations:

the headless build creates and scripts named assemble.PLUGINNAME.win32.win32.x86.xml ....
those files contain an entry

<property name="assemblyTempDir" value="${buildDirectory}/tmp"/>

At the end of the script there is a
<delete dir="${assemblyTempDir}"/>

This is quite deadly, if two such scripts execute in parallel.
Such parallel execution however is quite common when using continous integration systems.

Way better would have been (for example) .....
<property name="assemblyTempDir" value="${buildDirectory}/tmp_PLUGINNAME"/>

Chris


Chris wrote:
> while executing a headless build the eclipse build process creates
> several temp files in ${buildDirectory}/tmp. These are used during
> internal ant tasks like "jarUp, jarIng, ..."
>
> We have like 20 plugins - some relating to each other, some not. They
> are checked out into
> ${buildDirectory}/plugins (as expected by the headleass build process).
>
> I have the following problem: Our C.I. server sometimes builds the
> plugins in parallel (depending on how many executors are available, and
> of course the plugins must not relate to each other).
>
>
> This leads to the situation, that the first plugin build finishing
> cleans the tmp directory, and the other build running in parallel fails.
>
>
> I think the main problem is to use the same tmp folder for different
> builds, and to delete it completely when finishing, thereby killing
> other builds running in parallel.
>
>
> I have 2 simple questions (where either of them could solve my problem):
>
> Q1) We are still on Eclipse 3.3 (both for build and runtime
> environment). Did this behavior change with Eclipse 3.4 - afaik. I could
> build 3.3 deliverables with eclipse 3.4.
>
> Q2) Is there a way to specify the "tmp" folder used by the internal ant
> tasks?
>
> Q3) .... well if !Q1 && !Q2 .... do you have another hint?
>
>
> Thanks,
> Chris
Re: headless build - problems with tmp folder [message #48246 is a reply to message #48100] Wed, 25 February 2009 17:55 Go to previous messageGo to next message
Chris Williams is currently offline Chris WilliamsFriend
Messages: 29
Registered: July 2009
Junior Member
Why aren't your separate builds using separate workspaces/build
directories?

This behavior won't change in the build. It's going to assume your not
running multiple builds on the same files simultaneously - as it should.

If you have separate builds, they should each maintain their own
buildDirectory.
Re: headless build - problems with tmp folder [message #48395 is a reply to message #48246] Wed, 25 February 2009 22:49 Go to previous messageGo to next message
Eclipse UserFriend
Originally posted by: birt.lewold.at

I am NOT running multiple builds on the same files.
I am running two builds on two totally different features.

We have the following feature/plugin folder structure (we don't use this
structure because we like it that way - we use it as the eclipse build
process forces us to use this structure)....

/buildDirectry/
/plugins
/pluginA
/pluginB
/pluginC
/features
/featureA
/featureB
/featureC

Btw - we have about 25 such plugins right now and we build them using a
continous build process (controlled by Hudson). Some of the features
relate to each other, others don't.

Now assume the build server detects changes on pluginA and pluginB at
the same time. (not too uncommon, as it polls our SCM in the same
intervall for various projects).

So the build server triggers build of featureA and featureB. They DON'T
relate to each other, have no common files or whatever

Now the eclipse build process creates a folder /buildDirectory/tmp - it
places tmp files for both builds in this directory.

Assume featureA build finishes earlier - it wipes tmp .... peng -
featureB build is killed.

.... please don't tell me this is working as intended ....

Btw - the solution would be very simple. The assemblyTempDir property
defined in ALL the generated assemble.feature.xml files just would have
to be unique ... that shouldn't be a difficult thing - it's just hidden
by the eclipse build process, and out of the developers control.


Chris
Re: headless build - problems with tmp folder [message #48554 is a reply to message #48395] Fri, 27 February 2009 15:07 Go to previous messageGo to next message
Chris Williams is currently offline Chris WilliamsFriend
Messages: 29
Registered: July 2009
Junior Member
Chris wrote:

> I am NOT running multiple builds on the same files.
> I am running two builds on two totally different features.

> We have the following feature/plugin folder structure (we don't use this
> structure because we like it that way - we use it as the eclipse build
> process forces us to use this structure)....

> /buildDirectry/
> /plugins
> /pluginA
> /pluginB
> /pluginC
> /features
> /featureA
> /featureB
> /featureC

...

> So the build server triggers build of featureA and featureB. They DON'T
> relate to each other, have no common files or whatever

The issue here is that you're using a shared buildDirectory for what are
separate builds. A buildDirectory is assumed by the build process to be a
modifiable workspace that is devoted just to that single build. If
featureA and featureB are not related, they should not use the same
workspace for their builds. If there's no overlap in the contents of the
features at all, then I suggest breaking out the structure to be more like:

/featureAbuildDirectory
/features
/featureA
/plugins
/pluginA_1
/pluginA_2
/featureBbuildDirectory
/features
/featureB
/plugins
/pluginB_1
/pluginB_2


If there is some overlap between features (multiple use the same plugin),
then retain your original structure, but have Hudson use separate
workspaces for each build that are just copies of your buildDirectory. Or
make Hudson use a lock for those builds that use the same buildDirectory
to avoid having multiple builds trying to modify the same shared
filesystem location.
Re: headless build - problems with tmp folder [message #49045 is a reply to message #48554] Wed, 04 March 2009 15:13 Go to previous message
Eclipse UserFriend
Originally posted by: birt.lewold.at

Hello Chris,

thanks for your response, sorry to say that I can't agree with this concept.
There are several reasons ....

a) what you are telling me is, that I would have to change the buildDirectory regularly -
depending on plugin dependency. On (not even very) large projects with 40+ plugins /
features this is simply not possible.

b) The build process already takes care of many things needed to build multiple features
within one buildDirectory. e.g.
b1) various temp folders are created right underneath the plugin directory, where they
don't conflicht
b2) the generated assemble and package scripts contain the feature name in order to avoid
conflicts etc.

The only (afaik) situation where there is a collision is at the end, when JARing up the
archives. The solution would be very simple - again use a tmp folder under e.g. the
feature directory, or create a tmp folder containing the feature name (similar to the
assemble script names).

This would allow to run multiple plugin builds of plugins belonging to the same project to
be executed in parallel.


Chris




Chris Williams wrote:
> Chris wrote:
>
>> I am NOT running multiple builds on the same files.
>> I am running two builds on two totally different features.
>
>> We have the following feature/plugin folder structure (we don't use
>> this structure because we like it that way - we use it as the eclipse
>> build process forces us to use this structure)....
>
>> /buildDirectry/
>> /plugins
>> /pluginA
>> /pluginB
>> /pluginC
>> /features
>> /featureA
>> /featureB
>> /featureC
>
> ..
>
>> So the build server triggers build of featureA and featureB. They
>> DON'T relate to each other, have no common files or whatever
>
> The issue here is that you're using a shared buildDirectory for what are
> separate builds. A buildDirectory is assumed by the build process to be
> a modifiable workspace that is devoted just to that single build. If
> featureA and featureB are not related, they should not use the same
> workspace for their builds. If there's no overlap in the contents of the
> features at all, then I suggest breaking out the structure to be more like:
>
> /featureAbuildDirectory
> /features
> /featureA
> /plugins
> /pluginA_1
> /pluginA_2
> /featureBbuildDirectory
> /features
> /featureB
> /plugins
> /pluginB_1
> /pluginB_2
>
>
> If there is some overlap between features (multiple use the same
> plugin), then retain your original structure, but have Hudson use
> separate workspaces for each build that are just copies of your
> buildDirectory. Or make Hudson use a lock for those builds that use the
> same buildDirectory to avoid having multiple builds trying to modify the
> same shared filesystem location.
>
Re: headless build - problems with tmp folder [message #592348 is a reply to message #48009] Wed, 25 February 2009 16:48 Go to previous message
Chris is currently offline ChrisFriend
Messages: 97
Registered: July 2009
Member
Here some more investigations:

the headless build creates and scripts named assemble.PLUGINNAME.win32.win32.x86.xml ....
those files contain an entry

<property name="assemblyTempDir" value="${buildDirectory}/tmp"/>

At the end of the script there is a
<delete dir="${assemblyTempDir}"/>

This is quite deadly, if two such scripts execute in parallel.
Such parallel execution however is quite common when using continous integration systems.

Way better would have been (for example) .....
<property name="assemblyTempDir" value="${buildDirectory}/tmp_PLUGINNAME"/>

Chris


Chris wrote:
> while executing a headless build the eclipse build process creates
> several temp files in ${buildDirectory}/tmp. These are used during
> internal ant tasks like "jarUp, jarIng, ..."
>
> We have like 20 plugins - some relating to each other, some not. They
> are checked out into
> ${buildDirectory}/plugins (as expected by the headleass build process).
>
> I have the following problem: Our C.I. server sometimes builds the
> plugins in parallel (depending on how many executors are available, and
> of course the plugins must not relate to each other).
>
>
> This leads to the situation, that the first plugin build finishing
> cleans the tmp directory, and the other build running in parallel fails.
>
>
> I think the main problem is to use the same tmp folder for different
> builds, and to delete it completely when finishing, thereby killing
> other builds running in parallel.
>
>
> I have 2 simple questions (where either of them could solve my problem):
>
> Q1) We are still on Eclipse 3.3 (both for build and runtime
> environment). Did this behavior change with Eclipse 3.4 - afaik. I could
> build 3.3 deliverables with eclipse 3.4.
>
> Q2) Is there a way to specify the "tmp" folder used by the internal ant
> tasks?
>
> Q3) .... well if !Q1 && !Q2 .... do you have another hint?
>
>
> Thanks,
> Chris
Re: headless build - problems with tmp folder [message #592385 is a reply to message #48100] Wed, 25 February 2009 17:55 Go to previous message
Chris Williams is currently offline Chris WilliamsFriend
Messages: 29
Registered: July 2009
Junior Member
Why aren't your separate builds using separate workspaces/build
directories?

This behavior won't change in the build. It's going to assume your not
running multiple builds on the same files simultaneously - as it should.

If you have separate builds, they should each maintain their own
buildDirectory.
Re: headless build - problems with tmp folder [message #592443 is a reply to message #48246] Wed, 25 February 2009 22:49 Go to previous message
Chris is currently offline ChrisFriend
Messages: 97
Registered: July 2009
Member
I am NOT running multiple builds on the same files.
I am running two builds on two totally different features.

We have the following feature/plugin folder structure (we don't use this
structure because we like it that way - we use it as the eclipse build
process forces us to use this structure)....

/buildDirectry/
/plugins
/pluginA
/pluginB
/pluginC
/features
/featureA
/featureB
/featureC

Btw - we have about 25 such plugins right now and we build them using a
continous build process (controlled by Hudson). Some of the features
relate to each other, others don't.

Now assume the build server detects changes on pluginA and pluginB at
the same time. (not too uncommon, as it polls our SCM in the same
intervall for various projects).

So the build server triggers build of featureA and featureB. They DON'T
relate to each other, have no common files or whatever

Now the eclipse build process creates a folder /buildDirectory/tmp - it
places tmp files for both builds in this directory.

Assume featureA build finishes earlier - it wipes tmp .... peng -
featureB build is killed.

.... please don't tell me this is working as intended ....

Btw - the solution would be very simple. The assemblyTempDir property
defined in ALL the generated assemble.feature.xml files just would have
to be unique ... that shouldn't be a difficult thing - it's just hidden
by the eclipse build process, and out of the developers control.


Chris
Re: headless build - problems with tmp folder [message #592496 is a reply to message #48395] Fri, 27 February 2009 15:07 Go to previous message
Chris Williams is currently offline Chris WilliamsFriend
Messages: 29
Registered: July 2009
Junior Member
Chris wrote:

> I am NOT running multiple builds on the same files.
> I am running two builds on two totally different features.

> We have the following feature/plugin folder structure (we don't use this
> structure because we like it that way - we use it as the eclipse build
> process forces us to use this structure)....

> /buildDirectry/
> /plugins
> /pluginA
> /pluginB
> /pluginC
> /features
> /featureA
> /featureB
> /featureC

...

> So the build server triggers build of featureA and featureB. They DON'T
> relate to each other, have no common files or whatever

The issue here is that you're using a shared buildDirectory for what are
separate builds. A buildDirectory is assumed by the build process to be a
modifiable workspace that is devoted just to that single build. If
featureA and featureB are not related, they should not use the same
workspace for their builds. If there's no overlap in the contents of the
features at all, then I suggest breaking out the structure to be more like:

/featureAbuildDirectory
/features
/featureA
/plugins
/pluginA_1
/pluginA_2
/featureBbuildDirectory
/features
/featureB
/plugins
/pluginB_1
/pluginB_2


If there is some overlap between features (multiple use the same plugin),
then retain your original structure, but have Hudson use separate
workspaces for each build that are just copies of your buildDirectory. Or
make Hudson use a lock for those builds that use the same buildDirectory
to avoid having multiple builds trying to modify the same shared
filesystem location.
Re: headless build - problems with tmp folder [message #592679 is a reply to message #48554] Wed, 04 March 2009 15:13 Go to previous message
Chris is currently offline ChrisFriend
Messages: 97
Registered: July 2009
Member
Hello Chris,

thanks for your response, sorry to say that I can't agree with this concept.
There are several reasons ....

a) what you are telling me is, that I would have to change the buildDirectory regularly -
depending on plugin dependency. On (not even very) large projects with 40+ plugins /
features this is simply not possible.

b) The build process already takes care of many things needed to build multiple features
within one buildDirectory. e.g.
b1) various temp folders are created right underneath the plugin directory, where they
don't conflicht
b2) the generated assemble and package scripts contain the feature name in order to avoid
conflicts etc.

The only (afaik) situation where there is a collision is at the end, when JARing up the
archives. The solution would be very simple - again use a tmp folder under e.g. the
feature directory, or create a tmp folder containing the feature name (similar to the
assemble script names).

This would allow to run multiple plugin builds of plugins belonging to the same project to
be executed in parallel.


Chris




Chris Williams wrote:
> Chris wrote:
>
>> I am NOT running multiple builds on the same files.
>> I am running two builds on two totally different features.
>
>> We have the following feature/plugin folder structure (we don't use
>> this structure because we like it that way - we use it as the eclipse
>> build process forces us to use this structure)....
>
>> /buildDirectry/
>> /plugins
>> /pluginA
>> /pluginB
>> /pluginC
>> /features
>> /featureA
>> /featureB
>> /featureC
>
> ..
>
>> So the build server triggers build of featureA and featureB. They
>> DON'T relate to each other, have no common files or whatever
>
> The issue here is that you're using a shared buildDirectory for what are
> separate builds. A buildDirectory is assumed by the build process to be
> a modifiable workspace that is devoted just to that single build. If
> featureA and featureB are not related, they should not use the same
> workspace for their builds. If there's no overlap in the contents of the
> features at all, then I suggest breaking out the structure to be more like:
>
> /featureAbuildDirectory
> /features
> /featureA
> /plugins
> /pluginA_1
> /pluginA_2
> /featureBbuildDirectory
> /features
> /featureB
> /plugins
> /pluginB_1
> /pluginB_2
>
>
> If there is some overlap between features (multiple use the same
> plugin), then retain your original structure, but have Hudson use
> separate workspaces for each build that are just copies of your
> buildDirectory. Or make Hudson use a lock for those builds that use the
> same buildDirectory to avoid having multiple builds trying to modify the
> same shared filesystem location.
>
Previous Topic:PDE Export and Debug Info
Next Topic:how to observe junit tests
Goto Forum:
  


Current Time: Tue Jul 27 13:12:22 GMT 2021

Powered by FUDForum. Page generated in 0.01929 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top