Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » Buckminster » Re: Buckminster build in hudson with precreated target platform
Re: Buckminster build in hudson with precreated target platform [message #721465] Thu, 01 September 2011 20:16 Go to next message
Thomas Hallgren is currently offline Thomas Hallgren
Messages: 3229
Registered: July 2009
Senior Member
On 2011-09-01 22:05, Lothar Werzinger wrote:
> I tried to set the property cbi.include.source to false as described in the buckminster book, but the build still looks
> for the unavailable org.apache.commons.logging.source bundle.
>
The cbi.include.source has no relevance when you do a resolve. It's only valid when you build a site.

Why are you doing two different resolves b.t.w.? What's the purpose of the second one?

- thomas
Re: Buckminster build in hudson with precreated target platform [message #721472 is a reply to message #721465] Thu, 01 September 2011 20:53 Go to previous messageGo to next message
Lothar Werzinger is currently offline Lothar Werzinger
Messages: 153
Registered: July 2009
Location: Bay Area
Senior Member
Thomas Hallgren wrote on Thu, 01 September 2011 13:16
On 2011-09-01 22:05, Lothar Werzinger wrote:
> I tried to set the property cbi.include.source to false as described in the buckminster book, but the build still looks
> for the unavailable org.apache.commons.logging.source bundle.
>
The cbi.include.source has no relevance when you do a resolve. It's only valid when you build a site.


That's what I am doing.
Using the precreated TP (from the test_tp job) I try to build a site (in the test_build job)

This are the commands I am using:
resolve ../mmt-tools.cquery
perform biz.tradescape.mmt.cdoserver.feature#site.p2


it's all in the downloadable zip file to be reproducable.

Thomas Hallgren wrote on Thu, 01 September 2011 13:16

Why are you doing two different resolves b.t.w.? What's the purpose of the second one?

- thomas


Not sure what two different resolves you are talking about.
What I am trying to do is to create a target platform that will be used for


  1. development in the IDE
  2. automatic builds in hudson


I want to make sure that both the development and the automatic builds use exactly the same plugins and don't try to do dynamic resolves against the internet while building.


P.S.
just curious; why did you create a new thread instead of replying to the existing one?
Re: Buckminster build in hudson with precreated target platform [message #721545 is a reply to message #721472] Fri, 02 September 2011 05:08 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas Hallgren
Messages: 3229
Registered: July 2009
Senior Member
On 2011-09-01 22:53, Lothar Werzinger wrote:
> Thomas Hallgren wrote on Thu, 01 September 2011 13:16
>> On 2011-09-01 22:05, Lothar Werzinger wrote:
>> > I tried to set the property cbi.include.source to false as described in the buckminster book, but the build still looks
>> > for the unavailable org.apache.commons.logging.source bundle.
>> >
>> The cbi.include.source has no relevance when you do a resolve. It's only valid when you build a site.
>
>
> That's what I am doing.
> Using the precreated TP (from the test_tp job) I try to build a site (in the test_build job)
>
> This are the commands I am using:
>
> resolve ../mmt-tools.cquery
> perform biz.tradescape.mmt.cdoserver.feature#site.p2
>
AFAICS, your errors occur during resolve. So I repeat, the cbi.include.source has no relevance when you do a resolve.

>> Why are you doing two different resolves b.t.w.? What's the purpose of the second one?
>>
>
> Not sure what two different resolves you are talking about.
> What I am trying to do is to create a target platform that will be used for
>
>
> development in the IDE automatic builds in hudson
>
I see the test_rcp.cquery/test_rcp.rmap. From the looks of it, they resolve/materialize both your target platform and
your workspace. Then there's a mmt-tools.cquery/mmt-tools.rmap. Not sure what they do but they seem redundant.

- thomas

> P.S.
> just curious; why did you create a new thread instead of replying to the existing one?

Not sure what you mean, this is my sixth reply in this thread. All postings, from your original posting, shows up as as
single thread in my news reader.
Re: Buckminster build in hudson with precreated target platform [message #721547 is a reply to message #721545] Fri, 02 September 2011 05:40 Go to previous messageGo to next message
Lothar Werzinger is currently offline Lothar Werzinger
Messages: 153
Registered: July 2009
Location: Bay Area
Senior Member
Thomas Hallgren wrote on Thu, 01 September 2011 22:08

I see the test_rcp.cquery/test_rcp.rmap. From the looks of it, they resolve/materialize both your target platform and
your workspace. Then there's a mmt-tools.cquery/mmt-tools.rmap. Not sure what they do but they seem redundant.


test_rcp.cquery/test_rcp.rmap are from the job that builds the TP
mmt-tools.cquery/mmt-tools.rmap is from the job that tries to buils site.p2 using the previouly materialized TP.

As outlined in another post the goal is to have a TP (immutable after creation) that is used for development in the IDE as well as for all automatic builds.
As I had trouble with the build part I reduced it to simply building a feature composed of other features in the TP, so that I didn't have to distribute code to compile in the sample I provided to reproduce the issue.

Thomas Hallgren wrote on Thu, 01 September 2011 22:08

> P.S.
> just curious; why did you create a new thread instead of replying to the existing one?

Not sure what you mean, this is my sixth reply in this thread. All postings, from your original posting, shows up as as
single thread in my news reader.


I am using the web forum and for whatever reasons it shows up as another thread there.
Re: Buckminster build in hudson with precreated target platform [message #721575 is a reply to message #721547] Fri, 02 September 2011 07:11 Go to previous messageGo to next message
Stephan  is currently offline Stephan
Messages: 36
Registered: July 2009
Member
Hi Lothar

We do something very similar:

> test_rcp.cquery/test_rcp.rmap are from the job that builds the TP

We have...
- one Hudson job that creates the target platform with some MSPEC/CQUERY/RMAP
- one target definition file (e.g., tp-37.target) that points to the result from the job above

> mmt-tools.cquery/mmt-tools.rmap is from the job that tries to buils site.p2 using the previouly materialized TP.

Then we have a different job that executes the following commands to build the site:
importtargetdefinition -A '${WORKSPACE}/......../tp-37.target'
import '${WORKSPACE}/......../source.cquery'
build
perform some.feature#site.p2

I missed importtargetdefinition in your command list above. Did you try that?

Cheers, Stephan
Re: Buckminster build in hudson with precreated target platform [message #721583 is a reply to message #721547] Fri, 02 September 2011 07:55 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas Hallgren
Messages: 3229
Registered: July 2009
Senior Member
On 2011-09-02 07:40, Lothar Werzinger wrote:
> Thomas Hallgren wrote on Thu, 01 September 2011 22:08
>> I see the test_rcp.cquery/test_rcp.rmap. From the looks of it, they resolve/materialize both your target platform and
>> your workspace. Then there's a mmt-tools.cquery/mmt-tools.rmap. Not sure what they do but they seem redundant.
>
>
> test_rcp.cquery/test_rcp.rmap are from the job that builds the TP
> mmt-tools.cquery/mmt-tools.rmap is from the job that tries to buils site.p2 using the previouly materialized TP.
>
There must be some misunderstanding then. You don't need to re-resolve before when you perform unless you switch
workspace between the attempts.

> As outlined in another post the goal is to have a TP (immutable after creation) that is used for development in the IDE
> as well as for all automatic builds.

The TP is never changed during a build.

> As I had trouble with the build part I reduced it to simply building a feature composed of other features in the TP, so
> that I didn't have to distribute code to compile in the sample I provided to reproduce the issue.
>
OK, that's fine. The problem I will look at is why you don't get both source bundles into your TP. I have what I need to
do that.

>
> I am using the web forum and for whatever reasons it shows up as another thread there.

The web-forum apparently has some flaws...

- thomas
Re: Buckminster build in hudson with precreated target platform [message #721585 is a reply to message #721575] Fri, 02 September 2011 08:00 Go to previous messageGo to next message
Thomas Hallgren is currently offline Thomas Hallgren
Messages: 3229
Registered: July 2009
Senior Member
On 2011-09-02 09:11, Stephan wrote:
> Hi Lothar
> We do something very similar:
>
>> test_rcp.cquery/test_rcp.rmap are from the job that builds the TP
>
> We have...
> - one Hudson job that creates the target platform with some MSPEC/CQUERY/RMAP
> - one target definition file (e.g., tp-37.target) that points to the result from the job above
>
>> mmt-tools.cquery/mmt-tools.rmap is from the job that tries to buils site.p2 using the previouly materialized TP.
>
> Then we have a different job that executes the following commands to build the site:
> importtargetdefinition -A '${WORKSPACE}/......../tp-37.target'
> import '${WORKSPACE}/......../source.cquery'
> build
> perform some.feature#site.p2
>
> I missed importtargetdefinition in your command list above. Did you try that?
>
The importtargetdefinition uses the PDE target definition construct. Lothar uses Buckminster to build the target
platform based in the cquery/rmap dependency resolution.

All our builds use Buckminster to resolve both the target platform and the workspace at the same time. I've found that
approach much more flexible and less error prone. Rebuilding the TP is never a problem unless the p2 repositories that
serve up the TP are very slow. And if they are, you can always create a local mirror of them instead of a pre-defined
target platform.

IMO, using p2 repositories rather than pre-defined TP's as input to your build is more flexible.

- thomas
Re: Buckminster build in hudson with precreated target platform [message #721744 is a reply to message #721575] Fri, 02 September 2011 17:49 Go to previous messageGo to next message
Lothar Werzinger is currently offline Lothar Werzinger
Messages: 153
Registered: July 2009
Location: Bay Area
Senior Member
Stephan wrote on Fri, 02 September 2011 00:11
Hi Lothar

We do something very similar:

Then we have a different job that executes the following commands to build the site:
importtargetdefinition -A '${WORKSPACE}/......../tp-37.target'
import '${WORKSPACE}/......../source.cquery'
build
perform some.feature#site.p2

I missed importtargetdefinition in your command list above. Did you try that?

Cheers, Stephan


The hudson buckminster plugin does some "magic" there. It put's the line
setpref targetPlatformPath="/var/lib/hudson/jobs/test_tp/builds/2011-09-01_12-38-00/archive//targetPlatform"

before the commands that I enter.

Not sure if that is equivalent to your importtargetdefinition command.
Re: Buckminster build in hudson with precreated target platform [message #721746 is a reply to message #721583] Fri, 02 September 2011 17:52 Go to previous messageGo to next message
Lothar Werzinger is currently offline Lothar Werzinger
Messages: 153
Registered: July 2009
Location: Bay Area
Senior Member
Thomas Hallgren wrote on Fri, 02 September 2011 00:55

On 2011-09-02 07:40, Lothar Werzinger wrote:
> Thomas Hallgren wrote on Thu, 01 September 2011 22:08
>> I see the test_rcp.cquery/test_rcp.rmap. From the looks of it, they resolve/materialize both your target platform and
>> your workspace. Then there's a mmt-tools.cquery/mmt-tools.rmap. Not sure what they do but they seem redundant.
>
>
> test_rcp.cquery/test_rcp.rmap are from the job that builds the TP
> mmt-tools.cquery/mmt-tools.rmap is from the job that tries to buils site.p2 using the previouly materialized TP.
>
There must be some misunderstanding then. You don't need to re-resolve before when you perform unless you switch
workspace between the attempts.


As the jobs have independent workspaces the build job won't be able to access anything unless the cquery materializes it out of the TP, right?

Thomas Hallgren wrote on Fri, 02 September 2011 00:55


> As outlined in another post the goal is to have a TP (immutable after creation) that is used for development in the IDE
> as well as for all automatic builds.

The TP is never changed during a build.

> As I had trouble with the build part I reduced it to simply building a feature composed of other features in the TP, so
> that I didn't have to distribute code to compile in the sample I provided to reproduce the issue.
>
OK, that's fine. The problem I will look at is why you don't get both source bundles into your TP. I have what I need to
do that.


Thanks!

Thomas Hallgren wrote on Fri, 02 September 2011 00:55


>
> I am using the web forum and for whatever reasons it shows up as another thread there.

The web-forum apparently has some flaws...

- thomas


No problem, just seemed strange at first sight.
Re: Buckminster build in hudson with precreated target platform [message #721754 is a reply to message #721585] Fri, 02 September 2011 18:16 Go to previous messageGo to next message
Lothar Werzinger is currently offline Lothar Werzinger
Messages: 153
Registered: July 2009
Location: Bay Area
Senior Member
Thomas Hallgren wrote on Fri, 02 September 2011 01:00

On 2011-09-02 09:11, Stephan wrote:
> I missed importtargetdefinition in your command list above. Did you try that?
>
The importtargetdefinition uses the PDE target definition construct. Lothar uses Buckminster to build the target
platform based in the cquery/rmap dependency resolution.


One big reason trying to use buckminster to build the TP is that I had severe problems creating stable TPs with the "Target definition editor". Especially for complicated targets that are composed from different update sites. Compared to that defining and materializing the TP with buckminster was easy. I especially hate that the target definition editor sometimes "forgets" what features and plugins you selected when you open the target definition for editing.

Using the buckminster created TP in the IDE is easy, too as I can create a simple target definition file that points to the directory containing the TP. I already did that and the IDE works fine with the generated TP, as it does not need the source plugins.

Using buckminster for build looks very attractive, too in comparison to the ugly PDE build based build that we use now. Especially as we would like to have p2 update sites in conjunction with zip based products in the future and not only zip product builds.

Thomas Hallgren wrote on Fri, 02 September 2011 01:00


All our builds use Buckminster to resolve both the target platform and the workspace at the same time. I've found that
approach much more flexible and less error prone. Rebuilding the TP is never a problem unless the p2 repositories that
serve up the TP are very slow. And if they are, you can always create a local mirror of them instead of a pre-defined
target platform.

IMO, using p2 repositories rather than pre-defined TP's as input to your build is more flexible.

- thomas



If I want to guarantee that the individual resolves resolve exactly the same plugins it seems that a local mirror is required. I don't want anyone starting with a new workspace to get different plugins to build against by chance. The precreated TP seems to solve the problem nicely.

Being new to buckminster I also don't understand yet how buckminster handles the TP during development. I know you can call buckminster to do the actual build of your product out of the IDE with the buckminster menu, but what about development and debugging with the regular incremental builds in the IDE?
Re: Buckminster build in hudson with precreated target platform [message #721930 is a reply to message #721585] Sat, 03 September 2011 16:49 Go to previous messageGo to next message
Stephan  is currently offline Stephan
Messages: 36
Registered: July 2009
Member
Hi Thomas

> The importtargetdefinition uses the PDE target definition construct.
> Lothar uses Buckminster to build the target
> platform based in the cquery/rmap dependency resolution.
> All our builds use Buckminster to resolve both the target
> platform and the workspace > at the same time. I've found
> that approach much more flexible and less error prone.

Sounds promising. We also have an application that uses Windows-specific OLE classes from SWT. To build this application on Linux/Hudson, we have a target definition where we set os/arch to win32/win32. Otherwise, it does not build. Would this also work with your approach or does this approach "only" build a target platform for the os/arch where the build is running on?

Regards, Stephan
Re: Buckminster build in hudson with precreated target platform [message #721979 is a reply to message #721930] Sat, 03 September 2011 21:48 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik Lindberg
Messages: 2500
Registered: July 2009
Senior Member
We typically include all platform specific bundles in the target
definition and then build the product several times with different
platform filters.

- henrik

On 9/3/11 6:49 PM, Stephan wrote:
> Hi Thomas
>
>> The importtargetdefinition uses the PDE target definition construct.
>> Lothar uses Buckminster to build the target
>> platform based in the cquery/rmap dependency resolution.
>> All our builds use Buckminster to resolve both the target platform and
>> the workspace > at the same time. I've found
>> that approach much more flexible and less error prone.
>
> Sounds promising. We also have an application that uses Windows-specific
> OLE classes from SWT. To build this application on Linux/Hudson, we have
> a target definition where we set os/arch to win32/win32. Otherwise, it
> does not build. Would this also work with your approach or does this
> approach "only" build a target platform for the os/arch where the build
> is running on?
>
> Regards, Stephan
Re: Buckminster build in hudson with precreated target platform [message #722316 is a reply to message #721979] Mon, 05 September 2011 11:15 Go to previous messageGo to next message
Stephan  is currently offline Stephan
Messages: 36
Registered: July 2009
Member
Hi Henrik

> include all platform specific bundles in the target definition

That is exactly what we do now. I assumed, the approach taken by Buckminster itself (resolve source and TP in one step) does not require an additional target definition file at all. Did I misunderstand you?

Cheers, Stephan
Re: Buckminster build in hudson with precreated target platform [message #722352 is a reply to message #722316] Mon, 05 September 2011 13:13 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik Lindberg
Messages: 2500
Registered: July 2009
Senior Member
On 9/5/11 1:15 PM, Stephan wrote:
>> include all platform specific bundles in the target definition
>
> That is exactly what we do now. I assumed, the approach taken by
> Buckminster itself (resolve source and TP in one step) does not require
> an additional target definition file at all. Did I misunderstand you?

I may have misread your earlier post.
> Sounds promising. We also have an application that uses Windows-specific
> OLE classes from SWT. To build this application on Linux/Hudson, we have
> a target definition where we set os/arch to win32/win32. Otherwise, it
> does not build. Would this also work with your approach or does this
> approach "only" build a target platform for the os/arch where the build
> is running on?

My point was - materialize everything in one operation; source to
workspace, and binaries (possibly with source bundles) to the TP. This
materialization should be made for all platforms (i.e. '*').

When building something using the (multi platform) TP, you build for one
platform at a time by using different property sets. Alternatively,
define one action per platform, and one that builds all to avoid having
to deal with remembering to use/set the correct properties on each build
invocation.

Does that make it clearer?
Regards
- henrik
Re: Buckminster build in hudson with precreated target platform [message #722368 is a reply to message #722352] Mon, 05 September 2011 14:16 Go to previous messageGo to next message
Stephan  is currently offline Stephan
Messages: 36
Registered: July 2009
Member
> My point was - materialize everything in one operation; source to
> workspace, and binaries (possibly with source bundles) to the TP. This
> materialization should be made for all platforms (i.e. '*').

Okay, understood.

> When building something using the (multi platform) TP, you build for one
> platform at a time by using different property sets.

What Buckminster command do you have in mind when you write "building". I know how to pass properties to the "perform" command but in my case "compile SWT-OLE on Linux" the Buckminster "build" command fails and I'm not aware of any possibility to activate a Windows-only TP for this command (except calling importtargedefinition with os/arch-properties BEFORE the "build" command). Is there any way to make the "build" command work in this case?

Thanks, Stephan
Re: Buckminster build in hudson with precreated target platform [message #722379 is a reply to message #722368] Mon, 05 September 2011 14:46 Go to previous messageGo to next message
Henrik Lindberg is currently offline Henrik Lindberg
Messages: 2500
Registered: July 2009
Senior Member
On 9/5/11 4:16 PM, Stephan wrote:
>> When building something using the (multi platform) TP, you build for one
>> platform at a time by using different property sets.
>
> What Buckminster command do you have in mind when you write "building".
> I know how to pass properties to the "perform" command but in my case
> "compile SWT-OLE on Linux" the Buckminster "build" command fails and I'm
> not aware of any possibility to activate a Windows-only TP for this
> command (except calling importtargedefinition with os/arch-properties
> BEFORE the "build" command). Is there any way to make the "build"
> command work in this case?
>

From the UI Buckminster > Invoke Action > 'some action'
You either invoke an action that sets the filter to the wanted platform,
or call an action where this must be set in a properties file that you
point to (at the bottom of the dialog).

IIRC, if an action is invoked without any information about platform, it
will be performed for the platform you are currently running.

So, does your "compile SWT-OLE on Linux" action set the platform?

Regards
- henrik
Re: Buckminster build in hudson with precreated target platform [message #722425 is a reply to message #722379] Mon, 05 September 2011 18:31 Go to previous messageGo to next message
Stephan  is currently offline Stephan
Messages: 36
Registered: July 2009
Member
Sorry, when I was too less specific: We develop on Windows and run headless builds on Linux/Hudson:
- Running an action within the IDE (Buckminster > Invoke action) on Windows is not a problem because the TP is already Windows-specific (when using the running platform).
- Building on Linux works a bit different and consists of the following headless commands:
1) importtargetdefinition -A '${WORKSPACE}trunk/com.diligentit.cctvnet.build/headlessLinux/tp/tp-linux-win.target'
2) import '${WORKSPACE}trunk/com.diligentit.cctvnet.workplace.build/site.cquery'
3) build
4) perform -P '${WORKSPACE}trunk/com.diligentit.cctvnet.build/headlessLinux/buildsite-win32.properties' com.diligentit.cctvnet.workplace.build#site.p2
5) perform -D target.os=win32 -D target.ws=win32 -D target.arch=x86 com.diligentit.cctvnet.workplace.build#create.product.zip

This all works pretty well. Now the whole exercise is to get rid of 1) and resolve source and TP in 2), correct? The problem is that "step 3) - build" fails because it cannot find the Windows-specific OLE classes. Should "step 2) - import" already set os/ws to win32/win32? Would this create a Windows-specific TP?

Thanks, Stephan
Re: Buckminster build in hudson with precreated target platform [message #722483 is a reply to message #722425] Mon, 05 September 2011 22:28 Go to previous message
Henrik Lindberg is currently offline Henrik Lindberg
Messages: 2500
Registered: July 2009
Senior Member
If step 2) does not specify any platform filter, it will be filtered for
the current platform. Here, you probably want to use '*' so you get a TP
for all platforms. e.g. os, ws and arch should all be * (or some sane
set that gets you everything that you want, i.e. os and ws are * and
arch = x86)

Then in 4 and 5 you (correctly) specify that you want a build for
windows but was probably already filtered in 2) so it fails.

- henrik

On 9/5/11 8:31 PM, Stephan wrote:
> Sorry, when I was too less specific: We develop on Windows and run
> headless builds on Linux/Hudson:
> - Running an action within the IDE (Buckminster > Invoke action) on
> Windows is not a problem because the TP is already Windows-specific
> (when using the running platform). - Building on Linux works a bit
> different and consists of the following headless commands:
> 1) importtargetdefinition -A
> '${WORKSPACE}trunk/com.diligentit.cctvnet.build/headlessLinux/tp/tp-linux-win.target'
>
> 2) import
> '${WORKSPACE}trunk/com.diligentit.cctvnet.workplace.build/site.cquery'
> 3) build
> 4) perform -P
> '${WORKSPACE}trunk/com.diligentit.cctvnet.build/headlessLinux/buildsite-win32.properties'
> com.diligentit.cctvnet.workplace.build#site.p2
> 5) perform -D target.os=win32 -D target.ws=win32 -D target.arch=x86
> com.diligentit.cctvnet.workplace.build#create.product.zip
>
> This all works pretty well. Now the whole exercise is to get rid of 1)
> and resolve source and TP in 2), correct? The problem is that "step 3) -
> build" fails because it cannot find the Windows-specific OLE classes.
> Should "step 2) - import" already set os/ws to win32/win32? Would this
> create a Windows-specific TP?
>
> Thanks, Stephan
>
Previous Topic:Headless build does not produce launcher
Next Topic:Buckminster build in hudson with precreated target platform
Goto Forum:
  


Current Time: Thu Oct 30 14:40:33 GMT 2014

Powered by FUDForum. Page generated in 0.03277 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software