Home » Eclipse Projects » Buckminster dev » New actor proposal - env
| New actor proposal - env [message #26776] |
Tue, 28 October 2008 17:30  |
 Guillaume Chatelet
Messages: 146
Registered: July 2009 |
Senior Member |
|
|
Hello,
I already proposed two new actors for Buckminster :
- executor : executes something (real program or scripts / sets execution dir / sets environment variables ).
- fetcher : gets a resource from an url and eventually uncompress it to a local folder.
Those two actors are now quite used in production here at MikrosImage and really helps us a lot getting rid of Ant scripts which adds a layer of complexity our C++ coders don't want to manage.
The typical workflow is a big C++ program with a lot of dependencies to libraries ( includes & libs ) which also have dependencies etc...
At the moment, the standard actions on components are build / clean / rebuild. This is ok for small projects but not so good when a lot of dependencies are involved. Because of the multiple targets triggered across the components when an action is fired, the upToDatePolicy has to be checked on product folders with a lot of files ( eg : 4k for boost includes ). This slows down the build quite a lot ( a few seconds before the build actually starts ) and usually this is just to retrieve the include or lib folder of another component. Accumulated over a day of work, this can be quite frustrating.
Before Buckminster, this problem of dependencies was solved statically by specifying environment variables referring to directories dispatched over the files system. The dependencies were compiled by hand and the environment variables set in the user's session.
This was not appropriate because the build couldn't be automated.
Now Buckminster solves the materialisation, independent and dynamic nature of the component, the glue between the component is guaranteed by the actions which isolates them. However this needs to be set only at the materialisation stage and not each time the actions are executed.
One way to alleviate the workflow is to let Buckminster write a shell script that would bake the particular component's folders.
Thus you just have to call this script to setup your environment and start getting productive.
Feedback would be greatly appreciated. I admit this is quite a particular problem and I think we are near the limits of a generic tool like Buckminster, anyway, any comments would be very valuable.
Best regards,
Guillaume
|
|
|
| Re: New actor proposal - env [message #27101 is a reply to message #26776] |
Wed, 05 November 2008 15:57  |
Guillaume Chatelet
Messages: 146
Registered: July 2009 |
Senior Member |
|
|
Dear Buckminster's developers,
To be more concrete, here is a specification for my new proposal :
http://wiki.eclipse.org/Non_Java_projects_Proposal#Script_Ac tor
This is the result of the reflexions of my team in it's everyday use of Buckminster.
I would enjoy hearing from you regarding this issue.
Best regards,
Guillaume
Guillaume CHATELET wrote:
> Hello,
>
> I already proposed two new actors for Buckminster :
> - executor : executes something (real program or scripts / sets
> execution dir / sets environment variables ).
> - fetcher : gets a resource from an url and eventually uncompress it to
> a local folder.
>
> Those two actors are now quite used in production here at MikrosImage
> and really helps us a lot getting rid of Ant scripts which adds a layer
> of complexity our C++ coders don't want to manage.
>
> The typical workflow is a big C++ program with a lot of dependencies to
> libraries ( includes & libs ) which also have dependencies etc...
> At the moment, the standard actions on components are build / clean /
> rebuild. This is ok for small projects but not so good when a lot of
> dependencies are involved. Because of the multiple targets triggered
> across the components when an action is fired, the upToDatePolicy has to
> be checked on product folders with a lot of files ( eg : 4k for boost
> includes ). This slows down the build quite a lot ( a few seconds before
> the build actually starts ) and usually this is just to retrieve the
> include or lib folder of another component. Accumulated over a day of
> work, this can be quite frustrating.
>
> Before Buckminster, this problem of dependencies was solved statically
> by specifying environment variables referring to directories dispatched
> over the files system. The dependencies were compiled by hand and the
> environment variables set in the user's session.
> This was not appropriate because the build couldn't be automated.
> Now Buckminster solves the materialisation, independent and dynamic
> nature of the component, the glue between the component is guaranteed by
> the actions which isolates them. However this needs to be set only at
> the materialisation stage and not each time the actions are executed.
>
> One way to alleviate the workflow is to let Buckminster write a shell
> script that would bake the particular component's folders.
> Thus you just have to call this script to setup your environment and
> start getting productive.
>
> Feedback would be greatly appreciated. I admit this is quite a
> particular problem and I think we are near the limits of a generic tool
> like Buckminster, anyway, any comments would be very valuable.
>
> Best regards,
> Guillaume
|
|
|
| Re: New actor proposal - env [message #27141 is a reply to message #27101] |
Thu, 06 November 2008 10:15 |
|
Hi Guillaume,
I have read the proposal. The executor and actor are both fine. Controlling environment
settings and shell seems natural. Perhaps the actor could trigger on properties that
starts with "env." instead of concatenating. Your example:
<cs:property key="env"
value=" TIFF_LIB=${tiff.lib};TIFF_INCLUDE=${tiff.include};VERSION=${ VERSION};OSGI_VERSION=${buckminster.version} "/>
would then be:
<cs:property key="env.TIFF_LIB" value="${tiff.lib}"/>
<cs:property key="env.TIFF_INCLUDE" value="${tiff.include}"/>
<cs:property key="env.VERSION" value="${VERSION}"/>
<cs:property key="env.OSGI_VERSION" value="${buckminster.version}"/>
which is a bit easier to edit, especially in the ComponentSpec editor.
Here are some thoughts concerning the script actor.
1. I'm not too keen on generating a script based on numbered properties. I think a better
approach would be to call on different external scripts. If scripts really should be
inlined into the cspec, then we should extend the cspec with an script element that allows
free text. Inlining does have drawbacks though, since there's no way to use a native
script editor that does syntax checking and other nice things.
2. One filter should be enough since it's an LDAP filter. You can use constructs like
(&(filter1)(filter2)) or (|(filter1)(filter2)) or even (&(|(filter1)(filter2))(filter3)).
Mixing and dicing AND and OR constructs will become very complicated very quickly if we
separate this into different properties.
3. As a consequence of my reasoning around #1, there's no real difference between a script
execution and the executor.
Put together, this means that most of the functionality is there already. If you want
different behavior on different platforms you can create an group that has prerequisites
that are filtered and then call on the group. I.e.
<cs:actions>
<cs:public name="my.bat.script" filter="(target.os=win32)" actor="executor">
...
</cs:public>
<cs:public name="my.shell.script" filter="(target.os=linux)" actor="executor">
...
</cs:public>
</cs:actions>
<cs:groups>
<cs:public name="my.script">
<cs:attribute name="my.bat.script"/>
<cs:attribute name="my.shell.script"/>
</cs:public>
</cs:groups>
Performing "my.script" will then execute the correct executor based on the setting of
"target.os".
Regards,
Thomas Hallgren
Guillaume CHATELET wrote:
> Dear Buckminster's developers,
>
> To be more concrete, here is a specification for my new proposal :
> http://wiki.eclipse.org/Non_Java_projects_Proposal#Script_Ac tor
>
> This is the result of the reflexions of my team in it's everyday use of
> Buckminster.
>
> I would enjoy hearing from you regarding this issue.
> Best regards,
> Guillaume
>
> Guillaume CHATELET wrote:
|
|
|
| Re: New actor proposal - env [message #27180 is a reply to message #27141] |
Thu, 06 November 2008 14:15 |
Guillaume Chatelet
Messages: 146
Registered: July 2009 |
Senior Member |
|
|
Hi Thomas,
Thank you for taking time to read my proposal and for commenting on it.
I fully agree regarding the environment suggestion. It seems much more natural to have several properties prefixed with "env." .
Regarding the script actor, I'm not sure you got my point with the filter issue so I'm going to explain a bit more.
The aim here is to provide a mean to append lines to the script. Usually the appended lines are common to all the os but sometimes you want a particular line to be set only on linux or windows or macos.
In fact, I think this is a more general issue, I found no way to reuse a piece of information ( let's say a property ) over a whole CSpec.
If a set of actions needs a property : version="1.35.0", you have to add it to each actions.
So this tricks of filtering lines was a way to write less actions and less common properties.
Regarding the point 3. this is actually what we are already doing. This works well but it's very, very slow. Imagine each time you want to compile your code you have to do : right click on the project => Buckminster => execute action => select the action => click ok. This means each time you want to compile you need 5 mouse clicks. Moreover, as explained in my first mail, the multiple dependencies triggered by the action ( and uptodate policies checking ) adds a few seconds before actually performing the compilation. So we had this idea of writing scripts directly from Buckminster so we could use it in either a terminal or an IDE.
Now thinking about all that and considering that script and executor are kind of related we had the following idea :
If a property is set in the executor ( say write.script = true ) it simply writes a script producing the same result than the execution.
It would remove this script actor proposal and just extends a bit the executor thus allowing us to be far more productive.
What do you think about this new suggestion ?
Best regards,
Guillaume
Thomas Hallgren wrote:
> Hi Guillaume,
> I have read the proposal. The executor and actor are both fine.
> Controlling environment settings and shell seems natural. Perhaps the
> actor could trigger on properties that starts with "env." instead of
> concatenating. Your example:
>
> <cs:property key="env"
> value=" TIFF_LIB=${tiff.lib};TIFF_INCLUDE=${tiff.include};VERSION=${ VERSION};OSGI_VERSION=${buckminster.version} "/>
>
>
> would then be:
>
> <cs:property key="env.TIFF_LIB" value="${tiff.lib}"/>
> <cs:property key="env.TIFF_INCLUDE" value="${tiff.include}"/>
> <cs:property key="env.VERSION" value="${VERSION}"/>
> <cs:property key="env.OSGI_VERSION" value="${buckminster.version}"/>
>
> which is a bit easier to edit, especially in the ComponentSpec editor.
>
> Here are some thoughts concerning the script actor.
>
> 1. I'm not too keen on generating a script based on numbered properties.
> I think a better approach would be to call on different external
> scripts. If scripts really should be inlined into the cspec, then we
> should extend the cspec with an script element that allows free text.
> Inlining does have drawbacks though, since there's no way to use a
> native script editor that does syntax checking and other nice things.
>
> 2. One filter should be enough since it's an LDAP filter. You can use
> constructs like (&(filter1)(filter2)) or (|(filter1)(filter2)) or even
> (&(|(filter1)(filter2))(filter3)). Mixing and dicing AND and OR
> constructs will become very complicated very quickly if we separate this
> into different properties.
>
> 3. As a consequence of my reasoning around #1, there's no real
> difference between a script execution and the executor.
>
> Put together, this means that most of the functionality is there
> already. If you want different behavior on different platforms you can
> create an group that has prerequisites that are filtered and then call
> on the group. I.e.
>
> <cs:actions>
> <cs:public name="my.bat.script" filter="(target.os=win32)"
> actor="executor">
> ...
> </cs:public>
> <cs:public name="my.shell.script" filter="(target.os=linux)"
> actor="executor">
> ...
> </cs:public>
> </cs:actions>
>
> <cs:groups>
> <cs:public name="my.script">
> <cs:attribute name="my.bat.script"/>
> <cs:attribute name="my.shell.script"/>
> </cs:public>
> </cs:groups>
>
> Performing "my.script" will then execute the correct executor based on
> the setting of "target.os".
>
> Regards,
> Thomas Hallgren
>
>
> Guillaume CHATELET wrote:
>> Dear Buckminster's developers,
>>
>> To be more concrete, here is a specification for my new proposal :
>> http://wiki.eclipse.org/Non_Java_projects_Proposal#Script_Ac tor
>>
>> This is the result of the reflexions of my team in it's everyday use
>> of Buckminster.
>>
>> I would enjoy hearing from you regarding this issue.
>> Best regards,
>> Guillaume
>>
>> Guillaume CHATELET wrote:
|
|
|
| Re: New actor proposal - env [message #27219 is a reply to message #27180] |
Thu, 06 November 2008 15:26 |
|
Hi Guillaume,
Guillaume CHATELET wrote:
> Hi Thomas,
>
> Thank you for taking time to read my proposal and for commenting on it.
>
> I fully agree regarding the environment suggestion. It seems much more
> natural to have several properties prefixed with "env." .
>
> Regarding the script actor, I'm not sure you got my point with the
> filter issue so I'm going to explain a bit more.
> The aim here is to provide a mean to append lines to the script. Usually
> the appended lines are common to all the os but sometimes you want a
> particular line to be set only on linux or windows or macos.
>
Perhaps what you need is a more generic 'echo' actor? Such an actor could then be executed
conditionally and you could have many actions that would form a flow of echo invocations
that would build the script for you that in turn could be used as input for the executor.
> In fact, I think this is a more general issue, I found no way to reuse a
> piece of information ( let's say a property ) over a whole CSpec.
> If a set of actions needs a property : version="1.35.0", you have to add
> it to each actions.
>
Having global properties in the ComponentSpec is something that has been discussed before.
I like that idea. Especially if those properties are propagated during resolution and
action execution. It calls for an extension of the CSPEC XML though, since it doesn't
permit this today.
The way I see it, we alredy have a mechanism that enables us to control a flow of filter
conditioned things. We don't need yet another mechanism to do that. Let's assume that we
add an 'echo' actor and fix so that a property can be set both from an attribute and from
an element. I.e.
<property key="hello" value="world"/>
or
<property key="hello">world</property>
Using a CDATA construct, you could then write things like:
<property key="text"><![CDATA[
Here you can write virtually anything. Constructs like
${some.key} would of course be expanded.
]]</property>
An 'echo' actor could recognize the properties. 'inputfile' or 'text' (mutually exclusive)
and 'append' (true/false) and it would require exactly one <product> element that would
appoint the resulting file.
Would that work?
- thomas
|
|
| |
Goto Forum:
Current Time: Wed Apr 24 15:08:26 GMT 2024
Powered by FUDForum. Page generated in 0.03929 seconds |
] 
Search
Help
Register
Login
Home
