| Supporting Parallel Builds [message #580960] |
Sun, 04 October 2009 15:58 |
 Henrik Lindberg
Messages: 2509
Registered: July 2009 |
Senior Member |
|
|
I think the b3 model should be capable of describing rules for parallel
builds.
If all actors are perfect and without implicit dependencies then a b3
build model could be executed in parallel since the dependencies (and
thus build order) is known.
In practice however, there are several cases where parallel building
will potentially fail:
- an actor reuses the same temporary files and can not be told to give
them unique names - invocations of this actor must be serialized
- an actor produces side effects that are not modeled as dependencies
(this should really be considered a bug in the build model)
- two completely separate actions make use of the same temporary files,
or directory - and need to be serialized.
Taking yet a step and also thinking about parallelism and distributed
builds:
- an actor is not 100% done when it returns control (e.g one machine
writes over NFS and output is not flushed when second machine thinks
result is there)
I think it is quite easy to extend the b3 model to allow it to express
where parallel execution should not take place.
I think we simply need to be able to declare the following as being
synchronous (or not):
- Actor - when known that a "compiler" can not be run concurrently.
- Parts - in particular actions (if the same action is used many times
and it rebuilds with different properties each time)
- BuildUnit - The entire build unit has complex implicit dependencies
and we want to block all parallel execution.
In addition to this, there may be issues with two completely separate
actions in two separate build units where they happen to use the same
temporary files or directory. To solve this a general mechanism is
needed. Here a pseudo declaration:
synchronize {
foo#actionA, bar#actionB;
fee#actionC, fum#actionD;
}
i.e. basically stating - don't run these at the same time.
An other alternative would be to force an order by making one depend on
the other.
Naturally, the synchronized flag can be set via advice - e.g. on an
Actor as it may be impossible to know which units/actions that makes use
of a particular actor. Also, the synchronize statements could allow
patterns - i.e. "org.myorg.*". Yet another possibility is to allow Xpath
expressions in the synchronize statements (shown above) to define the
collections of things that needs to be synchronized.
A synchronize declaration can result in deadlock, but this should be
possible to detect before starting as all synchronize declarations are
available, and the path of execution is known. A deadlock can then be
avoided by serializing.
I propose that we at least add a boolean "Synchronized" to the elements
in the model, but that the first implementation of an Engine performs
serial builds, and that we wait with the synchronized section in the
build file until syntax is nailed down.
Thoughts?
- henrik
|
|
|
Powered by
FUDForum. Page generated in 0.03007 seconds
] 
Search
Help
Register
Login
Home
