[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] [List Home]
Re: Re[8]: [equinox-dev] OBR2

You are right Jeff, the point is not to boil the ocean.
The point is to try to bring to general availability some of the finer points about provisioning bundles and groups
that we discovered through the years. We do not want to dismiss or forget issues or problems that
we already know about. This is hardly boiling the ocean.

The issue of sharing

As we both know, the question of grouping introduces the issue of sharing...
I agree with you that allowing sharing between groups is more preferable.
But it is not as simple as you put it, unless one has simplifying assumptions.
This is fine, but we need to be really clear about these.

You are correct, deployment packages in MEG owns their bundle, it is somewhat unfortunate, but it is
quite unavailable in the MEG context. You see, groups "configure" their bundles, which includes
both creating configurations for services as well as potentially having side effects on the contents
of the bundle themselves (such as creating and initializing files or local databases in bundles).
Unless I am mistaken, both MEG and Eclipse are potentially in this situation.

If bundles would not be "configured" or "modified" in any way during installation, and there would be no side
effects to the environment because a bundle is installed as part of a group, then I would agree sharing is no
real issue... a simple reference counting scheme between the groups and individual bundles would do the trick
of knowing when a bundle can be uninstalled.

Notice though that this suggests a local shared repository for downloaded bundles (where the reference
counting or some other garbage collection form happens)...

Notice also that we do not have that in Eclipse, as far as I know---but please correct me if I am wrong.

Let's look at an example. I will use one that I know very well, but I am sure we can create more Eclipse-centric
ones through the scripting abilities of features...

Configuring OSGi services is one typical case... that Eclipse is not too accustomed to for it seldom uses or promotes services.
With services, the configuration happens at the level of a group... because several bundles need to be configured together...
if any bundle is shared, then configuring becomes a more complicated issue. It is a problem when installing a new feature
that needs to reconfigure... but also when uninstalling one of the features that had done some configuration...

Beyond that point about sharing, I like the scenario that you outline: discover - acquisition - develop.

Discovering... the first question is about what a developer is trying to discover... is it a bundle?
Well it depends, and in most of the case, it is not really a bundle... He or she is trying to fix
missing dependencies. Eclipse has a simpler environment because it promotes mostly dependencies
between bundles. This is a very usefull case, but it is a specific and simpler case. The same is true
about dependencies on extensions or extension points, since an extension point name is prefixed
by the bundle name.

But for OSGi, with services for example, or even with most Java package dependencies (such as interfaces,
specifications, etc.), the dependencies are not on bundles. One may need a Java package or a service,
not knowing anything about bundles. Therefore, the dependencies need to be translated at
some points to individual bundles or groups of bundles. Independently from groups or bundles,
we need a discovery infrastructure going from a dependency to a bundle or a group of bundles.

This is one point I was making about we still have work at the level of bundles, not even considering
groups. By the way, groups do not really help discovery... it is a matter of scale... for a few thousands
bundles, one may hope few-enough groups for humans to handle. But in a world-wide ecosystem, with
an environment as successful as Eclipse and OSGi, there will be thousands of groups, if not more.
The numbers are a loosing game, with bundles or groups.

So groups do not help discovery... but they help packaging. Groups do help in downloading
coherent sets of bundles. You are correct. It is a useful concept for packaging bundles together into one
coherent unit. They carry a constraint of installing or uninstalling all the parts of the group.

Groups therefore helps acquisition... although they also increase the risk of conflicts.

Acquisition, you are correct, this strongly suggests a resolver of some sort, working hand-in-hand
with the discovery process. Indeed, knowing what group of bundles or individual bundles handle
a dependency is not enough. Conflicts may arise (between versions) or between service
configurations. Also, licensing issues may prevent using a given bundle or bundle group.

Again, the fact that we have groups is somewhat orthogonal to this. We have the same problems
if we consider bundles alone. While groups help through carrying de-facto assembly constaints
and by a matter of granularity, they also make conflicts more likely if bundle sharing happens.
I am not sure any one knows where the right balance is.

Develop, I totally agree with you... that acquisition shall not mean installation... in fact, I would even go
further... that acquisition of the different forms of bundles should appear more seamless and more
integrated than they are today. We have CVS-like path for sources. We have import/export of
binary or binary with sources. We also have the update manager... to "install" binary plugins...
some of which carry sources for other binary plugins...

Having one concept of acquisition would be great.
Having one place to put acquired bundles would also be great, which you call "target area".
Being able to acquire different facets of a bundle would also be great...
Like for instance first acquire a binary version of a bundle... to compile against...
and then easily acquire the source for that binary bundle to be able to debug...

Hope this clarifies things a bit further.
Best regards,

Olivier Gruber, Ph.D.
Persistent & Distributed Object Platforms and Frameworks
IBM TJ Watson Research Center

Jeff McAffer <Jeff_McAffer@xxxxxxxxxx>
Sent by: equinox-dev-bounces@xxxxxxxxxxx

10/02/2005 02:34 PM
Please respond to Equinox development mailing list

        To:        Equinox development mailing list <equinox-dev@xxxxxxxxxxx>
        Subject:        Re: Re[8]: [equinox-dev] OBR2

Fine points Olivier.  We should be careful to not "boil the ocean" as they say.  There are some very real pragmatic problems of consumability here.  A repo of 1000 bundles with no grouping and/or categorization will not acheive the mission we need to acheive.  It seems the basic problem is that developers cannot easily discover, acquire and code against the bundles that are available.  As such, it is not apparent that OSGi is a happening place and the barrier to entry is higher.  The runtime deployment and updating of bundles is certainly interesting but IMHO we are not that far yet and the solution to that problem may will be domain specific.

With that in mind, consider the workflow followed by a developer on a quest for function.

1) discover - They need to browse/search for bundles that meet their requirements or given a bundle, find all the prerequisites.  Catalogs, queries, categories, groupings, etc figure large here as do constraint based traversals.

2) acquisition - The developer wants to get coherent sets of bundles against which he can compile and with which he can test.  This implies some sort of dependency validator or resolver.  Even if grouping mechanisms are used validation is still needed.  

3) develop - Source must be easily obtained and acquisition should not imply installation.  Acquired bundles and source should be put into a "target area" whatever that means for your development setup.

The problems Olivier outlined are real but (thankfully) many do not apply in this usecase.  

As for groups, I am not proposing groups be standardized or imposed on anyone.  Rather groups should be available as *a* means of describing sets.  Another alternative would be JSR124 data and a resolver.  These approaches (and others) should be able to co-exist.  The key is the use of structures of metadata that "talk about" the bundles rather than structures that "contain"/hide the bundles.  For example, multiple Eclipse features can "include" the same bundle.  Features are just metadata about bundles.  Further, the repository structure (update sites) are open and, for example, you are free to go to an update site and get individual bundles.  (I'm not saying features or update sites should be used, but they are an example).  MEG DeploymentPackages on the other hand are not just metadata as they *contain* their bundles.  This is unfortunate.

I hope that clears up some things.


Olivier Gruber <ogruber@xxxxxxxxxx>
Sent by: equinox-dev-bounces@xxxxxxxxxxx

09/29/2005 06:19 PM

Please respond to
Equinox development mailing list

Equinox development mailing list <equinox-dev@xxxxxxxxxxx>
Re: Re[8]: [equinox-dev] OBR2

Peter, Jeff,

Groups have value, but they have problems. But I would like to make sure that we don't impose groups.

We may want to promote one model for groups, through tool support in Eclipse for e.g., but I don't feel

we are in a position to impose a concept of groups, nor that we have a good general solution.

First of all, even we would all agree that groups have value, we have different grouping models.

At least, Eclipse features and MEG deployment packages. I also know of Linux distribution

using RPM deployment to bring down plugins (after all, this is just about downloading files).

I don't think we are in any position to impose a grouping... You are right Peter, there is a lot

of semantics associated with grouping... and it is mostly a concern for packaging and
management tools.

As we all remember, grouping introduces many headaches, especially when sharing
between groups are entertained. Installing and uninstalling groups with non empty
intersections is tricky. As far as I know, no one has a decent and robust solution so far.

This is especially true because groups never stay simple about sets of bundles...

Sooner or later scripting appears... configuration data... etc...

It is the case for Eclipse and it is the case for MEG Deployment Packages...
Then groups get into fragments... and optional downloads based on environment variables...
because applications are formed out of different  configurations in different environments...
and this is not to consider different offerings (different prices for different configurations).

So it is hard to reason about groups in these contexts... to even predict if it can be installed or not.

Then again, you are right Peter... dependencies between groups open up again the semantic
question about flexibility (import-export versus provide-require). Usually dependencies are more like

provide-require... but we need the import-export flexibility... While for Java packages, we can
rely on the specification versus implementation semantics, this is no longer true for groups...

However, there are very good reasons for not using bundles instead of groups...

Admin granule is usually much larger than the runtime bundle granule...

You want to allow flexibility in the configuration of a product or a product suite...

You want fine-grain ability to update... fine-grain security control...
fine-grain lifecyle and lazy activation...

All of which demonstrates that the runtime modularity of a software does not match

very well the admin modularity of it... or its provisioning modularity...
Something that we should may think about more... :-)

But here is my point,
we may allow groups... but it seems that we don't want to specify
them in the OSGi standard. In fact, I like that OSGi is mostly about runtime artifacts, and not

about provisioning. I would argue that we don't want to force the adoption of a group semantics.

I would argue that we need to fit in existing provisioning efforts... like JSR 124... or like OMA...

I would say that we need the following, just for bundles, without talking about groups.

We need to have a provisioning infrastructure for bundles, and bundle alone.

We need to be able to find where a bundle can be downloaded from (technically easy

but not that easy in practice to put in place...) We need to be able to override
a bundle's definition of its dependencies,  so to make it fit in a configuration
when provisioning it (a flexibility we don't have so far).

Above this layer, it should be easy to build the notion of groups, if one wants to.

There is a lot to do at the level of bundles alone...

We have to consider extensibility there for dependencies...

We already have import-export statements... and require-provide...

We also have declarative services... and extensions and extension points
(I know some people that may not want to deploy a plugin that provides extensions
to non-available extension points)...

I like the JSR 124 approach...
where we talk about resources, providing and requiring capabilities.

The approach is extensible with the ability to define new capabilities... with new matching rules.

It would be interesting to see if we can model

      - RFC 79

      - SCR

      - extensions and extension points

What's missing I believe would be a discovery infrastructure...
mapping from capabilities to resources...
some of which will be OSGi bundles.... some of which may be other things.

I feel that OSGi provisioning needs to fit in a larger story that allows to deploy not only bundles...

I don't feel like we will be able to impose a unique grouping model... but through an open approach

we may allow OSGi to be a premier choice for Java component models.

Groups are surely important, and I would suggest that we promote one model through Eclipse

and a good tooling support... mostly as a proof that we can built it on top of the bundle-level
infrastructure... but I don't see why we would want the provisioning infrastructure to be imposing
a grouping model. Promote... but accomodate... in other words... apply modularity and flexibility

to our own designs and architectures... :-)

Best regards,

Olivier Gruber, Ph.D.
Persistent & Distributed Object Platforms and Frameworks
IBM TJ Watson Research Center

Peter Kriens <Peter.Kriens@xxxxxxxx>
Sent by: equinox-dev-bounces@xxxxxxxxxxx

09/29/2005 08:58 AM
Please respond to Equinox development mailing list

      To:        Jeff McAffer <Jeff_McAffer@xxxxxxxxxx>

      cc:        Equinox development mailing list <equinox-dev@xxxxxxxxxxx>

      Subject:        Re[8]: [equinox-dev] OBR2

I think this is the discussion require-bundle versus Import-Package
all over again ... :-)

a priori grouping will favor popular platforms until the extent that
there will be only one dominant platform because is too hard to
support all. (Law of increasing returns). E.g. Microsoft windows.
Without grouping I think it is easier for a management system to
compose a set that is needed, allowing third parties to provide the
glue components.

And I never understood why things are not just merged in a single
bundle if they really have to be together ...

The diversity of the embedded market requires imho a flatter model.
But then again, I guess I am only a theoretician :-(

Kind regards,

  Peter Kriens

JM> For the most part.  I believe that explicit groups are
JM> necessary.  It is natural to group things and then group the
JM> groups etc.  Doing this with queries is possible but ends up being
JM> roundabout.  Your point about explosion is valid but queries don't
JM> solve the problem. For example, who is going to manage the
JM> queries? How do I tell you about my group?  Do I send you my
JM> query?  What if someone adds something that matches the query but
JM> I did not intend for that to be part of the group?  How do you
JM> compose multiple query-based groups to make a composite group?

JM> Groups can be done in a way that allows you to have your
JM> groups and me to have mine.  We can share the group definitions if
JM> we want but do not have to.  If shared they can be inthe repo as
JM> first class enties.  They (perhaps) can be versioned and express
JM> dependencies of their own.

JM> Jeff

JM> Peter Kriens <Peter.Kriens@xxxxxxxx>
JM> Sent by: equinox-dev-bounces@xxxxxxxxxxx
JM> 09/25/2005 11:14 AM
JM> Please respond to
JM>  Equinox development mailing list

JM> To
JM> Jeff McAffer/Ottawa/IBM@IBMCAcc
JM> Equinox development mailing list <equinox-dev@xxxxxxxxxxx>Subject
JM> Re[6]: [equinox-dev] OBR2

JM> I think we agree ...

JM>  The repo metadata should have no semantic knowledge of its contents.
JM>  The requirements/capabilities should be generic. When you import you
JM>  translate the specific (import-package, require-bundle, scree=128x256,
JM>  etc) requirements to generics. This means that the repo does not need
JM>  to have semantic knowledge.

JM>  However, a group should be the result of a query and not explicitly
JM>  submitted to the repo. If you do that, you always end up in a
JM>  combinatorial explosion of groups. The diversity out there is so high
JM>  ... which en then usually means only 1 configuration is supported.
JM>  Works maybe for Windows, but OSGi needs something more flexible imho
JM>  because portability is at the root.

JM>  Kind regards,

JM>      Peter Kriens

JM>> Just to clarify what I after is not having to author two sets
JM>> of metadata.  From a repository USER's point of view I have a
JM>> bundle, I add it to the repo.  I need a bundle, I query the repo
JM>> and get the thing I need/want.  When adding a bundle to the repo
JM>> the repo itself should interrogate the bundle and find any
JM>> dependencies.  

JM>> There is a challenge in that there may be dependencies that
JM>> cannot be detected.  For example, we have this in our Help setup
JM>> where help needs an appserver.  Typically Tomcat is supplied but
JM>> there may well be others.  You would not want to spec help to
JM>> depend on Tomcat.  If we were using services then you would spec
JM>> Help to need the webapp service.  Perhaps when we have declarative
JM>> services that will be a possibility but there will still be such
JM>> scenarios where the public contract implied by services is not
JM>> appropriate and alternative mechanisms (e.g., the extension
JM>> registry) are used.

JM>> Grouping bundles is an interesting notion that solves this
JM>> issue as well as allows function producers to explicitly declare
JM>> sets of bundles that go together.  A bundle group can simply be
JM>> viewed as a bundle that aggregates the constituent parts.  However
JM>> it is actually represented it is just a unit that exports the
JM>> union of the exports in its contents and imports/required the
JM>> union of its prerequisites. etc.  

JM>> Expressing or representing these capabilties and requirements
JM>> in an an open, extensible way is goodness.

JM>> Jeff

JM>> Peter Kriens <Peter.Kriens@xxxxxxxx>
JM>> Sent by: equinox-dev-bounces@xxxxxxxxxxx
JM>> 09/23/2005 12:16 PM
JM>> Please respond to
JM>>  Equinox development mailing list

JM>> To
JM>> Thomas Watson <tjwatson@xxxxxxxxxx>cc
JM>> Equinox development mailing list
JM>> <equinox-dev@xxxxxxxxxxx>, heavy@xxxxxxxxxxxxxxxxxxxxx
JM>> Re[4]: [equinox-dev] OBR2

JM>> Good idea, but I assume it can be used to generate the meta data?

JM>>  I think this is an example where the generic requirements capability
JM>>  model shines. We will have lots of dependencies that need to be
JM>>  modeled. Modeling them semantically in the repo format will make the
JM>>  repo format a major spec work undertaking.

JM>>  Kind regards,

JM>>      Peter Kriens

TW>>> Can we use SCR's component XML definition for service
TW>>> dependancies?  Even if the bundle does not use SCR directly can we
TW>>> still use this schema to define service dependancies?  I would not
TW>>> want to define a whole new schema for this.

TW>>> Richard, are you subsribed the the equinox-dev mailing list?
TW>>>  If not, please do so we don't have to keep cc'ing you on the
TW>>> mails :)

TW>>>  Thomas Watson
TW>>>  Pervasive Development
TW>>>  Phone: 512-838-4533 Tie: 678-4533
TW>>>  tjwatson@xxxxxxxxxx

TW>>> equinox-dev-bounces@xxxxxxxxxxx wrote on 09/23/2005 08:34:11 AM:

>>>> Agree ... if it is not in the manifest, we can't generate it. But it
>>>> is nice to populate a repo with only the data from the manifests and
>>>> get reasonable results.
>>>> Kind regards,
>>>>      Peter Kriens
>>>> RSH> Peter Kriens wrote:
>>>> >>It absolutely MUST be possible to generate the metadata from the
>>>> >>manifest.
>>>> >>  
>>>> >>
>>>> RSH> Well, it is not possible given the current state of bundle manifests to
>>>> RSH> generate all repository metadata. For example, there is no way to
>>>> RSH> generate service dependencies.
>>>> RSH> If someone invents a new capability/requirement pair, there will be no
>>>> RSH> way to generate it either.
>>>> RSH> For the standard package/module-related stuff, we can generate them from
>>>> RSH> the manifest.
>>>> ->> richard
>>>> RSH> _______________________________________________
>>>> RSH> equinox-dev mailing list
>>>> RSH> equinox-dev@xxxxxxxxxxx
>>>> RSH> https://dev.eclipse.org/mailman/listinfo/equinox-dev
>>>> --
>>>> Peter Kriens                              Mob +33633746480
>>>> 9C, Avenue St. Drézéry                    Tel +33467542167
>>>> 34160 Beaulieu, France                    Tel +15123514821
>>>>                                           Tel +33870447986
>>>> AOL,Yahoo, Skype pkriens                  ICQ 255570717
>>>> _______________________________________________
>>>> equinox-dev mailing list
>>>> equinox-dev@xxxxxxxxxxx
>>>> https://dev.eclipse.org/mailman/listinfo/equinox-dev

JM>>  --
JM>>  Peter Kriens                              Mob +33633746480
JM>>  9C, Avenue St. Drézéry                    Tel +33467542167
JM>>  34160 Beaulieu, France                    Tel +15123514821
JM>>                                           Tel +33870447986
JM>>  AOL,Yahoo, Skype pkriens                  ICQ 255570717

JM>>  _______________________________________________
JM>>  equinox-dev mailing list
JM>>  equinox-dev@xxxxxxxxxxx
JM>>  https://dev.eclipse.org/mailman/listinfo/equinox-dev


JM>  --
JM>  Peter Kriens                              Mob +33633746480
JM>  9C, Avenue St. Drézéry                    Tel +33467542167
JM>  34160 Beaulieu, France                    Tel +15123514821
JM>                                           Tel +33870447986
JM>  AOL,Yahoo, Skype pkriens                  ICQ 255570717

JM>  _______________________________________________
JM>  equinox-dev mailing list
JM>  equinox-dev@xxxxxxxxxxx
JM>  https://dev.eclipse.org/mailman/listinfo/equinox-dev

Peter Kriens                              Mob +33633746480
9C, Avenue St. Drézéry                    Tel +33467542167
34160 Beaulieu, France                    Tel +15123514821
                                       Tel +33870447986
AOL,Yahoo, Skype pkriens                  ICQ 255570717

equinox-dev mailing list
equinox-dev mailing list

equinox-dev mailing list