Home » Eclipse Projects » Equinox » Testing bundles from the outside
| Testing bundles from the outside [message #89415] |
Fri, 01 June 2007 15:02  |
Eclipse User |
|
|
|
Originally posted by: sangamon.netcologne.de
Hi,
I'm pretty sure that the answer for this already is out there somewhere,
but I don't seem to get the search right. Please bear with me.
I'm wondering what is the best practice for testing an OSGi bundle with
Equinox from an Ant script that is run in a continuous build loop.
I would imagine something like: Start up (clean) Equinox server, install
target bundle, install test bundle. somehow call a test service method,
after completion shut down bundles and server nicely and pass the
service method's return value as the process return value to the
outside, so Ant knows whether there were failures. Or is that completely
off track? How else would it be done, then?
If possible, I would like to get this done with vanilla Equinox, i.e.
without any Eclipse specific stuff on top of it, or other 3rd party
dependencies.
Can somebody please point me into the direction? A link, or probably
just an appropriate search phrase, might already do the trick. Thanks!
Best regards,
Patrick
|
|
|
| Re: Testing bundles from the outside [message #89445 is a reply to message #89415] |
Fri, 01 June 2007 16:10  |
| Eclipse User |
|
|
|
Originally posted by: alex_blewitt.yahoo.com
This is done for the Eclipse build process, and you can see the way that it invokes a test method in a test bundle. However, there's nothing that is especially eclipse specific here; you could use it to invoke any test case in a bundle. There might be some extra dependencies to make it work, but this shouldn't be a problem and/or affect how your bundle works at all.
If you want to craft it yourself, you'd have to see how it works.
Basically, it's going to look like:
public void start(BundleContext context)
{
this.context = context;
}
bundles = context.getBundles();
for(int i=0;i<bundles;i++)
{
if(bundles[i].getName() == System.getProperty("test.bundle") {
class = bundle.loadClass(System.getProperty("test.case");
Test test = (Test)class.newInstance();
TestResult result = new TestResult();
test.run(result);
}
};
Best practice would be to use what someone else has already done, so I'd use the Eclipse stuff if I were you. There's documentation and articles on the eclipse.org site, and I put together a presentation for EclipseCon06 at http://www.rcpapps.org/EclipseCon2006/
Alex.
|
|
|
| Re: Testing bundles from the outside [message #89520 is a reply to message #89445] |
Mon, 04 June 2007 20:27 |
| Eclipse User |
|
|
|
Originally posted by: sangamon.netcologne.de
Responding to Alex Blewitt:
Thanks for your answer! I fear that I don't seem to get to a real
lightweight solution so far, probably I'm still missing some basics.
>
> public void start(BundleContext context)
> {
> this.context = context;
> }
>
> bundles = context.getBundles();
> for(int i=0;i<bundles;i++)
> {
> if(bundles[i].getName() == System.getProperty("test.bundle") {
> class = bundle.loadClass(System.getProperty("test.case");
> Test test = (Test)class.newInstance();
> TestResult result = new TestResult();
> test.run(result);
> }
> };
>
....but this code is located in a bundle again, which just leaves me with
the question: How do I call this code from the outside, wrapped inside
Equinox setup/teardown, and how do I pass the result of the test run out
to the calling process?
However, sticking to your advice...
> Best practice would be to use what someone else has already done, so
> I'd use the Eclipse stuff if I were you.
....I've come to a solution, but I still wonder whether I can achieve the
same effect in a more lightweight fashion. I've created a very simple
Eclipse application wrapper around the test bundle service call like this.
public class TestStarterApplication implements IApplication {
public Object start(IApplicationContext context) throws Exception {
Bundle bundle = startBundle(TestActivator.BUNDLE_ID);
BundleContext bundleContext = bundle.getBundleContext();
ServiceReference serviceRef =
bundleContext.getServiceReference(TestService.class.getName( ));
TestService testService =
(TestService)bundleContext.getService(serviceRef);
int numFailures = testService.runTests();
return new Integer(numFailures);
}
public void stop() {
}
}
(I'm not sure about all this, of course - if there are any problems with
this approach, please let me know.)
It looks like this does what I want, but using an application gives me
quite some plugin dependencies rooted in org.eclipse.core.runtime, and
this still seems a bit like overkill to me. Is there really no easier
(and perhaps more Eclipse agnostic) way to invoke such a (pure OSGi, non
Eclipse dependent) service from a continuous build script and pass a
result through the teardown to the outside? Ideally I would like to get
away with a single equinox.jar as the test execution environment. :)
> There's documentation and articles on the eclipse.org site, and I put
> together a presentation for EclipseCon06 at
> http://www.rcpapps.org/EclipseCon2006/
I have seen that in-container testing is mentioned in the presentation
slides, however, to me it reads like my approach described above is
already at the lightweight end and that you've brought in even heavier
guns in your project. Is that impression justified, or should I just
take a closer look at the svn project?
Thanks and best regards,
Patrick
|
|
|
| Re: Testing bundles from the outside [message #89562 is a reply to message #89520] |
Tue, 05 June 2007 10:48 |
| Eclipse User |
|
|
|
Originally posted by: alex_blewitt.yahoo.com
Yes, you can use an IApplication to kick off the process. That's what the JUnit runner does (or the previous version of the -application using PlatformRunnable, but the IApplication is the 3.3+ way of doing it)
> It looks like this does what I want, but using an application
> gives me quite some plugin dependencies rooted in
> org.eclipse.core.runtime, and this still seems a bit like
> overkill to me. Is there really no easier (and perhaps more
> Eclipse agnostic) way to invoke such a (pure OSGi, non Eclipse
> dependent) service from a continuous build script and pass a
> result through the teardown to the outside? Ideally I would like
> to get away with a single equinox.jar as the test execution
> environment.
There's no reason why you can't do anything you want to do. I'd argue that it is in fact much, much easier to use the Eclipse JUnit runner (even if that has additional dependencies) than to write everything of your own; so we've got different interpertations of 'overkill'.
Anyway, yes, this bundle would be needed to be started. You acn do that with java -jar equinox.jar -Dtest.bundle=xxx -Dtest.case=xxx -Dosgi.bundles=file://path/to/your/bundle.jar if you really want, or you could bodge the eclipse.ini file to have the same with the right startup environment. You'd also need to make sure that if you were testing bundles in that OSGi instance that you install/started the right bundles (that's what the eclipse update configurator does), which is why it's in the standard eclipse.ini list.
NB you'd almost certainly want more than just 'number of failures' being reported. The way the JUnit runner works is to dump tests out to a file name (e.g. results.xml) which is then passed back and can be processed by one of the JUnitReport style plugins. The name of the file is parameterised in the eclipse JUnit runner.
> Ideally I would like to get away with a single equinox.jar as
> the test execution environment.
I somewhat suspect that you're missing the point of OSGi in general. OSGi systems are made up from a collection of bundles, and it seems that you're fighting really, really hard to put everything into one giant bundle. Why? What's the point? Regardless of choice-of-OSGi environment, they'll all have the same type of bundles inside.
In the case of the core.runtime or even the JUnit test runner; these are both OSGi bundles. It doesn't matter whether the framework you're using is Equinox or Felix or Prosyst or ... every bundle has dependencies on other bundles, and all you need is the minimal set to make something happen. The core.runtime will work fine on other environments, and although I haven't tried, the JUnit runtime is likely to be the same.
What's different about the runtimes is the way taht you start them up, specify where the bundles live etc. I think there's something in the pipeline to standardise how that works, but I've not heard of anything definite in that area.
My point is that the concept of 'pure OSGi that doesn't have any dependencies on Eclipse' is misleading. None of the stuff has any particular dependency on Eclipse; they have dependencies on OSGi bundles. And those bundles will run on whatever OSGi framework you happen to have, regardless of whether the prefix is 'org.eclipse' or 'com.foo'.
It sounds like you're going out of your way to re-write all this stuff to fit into one bundle so that you have zero dependencies on any other bundles. I guess I just don't see the point in that. I also don't believe that one-big-bundle is any more lightweight than a-few-smaller-bundles.
Alex.
|
|
|
| Re: Testing bundles from the outside [message #89655 is a reply to message #89562] |
Tue, 05 June 2007 13:11 |
| Eclipse User |
|
|
|
Originally posted by: sangamon.netcologne.de
Responding to Alex Blewitt:
> NB you'd almost certainly want more than just 'number of failures' being
> reported. The way the JUnit runner works is to dump tests out to a file
> name (e.g. results.xml) which is then passed back and can be processed
> by one of the JUnitReport style plugins. The name of the file is
> parameterised in the eclipse JUnit runner.
The test cases I'm running are not JUnit, but based on a (xUnit style)
custom framework. (No NIH syndrome here, the code and the tests have to
run on a variety of Java and .NET platforms, which pretty much ruled out
vanilla JUnit, plus we needed some additional features we would have had
to hack into JUnit, anyway.) I certainly want reporting/logs from that,
but that's not a problem. The interesting part is that I want the Ant
build (and thus this iteration of the CruiseControl build loop) to fail
if there have been any failing tests.
> I somewhat suspect that you're missing the point of OSGi in general.
> OSGi systems are made up from a collection of bundles, and it seems
> that you're fighting really, really hard to put everything into one
> giant bundle. Why? What's the point? Regardless of choice-of-OSGi
> environment, they'll all have the same type of bundles inside.
I'm not fighting to put everything into one giant bundle, I'm fighting
to keep the environment that needs to be checked out to run the tests to
a minimum. And I really don't understand why I should have the whole of
Eclipse sitting on my lap to run the test suite from CruiseControl, when
I can see it running from a bare equinox.jar in interactive console mode.
Let me try to explain... We have a completely OSGi agnostic product. Now
we have put the product jar inside an OSGi bundle and wrapped it inside
a very thin factory service that basically just takes care that the
product setup is aware of OSGi class loading specifics (that is, it is
configured to use Bundle#getClass()). We have another OSGi bundle that
contains our test suite as a jar and starts it with a fixture based on
the factory service. When I trigger the test runner in the test bundle's
#start() method and run equinox.jar in console mode with only those two
plugins added, the test suite output goes to the console and shows that
it runs green. So far, so good.
Now I want to integrate this into our continuous build process. The Ant
build would have to start up an OSGi environment, trigger the test
suite, cleanly shut down the OSGi environment upon completion, and let
the whole build fail if there have been any test failures. Ideally,
everything required to run the build (including the test environment)
should be checked out of the repository. This is why I'd like to keep
the number of plugin dependencies as small of possible, ideally to zero,
so I only have to keep the equinox.jar around. I wouldn't have thought
that this idea of "in-container" testing in a minimal environment is so
exotic.
> In the case of the core.runtime or even the JUnit test runner; these
> are both OSGi bundles. It doesn't matter whether the framework you're
> using is Equinox or Felix or Prosyst or ... every bundle has
> dependencies on other bundles, and all you need is the minimal set to
> make something happen.
"Minimal set" is the keyword here. :) I'm just surprised that the
minimal set for something I would think of as a very basic unit testing
scenario should consist of at least 10 plugins that don't even belong to
the core Equinox implementation, but rather to a framework on top of it.
(Yes, I start to understand that the separation between OSGi/Equinox and
Eclipse as a framework on top of that seems to be rather blurry.)
> It sounds like you're going out of your way to re-write all this stuff
> to fit into one bundle so that you have zero dependencies on any other
> bundles. I guess I just don't see the point in that. I also don't
> believe that one-big-bundle is any more lightweight than
> a-few-smaller-bundles.
I certainly don't believe that, either. :)
Best regards,
Patrick
|
|
|
| Re: Testing bundles from the outside [message #89699 is a reply to message #89655] |
Tue, 05 June 2007 14:03 |
| Eclipse User |
|
|
|
Originally posted by: alex_blewitt.yahoo.com
> > NB you'd almost certainly want more than just
> > 'number of failures' being reported.
>
> The test cases I'm running are not JUnit, but based
> on a (xUnit style)
> custom framework.
OK. I was just mentioning how the Eclipse JUnit runner works, so that if you wanted to communicate more detail about what the failures were, you know how that is done. Of course, you could use a non-zero return code to indicate that too if you just wanted to stop the build.
> I'm not fighting to put everything into one giant
> bundle, I'm fighting
> to keep the environment that needs to be checked out
> to run the tests to
> a minimum. And I really don't understand why I should
> have the whole of
> Eclipse sitting on my lap to run the test suite from
> CruiseControl, when
> I can see it running from a bare equinox.jar in
> interactive console mode.
Yeah, but 'what you need for a build' is separate from 'what you need to run the product', surely? I mean, you don't ship cruisecontrol as part of your product; and any testing harness is going to be in addition of your product, too. It seems somewhat odd to want the testing harness to be as small as possible.
>
> When I trigger the test runner
> in the test bundle's
> #start() method and run equinox.jar in console mode
> with only those two
> plugins added, the test suite output goes to the
> console and shows that
> it runs green. So far, so good.
>
> Now I want to integrate this into our continuous
> build process. The Ant
> build would have to start up an OSGi environment,
> trigger the test
> suite, cleanly shut down the OSGi environment upon
> completion, and let
> the whole build fail if there have been any test
> failures. Ideally,
> everything required to run the build (including the
> test environment)
> should be checked out of the repository. This is why
> I'd like to keep
> the number of plugin dependencies as small of
> possible, ideally to zero,
> so I only have to keep the equinox.jar around. I
> wouldn't have thought
> that this idea of "in-container" testing in a minimal
> environment is so
> exotic.
No, clearly you can put whatever you want into the test bundle, but your 'equinox' isn't the only dependency; it's also the test harness, too.
> "Minimal set" is the keyword here. :) I'm just
> surprised that the
> minimal set for something I would think of as a very
> basic unit testing
> scenario should consist of at least 10 plugins that
> don't even belong to
> the core Equinox implementation, but rather to a
> framework on top of it.
Why? They're all bundles. And testing isn't in the core spec, so I wouldn't expect it in the main osgi bundle. Whether you want to use the existing setup or you own is up to you, of course, and you have made the decision (both by not wanting JUnit and also by not wanting any other bundles).
So your test harness has to be something on top of equinox, and you're writing your own for your xUnit case as well as compressing it down into one bundle.
|
|
|
| Re: Testing bundles from the outside [message #89728 is a reply to message #89699] |
Tue, 05 June 2007 18:07 |
| Eclipse User |
|
|
|
Originally posted by: alex_blewitt.yahoo.com
Sorry, I got cut off mid post as I had to run to something else.
My point was that there is testing in Eclipse, but using JUnit. So if you want to do something else, you're on your own (figuratively speaking).
Also, any such test harness is going to have dependencies (e.g. 'xUnit.jar') as well as the test harness (e.g. 'xUnitOSGiHarness.jar') and your code under test (which may be one or more bundles). It would be natural to assume, therefore, that these would be provided as separate bundles and as such any test would need all of these to work. If there is common code in the xUnitOSGiHarness, then it would make sense for that to be factored out into a common bundle (so that it could be used in your project, for example).
The thing is, the more modular you get, the more bundles you have. There's nothing specific about the Eclipse JUnit harness; it has dependencies which it needs to run, and that needs to run on an OSGi framework (of which Equinox might be one of them).
Number of bundles isn't a number you should try to minimise just for the heck of it.
So, you'll need more than just equinox.jar in any case; and the fact that you have several dependencies for the test harness in no way makes it more difficult, slow or any less OSGi-based.
In any case, the code I posted originally showed you how to go about doing the testing without any dependencies so I'm sure you could use that to try and build your uber-bundle with everything in it if you wanted to.
Hope that helps,
Alex.
|
|
|
| Re: Testing bundles from the outside [message #89886 is a reply to message #89728] |
Wed, 06 June 2007 16:49 |
| Eclipse User |
|
|
|
Originally posted by: sangamon.netcologne.de
Responding to Alex Blewitt:
First of all, thanks for bearing with me and taking the time for your
very detailed answers.
> Number of bundles isn't a number you should try to minimise just
> for the heck of it.
I'm certainly not trying that. :) I am very much in favor of having
multiple, small modules, and I wouldn't see any problem in moving the
test framework lib to a bundle separate from the actual test cases, for
example. But I still fail to see why I should have to depend on 10+
additional modules just to control the lifecycle of a framework instance
for in-container testing. (Just to make sure, this is not about the
testing framework at all, it's only about orchestrating the sequence
"framework setup -> bundle test run -> framework shutdown -> interpret
result and fail build if needed" from an Ant script.)
> So, you'll need more than just equinox.jar in any case; and the
> fact that you have several dependencies for the test harness in
> no way makes it more difficult, slow or any less OSGi-based.
It's certainly not less OSGi based. But I came to think of OSGi/Equinox
and Eclipse as layers, Eclipse residing on top of OSGi/Equinox. If I'm
told that I should use stuff from the higher layer to test stuff that
depends on the lower layer only, I start wondering.
I'll try the latest suggestion I have from another source, which is to
trigger the test suite from the test plugin's #start() (preferrably in a
dedicated thread), then either brutally call System.exit() with the test
result (0:green/1:red) as the argument, or write the result to a file,
call context.getBundle(0).stop() for a marginally cleaner shutdown
procedure, and inspect the output file from the ant script. This both
sounds rather hackish to me, but if it works, it should hopefully give
me what I want: Minimal requirements for the testing environment.
> In any case, the code I posted originally showed you how to go about
> doing the testing without any dependencies so I'm sure you could use
> that to try and build your uber-bundle with everything in it if you
> wanted to.
As indicated in an earlier post, I'm afraid I still don't see how this
code should help me to solve my actual problem - controlling the life
cycle of an OSGi framework instance by letting it execute exactly one
"service" before shutting down, and passing the result of this "service"
call to the calling process. The code you posted just uses a bundle for
class loading, it runs inside a bundle and it is not involved with
controlling the framework instance life cycle and communication to the
outside at all, AFAICS. (And your assumption that I'm trying to squeeze
everything into one uber-bundle still leaves me with the impression that
I failed to properly describe my intention.)
Best regards,
Patrick
|
|
|
| Re: Testing bundles from the outside [message #89914 is a reply to message #89886] |
Wed, 06 June 2007 19:41 |
| Eclipse User |
|
|
|
Originally posted by: alex_blewitt.yahoo.com
> But I still fail to see why I should have to depend on 10+
> additional modules just to control the lifecycle of a framework
> instance for in-container testing.
You don't, and I never said that you did. In fact, the code I posted showed you how to do this in a single class. In fact, you don't even need to do this from inside the framework; you can do it from outside with the EclipseStarter to get hold of the system BundleContext. You still need *something* though.
> But I came to think of OSGi/Equinox and Eclipse as layers,
> Eclipse residing on top of OSGi/Equinox. If I'm told that I
> should use stuff from the higher layer to test stuff that
> depends on the lower layer only, I start wondering.
This is probably your problem; there's no such concept of layers. It's a tree of dependent bundles; in effect, each bundle is its own layer. Trying to draw an arbitrary line between one-set-of-bundles and another-set-of-bundles doesn't help anything. (Actually, the OSGi spec does talk of layers but they are orthogonal to bundles; in fact, the modularity layer is what makes bundles work, then there's the security layer, the service layer etc.)
Secondly, thing-under-test depends on test-framework and test-harness. In a modularised environment (such as OSGi) these are separate bundles, and that's exactly the case for the JUnit runner stuff (which granted, doesn't serve your needs since you're going down the xUnit framework).
You need *something* in addition to thing-under-test; your test code, the libs that your test code needs (e.g. mocklibraries). Why do/should you care that mocklibraries is one big bundle or several little bundles? It's going to be the same size anyhow.
The test-framework (thing that starts the OSGi layer, loads the needed bundle(s), runs the tests, collects the test data) is again a set of bundles. In fact, it really doesn't matter whether what it's testing is a single bundle with no dependencies or the largest bundle possible that may have thousands of dependencies, such as org.eclipse.jdt.ui which directly or indirectly probably depends on most of the Eclipse SDK install.
Anyway, unfortunately we have seemed to hit an impasse, so whilst it's actually really easy to solve the problem you're talking about we're no nearer to getting our individual points across.
FWIW here's how someone else did it :
http://www.eclipsezone.com/eclipse/forums/m92129768.html
Alex.
|
|
|
| Re: Testing bundles from the outside [message #90655 is a reply to message #89914] |
Thu, 14 June 2007 13:33 |
| Eclipse User |
|
|
|
Originally posted by: sangamon.netcologne.de
Responding to Alex Blewitt:
> FWIW here's how someone else did it :
>
> http://www.eclipsezone.com/eclipse/forums/m92129768.html
Nice. I have modified this a bit to call a service via reflection. This
really looks like the best solution so far: Clean setup/teardown of the
framework, clean, unrestricted transmission of the test result, no
dependencies other than equinox.jar and my bundles. I think I'll go with
that. Sorry if that's what you have been suggesting with the code
snippet in your initial posting - I didn't get it (and still don't), it
looked like bundle-internal code to me.
Thanks a lot for your help and patience!
Best regards,
Patrick
|
|
|
Goto Forum:
Current Time: Fri Apr 26 09:45:14 GMT 2024
Powered by FUDForum. Page generated in 0.05104 seconds |
] 
Search
Help
Register
Login
Home
