Home » Archived » Buckminster » Problems compiling and running tests the Buckminster way.
| Problems compiling and running tests the Buckminster way. [message #378948] |
Wed, 30 July 2008 18:59  |
 James H
Messages: 5
Registered: July 2009 |
Junior Member |
|
|
Hi,
I'm using the buckminster headless (3.4) to manage the build process of
an RCP and related artefacts (update site, etc). I'm running buckminster
via ant, and would like to utilize the generation of CSPEC from whatever
metadata PDE needs (i.e. as much as possible buckminster is configured
externally from the code it builds).
I have a couple of questions with regard to the compiling and running of
JUnit tests.
I hope this is the right place to post this question.
First issue - I had a lot of difficulty getting the workspace to
compile, as my projects were arranged with unit tests (JUnit) under a
test source folder under each project:
org.acme.model
+ src
+ test
` META-INF
This is a convenient way to keep code and tests together, with the
intention to avoid bundling the tests. The way we have done it in the
past is to add JUnit to the .classpath, so we can run the tests
manually, and then a complicated ant behemoth which we're hoping to
replace with Buckminster/PDE.
I found that the source folder entries in the .classpath file was being
respected (ooh, very clever!), but it didn't seem to be picking up the
org.eclipse.jdt.junit.JUNIT_CONTAINER/3 container (ah).
Ideally, I don't want to move the these tests into a seperate test
bundle fragment, as the tests should be testing isolated classes, and
perhaps interactions between small numbers of classes from the same
bundle (so no external dependencies).
Having said this, this was the only way I could get the compilation to
complete successfully.
So first question: is there a fix so I can compile these tests in place
within the bundle?
My second question is to do with buckminster.test.
I am a little bit confused; I haven't been able to get anything out of
it at all:
My ant script gives output like this:
[java] test '--directory' 'build/test-results'
'runtime:org.acme.model.test:/plugin-tests.lst'
[java] Running test suite: org.acme.model.test:/plugin-tests.lst
BUILD SUCCESSFUL
Total time: 3 seconds
Though looking at the results-*.xml I see this XML:
<testsuites>
<testsuite
name="org.acme.model.test:/plugin-tests.lst"
tests="0" failures="0" errors="0" timestamp="1217438215287" time="0.0">
<error message="No such bundle: org.acme.model.test"
type="java.lang.RuntimeException">java.lang.RuntimeException: No such
bundle: org.acme.model.test
at
org.eclipse.buckminster.test.junit.TestCommand$TestLocationR esolver$UnresolvedTestSuiteDescriptor.<init>(T estCommand.java:100)
at
org.eclipse.buckminster.test.junit.TestCommand$TestLocationR esolver.resolveLocation(TestCommand.java:367)
at
org.eclipse.buckminster.test.junit.TestCommand$TestLocationR esolver.<init>(TestCommand.java:185)
at
org.eclipse.buckminster.test.junit.TestCommand.run(TestComma nd.java:523)
at
org.eclipse.buckminster.cmdline.AbstractCommand.basicRun(Abs tractCommand.java:80)
at org.eclipse.buckminster.cmdline.Headless.run(Headless.java:1 88)
at org.eclipse.buckminster.cmdline.Headless.run(Headless.java:1 22)
at org.eclipse.buckminster.cmdline.Headless.start(Headless.java :113)
at
<!-- snip -->
I am running every buckminster command through the same ant macro so
running checkout, build, test, package should all use the same
workspace, buckminster.output.root etc:
<macrodef name="buckminster">
<attribute name="command" />
<attribute name="loglevel" default="INFO" />
<element name="arg-elements" optional="true" implicit="true" />
<sequential>
<java jar="${startup.jar}" fork="true">
<sysproperty key="buckminster.output.root"
value="${buckminster.root}" />
<arg value="-data" />
<arg value="${workspace}" />
<arg value="-consoleLog" />
<arg value="--displaystacktrace" />
<arg value="--loglevel" />
<arg value="@{loglevel}" />
<arg value="@{command}" />
<arg-elements />
</java>
</sequential>
</macrodef>
I can see the bundle in both ${workspace}/plugins and
${buckminster.root}/org.acme.model.test_1.0.0-osgi-bundle.
plugin-tests.lst is also listed in the build.properties on
org.acme.model.test.
Looking at the source code, the TestCommand looks for the bundle under
test with the PackageAdmin -
http://www2.osgi.org/javadoc/r4/org/osgi/service/packageadmi n/PackageAdmin.html
, which I assume will be looking at the host framework - i.e. the one
that the buckminster test is running on.
Again, my assumption is that I've set this up wrong. Asking the
community is the next step to my enlightenment. :)
Just for information:
$ ./buckminster listsite
Features:
org.eclipse.buckminster.core.headless.feature_1.1.0.r09455(B uckminster -
Core Headless)
org.eclipse.buckminster.pde.headless.feature_1.1.0.r09455
(Buckminster - PDE support for headless operation)
org.eclipse.buckminster.product_1.1.0.r09455 (Buckminster Product)
org.eclipse.buckminster.subclipse.headless.feature_1.0.0.r09 466
Buckminster - Subclipse based SVN support for headless operation)
org.eclipse.buckminster.test.headless.feature_1.0.0 (Feature Feature)
org.eclipse.equinox.executable_3.3.100.v20080606-7H-ELQEDdVv 0SVkW65GwAA
(Eclipse Platform Launchers)
Many thanks, and regards,
James
|
|
| |
| Re: Problems compiling and running tests the Buckminster way. [message #379282 is a reply to message #378948] |
Wed, 06 August 2008 12:53  |
Michal Ruzicka
Messages: 23
Registered: July 2009 |
Junior Member |
|
|
Hi James,
I hope you were able to sort out your compilation problems with Thomas,
in this post I try to answer your testing related questions.
James H napsal(a):
> Hi,
>
> I'm using the buckminster headless (3.4) to manage the build process of
> an RCP and related artefacts (update site, etc). I'm running buckminster
> via ant, and would like to utilize the generation of CSPEC from whatever
> metadata PDE needs (i.e. as much as possible buckminster is configured
> externally from the code it builds).
>
> I have a couple of questions with regard to the compiling and running of
> JUnit tests.
>
> I hope this is the right place to post this question.
>
> First issue - I had a lot of difficulty getting the workspace to
> compile, as my projects were arranged with unit tests (JUnit) under a
> test source folder under each project:
>
> org.acme.model
> + src
> + test
> ` META-INF
>
> This is a convenient way to keep code and tests together, with the
> intention to avoid bundling the tests. The way we have done it in the
> past is to add JUnit to the .classpath, so we can run the tests
> manually, and then a complicated ant behemoth which we're hoping to
> replace with Buckminster/PDE.
>
> I found that the source folder entries in the .classpath file was being
> respected (ooh, very clever!), but it didn't seem to be picking up the
> org.eclipse.jdt.junit.JUNIT_CONTAINER/3 container (ah).
>
> Ideally, I don't want to move the these tests into a seperate test
> bundle fragment, as the tests should be testing isolated classes, and
> perhaps interactions between small numbers of classes from the same
> bundle (so no external dependencies).
>
> Having said this, this was the only way I could get the compilation to
> complete successfully.
>
> So first question: is there a fix so I can compile these tests in place
> within the bundle?
>
> My second question is to do with buckminster.test.
>
> I am a little bit confused; I haven't been able to get anything out of
> it at all:
>
> My ant script gives output like this:
> [java] test '--directory' 'build/test-results'
> 'runtime:org.acme.model.test:/plugin-tests.lst'
> [java] Running test suite: org.acme.model.test:/plugin-tests.lst
> BUILD SUCCESSFUL
> Total time: 3 seconds
>
> Though looking at the results-*.xml I see this XML:
> <testsuites>
> <testsuite
> name="org.acme.model.test:/plugin-tests.lst"
> tests="0" failures="0" errors="0" timestamp="1217438215287"
> time="0.0">
> <error message="No such bundle: org.acme.model.test"
>
> type="java.lang.RuntimeException">java.lang.RuntimeException: No such
> bundle: org.acme.model.test
> at
> org.eclipse.buckminster.test.junit.TestCommand$TestLocationR esolver$UnresolvedTestSuiteDescriptor.<init>(T estCommand.java:100)
>
> at
> org.eclipse.buckminster.test.junit.TestCommand$TestLocationR esolver.resolveLocation(TestCommand.java:367)
>
> at
> org.eclipse.buckminster.test.junit.TestCommand$TestLocationR esolver.<init>(TestCommand.java:185)
>
> at
> org.eclipse.buckminster.test.junit.TestCommand.run(TestComma nd.java:523)
>
> at
> org.eclipse.buckminster.cmdline.AbstractCommand.basicRun(Abs tractCommand.java:80)
>
> at org.eclipse.buckminster.cmdline.Headless.run(Headless.java:1 88)
> at org.eclipse.buckminster.cmdline.Headless.run(Headless.java:1 22)
> at
> org.eclipse.buckminster.cmdline.Headless.start(Headless.java :113)
> at
> <!-- snip -->
>
> I am running every buckminster command through the same ant macro so
> running checkout, build, test, package should all use the same
> workspace, buckminster.output.root etc:
>
> <macrodef name="buckminster">
> <attribute name="command" />
> <attribute name="loglevel" default="INFO" />
> <element name="arg-elements" optional="true" implicit="true" />
> <sequential>
> <java jar="${startup.jar}" fork="true">
>
> <sysproperty key="buckminster.output.root"
> value="${buckminster.root}" />
> <arg value="-data" />
> <arg value="${workspace}" />
> <arg value="-consoleLog" />
> <arg value="--displaystacktrace" />
> <arg value="--loglevel" />
> <arg value="@{loglevel}" />
> <arg value="@{command}" />
> <arg-elements />
>
> </java>
> </sequential>
> </macrodef>
>
> I can see the bundle in both ${workspace}/plugins and
> ${buckminster.root}/org.acme.model.test_1.0.0-osgi-bundle.
>
> plugin-tests.lst is also listed in the build.properties on
> org.acme.model.test.
>
> Looking at the source code, the TestCommand looks for the bundle under
> test with the PackageAdmin -
> http://www2.osgi.org/javadoc/r4/org/osgi/service/packageadmi n/PackageAdmin.html
> , which I assume will be looking at the host framework - i.e. the one
> that the buckminster test is running on.
Yes, that is right. Buckminster test command really expects the bundles
to be already part of the host framework.
The reason behind that was to make the testing process as simple as
possible, preferably by executing the tests within a single framework
i.e. without starting other frameworks. (This was due to my frustration
from running tests using TPTP where there are three frameworks involved
in test execution which doesn't make the process exactly straightforward.)
On the other hand I see the relevance of your expectation that the
Buckminster test command should look for tests in the current workspace
similar to the other Buckminster commands which also mostly operate on
things in the current workspace, so I've entered a bugzilla issue on that:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=243293
The solution to the problem can be easily described as follows:
1) By using functionality already present in Buckminster create a new
product (= new framework) containing all the bundles to be tested and
their dependencies together with a few Buckminster bundles that will
drive the test execution.
2) Execute the tests in the newly created product / framework using the
Buckminster's test command in its current form.
As simple as the description seems it is not as simple when we look at
it in more detail. Specifically there are these problems that need to be
addressed:
1) Is the set of bundles described as "all the bundles to be tested and
their dependencies" the right set to run the tests on, shouldn't there
be some more bundles included - perhaps all the bundles in the workspace?
2) Which workspace to run the new framework on? Note that the workspace
the Buckminster is running on can't be used just because the Bucminster
is already running there. Should a new workspace be created?
The good news is that it is possible to implement something very similar
to what has been described with the current Buckminster.
All you need to do is to:
1) Create a product description (for an example see
org.eclipse.buckminster.product) containing all the bundles that you
intend to test plus its dependencies and the
org.eclipse.buckminster.test.headless.feature feature and its dependencies.
2) Build the product by executing create.<product name> Buckminster
action on the product description.
3) Execute the test by invoking the Buckminster test command on the
newly created product.
Note that by following these instructions you have the flexibility to
define all the details - namely:
- the set of bundles that should be included in the product to be tested
- the workspace to execute the tests on
Hope this was helpful.
Regards,
Michal
|
|
|
| Re: Problems compiling and running tests the Buckminster way. [message #379286 is a reply to message #379282] |
Tue, 12 August 2008 11:07 |
James H
Messages: 5
Registered: July 2009 |
Junior Member |
|
|
Hi, Michal
Thanks for your comments. I've inlined my replies below.
Thanks, and regards,
James
Michal Ruzicka wrote:
> I hope you were able to sort out your compilation problems with Thomas,
> in this post I try to answer your testing related questions.
In the end, I split the tests out into a seperate fragment. This is
sub-optimal, since the unit tests seem to naturally live with the src.
>> Looking at the source code, the TestCommand looks for the bundle under
>> test with the PackageAdmin -
>> http://www2.osgi.org/javadoc/r4/org/osgi/service/packageadmi n/PackageAdmin.html
>> , which I assume will be looking at the host framework - i.e. the one
>> that the buckminster test is running on.
>
> Yes, that is right. Buckminster test command really expects the bundles
> to be already part of the host framework.
Ah, ok. This isn't ideal, but I understand how this implementation came
about.
> The reason behind that was to make the testing process as simple as
> possible, preferably by executing the tests within a single framework
> i.e. without starting other frameworks. (This was due to my frustration
> from running tests using TPTP where there are three frameworks involved
> in test execution which doesn't make the process exactly straightforward.)
I don't see starting new frameworks as particularly problematic, but
only if starting a new framework is easy for the bundle developer to do
as part of the writing of tests.
> On the other hand I see the relevance of your expectation that the
> Buckminster test command should look for tests in the current workspace
> similar to the other Buckminster commands which also mostly operate on
> things in the current workspace, so I've entered a bugzilla issue on that:
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=243293
Yes this is exactly what I'd be expecting.
I see a number of different tests that could come under the scope of the
buckminster test bundle:
* unit tests and integration tests at the class level - so using mock
classes to stub out dependencies for classes and packages under test.
These should have no dependency on an OSGi framework and should be run
in a standalone manner. I expect this is low hanging fruit for a test
bundle in the buckminster OSGi framework.
* unit and integration tests at the bundle level - I'm thinking
mock/stub bundles/services. These would almost certainly involve the
launching of OSGi frameworks, though it's unclear who is responsible for
that.
As if this requirment set isn't enough, we'd also like to be able to
instrument the tests to generate coverage reports e.g. Emma,
Cobertura... though I guess you have hooks for this sort of thing.
> The solution to the problem can be easily described as follows:
>
> 1) By using functionality already present in Buckminster create a new
> product (= new framework) containing all the bundles to be tested and
> their dependencies together with a few Buckminster bundles that will
> drive the test execution.
>
> 2) Execute the tests in the newly created product / framework using the
> Buckminster's test command in its current form.
> As simple as the description seems it is not as simple when we look at
> it in more detail. Specifically there are these problems that need to be
> addressed:
> 1) Is the set of bundles described as "all the bundles to be tested and
> their dependencies" the right set to run the tests on, shouldn't there
> be some more bundles included - perhaps all the bundles in the workspace?
There's also the point of interactions between different bundles you're
trying to test in isolation. e.g. Bundles A and B have test fragments At
and Bt. B happens to contribute a service which A uses. So even though A
and B are loosely coupled, you're still having an A-B interaction when
you only want to test A, which may cause tests in At to fail, or even
difficult or impossible to write.
> 2) Which workspace to run the new framework on? Note that the workspace
> the Buckminster is running on can't be used just because the Bucminster
> is already running there. Should a new workspace be created?
Yes, I see the problem you're alluding to. But I think it's even worse
than that.
Ideally, each /test/ should be launching its own specially configured
framework, to make the dependency graph that you're testing as small as
possible.
I'd like to be able to programmatically generate the configurations and
mock services/bundles/plugin.xml at test time, to be deployed into a
testing OSGi framework.
We're currently investigating
http://static.springframework.org/osgi/docs/1.1.0/reference/ html/testing.html
which you are probably already aware of.
Of course, there is still the difficulty of expressing which tests
should be invoked as a standalone, and which should be invoked with a
bundle framework, and of course trying to keep the burden of maintaining
the metadata to a minimum.
I'm sorry I couldn't be more helpful.
Many thanks, and regards,
James
|
|
|
Goto Forum:
Current Time: Thu Apr 25 07:09:44 GMT 2024
Powered by FUDForum. Page generated in 0.03693 seconds |
] 
Search
Help
Register
Login
Home
