Eclipse Community Forums: RCP Testing Tool » Running test runner against sources

Help

Home

Home » Eclipse Projects » RCP Testing Tool » Running test runner against sources

Show: Today's Messages :: Show Polls :: Message Navigator

Running test runner against sources [message #1719475]

Fri, 08 January 2016 09:25

Joseph Benavides

Messages: 47
Registered: February 2015

Member

Hi Guys,

Happy new year to all!

I was wondering if it is possible to run the test runner against sources instead of an actual RCP application?

Asking because we're deploying a project as plugins not as an RCP application.

cheers!

Joseph

Report message to a moderator

Re: Running test runner against sources [message #1719668 is a reply to message #1719475]

Mon, 11 January 2016 09:35

Krzysztof Jozefowicz

Messages: 19
Registered: May 2015

Junior Member

Interesting question, on our project ( https://github.com/nokia/RED ) we have intermediate step of integration with clean Eclipse prior RCPTT tests.
This might seems as unnecessary step but we threat RCPTT as functional/acceptance tests done on integrated feature (running from code might hide issues).
As a note we found out that one of the flavors of Eclipse ( EE edition ) behaved differently than others.

Report message to a moderator

Re: Running test runner against sources [message #1719792 is a reply to message #1719668]

Tue, 12 January 2016 05:51

Joseph Benavides

Messages: 47
Registered: February 2015

Member

Hi Krzysztof, thanks for your response. I was afraid that it would have to be something like that as well. I was just wondering if it could be possible at all. I wanted to run it against our master branch without having to do an integration with a clean Eclipse. Anyway, in case we really have to go through that route, I'd like to ask for some guidance on how you did the integration step?

cheers!

Report message to a moderator

Re: Running test runner against sources [message #1719805 is a reply to message #1719792]

Tue, 12 January 2016 08:28

Krzysztof Jozefowicz

Messages: 19
Registered: May 2015

Junior Member

Hi Joseph

Our workflow looks as follows ( done on Jenkins unattended using multiple nodes, Linux & Windows for testing ) :

1. build from sources, when successful:
2. create custom test suites for tests,bugs,debug and unstable test ( this way we can run in parallel group of tests plus tester does not need to manage main test suite with all tests - just add proper tags. check github for python scripts )
3. execute in parallel on multiple nodes ( win and linux) separate suites
- during this step, clean eclipse is unziped, cli silent install is performed:

eclipse/eclipse -application org.eclipse.equinox.p2.director -nosplash -consoleLog --launcher.suppressErrors -repository jar:file:$RedFeatureFile\!/,http://download.eclipse.org/releases/mars/   -installIU org.robotframework.ide.eclipse.main.feature.feature.group -vmargs -Dosgi.requiredJavaVersion=1.7

4. after each group ends, html reports are updated with paths to failed test within RCPTT project (for our convenience).

Whole tests ( around 150 top level scenarios ) takes 50 minutes (we could shave it by 30-40% by profiling RED & RCPTT or including more parallel sets), additionally we have jobs to execute user selected test (takes ~60s to execute together with eclipse integration) .

Report message to a moderator

Re: Running test runner against sources [message #1720662 is a reply to message #1719805]

Wed, 20 January 2016 08:39

Joseph Benavides

Messages: 47
Registered: February 2015

Member

Hi Krzysztof,

Thank you for that information. That helps a lot. I'm not fairly well versed with all the features for Jenkins but how are some of these steps done in Jenkins? I would suppose 2 and 3 are done as separate Jenkins jobs and the silent install step is done via a script?

cheers!

Joseph

Report message to a moderator

Re: Running test runner against sources [message #1720730 is a reply to message #1720662]

Wed, 20 January 2016 14:54

Krzysztof Jozefowicz

Messages: 19
Registered: May 2015

Junior Member

Hi

Each point which I mentioned is a separate job/jobs . This can be done on several different ways, our approach is as follows :
separate jobs for:
1. build-job -> produces feature zip as job artefact
2. create custom suites job
3. execute test jobs - get #1 artefact (zip file with feature) + get tests from repo with #2 changes, in consecutive execute shell blocks "install on local eclipse #1" , "execute custom testcases from #2" etc.

Another, top level job manages above, with parallel execution (simple SDL by https://wiki.jenkins-ci.org/display/JENKINS/Build+Flow+Plugin ) . it starts when svn repo change is indicated. Body of SDL script starts #1 ,#2, than parallel #3 variations (each tests subsets generated by #2).

In our CI this allow us to have automated build&testing but also a way to manual trigger any of the CI process if needed.

Report message to a moderator

Re: Running test runner against sources [message #1720839 is a reply to message #1720730]

Thu, 21 January 2016 11:01

Joseph Benavides

Messages: 47
Registered: February 2015

Member

Hi Krzysztof,

Stupid question, how do you run shell blocks for the "install on local eclipse" step.

cheers!

Joseph

Report message to a moderator

Re: Running test runner against sources [message #1721005 is a reply to message #1720839]

Fri, 22 January 2016 13:36

Krzysztof Jozefowicz

Messages: 19
Registered: May 2015

Junior Member

Hi

I may wrote it in wrong form, "local" is in my case a eclipse binary on CI node, not as "local" on my personal laptop.
On each CI node, we have a folder with binaries in "clean state"as zips, job unzips those binaries ( eclipse,rcptt runner) to job workspace ,does the feature install to eclipse and execute rcptt tests. After job is completed, workspace is deleted to preserve disk space.

We use consecutive execute shell blocks for our convenience and maintainability , all actions (copy bins, install feature, run tests, update report ) can be run within one shell block.

Report message to a moderator

Re: Running test runner against sources [message #1721528 is a reply to message #1721005]

Thu, 28 January 2016 04:23

Joseph Benavides

Messages: 47
Registered: February 2015

Member

Hi Krzysztof,

Thanks for the response. I actually understood what you meant by "local". I do have some other questions if you don't mind.

1. Is the clean eclipse binary already located in your CI node or is it downloaded everytime the job is run?
2. Since the concept of shell blocks in Jenkins is fairly new to me (I've only used Maven projects the entire time), how is it done? I did see I could add consecutive shell blocks in a Jenkins free-style project. Is that the correct one to use?

cheers!

Joseph

Report message to a moderator

Re: Running test runner against sources [message #1721541 is a reply to message #1721528]

Thu, 28 January 2016 08:37

Krzysztof Jozefowicz

Messages: 19
Registered: May 2015

Junior Member

#1 It depends on you setup and testing strategy. Even with good connection, it takes around 5-10 minutes to download Eclipse binary. When you need to use latest stable Eclipse release, it's better to cache it locally (either on node or in LAN).
In our setup, we preferred to store the eclipse zip on each CI node. As each node can run multiple jobs at the same time, we copy eclipse to each job workspace at pre setup, when job finishes, workspace with binaries is cleared. There was not big overhead (max 10s) of unzipping eclipse compared to using the old one from previous build so we decided to unzip clean on each build pass. It up to you.

#2 each job can consists of steps, with pre and post job steps as well. When using multiple run shell blocks just remember that any environments parameters and changes are visible in that block only. So for instance when you follow running RCPTT on linux tutorial, pay attention that metacity initialization should be in the same block as running Rcptt.

Report message to a moderator

Previous Topic:	Error running multiple sessions on Mac
Next Topic:	eclExecutionDelay has no effect in pom.xml

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Sat Apr 27 00:22:57 GMT 2024

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter