Eclipse Community Forums: SWTBot

Help

Home

Home » Eclipse Projects » SWTBot » Flaky Tests(How to handle Flaky Tests with SWTBot)

Show: Today's Messages :: Show Polls :: Message Navigator

Flaky Tests [message #1742861]

Thu, 08 September 2016 09:25

Vivek Veera

Messages: 18
Registered: July 2015

Junior Member

Team,

Does SWTBot support Flaky Tests.
I have some tests which fail at times and run fine in next execution. It could be due to timing issues. I am yet to identify the exact reason but tests run fine without any changes next time. Therefore, to handle such tests can I use @Flaky annotation (org.eclipse.xtext.junit4.Flaky) so that tests are executed more than once.

Does SWTBot support @Flaky annotation or we have any other option to handle such test cases in SWTBot. Please let me know if you have any information on this.

Appreciate your time.

Thanks.
Vivek

Report message to a moderator

Re: Flaky Tests [message #1742928 is a reply to message #1742861]

Thu, 08 September 2016 15:36

Patrick Tasse

Messages: 85
Registered: July 2009

Member

Hi Vivek,

In our project we are also affected by intermittent build failures. Some are due to flaky SWTBot handling, but some are due to other reasons (race conditions, connection issues, etc.).

I think we prefer the annoyance of having to retrigger the builds to motivate us to troubleshoot and fix the issue. We also track them with bugs to increase visibility (see http://eclip.se/486207). If the failure rate is so high that it interferes with our ability to merge patches, then in rare occasions we annotate the test with @Ignore but that is more an indication that the test was not in good enough condition to be ready for use.

If we annotate tests with @Ignore or @Flaky, there is less motivation to investigate the problem. If the issue is with SWTBot code itself, it would benefit all its users if the test framework was corrected or made more robust.

Patrick

Report message to a moderator

Re: Flaky Tests [message #1742980 is a reply to message #1742928]

Fri, 09 September 2016 02:29

Vivek Veera

Messages: 18
Registered: July 2015

Junior Member

Thanks Patrick.

I understand and agree with what you said. Using @Flaky will hide the real issue.

In fact, it will be very helpful if we have a 'Test Environment Checklist' to avoid test failures due to SWTBot timing issues. Something like;

Test Env Checklist:
1> Stop other UI applications while running tests as this might shift the focus.
2> Stop Keyboard activity as this might shift the focus.
3> Do not run build while running SWTBot test.
4> Avoid...etc.
5> ...

I could not get much on this from the SWTBot site and user guide. I believe its very important to have such checklist or guidelines to avoid test failures due to environment and other factors not related to test code mainly.

Thanks for sharing the information. Appreciate your time.

Vivek

Report message to a moderator

Previous Topic:	news.eclipse.org is shutting down.
Next Topic:	[solved] Eclipse-Plugin-Test: Log4J Error by using SWTBot

Goto Forum:

-=] Back to Top [=-

[ Syndicate this forum (XML) ] [

]

Current Time: Thu Sep 12 22:06:18 GMT 2024

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter