Skip to main content


Eclipse Community Forums
Forum Search:

Search      Help    Register    Login    Home
Home » Eclipse Projects » SWTBot » Flaky Tests(How to handle Flaky Tests with SWTBot)
Flaky Tests [message #1742861] Thu, 08 September 2016 09:25 Go to next message
Vivek Veera is currently offline Vivek VeeraFriend
Messages: 18
Registered: July 2015
Junior Member
Team,

Does SWTBot support Flaky Tests.
I have some tests which fail at times and run fine in next execution. It could be due to timing issues. I am yet to identify the exact reason but tests run fine without any changes next time. Therefore, to handle such tests can I use @Flaky annotation (org.eclipse.xtext.junit4.Flaky) so that tests are executed more than once.

Does SWTBot support @Flaky annotation or we have any other option to handle such test cases in SWTBot. Please let me know if you have any information on this.

Appreciate your time.

Thanks.
Vivek
Re: Flaky Tests [message #1742928 is a reply to message #1742861] Thu, 08 September 2016 15:36 Go to previous messageGo to next message
Patrick Tasse is currently offline Patrick TasseFriend
Messages: 53
Registered: July 2009
Member
Hi Vivek,

In our project we are also affected by intermittent build failures. Some are due to flaky SWTBot handling, but some are due to other reasons (race conditions, connection issues, etc.).

I think we prefer the annoyance of having to retrigger the builds to motivate us to troubleshoot and fix the issue. We also track them with bugs to increase visibility (see http://eclip.se/486207). If the failure rate is so high that it interferes with our ability to merge patches, then in rare occasions we annotate the test with @Ignore but that is more an indication that the test was not in good enough condition to be ready for use.

If we annotate tests with @Ignore or @Flaky, there is less motivation to investigate the problem. If the issue is with SWTBot code itself, it would benefit all its users if the test framework was corrected or made more robust.

Patrick
Re: Flaky Tests [message #1742980 is a reply to message #1742928] Fri, 09 September 2016 02:29 Go to previous message
Vivek Veera is currently offline Vivek VeeraFriend
Messages: 18
Registered: July 2015
Junior Member
Thanks Patrick.

I understand and agree with what you said. Using @Flaky will hide the real issue.

In fact, it will be very helpful if we have a 'Test Environment Checklist' to avoid test failures due to SWTBot timing issues. Something like;

Test Env Checklist:
1> Stop other UI applications while running tests as this might shift the focus.
2> Stop Keyboard activity as this might shift the focus.
3> Do not run build while running SWTBot test.
4> Avoid...etc.
5> ...

I could not get much on this from the SWTBot site and user guide. I believe its very important to have such checklist or guidelines to avoid test failures due to environment and other factors not related to test code mainly.

Thanks for sharing the information. Appreciate your time.

Vivek
Previous Topic:news.eclipse.org is shutting down.
Next Topic:[solved] Eclipse-Plugin-Test: Log4J Error by using SWTBot
Goto Forum:
  


Current Time: Fri Sep 21 03:40:45 GMT 2018

Powered by FUDForum. Page generated in 0.01383 seconds
.:: Contact :: Home ::.

Powered by: FUDforum 3.0.2.
Copyright ©2001-2010 FUDforum Bulletin Board Software

Back to the top