Eclipse Community Forums: SWTBot

Home » Eclipse Projects » SWTBot » Flaky Tests(How to handle Flaky Tests with SWTBot)

Thu, 08 September 2016 05:25

Eclipse User

Team,

Does SWTBot support Flaky Tests.
I have some tests which fail at times and run fine in next execution. It could be due to timing issues. I am yet to identify the exact reason but tests run fine without any changes next time. Therefore, to handle such tests can I use @Flaky annotation (org.eclipse.xtext.junit4.Flaky) so that tests are executed more than once.

Does SWTBot support @Flaky annotation or we have any other option to handle such test cases in SWTBot. Please let me know if you have any information on this.

Appreciate your time.

Thanks.
Vivek

Re: Flaky Tests [message #1742928 is a reply to message #1742861]

Thu, 08 September 2016 11:36

Eclipse User

Hi Vivek,

In our project we are also affected by intermittent build failures. Some are due to flaky SWTBot handling, but some are due to other reasons (race conditions, connection issues, etc.).

I think we prefer the annoyance of having to retrigger the builds to motivate us to troubleshoot and fix the issue. We also track them with bugs to increase visibility (see http://eclip.se/486207). If the failure rate is so high that it interferes with our ability to merge patches, then in rare occasions we annotate the test with @Ignore but that is more an indication that the test was not in good enough condition to be ready for use.

If we annotate tests with @Ignore or @Flaky, there is less motivation to investigate the problem. If the issue is with SWTBot code itself, it would benefit all its users if the test framework was corrected or made more robust.

Patrick

Re: Flaky Tests [message #1742980 is a reply to message #1742928]

Thu, 08 September 2016 22:29

Eclipse User

Thanks Patrick.

I understand and agree with what you said. Using @Flaky will hide the real issue.

In fact, it will be very helpful if we have a 'Test Environment Checklist' to avoid test failures due to SWTBot timing issues. Something like;

Test Env Checklist:
1> Stop other UI applications while running tests as this might shift the focus.
2> Stop Keyboard activity as this might shift the focus.
3> Do not run build while running SWTBot test.
4> Avoid...etc.
5> ...

I could not get much on this from the SWTBot site and user guide. I believe its very important to have such checklist or guidelines to avoid test failures due to environment and other factors not related to test code mainly.

Thanks for sharing the information. Appreciate your time.

Vivek

Previous Topic:	news.eclipse.org is shutting down.
Next Topic:	[solved] Eclipse-Plugin-Test: Log4J Error by using SWTBot

Goto Forum:

-=] Back to Top [=-

Current Time: Fri May 16 06:44:20 EDT 2025

.:: Contact :: Home ::.

Breadcrumbs

Sign up to our Newsletter