Have you ever seen the test sometimes passed and sometime failed with the same code and configuration during testing ?
I have facing this in every projects I worked. Lets talk about what is flaky tests and how to handle it in your automation test suite.
“Flaky tests are tests that fail intermittently. This can happen if there is an underlying bug that only happens intermittently. Or it could also happen if a test is timing-dependent and affected by environmental conditions such as the speed of the machine running the test.” ___ https://github.com/nodejs/node/wiki/Flaky-tests ___
When the test that are marked as flaky, its status is set to unstable (yellow) instead of failed (red). If we already know which tests in our test suite are Flaky test, the failure may be considered to reduce from fatal to warning level.
In my project which using BDD to write test scenarios, we run the feature file first then we have a step to rerun all failed tests in that feature if any. It makes sure some failures like timing will be rerun and the status will be updated.
I faced some flaky tests that i dont even know the reason why it is failed sometimes. It may because of computer resources. i dont know… In this case, i manually check the functionality of that test and make sure nothing is broken and ignore the flaky test status.
What to do when you encounter a new flaky test.
Flaky tests should be fixed to be made non-flaky. Usually this means fixing the underlying bug, or changing the test so that it doesn’t depend on timing.
Below is nice post on Google Testing blog: https://testing.googleblog.com/2016/05/flaky-tests-at-google-and-how-we.html
There are some few tips that we can make our UI test more stable and less Flaky:
- Navigating to the page by go to (open) the URL directly instead of click some elements/links/menu to access the page.
- Utilize API to setup or teardown UI tests (login, set data state, create/delete data …)
- Write an atomic automation test to make sure each test will not depend on the others.
Step Up Automation 