False-fail and False-pass in Test Automation

There are many metrics that can be used to monitor the test automation strategy, the effectiveness and efficiency of the Test Automation Solution (TAS). In this post I will discuss about 2 metrics: False-Fail(False-positive) and False-Pass(False-negative).

False-Fail, also called False-Positive

Normally, when one test is failed in execution, we need to take some time to analyze test failures. This is even more frustrating when it turns out to be a false alarm.

This happens when the problem is in the TAS or test case but not in the SUT. It is important that the number of false alarms (and the potential wasted effort) are kept low. We call it “False-fail”. False-fail can reduce confidence in the TAS.

False-Pass, also called False-Negative

Conversely, False-Pass happens when there was a failure in the SUT but it was not identified by the test automation so a pass result was reported. In this case, a potential defect may escape detection. This can be occur because the verification of the outcome was not done properly, an invalid test oracle was used or the test case was expecting the wrong result. The False-Pass result may be more dangerous.

The false alarm also can be caused by an unstable SUT that is behaving in an unpredictable manner. eg: timing out. Test hooks can cause false alarm as well due to the level of intrusion they are causing.

How to avoid False-Fail result

1. Stabilize or remove the Flaky tests
   • Monitor the repeat-ability of tests, If a test keeps failing for some test cycles, you should spend effort to analyze the root cause and fix it if possible, otherwise, remove them from the test suite.

How to avoid False-Pass result

1. Parallel Effort
   • When you add new test script for new scenario into automation test suite and for the first time you execute it with all test suite, You should manually perform the test and compare it with the automation test result to check if your test script is working as expected. It called “parallel effort”.
2. Clarify the expected result/what need to be verified for the scenario
   • When you get the test scenario from manual team, you need to clarify with them to understand clearly what the test is about? can this test be automated ? what function do the test target ? What need to be checked/verified? You need to assure you did assert all important verification points in the test script.
   • One manual test case can be covered by several test scenarios in automation test suites as long as the test coverage is met.
3. If the verification data is dynamic, It changes following some logic, You need to make sure the logic in the automation is correct, make sure the expected outcome is correct at the moment and will correct in the feature.