Automation testing is increasingly recognized and plays an important role in many projects. It is a part of QA team, along with manual testers, help the team to ensure the quality of the product.
When the feature is automated, the test cases for that feature is executed by automation framework, a computer, no more manual effort on this functionality, it means no more human involved. So how we can assure that the automation test is good and reliable, automation test will catch the bugs if it is introduced in the application ? will the test failed and warn the team when code changes and causes the system to break ? We needs test our test code first to ensure that !
Below is some ideas that can help you test your automation test code, both during developing it as well as monitoring the test afterwards to make sure the test is correct and reliable:
1. Make sure the test is stable
if you write a flaky test, it will easy to be failed again and again. You will need to waste time to check it to make sure it is not a bug. Flaky test is likely to pass when rerun but when you keep doing that, you can miss some bugs (intermittent or some bugs which only reproduced at a specific condition)
2. Keep monitoring the automation tests
Monitor the result of the tests and also the result history. If it keeps failing in recent reports and it is because the test is not stable, it shows that this is not a good test. You need to fix/stabilize it to be more reliable.
If your team found a bug which you think it should be catch by existing automation test but it did not, it shows that something wrong with your test. Check if the test is correct, enough assertions had been added, the logic in the test, … Do analysis and fix it asap.
3. Use IDE debugging to inspect the response data when adding any assertion for the test
Try to take full advantage of IDE to support you develop a good test script. when you assert actual_result with expected_result , you need to debug to inspect what is the returned value of actual_result, what is the value of expected_result. Do not just write code and run it to see if it is pass. Even the test is passed right after you finish the code, try to debug to inspect the data you are using for assertion.
A fun story, when you have been working in automation testing for a while , you will may want to achieve an ultimate skill, that is your developed test code will be passed at the first run. One day, my team member was very happy to tell me: hey, take a look, I have done a quite complex automation test script and it is passed at the first run. Awesome !
I quickly reviewed his code and told him to debug the test script to inspect it and we found he was doing this assertion assert_that(order_info['orderPlaced'], is_not(None)) , however, the response data order_info['orderPlaced'] actually returned empty instead of None. the test is passed but just because wrong assertion is used.
He updated the test script and see more work need to be done for it.
anyway, good luck with archiving ultimate automation test skill. just remember to run the test the first time in debug mode, so you can inspect the data along the test run. Hope it PASSED 🙂
4. Make it failed and see if your assertions work
If the automation test is good, when code changes and causes the system to break, the test should be failed. So when develop your test script, you can try to simulate that situation, let’s make the test failed and see if your assertions can catch it. e.g. changing if-statements in the code then run the tests to see if the tests fail. You’ve got work to do when they don’t. This process is also called “mutation testing” and can be done manually or by using frameworks.
5. Clarify the expected result/what need to be verified for the scenario
When you get the test scenario from manual team, you need to clarify with them to understand clearly what the test is about? can this test be automated ? what function do the test target ? What need to be checked/verified? You need to assure you did assert all important verification points in the test script.
One manual test case can be covered by several test scenarios in automation test suites as long as the test coverage is met.
6. Make sure logic in automation test is correct
If the verification data is dynamic, It changes following some logic, You need to make sure the logic in the automation is correct, make sure the expected outcome is correct at the moment and will correct in the feature.
7. Run your test in all environments that it is supposed to run
Make sure your automation test script works on all environments that you target it will be run such as TEST, STAGE, Production.
8. Be care with Conditional Statements
- Make sure the If/else condition is variable . I had seen my member to define a If condition that always True and the test will never go to the Else code
- Handle both If and else. e.g.
If (x=y), assert A, what if x # y ? make sure you handle for both side of the If and Else like this:If (x=y), assert A Else assert B
9. Parallel effort
When you add new test script for new scenario into automation test suite and for the first time you execute it with all test suite, You should manually perform the test and compare it with the automation test result to check if your test script is working as expected. It called “parallel effort”.
Let’s see how ISTQB syllabus explain it:
“As a part of transition activities, many organizations create a parallel team to begin the process of automating existing manual test scripts. The new automated scripts are then incorporated into the testing effort, replacing the manual scripts. However, prior to doing so, it is often recommended to compare and validate that the automated script is performing the same test and validation as the manual script it is replacing.
In many instances, an assessment of the manual scripts will be made prior to conversion to automation. As a result of such an assessment, it might be determined that there is a need to restructure existing manual test scripts to a more efficient and effective approach under automation.” ___Advanced Level Syllabus – Test Automation Engineer ___
If you have any ideas to test the automation test code, Please share me, I would like to hear more from you.
Reference Links:
Step Up Automation 