Prior to commencing any automated effort, We needs to consider the applicability and viability of creating automated vs. manual tests. The suitability criteria may include but not limited to:
- Frequency of use
- Complexity to automate
- Compatibility of tool support
- Suitability of automation for the stage of the software product lifecycle
- Sustainability of the automated environment
- Controllability of the SUT
Frequency of use
How often a test needs to be run is one consideration as to the viability of whether or not to automate. Tests that are run more regularly, as a part of a major or minor release cycle, are better candidates for automation as they will be used frequently. As a general rule, the greater the number of application releases—and therefore corresponding test cycles—the greater the benefit of automating tests.
As functional tests become automated, they can be used in subsequent releases as a part of regression testing. Automated tests used in regression testing will provide high return on investment (ROI) and risk mitigation for the existing code base.
If a test script is run once a year, and the SUT changes within the year, it may not be feasible or efficient to create an automated test. The time it might take to adapt the test on a yearly basis to conform to the SUT might be best done manually.
Complexity to automate
In the cases where a complex system needs to be tested, there may be a tremendous benefit from automation to spare the manual tester the difficult task of having to repeat complex steps which are tedious, time-consuming, and error-prone to execute.
However, certain test scripts may be difficult or not cost-effective to automate. There is a range of factors that might affect this, including: an SUT that is not compatible with existing available automated test solutions; the requirement to produce substantial program code and develop calls to APIs in order to automate; the multiplicity of systems that need to be addressed as part of a test execution; the interaction with external interfaces and/or proprietary systems; some aspects of usability testing; the amount of time needed to test the automation scripts, etc.
Suitability of automation for the stage of the software product lifecycle
An SUT has a product lifecycle which can span from years to decades. As the development of a system begins, the system changes and expands to address defects and add refinements to meet end user needs.
In the early stages of a system’s development, change may be too rapid to implement an automated testing solution. As screen layouts and controls are optimized and enhanced, creating automation in a dynamically changing environment may require continuous re-work, which is not efficient or effective. This would be similar to trying to change a tire on a moving car; it’s better to wait for the car to stop. For large systems in
a sequential development environment, when a system has stabilized and includes a core of functionality, this then becomes the best time to begin the implementation of automated tests.
Over time, systems reach the end of their product lifecycles, and are either retired or redesigned to use newer and more efficient technology. Automation is not recommended for a system nearing the end of its
lifecycle as there will be little value in undertaking such a short-lived initiative. However, for systems that are being redesigned using a different architecture but preserving the existing functionality, an automated
testing environment which defines data elements will be equally useful in the old and new systems. In this case, reuse of test data would be possible and recoding of the automated environment to be compatible with the new architecture would be necessary.
Reference resources:
Step Up Automation 