(TO BE UPDATED…)
Test Data Management is an interesting topic that I always love to know more. It is a very important factor to help us maintain a good automation test suite. Along this post, I will also attach some great articles about this topic that I came across.
I. What is Test Data in Test Automation
Firstly, we need to understand what is test data.
For me, test data is any data you are using in the tests. It is the input data to the application which then combine with actions in the test to drive the behavior of the test cases.
Test data can be URL of the application. different URLs represented for different environments or different applications.
Test data can be Users/Password which the test can use to login to application and then perform various test scenarios.
It can be Credit Card numbers, address.
Or system data such as DB connection information.
Data Strategy is the combination of code, procedure, and infrastructure that affects how tests interact with data to stimulate the system under test.
It has two main parts: a creational piece and a cleanup piece.
- The Creation piece is all about how and when data is created.
- The Cleanup piece is the method you use to delete the test data or set the data source back to a previous state.
Here are some questions that you should answer before creating a test data:
- which environments do you need to run this test ?
- can you create an independent test data for this test ?
- how expensive the test data created in this test is ?
- can the test be still execute properly with this test data in next execution ? will the test data be at known state before running ?
- Can you clean up your created test data after test execution ?
- is it safe to run along with other tests in parallel ?
- Is this test data can be reused for other tests ?
Answering these questions may help you to choose the correct approach for test data which we will discuss below.
Best practice, you should create a test data so that the test is atomic, independent from other tests and the tests itself should be responsible for managing their own data.
There are two approaches that I often use to create test data for automation tests: Static Test Data and Dynamic Test Data Creation
II. Static Test Data
With this approach, we need to create a data manually before adding tests. This data usually can be reused for many tests and the most important is that no test will update or delete this data during execution.
Popular static data in an automation framework is Users.
Static test data only has creational piece. It only needs to be created one time. No cleanup needed for this data.
When use Static data:
- No API supported for creating new data during execution.
- Or No cleanup API and the test data must be clean after test (e.g: keep this data may spam the DB)
- Or The data is very complex to setup by calling APIs
- And Data can be reused for many other tests and this data is not likely to be changed during test execution.
How to manage Static test data
- Using Constants class and define public static variables
- DONOT use interface to define test constants
- Store in a file such as properties, JSON, XML
- DONOT use excel as test data source for automation as it requires external library and complex setup
III. Dynamic Test Data Creation
With this approach, The test data will be created during test execution. Most of automation tests should use this approach.
1. Create/Cleanup test data by API for single test.
If the test data is specifically created for a single test and it cannot be reuse for other tests, we can call relevant API endpoints to create the data the test need and also delete that data after test execution is done.
2. Create/Cleanup a test data by API for multi tests.
If one test data can be used for multiple tests, we can create that test data with wider scope.
In Pytest framework, you can use fixture to define test data with different scopes such as function/module/session.
3. Create/Cleanup Test data by API and DB
Sometimes, you may need to access DB to update the test data to set it to a specific condition.
Be careful with this approach, You need to know clearly what you are trying to insert/update in DB.
Make sure the test data is deleted or reset after test execution, otherwise you may leave an invalid test data in DB and may cause issues later.
Do Not Delete the data directly in DB unless you know exactly where your data is stored in every table of the DB. you should use API to do that.
an example that can use this approach is :
Testing a user cannot use Coupon when it reaches the expire date. To setup test data for this scenario, you will need to create a new Coupon. After created, this Coupon status would be Draft or Active status. then you need to access DB and update its end date to a future date and verify that the Coupon status is now moved to “EXPIRED” status.
4. Selfish Data Creation
I got this term from Paul Merrill, sometimes, you may consider to use an approach that creating the test data by API during test execution without doing any data cleanup afterwards.
The reason may be that the cleanup API is not available or not allowed. And not clean up the data is not a big problem. Benefit of automating the scenario outweighs the problem of generating too much data.
Remember to create unique data for each execution with this approach. e.g. append timestamp to email pattern to generate a unique email.
IV. How do we know which data to clean up
- Store data generated in test into Data Storage
- Signed Data: created data in a specific way so as to be easily recognizable. e.g. name with prefix “AutoTest”
- Marked Data: additional field in DB, e.g. isTestData = true/false
Others:
- Manage Test Data for Performance test
- Sharing test data between Test Steps in Cucumber framework
- Reset the test data state before or after a test run (be careful to use this approach. the test data which reset the state should only be used in one test, otherwise it can cause the conflict)
Reference Links:
https://www.ontestautomation.com/managing-test-data-in-end-to-end-test-automation/
https://techbeacon.com/app-dev-testing/3-highly-effective-strategies-managing-test-data
Step Up Automation 