Test Automation

Test Automation refers to the use of software to control the execution of tests, compare the actual outcomes with the predicted outcomes, manage test data, and utilise results to improve software quality.

Goal

The goal of automated testing is to catch bugs early in the development cycle, reduce the time and effort required for manual testing, and increase the scope and depth of tests thereby improving software quality, reducing the time to market, and enabling frequent and consistent tests.

Context

When building software it is easy to make mistakes, which cause bugs in the system. The later you catch these bugs the more expensive they are to fix because the developers will have forgotten the details of the implementation as well as the risk of cascading problems as additional code layered on top of the original mistake.

In addition, while a feature is written once, it could be broken at any time by a change in the code. This means that the same feature needs to be continually tested to ensure that it is still working as expected. Manual testing would make frequent releases impossible both from a time delay perspective as well as cost. Test automation is required to enable quick releases to capture fast feedback from users.

Test Types

Test Type	Description
Unit Tests	Automated tests that validate the functionality of individual components or units of code in isolation.
Integration Tests	Automated tests that verify the interactions between components or systems.
End-to-End Tests	Automated tests that assess the system's functional requirements from an end-user perspective.
Performance Tests	Automated tests that evaluate the performance characteristics of the application, such as responsiveness and scalability.

Test Data

Test data is essential for validating the behaviour of the software under various conditions. There are different approaches to managing test data, each with its own pros and cons.

Method	Description	Benefits	Considerations
Static Test Data	Pre-defined data sets stored in files or databases.	Easy to create and use Ensures consistency.	Can become outdated May not reflect current scenarios.
Dynamic Test Data	Data generated in real-time or just before test execution.	Covers more scenarios Reflects realistic use cases.	More complex to generate May increase test execution time.
Data Pooling	Centralised repository of test data that can be shared across multiple tests.	Promotes reusability and efficiency Reduces redundancy.	Requires maintenance to keep data relevant.
Synthetic Data Generation	Use of algorithms or tools to generate data mimicking real scenarios.	Useful for sensitive data Can scale for large volumes.	May not accurately reflect real user behaviour.
Database Cloning or Subsetting	Creating a complete copy or subset of the production database.	Accurately reflects real-world scenarios.	Resource-intensive for cloning Subsetting requires careful selection.
Mocking and Stubbing	Simulating the behaviour of complex data sources or systems.	Allows component isolation Quick and efficient.	May oversimplify interactions or not capture behaviours fully.

Inputs

Artifact	Description
Test Suite	A collection of tests that validate the functionality of the software.
Test Data	Data sets used for testing scenarios to validate the application behaviour under various conditions.

Outputs

Artifact	Description	Benefits
Test Results	Reports of tests, including details of failures and the environment in which it occurred.	Identifies defects early Provides feedback for improvement Can stop a build from being deployed

Anti-patterns

Flaky Tests: Automating unstable or frequently changing tests, leading to maintenance challenges.
UI Test Overload: Over-reliance on UI tests for functionality that could be tested at lower levels.
Long Test Cycles: Teams need quick feedback to make fixes before moving onto the next feature.