Testing in Production
Testing in production refers to the process of testing new features, bug fixes, and updates in the live environment where the end-users operate. This approach complements pre-release testing by exposing new changes to real-world conditions.
Goal
The goal of testing in production is to ensure that new changes perform as expected under live conditions, ensuring that your testing effort is based on real-world scenarios and user interactions.
Context
Tests are traditionally performed in different environments such as development, staging, and pre-production. However, these environments may not fully replicate the real-world conditions, data and user interactions. In addition, we do not want to block the release process by having a long testing phase. Testing in production lets us achieve both aims.
Case study: Hudson's Bay Company
Hudson's Bay Company (HBC) shifted from weekly to daily deploys by moving most of their testing into production. Their journey shows the typical pattern teams hit when they make this shift.
- Started with slow delivery. It took months to release new features. The bottleneck was not coding speed but the release pipeline itself.
- Replaced the monolith with microservices. Development speed went up, but staging became the new bottleneck. Keeping the staging environment in sync with production was expensive, fragile, and slowed down every team that relied on it.
- Adopted dark deployments. New features were deployed to production with their feature flags switched on for one new brand only. Integration tests ran against real production data and traffic. Once the feature was stable for that brand, the flag was extended to the other brands using a phased rollout.
- Invested in easy rollbacks. Key business metrics were monitored continuously, and the team could revert any deployment quickly if a metric moved the wrong way.
HBC's journey illustrates two things. Every staging environment will diverge from production over time, and the cost of keeping it in sync grows with the size of the codebase. Once HBC stopped fighting that divergence and used production itself as the test environment, the team relied on feature flags and quick rollbacks as the safety net rather than on staging fidelity.
Inputs
| Artifact | Description |
|---|---|
| Live Product | The code changes deployed to the production environment. |
| Feature Toggles | We use feature toggles to control the visibility of new features in the live environment to enable the testing of new features in production without exposing them to all users. |
Outputs
| Artifact | Description | Benefits |
|---|---|---|
| Defects Log | A compilation of defects, usability issues, and observations made during exploratory testing. | Provides immediate feedback and insights into application quality. |
Anti-patterns
- Skipping Pre-production Testing: Relying solely on testing in production without adequate pre-release validation can lead to critical issues affecting all users.
- Poor Monitoring Setup: Without comprehensive monitoring, identifying and diagnosing issues becomes challenging, potentially leading to prolonged user exposure to bugs.
- Exposing features to end users: Deploying new features to all users without proper testing can lead to a poor user experience and potential business impact.