Experiment Plan
An Experiment Plan is a structured document that outlines the methodology for conducting experiments to test hypotheses related to product features, user behaviour, market trends, or other areas of interest in product development. It specifies the objectives, methods, metrics, and expected outcomes of an experiment.
Purpose
The purpose of an experiment plan is to agree upfront on the methods, users and success criteria for the experiments we are going to run. This ensures that we are not biased towards our own ideas and create excuses for negative results.
Format
| Element | Description |
|---|---|
| Audience | Defines the participants or jobs-to-be-done involved in the experiment, ensuring the results are relevant to the target users or consumers. |
| Hypothesis | A clear statement predicting the outcome of the experiment based on assumptions or existing knowledge. It guides the experiment's direction and objectives. |
| Method | The approach or techniques used to conduct the experiment. |
| Success Criteria | Specific, measurable objectives that determine what constitutes a successful outcome for the experiment. We need to make our success criteria hard to achieve so that if we achieve them we can be confident in the results. There is nothing worse than a result ending in a maybe. |
| Materials/Tools | Lists the resources, tools, and materials required to conduct the experiment. This ensures all necessary preparations are made in advance. |
| Procedure | Step-by-step instructions on how the experiment will be conducted, including the setup, execution, and data collection methods. |
| Timeline | A schedule detailing the start and end dates of the experiment and any important milestones. It helps in tracking progress and ensuring the experiment stays on course. |
| Budget | An estimate of the financial resources required for the experiment, covering materials, tools, and any compensations for participants. |
Anti-patterns
- Changing Metrics Mid-Experiment: Stopping an experiment early or adjusting success metrics during the experiment can lead to biased interpretations of the results.
- Arguing over Outcomes: Teams can become defensive or argumentative when results don't align with their expectations, leading to unproductive discussions. Defined success criteria can help mitigate this.