Developing Experiment Plan

Developing an experiment plan involves outlining the strategies and methodologies to be used in evaluating the assumptions about a solution.

Goal

The purpose of developing experiment plans is to agree upfront on the methods, users and success criteria for the experiments we are going to run. This ensures that we are not biased towards our own ideas and create excuses for negative results.

Context

Bad experiments don't teach us anything. By planning the people to get involved, the methods to use and the success criteria we can avoid the endless debates about whether we should continue with a solution idea or not.

Plan Elements

ElementDescription
AudienceDefines the participants or jobs-to-be-done involved in the experiment, ensuring the results are relevant to the target users or consumers.
HypothesisA clear statement predicting the outcome of the experiment based on assumptions or existing knowledge. It guides the experiment's direction and objectives.
MethodThe approach or techniques used to conduct the experiment.
Success CriteriaSpecific, measurable objectives that determine what constitutes a successful outcome for the experiment. We need to make our success criteria hard to achieve so that if we achieve them we can be confident in the results. There is nothing worse than a result ending in a maybe.
Materials/ToolsLists the resources, tools, and materials required to conduct the experiment. This ensures all necessary preparations are made in advance.
ProcedureStep-by-step instructions on how the experiment will be conducted, including the setup, execution, and data collection methods.
TimelineA schedule detailing the start and end dates of the experiment and any important milestones. It helps in tracking progress and ensuring the experiment stays on course.
BudgetAn estimate of the financial resources required for the experiment, covering materials, tools, and any compensations for participants.

Experiment Methods

PracticeDescriptionBenefitsConsiderationsBest Suited For
Prototype TestingCreating early models of a product to gather feedback and iterate before full-scale development.
  • Early insight into usability and design issues.
  • May not fully represent the final product experience.
  • Early product development stages.
Landing Page TestingCreating a page to describe a potential product or feature to gauge user interest through actions like sign-ups.
  • Quickly assesses user interest.
  • Low cost and easy to implement.
  • May not capture depth of user engagement or feedback.
  • High traffic needed for significant data.
  • Validating interest in a new product or feature before full development.
Wizard of Oz TestingSimulating the functionality of a product or feature that doesn't yet exist to test user reactions.
  • Allows testing of concepts without full development.
  • Can provide valuable insights into user interest and behaviour.
  • May mislead users if not clearly communicated.
  • Requires manual backend work to simulate automation.
  • Validating product concepts before committing development resources.
Concierge TestingManually providing services or features to users that you plan to automate in the future, to validate demand.
  • Lowers initial development costs by validating ideas manually first.
  • Provides deep insights into user needs and service delivery.
  • Not scalable; labour-intensive.
  • May not reflect true user experience of the automated product.
  • Early-stage startups testing service concepts or features.
Fake Door TestingPresenting the option for a non-existent product or feature to measure user interest based on engagement.
  • Quickly gauges user interest without developing the feature.
  • Easy to implement and analyse.
  • Potential to disappoint or frustrate users.
  • Engagement metrics may not translate directly to actual usage or value.
  • Assessing user demand for new features or products.
Crowdfunding CampaignsUsing platforms to present product ideas to potential customers, gauging interest through financial pledges.
  • Directly measures market demand and can fund development.
  • Builds a community of early adopters.
  • Requires significant marketing effort.
  • Success depends on the appeal of the campaign, not just the product.
  • Early-stage products seeking validation and funding.
A/B TestingComparing two versions (A and B) to see which one performs better.
  • Direct feedback on preferences; easy to implement.
  • May require large sample sizes to be significant.
  • Web page designs, feature evaluations.
Multivariate TestingTesting multiple variables simultaneously to see how they interact and affect outcomes.
  • Can explore complex interactions.
  • Complex to set up and analyse.
  • Advanced product features, user interfaces.

Inputs

ArtifactDescription
Prioritised AssumptionsThe assumption that the team is trying to validate.
Jobs to be Done (JTBD)A list of job stories that represent the core tasks, emotional needs, and social roles that users seek to fulfill.

Outputs

ArtifactDescriptionBenefits
Experiment PlanA detailed document outlining the experiment objectives, methods, timeline, and success criteria.Ensures that experiments accurately evaluate the assumptions and provide actionable insights.

Anti-patterns

  • Over-Planning: Spending excessive time on planning without moving into action, resulting in delays in gaining insights.
  • Underestimating Resources: Failing to allocate adequate time, budget, and personnel for experiments, compromising the quality and depth of insights.
  • Choosing methods based on convenience, not suitability: Selecting experiment methods based on ease of execution rather than their effectiveness in addressing the experiment questions.

Was this page helpful?

Previous
Identifying Target Audience
© ZeroBlockers, 2024. All rights reserved.