Running Experiments

Running experiments involves systematically testing hypotheses by conducting evaluative research with target users. This activity is crucial in validating assumptions about user behaviour, product features, and market needs.


The purpose of running experiments is to validate or invalidate assumptions about user behaviour, product features, and market needs. It helps teams make informed decisions based on data rather than assumptions.


Industry Context

Most features fail because we make assumptions about how users will behave, but unfortunately we are often wrong. Running experiments is a way to validate these assumptions before we invest too much time and effort into building a solution.

ZeroBlockers Context

We are accountable for outcomes, not just outputs so we need to actively overcome our biases if we want to achieve the desired results.


PracticeDescriptionBenefitsConsiderationsBest Suited For
Prototype TestingCreating early models of a product to gather feedback and iterate before full-scale development.
  • Early insight into usability and design issues.
  • May not fully represent the final product experience.
  • Early product development stages.
Landing Page TestingCreating a page to describe a potential product or feature to gauge user interest through actions like sign-ups.
  • Quickly assesses user interest.
  • Low cost and easy to implement.
  • May not capture depth of user engagement or feedback.
  • High traffic needed for significant data.
  • Validating interest in a new product or feature before full development.
Wizard of Oz TestingSimulating the functionality of a product or feature that doesn't yet exist to test user reactions.
  • Allows testing of concepts without full development.
  • Can provide valuable insights into user interest and behaviour.
  • May mislead users if not clearly communicated.
  • Requires manual backend work to simulate automation.
  • Validating product concepts before committing development resources.
Concierge TestingManually providing services or features to users that you plan to automate in the future, to validate demand.
  • Lowers initial development costs by validating ideas manually first.
  • Provides deep insights into user needs and service delivery.
  • Not scalable; labour-intensive.
  • May not reflect true user experience of the automated product.
  • Early-stage startups testing service concepts or features.
Fake Door TestingPresenting the option for a non-existent product or feature to measure user interest based on engagement.
  • Quickly gauges user interest without developing the feature.
  • Easy to implement and analyse.
  • Potential to disappoint or frustrate users.
  • Engagement metrics may not translate directly to actual usage or value.
  • Assessing user demand for new features or products.
Crowdfunding CampaignsUsing platforms to present product ideas to potential customers, gauging interest through financial pledges.
  • Directly measures market demand and can fund development.
  • Builds a community of early adopters.
  • Requires significant marketing effort.
  • Success depends on the appeal of the campaign, not just the product.
  • Early-stage products seeking validation and funding.
A/B TestingComparing two versions (A and B) to see which one performs better.
  • Direct feedback on preferences; easy to implement.
  • May require large sample sizes to be significant.
  • Web page designs, feature evaluations.
Multivariate TestingTesting multiple variables simultaneously to see how they interact and affect outcomes.
  • Can explore complex interactions.
  • Complex to set up and analyse.
  • Advanced product features, user interfaces.


  • Lack of Clear Hypotheses: Conducting experiments without a clear understanding of what is being tested.
  • Confirmation Bias: Designing experiments in a way that may inadvertently favour a desired outcome.
  • Ignoring Negative Results: Disregarding or rationalising away results that do not support initial assumptions.
  • Overgeneralisation: Extrapolating findings from a limited dataset to a broader context without sufficient evidence.

Case Studies

Was this page helpful?

Preparing Experiment Materials
© ZeroBlockers, 2024. All rights reserved.