Concierge Testing
Concierge Testing is a method where a service or product is manually delivered to users as if it were fully operational and automated. This hands-on approach allows teams to validate the demand and feasibility of a product concept by providing personalised service without the initial need for extensive technological development.
Goal
The primary goal of Concierge Testing is to understand user needs, validate product-market fit, and gather qualitative feedback on a service or product idea before investing in full-scale development.
Context
Most features fail to deliver the expected value because users don't want them as much as we think they will. But it can be hard to validate whether customers want something from prototypes that describe a process. Enabling them to experience the process is a much more reliable way to validate our assumptions, but if we invest the full cost in building the solution before we test it, we risk wasting time and money. Concierge testing lets people experience the full service with very limited development effort.
Difference compared to Wizard of Oz Testing
The main difference between Wizard of Oz Testing and Concierge Testing is that Wizard of Oz Testing is focused on simulating the experience of a product or feature, while Concierge Testing actually delivers the service and uses the insights gathered from delivery to influence the technology that they will build.
Inputs
| Artifact | Description |
|---|---|
| Experiment Plan | A detailed document outlining the experiment objectives, methods, timeline, and success criteria. |
| Process Flow | The process flow or user journey to be evaluated in a Wizard of Oz or Concierge test, including all relevant touchpoints and interactions. |
Outputs
| Artifact | Description | Benefits |
|---|---|---|
| Validated Assumptions | A list of assumptions that have been validated. |
|
| Invalidated Assumptions | A list of assumptions that have been invalidated. |
|
Anti-patterns
- Over-engineering the Solution: Investing too much in tools or processes for the concierge service that won’t scale in a fully automated solution.
- Scaling Prematurely: Attempting to scale the concierge model before validating the concept or understanding operational challenges.
- Failing to plan the transition: Not planning for the transition from concierge to automated service, leading to a disjointed user experience.