Principle 1 - We don't know exactly what customers want
Symptom
90% of features fail to deliver the expected business value. Research by Ron Kohavi, Alex Deng and Lukas Vermeer found that the success rate of new features is much lower than we think.
| Company/Source | Success Rate | False Positive Rate | Reference |
|---|---|---|---|
| Microsoft | 33% | 5.9% | Kohavi, Crook and Longbotham 2009 |
| Avinash Kaushik | 20% | 11.1% | Kaushik 2006 |
| Bing | 15% | 15.0% | Kohavi, Deng and Longbotham, et al. 2014 |
| Booking.com, Google Ads, Netflix | 10% | 22.0% | Manzi 2012, Thomke, Experimentation Works: The Surprising Power of Business Experiments 2020, Moran 2007 |
| Airbnb Search | 8% | 26.4% | Ronny Kohavi LinkedIn |
We spend a lot of time trying to improve the efficiency of how we build software products but it doesn't matter how efficient we are if we don't get a return on our investment.
Cause
We're experts and we have years of experience so of course we know what our customers want. Our ideas are based on our experience, knowledge of the market and direct feedback from customers so we have high confidence when we lock in our solutions during business case sign off.
The problem is that we are creating our ideas in the presence of three gaps:
- The knowledge gap: The difference between what we would like to know and what we actually know.
- The alignment gap: The difference between what we think in our heads and what the team who are building the product think. Everyone has their inbuilt assumptions and biases so what gets communicated is not what is received.
- The effects gap: The difference between how we expect people to behave versus how they actually behave. Our products, the market and customer behaviours are a complex adaptive system. Things are constantly reacting and changing in unpredictable ways.
These gaps, particularly the effects gap, are why our features fail to deliver the expected value.
"No business plan survives first contact with a customer" Steve Blank
The Context Change
In the mass manufacturing era, the products being produced already existed such as furniture, automobiles, and appliances. These products had known, and elastic, demand. Therefore the goal was to reduce the cost of production to increase demand.
Today we are in a different place. We are always building new products, so the demand is not certain. In addition, the managers defining the processes in the factories were experts in production techniques. Today, technology moves so fast that knowing what is possible or the best way to design the user experience are not widely distributed skills. Yet we ask business people to provide a detailed specifications of what they want to build.
Solution
We need to acknowledge that we don't know how people will behave. When you accept that work may fail the most important thing is to reduce the risk as much as possible. We can do this by identifying the core assumptions underpinning our solutions and gathering feedback from customers early and often to validate these assumptions across four dimensions:
- Desirability - do people want it?
- Feasibility - can we build it?
- Viability - can we make money from it?
- Usability - can people use it?
To validate these assumptions, teams need input from multiple disciplines such as product management, research, design and engineering. By adopting a process of research and solution validation before we invest in the expensive delivery phase we can reduce the risk of building the wrong thing.
Impact
This changes how we fund new product development. We need to invest more in the upfront analysis and validation phase to reduce the risk of building the wrong thing. But this shifts investment from the delivery phase to the research and validation phase.
There are two types of financial investment - CapEx and OpEx. CapEx is for investing in assets that will provide value over time, like a new factory. OpEx is for the costs of operating the business, like salaries and raw materials. The markets treat CapEx as an investment in the future so a company looks more efficient when it can shift investments from OpEx to CapEx.
From an accounting perspective, Research and Development (R&D) is OpEx but software development is CapEx. Investing more time, and including more disciplines, in the upfront research and validation phase would shift salaries from CapEx to OpEx, the opposite way to how finance teams want to move, even though the move will improve the performance of the company in the long term.
Mitigating the Impact
This shift is not without precedence. The move from on-premise to cloud computing was a similar shift from CapEx to OpEx. Because you are renting computing space in the cloud it is considered an operating expense instead of the capital investment of buying servers.
Market analysts understood the value proposition so they did not penalise companies who made the shift to the cloud. As the problems of capitalising software based on costs, rather than value, become more widely known the market will start to reward companies who are seen to reduce their long-term risks. Including an accounting policy note in the financial statements that explains the shift from CapEx to OpEx can remove any impact during the transition period.
Next Steps
But if reducing early risk was all we needed to improve our effectiveness then the companies listed, Microsoft, Airbnb, Booking.com and more would have much higher success rates. The truth is that investing in more research reduces our risk but it does not eliminate it because we can only know for certain when customers have the product in their hands.