Reprinted by permission of Harvard Business Review Press. Excerpted from Experimentation Works: The Surprising Power of Business Experiments. Copyright 2020 Stefan H. Thomke. All rights reserved.
Experimentation Works
Our success at Amazon is a function of how many experiments we do per year, per month, per week, per day. —JEFF BEZOS, CEO, AMAZON
Innovation is important because it drives profitable growth and creates shareholder value. But here is the dilemma: despite being awash in information coming from every direction, today’s managers operate in an uncertain world where they lack the right data to inform strategic and tactical decisions.
Consequently, for better or worse, our actions tend to rely on experience, intuition, and beliefs. But this all too often doesn’t work. And all too often, we discover that ideas that are truly innovative go against our experience and assumptions, or the conventional wisdom. Whether it’s improving customer experiences, trying out new business models, or developing new products and services, even the most experienced managers are often wrong, whether they like it or not.
Business experiments raise the innovation game dramatically.
Today, Microsoft and several other leading companies—including Amazon, Booking.com, Facebook, and Google—each conduct more than ten thousand online controlled experiments annually, which individually engage millions of users. Startups and companies without digital roots, such as Walmart, State Farm Insurance, Nike, FedEx, the New York Times Company, and the BBC, also run them regularly, though on a smaller scale. These organizations have discovered that an “experiment with everything” approach has surprisingly large payoffs.
In 2016, Jeff Bezos gave shareholders a rare insight into Amazon’s innovation engine. In his annual letter, he explained: “One area where I think we are especially distinctive is failure. I believe we are the best place in the world to fail (we have plenty of practice!), and failure and invention are inseparable twins. To invent you have to experiment, and if you know in advance that it’s going to work, it’s not an experiment. Most large organizations embrace the idea of invention, but are not willing to suffer the string of failed experiments necessary to get there.”
Bezos didn’t stop there. For him, the business logic for tolerating, even inviting, failure came from the outsized economic returns of winning. He explained why experiments have been so important to Amazon’s growth model:
“Outsized returns often come from betting against conventional wisdom, and conventional wisdom is usually right. Given a ten percent chance of a 100 times payoff, you should take that bet every time. But you’re still going to be wrong nine times out of ten. We all know that if you swing for the fences, you’re going to strike out a lot, but you’re also going to hit some home runs. The difference between baseball and business, however, is that baseball has a truncated outcome distribution. When you swing, no matter how well you connect with the ball, the most runs you can get is four. In business, every once in a while, when you step up to the plate, you can score 1,000 runs. Long-tailed distribution of returns is why it’s important to be bold.”
Business experimentation has become part and parcel of how Amazon makes decisions, through a process Bezos calls “the unnatural thing of trying to disconfirm our beliefs.” After all, humans strongly prefer evidence that confirms their preexisting beliefs. But such confirmation bias gets in the way of making decisions about innovation, an arena in which most ideas don’t work.
So it should not have come as a surprise when, partially in an effort to kick its experiments up a notch, Amazon acquired Whole Foods about fourteen months after Bezos wrote the letter. Industry observers felt that its physical supermarkets could become laboratories for radical experiments. Correspondingly, share prices of competing grocery chains plummeted after the announcement. Amazon’s reputation for fearless innovation was fueled by such radical business experiments—so-called big swings—and, equally important, the tens of thousands smaller and disciplined experiments that have led to a highly optimized user experience in its web store.
If the case for business experimentation is so compelling, then why don’t more companies conduct rigorous tests of their risky overhauls and expensive innovation proposals in order to make better decisions? Why do executives rely on hierarchy, persuasion, or PowerPoints instead of demanding that teams present experimental evidence before making business decisions?
Clearly, there are cultural obstacles that inhibit experimentation. It’s also true that managers often misuse the term, saying “We experiment” in lieu of “We are trying something new,” but without putting enough thought into the discipline and rigor needed to get useful test results. In the most egregious cases, projects or business initiatives become “experiments” after they are finished, in an effort to excuse poor execution.
Many organizations have considerable difficulty executing good experiments. Although the process of experimentation seems straightforward, it is surprisingly hard in practice, owing to myriad organizational, management, and technical challenges. Moreover, most tests of new business initiatives are too informal. They are not based on proven scientific and statistical methods, and so executives end up misinterpreting statistical noise as causation—and make bad decisions.
Companies can run good experiments by systematically following a clear set of principles.
In an ideal experiment, testers separate an independent variable (the presumed cause) from a dependent variable (the observed effect) while holding all other potential causes constant. They then manipulate the former to study changes in the latter. The manipulation, followed by careful observation and analysis, yields insight into the relationships between cause and effect, which ideally can be applied and tested in other settings.
To obtain that kind of learning—and ensure that each experiment yields better decisions—companies should ask themselves seven important questions:
- Does the experiment have a testable hypothesis?
- Have stakeholders made a commitment to abide by the results?
- Is the experiment doable?
- How can we ensure reliable results?
- Do we understand cause and effect?
- Have we gotten the most value out of the experiment?
- Are experiments really driving our decisions?
Although some of the questions seem obvious, many companies conduct tests without fully addressing them. As a result, they miss out.
As the Amazon example shows, business is of absolute importance to a company’s ability to compete. Experimentation helps us begin to answer the kinds of questions that all organizations confront, from what products to make and what customer experiences to offer, to what information we need to make those decisions.
Reprinted by permission of Harvard Business Review Press. Excerpted from Experimentation Works: The Surprising Power of Business Experiments. Copyright 2020 Stefan H. Thomke. All rights reserved.
