by John A. List
May 05, 2022
Why do some products, companies, and social programs thrive as they grow while others peter out? According to the author, there are five causes: 1) False positives, or inaccurately interpreting a piece of evidence or data; 2) Biased representativeness of population, or not making sure your samples reflect the larger population at scale; 3) Non-negotiables that can’t grow or be replicated; 4) Negative spillovers, or unintended outcomes; and 5) Cost traps. Here, he explains and offers examples of each cause, as well as how to anticipate or avoid them.
For the last several years, I’ve been at the forefront of a movement known as implementation science, or the science of scaling. In this work, we are trying to understand why some products, companies, and social programs thrive as they grow, while others peter out.
When a seemingly promising idea loses efficacy or profitability as it expands, we call it a “voltage drop.” These failures to scale never happen because of one single reason.
Over the last 25 years as a behavioral economist, consultant to companies large and small, and former White House economic adviser, I’ve identified five causes, what I call “vital signs,” of voltage drops.
1. False Positives
This occurs when you interpret a piece of evidence or data as proof that something is true, when in fact it isn’t — for example, as we’ve seen with inaccurate Covid test results. For scaling, a false positive is an erroneous sign that an idea has voltage when it really doesn’t.
Sometimes a false positive occurs because of a statistical error, as was the case with the famous drug abuse prevention program, D.A.R.E. After an independent study showed promising short-term results, the program received an influx of funding from the U.S. Department of Justice. However, there were several problems with the study: It excluded drugs like alcohol and marijuana, focusing on tobacco; it was based on a small sample size; and later studies and even metanalyses could not replicate the results.
In other cases, false positives result from intentional lying. Think of Elizabeth Holmes and the purportedly groundbreaking blood-testing technology of Theranos, which didn’t actually exist.
When possible, the solution for rooting out false positives is to have at least three independent replications of the idea that show early promise. In companies with confidential research, employees must be incentivized with financial rewards that encourage them to question results.
2. Biased Representativeness of Population
Once you’ve reliably demonstrated the efficacy of the endeavor you hope to scale, the next step is to answer the question “How broadly will the idea work?”
All ventures must understand their potential audience. The first way to do this is by making sure your test samples in the small scale reflect the larger population at scale. Otherwise, you’ll be like McDonald’s, which fell victim to selection bias when it launched the unsuccessful Arch Deluxe. Focus group participants loved the new product, but they weren’t representative of the majority of Americans, who simply wanted to keep eating their Big Macs.
To weed out such biases, make sure your early adopters are a random sample. You should also make sure that your survey respondents have appropriate incentives to tell you the truth. A focus group participant who says they would purchase a product if it was introduced could simply be saying, “I would love the option to consider that product in the future,” as opposed to “I will be purchasing the product in the future.”
3. Non-Negotiables That Can’t Grow or Be Replicated
For an idea or enterprise to hold strong at scale, you need to know whether your “non-negotiables” — the drivers of your success — can be replicated at scale. In other words, is your secret sauce the “chef” or the “ingredients”? Since people don’t scale well (i.e., they can’t be cloned), talent-centric ventures often don’t either. You can’t afford all the talent you need as you grow, so you hire fewer high-performers and quality suffers at scale — a cruel voltage drop.
But, this vital sign is about much more than just people. As you scale, regulatory constraints, resource constraints, fidelity concerns, and a host of other issues might arise. In the end, we must bring these scaling constraints back to the petri dish and make sure the idea works with them in place.
4. Negative Spillovers
A spillover effect is the unintended impact one event or outcome can have on another event or outcome. A classic example is when a city opens a new factory, and the air pollution it produces impacts the health of nearby residents.
As you scale, the likelihood of spillovers increases dramatically. General equilibrium effects, or natural readjustments of the market, are one chief cause. I saw this firsthand when I was the chief economist at Uber. A coupon that led to more riders in one area of Seattle failed when we scaled it to the whole city because surge prices kicked in, and users found cheaper ways to get around that night.
Positive spillovers exist too, like network effects that make a social-media platform more valuable as more people join it. When designing your idea early on, you must anticipate negative spillovers and look for opportunities to engineer and benefit from positive ones.
5. The Cost Trap
To scale successfully, you need to determine not only how many people like your idea, but also what they’re willing to pay for it and, crucially, how much it will cost to provide.
When designing your enterprise, you must account for two types of costs: upfront fixed costs, like the one-time investment for the research and development to create a new product or service, and your ongoing operating expenses. Upfront costs can be recouped, but operating ones can bleed you and lead to a voltage drop, as happened to the innovative scientific wellness startup Arivale, which was poised to change preventative health care, only to go bankrupt a few years later, because it couldn’t find a viable price point for its services.
One strategy to escape the cost trap of scaling is to make sure you benefit from economies of scale, a skill Elon Musk excels at in all his ventures. Ever since he helped transform the world of online banking at PayPal, each major innovation he has undertaken thrives on scale economies. Consider Tesla. Its massive success can be traced to economies of scale of its two most important components: batteries and solar power generation cells, both of which can be manufactured significantly cheaper in higher numbers. In addition, everything at Tesla is geared toward increasing the efficiency of “the machine that makes the machines,” or what Musk affectionately calls his “Alien Dreadnought” — that is, a highly advanced, fully automated production facility.
Another strategy is to create models that don’t rely on top-tier talent. As you scale, finding and paying high-performers will become prohibitive. The solution is to create products that can give their full value to customers even with average performers delivering it.
When I give talks on this topic, I like to invoke the famous opening line of Leo Tolstoy’s novel Anna Karenina: “Happy families are all alike; every unhappy family is unhappy in its own way.” Similarly, scalable ideas are all alike; every unscalable idea is unscalable in its own way. The difference with scaling is there are only five main obstacles you face. And once you anticipate and avoid them, you can scale your idea for the highest voltage possible.
John A. List is the Kenneth C. Griffin Distinguished Service Professor in Economics at the University of Chicago and Distinguished Professor of Economics at the Australian National University, as well as the chief economist at Lyft and, previously, at Uber. He has served on the Council of Economic Advisers and is the recipient of numerous awards and honors, including the AAEA’s Galbraith Award. His work has been featured in The New York Times, The Economist, Harvard Business Review, Fortune, Slate, and The Washington Post, and on NPR, NBC, and Bloomberg. List has authored over 250 peer-reviewed journal articles, several academic books, including national bestseller The Voltage Effect.