Uncategorized
Tech giants dominate research but the line between real breakthrough and product showcase can be fuzzy. Some scientists have had enough.
by Will Douglas Heaven
November 12, 2020
Last month Nature published a damning response written by 31 scientists to a study from Google Health that had appeared in the journal earlier this year. Google was describing successful trials of an AI that looked for signs of breast cancer in medical images. But according to its critics, the Google team provided so little information about its code and how it was tested that the study amounted to nothing more than a promotion of proprietary tech.
“We couldn’t take it anymore,” says Benjamin Haibe-Kains, the lead author of the response, who studies computational genomics at the University of Toronto. “It’s not about this study in particular—it’s a trend we’ve been witnessing for multiple years now that has started to really bother us.”
Haibe-Kains and his colleagues are among a growing number of scientists pushing back against a perceived lack of transparency in AI research. “When we saw that paper from Google, we realized that it was yet another example of a very high-profile journal publishing a very exciting study that has nothing to do with science,” he says. “It’s more an advertisement for cool technology. We can’t really do anything with it.”
Science is built on a bedrock of trust, which typically involves sharing enough details about how research is carried out to enable others to replicate it, verifying results for themselves. This is how science self-corrects and weeds out results that don’t stand up. Replication also allows others to build on those results, helping to advance the field. Science that can’t be replicated falls by the wayside.
At least, that’s the idea. In practice, few studies are fully replicated because most researchers are more interested in producing new results than reproducing old ones. But in fields like biology and physics—and computer science overall—researchers are typically expected to provide the information needed to rerun experiments, even if those reruns are rare.
Ambitious noob
AI is feeling the heat for several reasons. For a start, it is a newcomer. It has only really become an experimental science in the past decade, says Joelle Pineau, a computer scientist at Facebook AI Research and McGill University, who coauthored the complaint. “It used to be theoretical, but more and more we are running experiments,” she says. “And our dedication to sound methodology is lagging behind the ambition of our experiments.”
The problem is not simply academic. A lack of transparency prevents new AI models and techniques from being properly assessed for robustness, bias, and safety. AI moves quickly from research labs to real-world applications, with direct impact on people’s lives. But machine-learning models that work well in the lab can fail in the wild—with potentially dangerous consequences. Replication by different researchers in different settings would expose problems sooner, making AI stronger for everyone.
AI already suffers from the black-box problem: it can be impossible to say exactly how or why a machine-learning model produces the results it does. A lack of transparency in research makes things worse. Large models need as many eyes on them as possible, more people testing them and figuring out what makes them tick. This is how we make AI in health care safer, AI in policing more fair, and chatbots less hateful.
What’s stopping AI replication from happening as it should is a lack of access to three things: code, data, and hardware. According to the 2020 State of AI report, a well-vetted annual analysis of the field by investors Nathan Benaich and Ian Hogarth, only 15% of AI studies share their code. Industry researchers are bigger offenders than those affiliated with universities. In particular, the report calls out OpenAI and DeepMind for keeping code under wraps.
Then there’s the growing gulf between the haves and have-nots when it comes to the two pillars of AI, data and hardware. Data is often proprietary, such as the information Facebook collects on its users, or sensitive, as in the case of personal medical records. And tech giants carry out more and more research on enormous, expensive clusters of computers that few universities or smaller companies have the resources to access.
To take one example, training the language generator GPT-3 is estimated to have cost OpenAI $10 to $12 million—and that’s just the final model, not including the cost of developing and training its prototypes. “You could probably multiply that figure by at least one or two orders of magnitude,” says Benaich, who is founder of Air Street Capital, a VC firm that invests in AI startups. Only a tiny handful of big tech firms can afford to do that kind of work, he says: “Nobody else can just throw vast budgets at these experiments.”
The rate of progress is dizzying, with thousands of papers published every year. But unless researchers know which ones to trust, it is hard for the field to move forward. Replication lets other researchers check that results have not been cherry-picked and that new AI techniques really do work as described. “It’s getting harder and harder to tell which are reliable results and which are not,” says Pineau.
What can be done? Like many AI researchers, Pineau divides her time between university and corporate labs. For the last few years, she has been the driving force behind a change in how AI research is published. For example, last year she helped introduce a checklist of things that researchers must provide, including code and detailed descriptions of experiments, when they submit papers to NeurIPS, one of the biggest AI conferences.
Replication is its own reward
Pineau has also helped launch a handful of reproducibility challenges, in which researchers try to replicate the results of published studies. Participants select papers that have been accepted to a conference and compete to rerun the experiments using the information provided. But the only prize is kudos.
This lack of incentive is a barrier to such efforts throughout the sciences, not just in AI. Replication is essential, but it isn’t rewarded. One solution is to get students to do the work. For the last couple of years, Rosemary Ke, a PhD student at Mila, a research institute in Montreal founded by Yoshua Bengio, has organized a reproducibility challenge where students try to replicate studies submitted to NeurIPS as part of their machine-learning course. In turn, some successful replications are peer-reviewed and published in the journal ReScience.
“It takes quite a lot of effort to reproduce another paper from scratch,” says Ke. “The reproducibility challenge recognizes this effort and gives credit to people who do a good job.” Ke and others are also spreading the word at AI conferences via workshops set up to encourage researchers to make their work more transparent. This year Pineau and Ke extended the reproducibility challenge to seven of the top AI conferences, including ICML and ICLR.
Another push for transparency is the Papers with Code project, set up by AI researcher Robert Stojnic when he was at the University of Cambridge. (Stojnic is now a colleague of Pineau’s at Facebook.) Launched as a stand-alone website where researchers could link a study to the code that went with it, this year Papers with Code started a collaboration with arXiv, a popular preprint server. Since October, all machine-learning papers on arXiv have come with a Papers with Code section that links directly to code that authors wish to make available. The aim is to make sharing the norm.
Do such efforts make a difference? Pineau found that last year, when the checklist was introduced, the number of researchers including code with papers submitted to NeurIPS jumped from less than 50% to around 75%. Thousands of reviewers say they used the code to assess the submissions. And the number of participants in the reproducibility challenges is increasing.
Sweating the details
But it is only a start. Haibe-Kains points out that code alone is often not enough to rerun an experiment. Building AI models involves making many small changes—adding parameters here, adjusting values there. Any one of these can make the difference between a model working and not working. Without metadata describing how the models are trained and tuned, the code can be useless. “The devil really is in the detail,” he says.
It’s also not always clear exactly what code to share in the first place. Many labs use special software to run their models; sometimes this is proprietary. It is hard to know how much of that support code needs to be shared as well, says Haibe-Kains.
Pineau isn’t too worried about such obstacles. “We should have really high expectations for sharing code,” she says. Sharing data is trickier, but there are solutions here too. If researchers cannot share their data, they might give directions so that others can build similar data sets. Or you could have a process where a small number of independent auditors were given access to the data, verifying results for everybody else, says Haibe-Kains.
Hardware is the biggest problem. But DeepMind claims that big-ticket research like AlphaGo or GPT-3 has a trickle-down effect, where money spent by rich labs eventually leads to results that benefit everyone. AI that is inaccessible to other researchers in its early stages, because it requires a lot of computing power, is often made more efficient—and thus more accessible—as it is developed. “AlphaGo Zero surpassed the original AlphaGo using far less computational resources,” says Koray Kavukcuoglu, vice president of research at DeepMind.
In theory, this means that even if replication is delayed, at least it is still possible. Kavukcuoglu notes that Gian-Carlo Pascutto, a Belgian coder at Mozilla who writes chess and Go software in his free time, was able to re-create a version of AlphaGo Zero called Leela Zero, using algorithms outlined by DeepMind in its papers. Pineau also thinks that flagship research like AlphaGo and GPT-3 is rare. The majority of AI research is run on computers that are available to the average lab, she says. And the problem is not unique to AI. Pineau and Benaich both point to particle physics, where some experiments can only be done on expensive pieces of equipment such as the Large Hadron Collider.
In physics, however, university labs run joint experiments on the LHC. Big AI experiments are typically carried out on hardware that is owned and controlled by companies. But even that is changing, says Pineau. For example, a group called Compute Canada is putting together computing clusters to let universities run large AI experiments. Some companies, including Facebook, also give universities limited access to their hardware. “It’s not completely there,” she says. “But some doors are opening.”
Haibe-Kains is less convinced. When he asked the Google Health team to share the code for its cancer-screening AI, he was told that it needed more testing. The team repeats this justification in a formal reply to Haibe-Kains’s criticisms, also published in Nature: “We intend to subject our software to extensive testing before its use in a clinical environment, working alongside patients, providers and regulators to ensure efficacy and safety.” The researchers also said they did not have permission to share all the medical data they were using.
It’s not good enough, says Haibe-Kains: “If they want to build a product out of it, then I completely understand they won’t disclose all the information.” But he thinks that if you publish in a scientific journal or conference, you have a duty to release code that others can run. Sometimes that might mean sharing a version that is trained on less data or uses less expensive hardware. It might give worse results, but people will be able to tinker with it. “The boundaries between building a product versus doing research are getting fuzzier by the minute,” says Haibe-Kains. “I think as a field we are going to lose.”
Research habits die hard
If companies are going to be criticized for publishing, why do it at all? There’s a degree of public relations, of course. But the main reason is that the best corporate labs are filled with researchers from universities. To some extent the culture at places like Facebook AI Research, DeepMind, and OpenAI is shaped by traditional academic habits. Tech companies also win by participating in the wider research community. All big AI projects at private labs are built on layers and layers of public research. And few AI researchers haven’t made use of open-source machine-learning tools like Facebook’s PyTorch or Google’s TensorFlow.
As more research is done in house at giant tech companies, certain trade-offs between the competing demands of business and research will become inevitable. The question is how researchers navigate them. Haibe-Kains would like to see journals like Nature split what they publish into separate streams: reproducible studies on one hand and tech showcases on the other.
But Pineau is more optimistic. “I would not be working at Facebook if it did not have an open approach to research,” she says.
Other large corporate labs stress their commitment to transparency too. “Scientific work requires scrutiny and replication by others in the field,” says Kavukcuoglu. “This is a critical part of our approach to research at DeepMind.”
“OpenAI has grown into something very different from a traditional laboratory,” says Kayla Wood, a spokesperson for the company. “Naturally that raises some questions.” She notes that OpenAI works with more than 80 industry and academic organizations in the Partnership on AI to think about long-term publication norms for research.
Pineau believes there’s something to that. She thinks AI companies are demonstrating a third way to do research, somewhere between Haibe-Kains’s two streams. She contrasts the intellectual output of private AI labs with that of pharmaceutical companies, for example, which invest billions in drugs and keep much of the work behind closed doors.
The long-term impact of the practices introduced by Pineau and others remains to be seen. Will habits be changed for good? What difference will it make to AI’s uptake outside research? A lot hangs on the direction AI takes. The trend for ever larger models and data sets—favored by OpenAI, for example—will continue to make the cutting edge of AI inaccessible to most researchers. On the other hand, new techniques, such as model compression and few-shot learning, could reverse this trend and allow more researchers to work with smaller, more efficient AI.
Either way, AI research will still be dominated by large companies. If it’s done right, that doesn’t have to be a bad thing, says Pineau: “AI is changing the conversation about how industry research labs operate.” The key will be making sure the wider field gets the chance to participate. Because the trustworthiness of AI, on which so much depends, begins at the cutting edge.
Article link: AI is wrestling with a replication crisis | MIT Technology Review
Technology companies have taken many aspects of tech governance from democratically elected leaders. It will take an international effort to fight back.by
September 29, 2020
MS TECH | GETTY
Should Twitter censor lies tweeted by the US president? Should YouTube take down covid-19 misinformation? Should Facebook do more against hate speech? Such questions, which crop up daily in media coverage, can make it seem as if the main technologically driven risk to democracies is the curation of content by social-media companies. Yet these controversies are merely symptoms of a larger threat: the depth of privatized power over the digital world.
Every democratic country in the world faces the same challenge, but none can defuse it alone. We need a global democratic alliance to set norms, rules, and guidelines for technology companies and to agree on protocols for cross-border digital activities including election interference, cyberwar, and online trade. Citizens are better represented when a coalition of their governments—rather than a handful of corporate executives—define the terms of governance, and when checks, balances, and oversight mechanisms are in place.
There’s a long list of ways in which technology companies govern our lives without much regulation. In areas from building critical infrastructure and defending it—or even producing offensive cyber tools—to designing artificial intelligence systems and government databases, decisions made in the interests of business set norms and standards for billions of people.
Increasingly, companies take over state roles or develop products that affect fundamental rights. For example, facial recognition systems that were never properly regulated before being developed and deployed are now so widely used as to rob people of their privacy. Similarly, companies systematically scoop up private data, often without consent—an industry norm that regulators have been slow to address.
Related Story
Why Congress should look at Twitter and Facebook
Since technologies evolve faster than laws, discrepancies between private agency and public oversight are growing. Take, for example, “smart city” companies, which promise that local governments will be able to ease congestion by monitoring cars in real time and adjusting the timing of traffic lights. Unlike, say, a road built by a construction company, this digital infrastructure is not necessarily in the public domain. The companies that build it acquire insights and value that may not flow back to the public.
This disparity between the public and private sectors is spiraling out of control. There’s an information gap, a talent gap, and a compute gap. Together, these add up to a power and accountability gap. An entire layer of control of our daily lives thus exists without democratic legitimacy and with little oversight.
Why should we care? Because decisions that companies make about digital systems may not adhere to essential democratic principles such as freedom of choice, fair competition, nondiscrimination, justice, and accountability. Unintended consequences of technological processes, wrong decisions, or business-driven designs could create serious risks for public safety and national security. And power that is not subject to systematic checks and balances is at odds with the founding principles of most democracies.
Today, technology regulation is often characterized as a three-way contest between the state-led systems in China and Russia, the market-driven one in the United States, and a values-based vision in Europe. The reality, however, is that there are only two dominant systems of technology governance: the privatized one described above, which applies in the entire democratic world, and an authoritarian one.
To bring globe-spanning technology firms to heel, we need something new: a global alliance that puts democracy first.
The laissez-faire approach of democratic governments, and their reluctance to rein in private companies at home, also plays out on the international stage. While democratic governments have largely allowed companies to govern, authoritarian governments have taken to shaping norms through international fora. This unfortunate shift coincides with a trend of democratic decline worldwide, as large democracies like India, Turkey, and Brazil have become more authoritarian. Without deliberate and immediate efforts by democratic governments to win back agency, corporate and authoritarian governance models will erode democracy everywhere.
Does that mean democratic governments should build their own social-media platforms, data centers, and mobile phones instead? No. But they do need to urgently reclaim their role in creating rules and restrictions that uphold democracy’s core principles in the technology sphere. Up to now, these governments have slowly begun to do that with laws at the national level or, in Europe’s case, at the regional level. But to bring globe-spanning technology firms to heel, we need something new: a global alliance that puts democracy first.
Teaming up
Global institutions born in the aftermath of World War II, like the United Nations, the World Trade Organization, and the North Atlantic Treaty Organization, created a rules-based international order. But they fail to take the digital world fully into account in their mandates and agendas, even if many are finally starting to focus on digital cooperation, e-commerce, and cybersecurity. And while digital trade (which requires its own regulations, such as rules for e-commerce and criteria for the exchange of data) is of growing importance, WTO members have not agreed on global rules covering services for smart manufacturing, digital supply chains, and other digitally enabled transactions.
What we need now, therefore, is a large democratic coalition that can offer a meaningful alternative to the two existing models of technology governance, the privatized and the authoritarian. It should be a global coalition, welcoming countries that meet democratic criteria.
The Community of Democracies, a coalition of states that was created in 2000 to advance democracy but never had much impact, could be revamped and upgraded to include an ambitious mandate for the governance of technology. Alternatively, a “D7” or “D20” could be established—a coalition akin to the G7 or G20 but composed of the largest democracies in the world.
Such a group would agree on regulations and standards for technology in line with core democratic principles. Then each member country would implement them in its own way, much as EU member states do today with EU directives.
What problems would such a coalition resolve? The coalition might, for instance, adopt a shared definition of freedom of expression for social-media companies to follow. Perhaps that definition would be similar to the broadly shared European approach, where expression is free but there are clear exceptions for hate speech and incitements to violence.
Or the coalition might limit the practice of microtargeting political ads on social media: it could, for example, forbid companies from allowing advertisers to tailor and target ads on the basis of someone’s religion, ethnicity, sexual orientation, or collected personal data. At the very least, the coalition could advocate for more transparency about microtargeting to create more informed debate about which data collection practices ought to be off limits.
The democratic coalition could also adopt standards and methods of oversight for the digital operations of elections and campaigns. This might mean agreeing on security requirements for voting machines, plus anonymity standards, stress tests, and verification methods such as requiring a paper backup for every vote. And the entire coalition could agree to impose sanctions on any country or non-state actor that interferes with an election or referendum in any of the member states.
Related Story
Why Facebook’s political-ad ban is taking on the wrong problem
Another task the coalition might take on is developing trade rules for the digital economy. For example, members could agree never to demand that companies hand over the source code of software to state authorities, as China does. They could also agree to adopt common data protection rules for cross-border transactions. Such moves would allow a sort of digital free-trade zone to develop across like-minded nations.
China already has something similar to this in the form of eWTP, a trade platform that allows global tariff-free trade for transactions under a million dollars. But eWTP, which was started by e-commerce giant Alibaba, is run by private-sector companies based in China. The Chinese government is known to have access to data through private companies. Without a public, rules-based alternative, eWTP could become the de facto global platform for digital trade, with no democratic mandate or oversight.
Another matter this coalition could address would be the security of supply chains for devices like phones and laptops. Many countries have banned smartphones and telecom equipment from Huawei because of fears that the company’s technology may have built-in vulnerabilities or backdoors that the Chinese government could exploit. Proactively developing joint standards to protect the integrity of supply chains and products would create a level playing field between the coalition’s members and build trust in companies that agree to abide by them.
The next area that may be worthy of the coalition’s attention is cyberwar and hybrid conflict (where digital and physical aggression are combined). Over the past decade, a growing number of countries have identified hybrid conflict as a national security threat. Any nation with highly skilled cyber operations can wreak havoc on countries that fail to invest in defenses against them. Meanwhile, cyberattacks by non-state actors have shifted the balance of power between states.
Right now, though, there are no international criteria that define when a cyberattack counts as an act of war. This encourages bad actors to strike with many small blows. In addition to their immediate economic or (geo)political effect, such attacks erode trust that justice will be served.
A democratic coalition could work on closing this accountability gap and initiate an independent tribunal to investigate such attacks, perhaps similar to the Hague’s Permanent Court of Arbitration, which rules on international disputes. Leaders of the democratic alliance could then decide, on the basis of the tribunal’s rulings, whether economic and political sanctions should follow.
These are just some of the ways in which a global democratic coalition could advance rules that are sorely lacking in the digital sphere. Coalition standards could effectively become global ones if its members represent a good portion of the world’s population. The EU’s General Data Protection Regulation provides an example of how this could work. Although GDPR applies only to Europe, global technology firms must follow its rules for their European users, and this makes it harder to object as other jurisdictions adopt similar laws. Similarly, non-members of the democratic coalition could end up following many of its rules in order to enjoy the benefits.
If democratic governments do not assume more power in technology governance as authoritarian governments grow more powerful, the digital world—which is a part of our everyday lives—will not be democratic. Without a system of clear legitimacy for those who govern—without checks, balances, and mechanisms for independent oversight—it’s impossible to hold technology companies accountable. Only by building a global coalition for technology governance can democratic governments once again put democracy first.
Marietje Schaake is the international policy director at Stanford University’s Cyber Policy Center and an international policy fellow at Stanford’s Institute for Human-Centered Artificial Intelligence. Between 2009 and 2019, Marietje served as a Member of European Parliament for the Dutch liberal democratic party.
Article meta
Share
Tagged
Author
Tech companies are setting norms and standards of all kinds that used to be set by governments.
Technology companies provide much of the critical infrastructure of the modern state and develop products that affect fundamental rights. Search and social media companies, for example, have set de facto norms on privacy, while facial recognition and predictive policing software used by law enforcement agencies can contain racial bias.
In this episode of Deep Tech, Marietje Schaake argues that national regulators aren’t doing enough to enforce democratic values in technology, and it will take an international effort to fight back. Schaake—a Dutch politician who used to be a member of the European parliament and is now international policy director at Stanford University’s Cyber Policy Center—joins our editor-in-chief, Gideon Lichfield, to discuss how decisions made in the interests of business are dictating the lives of billions of people.
Also this week, we get the latest on the hunt to locate an air leak aboard the International Space Station—which has grown larger in recent weeks. Elsewhere in space, new findings suggest there is even more liquid water on Mars than we thought. It’s located in deep underground lakes and there’s a chance it could be home to Martian life. Space reporter Neel Patel explains how we might find out.
Back on Earth, the US election is heating up. Data reporter Tate Ryan-Mosley breaks down how technologies like microtargeting and data analytics have improved since 2016.
Check out more episodes of Deep Tech here.
Megaphone: A Modern Podcasting Platform
Show notes and links:
- How democracies can claim back power in the digital world September 29, 2020
- The technology that powers the 2020 campaigns, explained September 28, 2020
- There might be even more underground reservoirs of liquid water on Mars September 28, 2020
- Astronauts on the ISS are hunting for the source of another mystery air leak September 30, 2020
Full episode transcript:
Gideon Lichfield: There’s a situation playing out onboard the International Space Station that sounds like something out of Star Trek…
Computer: WARNING. Hull breach on deck one. Emergency force fields inoperative.
Crewman: Everybody out. Go! Go! Go!
*alam blares*
Gideon Lichfield: Well, it’s not quite that bad. But there is an air leak in the space station. It was discovered about a year ago, but in the last few weeks, it’s gotten bigger. And while NASA says it’s still too small to endanger the crew… well… they also still can’t quite figure out where the leak is.
Elsewhere in space, new findings suggest there is even more liquid water on Mars than we thought. It’s deep in underground lakes. There might even be life in there. The question is—how will we find out?
Here on Earth, meanwhile, the US election is heating up. We’ll look at how technologies like microtargeting and data analytics have improved since 2016. That means campaigns can tailor messages to voters more precisely than ever.
And, finally, we’ll talk to one of Europe’s leading thinkers on tech regulation, who argues that democratic countries need to start approaching it in an entirely new way.
I’m Gideon Lichfield, editor-in-chief of MIT Technology Review, and this is Deep Tech.
The International Space Station always loses a tiny bit of air, and it’s had a small leak for about a year. But in August, Mission Control noticed air pressure on board the station was dropping—a sign the leak was expanding.
The crew were told to hunker down in a single module and shut the doors between the others. Mission Control would then have a go at pressurizing each sealed module to determine where the leak was.
As our space reporter Neel Patel writes, this process went on for weeks. And they didn’t find the leak. Until, one night…
Neel Patel: On September 28th, in the middle of the night, the astronauts are woken up. Two cosmonauts and one astronaut that are currently on the ISS. And mission control tells them, “Hey, we think we know where the leak is, finally. You guys have to go to the Russian side of the station in the Svezda module and start poking around and seeing if you can find it.”
Gideon Lichfield: Okay. And so they got up and they got in the, in the module and they went and poked around. And did they find it?
Neel Patel: No, they have still not found that leak yet. These things take a little bit of time. It’s, you know, you can’t exactly just run around searching every little wall in the module and, you know, seeing if there’s a little bit of cool air that’s starting to rush out.
The best way for the astronauts to look for the leak is a little ultrasonic leak detector. That kind of spots frequencies that air might be rushing out. And that’s an indication of where there might be some airflow where there shouldn’t be. And it’s really just a matter of holding that leak detector up to sort of every little crevice and determining if things are, you know, not the way they should be.
Gideon Lichfield: So as I mentioned earlier, the space station always leaks a little bit. What made this one big enough to be worrying?
Neel Patel: So..the.. you know, like I said before, the air pressure was dropping a little bit. That’s an indication that the hole is not stable, that there might be something wrong, that there could allegedly be some kind of cracks that had been growing.
And if that’s the case, it means that the hull of the spacecraft at that point is a little bit unstable. And if the leak is not taken care of as soon as possible, if the cracks are not repaired, as soon as possible, things could grow and grow and eventually reach a point where something might break. Now, that’s a pretty distant possibility, but you don’t take chances up in space.
Gideon Lichfield: Right. And also you’re losing air and air is precious…
Neel Patel: Right. And in this instance, there was enough air leaking that there started to be concerns from both the Russian and US sides that they may need to send in more oxygen sooner than later.
And, you know, the way space operations work, you have things planned over for years in advance. And of course, you know, you still have a leak to worry about.
Gideon Lichfield: So how do leaks actually get started on something like the ISS?
Neel Patel: So that’s a good question. And there are a couple ways for this to happen. Back in 2018, there was a two millimeter hole found on the Russian Soyuz spacecraft.
That was very worrisome and no one understood initially how that leak might’ve formed. Eventually it was determined that a drilling error during manufacturing probably caused it. That kind of leak was actually sort of good news because it meant that, with a drilling hole, things are stable. There aren’t any kind of like aberrant cracks that could, you know, get bigger and start to lead to a bigger destruction in the hull. So that was actually good news then, but other kinds of leaks are mostly thought to be caused by micro meteoroids.
Things in space are flying around at. Over 20,000 miles per hour, which means even the tiniest little object, even the tiniest little grain or dust could you know, just whip a very massive hole inside the hull of the space station.
Gideon Lichfield: Ok so those are micro meteoroids that are probably causing those kinds of leaks, but obviously there’s also a growing problem of space debris. Bits of spacecraft and junk that we’ve been thrown up into orbit that is posing a threat.
Neel Patel: Absolutely space debris is a problem. It’s only getting worse and worse with every year. Probably the biggest, most high profile, incident that caused the most space debris in history was the 2009 crash between two satellites, Iridium 33 and cosmos 2251. That was the first and only satellite crash between two operational satellites that we know of so far. And the problem with that crash is it ended up creating tons and tons of debris that were less than 10 centimeters in length. Now objects greater than 10 centimeters are tracked by the Air Force, but anything smaller than 10 centimeters is virtually undetectable so far. That means that, you know, any of these little objects that are under 10 centimeters, which is, you know, a lot of different things are threats to the ISS. And as I mentioned before at the speed that these things are running at, they could cause big destruction for the ISS or any other spacecraft in orbit.
Gideon Lichfield: So it’s basically a gamble? Yeah? They’re just hoping that none of these bits crashes into it, because if it does, there’s nothing they can do to spot it or stop it.
Neel Patel: No, our radar technologies are getting better. So we’re able to spot smaller and smaller objects, but this is still a huge problem that so many experts have been trying to raise alarms about.
And unfortunately, the sort of officials that be, that control, you know, how we manage the space environment still haven’t come to a consensus about what we want to do about this, what kind of standards we want to implement and how we can reduce the problem.
Gideon Lichfield: So… They still haven’t found this leak. So what’s going on now?
Neel Patel: Okay. So according to a NASA spokesperson quote, there have been no significant updates on the leak since September 30th. Roscosmos, the Russian space agency, released information that further isolated the leak to the transfer chamber of the Svezda service module. The investigation is still ongoing and poses no immediate danger to the crew.
Gideon Lichfield: All right, leaving Earth orbit for a bit. Let’s go to Mars. People have been looking for water on Mars for a long time, and you recently reported that there might be more liquid water on Mars than we originally thought. Tell us about this discovery.
Neel Patel: So in 2018, a group of researchers used radar observations that were made by the European Space Agency’s Mars Express orbiter to determine that there was a giant, subsurface lake sitting 1.5 kilometers below the surface of Mars underneath the glaciers near the South pole. The lake is huge. It’s almost 20 kilometers long and is, you know, liquid water. We’re not talking about the frozen stuff that’s sitting on the surface. We’re talking about liquid water. Two years later, the researchers have come back to even more of that radar data. And what they found is that neighboring that body of water might be three other lakes. Also nearby, also sitting a kilometer underground.
Gideon Lichfield: So how does this water stay liquid? I mean Mars is pretty cold, especially around the poles.
Neel Patel: So the answer is salt. It’s suspected that these bodies of waters have been able to exist in a liquid form for so long, despite the frigid temperatures, because they’re just caked in a lot of salt. Salts, as you might know, can significantly lower the freezing point of water. On Mars it’s thought that there might be calcium, magnesium, sodium, and other salt deposits.
These have been found around the globe and it’s probable that these salts are also existing inside the lakes. And that’s what allowed them to have stayed as liquid instead of a solid for so long.
Gideon Lichfield: So what would it take to get to these underground lakes? If we could actually be on Mars and what might we find when we got there?
Neel Patel: These lakes, as I’ve mentioned, are sitting at least one kilometer sometimes further, deeper, underground. Uh, there’s not really a chance that any kind of future Martian explorers in the next generation or two are going to have the type of equipment that are gonna allow them to drill all the way that deep.
Which is not really a problem for these future colonists. There’s plenty of surface ice at the Martian poles that’s easier to harvest in case they want to create drinking water or, you know, turn that into hydrogen oxygen, rocket fuel.
The important thing to think about is do these underground lakes perhaps possess Martian life. As we know on Earth, life can exist in some very extreme conditions and it’s, you know, at least a non zero chance that these lakes perhaps also possess the same sort of extreme microbes that can survive these kinds of frigid temperatures and salty environments.
Gideon Lichfield: Alright so maybe we don’t want to try to drink this water, but it would be great if we could explore it to find out if there is in fact life there. So is there any prospect that any current or future space mission could get to those leaks and find that out?
Neel Patel: No, not anytime soon. Drilling equipment is very big, very heavy. There’s no way you’re going to be able to properly fit something like that on a spacecraft. That’s going to Mars. But one way we might be able to study the lakes is by measuring the seismic activity around the South pole.
If we were to place a small little Lander on the surface of Mars, have it drill just a little ways into the ground. It could measure the vibrations coming out of Mars. It could use those, use that data to characterize how big the lakes are, what their shape might be. And by extension, we may be able to use that data to determine, you know, how… in what locations of the lakes life might exist and, you know, figure out where we want to probe next for further study.
Gideon Lichfield: Technology has been an increasingly important part of political campaigns in the US, particularly since Barack Obama used micro-targeting and big data to transform the way that he campaigned. With every election since then, the techniques have gotten more and more sophisticated. And in her latest story for MIT technology review, Tate Ryan-Mosley looks at some of the ways in which the campaigns this time round are segmenting and targeting voters even more strategically than before. So Tate, can you guide us through what is new and improved and how things have changed since the 2016 election?
Tate Ryan-Mosley: Yeah. So I’ve identified kind of four key continuations of trends that have started and in prior presidential elections, and all, kind of all of the trends are pushing towards this kind of new era of campaigning where all of the messages, the positioning, the presentation of their candidates is really being, you know, personalized for each individual person in the United States. And so, the key things driving that are really, you know, data acquisition. So the amount of data that these campaigns have on every person in the United States. Another new thing is data exchanges which is kind of the structural mechanism by which all of this data is aggregated and shared and used.
And then the way that that data kind of gets pushed into the field and into strategy is of course microtargeting. And this year, you know, we’re seeing campaigns employ things with much more granularity, like using SMS as one of the main communication tools to reach prospective voters. Actually uploading lists of profile names into social media websites. And lastly, kind of a big shift in 2020 is a more clear move away from kind of the traditional opinion polling mechanisms into AI modeling. So instead of having, you know, these big polling companies call a bunch of people and try to get a sense of the pulse of the election, you’re really seeing AI being leveraged to predict the outcomes of elections and in particular segments.
Gideon Lichfield: So let’s break a couple of those things down. One of the areas that you talked about is data exchanges, and there’s a company that you write about in your story called Data Trust. Can you tell us a bit about who they are and what they do?
Tate Ryan-Mosley: So data trust is the Republican’s kind of main data aggregation technology. And so what it enables them to do is collect data on all prospective voters, host that data, analyze the data, and actually share it with, politically aligned PACs, 501(c)(3)’s and 501(c)(4)’s. And previously because of FEC regulations, you’re not allowed to kind of cross that wall between campaign and 501(c)(3)’s, 501(c)(4)’s and PACs. And the way that these data exchanges are set up is it’s enabling data sharing between those groups.
Gideon Lichfield: How does that not cross the wall?
Tate Ryan-Mosley: Right. So basically the, what they say is the data is anonymized to the point that you don’t know where the data is coming from. And that is kind of the way that they’ve been able to skirt the rules. The Democrats actually sued the Republicans after the 2016 election, and then they lost. And so what’s really notable is that this year the Democrats have created their own data exchange, which is called DDX. And so this is the first year that the Democrats will have any type of similar technology. And since the Democrats have come online, they’ve actually collected over 1 billion data points, which is a lot of data.
Gideon Lichfield: So these data exchanges allow basically a campaign and everyone that is aligned with it, supporting it, to share all the same data. And what is that enabling them to do that they couldn’t do before?
Tate Ryan-Mosley: Yeah,that’s a good question. And what it’s really doing is it’s kind of enabling a lot of efficiency and the way that voters are being reached. So there’s a lot of double spend on voters who are already decided. So for example, the Trump campaign might be reaching out to a particular, you know, voter that has already been decided by a group like the NRA to be, you know, conservatively aligned and very likely to vote for Trump. But the Trump campaign doesn’t know that in their data set. So this would enable the Trump campaign to not spend money reaching out to that person. And it makes kind of the efficiency and the comprehensiveness of their outreach kind of next level.
Gideon Lichfield: So let’s talk about micro-targeting. The famous example of micro-targeting of course, is Cambridge Analytica, which illicitly acquired a bunch of people’s data from Facebook in the 2016 campaign, and then claimed that it could design really specific messages aimed at millions of American voters. And a lot of people, I think called that ability into question, right. But where are we now with microtargeting?
Tate Ryan-Mosley: There’s kind of this misconception around the way in which microtargeting is impactful. What Cambridge Analytica claimed to do was use data about people’s opinions and personalities to profile them and create messages that were really likely to persuade a person about a specific issue at a particular time. And that’s kind of what’s been debunked. That, you know, political ads, political messages are not actually significantly more persuasive now than they’ve ever been. And really you can’t prove it. There’s no way to attribute a vote to a particular message or a particular ad campaign.
Tate Ryan-Mosley: So what’s really become the consensus about, you know, why micro-targeting is important is that it increases the polarization of the electorate or the potential electorate. So basically it’s really good at identifying already decided voters and making them either more mobile. So you know, more vocal about their cause and their position or bringing them increasingly into the hard line and even getting them to donate. So we saw this pretty clearly with the Trump campaigns app that they have put out this year.
So there’s a lot of surveillance kind of built into the structure of the app that is meant to micro target their own supporters. and the reason they’re doing that is that’s kind of seen as the number one fundraising mechanism. If we can convince somebody who agrees with Trump to get really impassioned about Trump, that really means, that means money.
Gideon Lichfield: Let’s talk about another thing, which is polling. Of course, the difficulty with polling that we saw in the 2016 election was people don’t answer their phones anymore and doing an accurate opinion poll is getting harder and harder. So how is technology helping with that problem?
Tate Ryan-Mosley: So what’s being used is now AI modeling, which basically takes a bunch of data and spits out a prediction about how likely a person is either to show up to vote, to vote in a particular way, or to feel a certain way about a particular issue. and so these AI models they’re also used in 2016 and it’s worth noting in 2016, AI models were about as accurate as traditional opinion polls in terms of, you know, really not predicting that Trump was going to win. But you know, as the data richness gets better, as data gets more, you know, becomes more real time, as the quality improves, we’re seeing an increased accuracy in AI modeling that kind of is signifying. It’s likely to take, you know, more and more, become a bigger part of how polling is done.
Gideon Lichfield: So what we’re seeing is that this election represents a new level in the use of technologies that we’ve seen over the past decade or more, that are us the ability, or giving campaigns the ability to target people ever more precisely to share data about people more widely and use it more efficiently. As well as to predict which way voters are going to go much more reliably. So what does all this add up to? What are the consequences for our politics?
Tate Ryan-Mosley: What we’re really seeing as is kind of a fragmentation of campaign messaging and the ability to kind of scale those fragments and those silos up. And so what’s happening is it’s becoming significantly easier for campaigns to say different things, to different groups of people and that kind of skirts some of the norms that we have and in public opinion and civic discourse around lying around, you know, switching positions around distortion that have in the past really been able to check public figures.
Gideon Lichfield: Because politicians can say one thing to one group of people, a completely different thing to a different group. And the two groups don’t know that they’re being fed different messages.
Tate Ryan-Mosley: Exactly. So, you know, the Biden campaign can easily send out a text message to a small group of, you know, 50 people in a swing county that say something really specific to their local politics. And most people wouldn’t ever know, or really be able to fact check them because they just don’t have access to the messages that campaigns are giving, you know, really specific groups of people.
And so that’s really kind of changing the way that we have civic discourse. And you know, it even allows some campaigns to kind of manufacture cleavages in the public. So it can actually kind of game out how they want to be viewed by a specific group of people and hit those messages home, you know, and kind of create cleavage that previously wasn’t there or wouldn’t be there organically.
Gideon Lichfield: Does that mean that American politics is just set to become irretrievably fragmented?
Tate Ryan-Mosley: I mean, that’s absolutely the concern. What’s interesting as I’ve talked to some experts that actually feel that this might indeed be the pinnacle of campaign technology and personalized campaigns because public opinion is really shifting on this. So Pew research group actually just did a survey that came out this month that showed that the majority of the American public does not think social media platforms should allow for any political advertisement at all.
And the large majority of Americans believe that political micro-targeting, especially on social media should be disallowed. And we’re starting to see that reflected in Congress. So there are a handful of bills actually that have bipartisan support that have been introduced to both the house and the Senate that are seeking to kind of address some of these issues. Obviously we won’t see the impact of that before the 2020 election, but a lot of experts are pretty hopeful that we’ll be able to see some legitimate regulation for the upcoming presidential in 2024.
Gideon Lichfield: Tech companies are setting norms and standards of all kinds that used to be set by governments. That’s the view of Marietje Schaake, who wrote an essay for us recently. Marietje is a Dutch politician who used to be a member of the European parliament and is now international policy director at Stanford University’s Cyber Policy Center. Marietje, What’s a specific example of the way in which the decisions that tech companies have made end up effectively setting the norms for the rest of us?
Marietje Schaake: Well, I think a good example is how, for example, facial recognition systems and even the whole surveillance model of social media and search companies has set de facto norms compromising the right to privacy. I mean, if you look at how much data is collected across a number of services, the fact that there’s data brokers renders the rights of privacy very, very fragile, if not compromised as such. And so I think that is an example, especially if there’s no laws to begin with where the de facto standard is very, very hard to roll back once it’s set by the companies.
Gideon Lichfield: Right. So how did we get to this?
Marietje Schaake: Yeah, that’s, that’s the billion dollar question. And I think we have to go back to the culture that went along with the rise of companies coming out of Silicon Valley that was essentially quite libertarian. And I think they, these companies, these, entrepreneurs, these innovators, may have had good intentions, may have hoped that their inventions and their businesses would have a liberating effect and they can lawmakers that the best support that they could give this liberating technology was to do nothing in the form of regulation. And effectively in the US and in the EU—even if the EU is often called a super regulator—there has been very, very little regulation to preserve core principles like non-discrimination or antitrust in light of the massive digital disruptions. And so the success of the libertarian culture from Silicon Valley, the power of big tech companies now that can lobby against regulatory proposals explains why we are where we are.
Gideon Lichfield: One of the things that you say in your essay is that there are actually two kinds of regulatory regimes in the world, for tech. There’s the privatized one, in other words, in Western countries the tech companies are really the ones setting a lot of the rules for how the digital space works. And then there’s an authoritarian one which is China, Russia, and other countries where governments are taking a very heavy handed approach to regulation. What are the consequences then of having a world in which it’s a choice between these two regimes?
Marietje Schaake: I think the net result is that the resilience of democracy and actually the articulation of democratic values, the safeguarding of democratic values, the building of institutions has lagged behind. And this comes at a time where democracy is under pressure globally anyway. We can see it in our societies. We can see it on the global stage where in multilateral organizations, it is not a given that the democracies have the majority of votes or, or voices. And so all in all it makes democracy and projected out into the future, the democratic mark on the digital world, very fragile. And that’s why I think there’s reason for concern.
Gideon Lichfield: Okay. So in your essay, you’re proposing a solution to all of this, which is a kind of democratic alliance of nations to create rules for tech governance. Why is that necessary?
Marietje Schaake: Right. I think it’s necessary for democracies to work together much more effectively, and to step up their role in developing a democratic governance model of technology. And I think it’s necessary because with the growing power of. Corporations and their, uh, ability to set standards and effectively to govern the digital world on the one hand.
And then on the other hand, a much more top down control oriented state led model that we would see in States like China and Russia. There, there’s just too much of a vacuum on the part of democracies. And I think if they work together, they’re in the best position to handle cross border companies and to have an effective way of working together to make sure that they leverage their collective scale, essentially.
Gideon Lichfield: Can you give an example of how this democratic coalition would work? What sorts of decisions might it take or where might it set rules?
Marietje Schaake: Well, let me focus on one area that I think needs a lot of work and attention. And that is the question of how to interpret laws of war and armed conflict but also the preservation of peace and accountability after cyber attacks.
So right now, because there is a vacuum in the understanding of how laws of armed conflict and thresholds of war apply in the digital world, attacks happen every day. But often without consequences. And the notion of accountability, I think is very important as part of the rule of law to ensure that there is a sense of justice also in the digital world. And so I can very well imagine that in this space that really needs to be articulated and shaped now with institutions and mechanisms, then the democracies could, could really focus on that area of war, of peace, of accountability.
Gideon Lichfield: So when you say an attack happens without consequences, you mean some nation state or some actor launches a cyber attack and nobody can agree that it should be treated as an act of war?
Marietje Schaake: Exactly. I think that that is happening far more often than people might realize. And in fact, because there is such a legal vacuum, it’s easy for attackers to sort of stay in a zone where they can almost anticipate that they will not face any consequences. And part of this is political. How willing are countries to come forward and point to a perpetrator. But it’s also that there’s currently a lack of proper investigation to ensure that there might be something like a trial, you know, a court of arbitration where different parties can, can speak about their side of the conflict and that there would be a ruling by an independent, judiciary-type of organization to make sure that there is an analysis of what happened but that there’s also consequences to clearly escalatory behavior.
And if the lack of accountability continues, I fear that it will play into the hands of nations and their proxies. So the current lack of holding to account perpetrators that may launch cyber attacks to achieve their geopolitical political or even economic goals is very urgent. So I would imagine that a kind of tribunal or a mechanism of arbitration could really help close this accountability gap.
Gideon Lichfield: That’s it for this episode of Deep Tech. This is a podcast just for subscribers of MIT Technology Review, to bring alive the issues our journalists are thinking and writing about.
Before we go, I want to quickly tell you about EmTech MIT, which runs from October 19th through the 22nd. It’s our flagship annual conference on the most exciting trends in emerging technology.
This year, it’s all about how we can build technology that meets the biggest challenges facing humanity, from climate change and racial inequality to pandemics and cybercrime.
Our speakers include the CEOs of Salesforce and Alphabet X, the CTOs of Facebook and Twitter, the head of cybersecurity at the National Security Agency, the head of vaccine research at Eli Lilly, and many others. And because of the pandemic, it’s an online event, which means it’s both much cheaper than in previous years and much, much easier to get to.
You can find out more and reserve your spot by visiting EmTechMIT.com – that’s E-M…T-E-C-H…M-I-T dot com – and use the code DeepTech50 for $50 off your ticket. Again, that’s EmTechMIT.com with the discount code DeepTech50.
Deep Tech is written and produced by Anthony Green and edited by Jennifer Strong and Michael Reilly. I’m Gideon Lichfield. Thanks for listening.
Article link: Podcast: How democracies can reclaim digital power | MIT Technology Review
How to improve your decision-making by Sir Andrew Likierman
From the Magazine (January–February 2020)
Idea in Brief
The Objective
A manager’s core function is to exercise judgment—to form views and interpret ambiguous evidence in a way that will lead to a good decision.
The Challenge
We have no clear framework for learning good judgment or recognizing it in others. To evaluate a leader’s judgment, we often rely on his or her track record, which can be misleading.
The Solution
This article identifies six components that contribute to good judgment: learning, trust, experience, detachment, options, and delivery. By working on each, leaders can improve their ability to make sense of an ambiguous situation.
A decision must be made. The facts have been assembled, and the arguments for and against the options spelled out, but no clear evidence supports any particular one. Now people around the table turn to the CEO. What they’re looking for is good judgment—an interpretation of the evidence that points to the right choice.
Judgment—the ability to combine personal qualities with relevant knowledge and experience to form opinions and make decisions—is “the core of exemplary leadership” according to Noel Tichy and Warren Bennis (the authors of Judgment: How Winning Leaders Make Great Calls). It is what enables a sound choice in the absence of clear-cut, relevant data or an obvious path. To some degree we are all capable of forming views and interpreting evidence. What we need, of course, is good judgment.
A lot of ink has been spilled in the effort to understand what good judgment consists of. Some experts define it as an acquired instinct or “gut feeling” that somehow combines deep experience with analytic skills at an unconscious level to produce an insight or recognize a pattern that others overlook. At a high level this definition makes intuitive sense; but it is hard to move from understanding what judgment is to knowing how to acquire or even to recognize it.
In an effort to meet that challenge, I’ve talked to CEOs in a range of companies, from some of the world’s largest right down to start-ups. I’ve approached leaders in the professions as well: senior partners at law and accountancy firms, generals, doctors, scientists, priests, and diplomats. I asked them to share their observations of their own and other people’s exercise of judgment so that I could identify the skills and behaviors that collectively create the conditions for fresh insights and enable decision makers to discern patterns that others miss. I have also looked at the relevant literatures, including leadership and psychology.
I’ve found that leaders with good judgment tend to be good listeners and readers—able to hear what other people actually mean, and thus able to see patterns that others do not. They have a breadth of experiences and relationships that enable them to recognize parallels or analogies that others miss—and if they don’t know something, they’ll know someone who does and lean on that person’s judgment. They can recognize their own emotions and biases and take them out of the equation. They’re adept at expanding the array of choices under consideration. Finally, they remain grounded in the real world: In making a choice they also consider its implementation.
Practices that leaders can adopt, skills they can cultivate, and relationships they can build will inform the judgments they make. In this article I’ll walk through the six basic components of good judgment—I call them learning, trust, experience, detachment, options, and delivery—and offer suggestions for how to improve them.
Learning: Listen Attentively, Read Critically
Good judgment requires that you turn knowledge into understanding. This sounds obvious, but as ever, the devil is in the detail—in this case your approach to learning. Many leaders rush to bad judgments because they unconsciously filter the information they receive or are not sufficiently critical of what they hear or read.
The truth, unfortunately, is that few of us really absorb the information we receive. We filter out what we don’t expect or want to hear, and this tendency doesn’t necessarily improve with age. (Research shows, for example, that children notice things that adults don’t.) As a result, leaders simply miss a great deal of the information that’s available—a weakness to which top performers are especially vulnerable because overconfidence so often comes with success.
Exceptions exist, of course. I first met John Buchanan early in a distinguished four-decade career during which he became the CFO at BP, the chairman of Smith & Nephew, the deputy chairman of Vodafone, and a director at AstraZeneca, Alliance Boots, and BHP Billiton. What struck me immediately and throughout our acquaintance was that he gave me and everyone else his undivided attention. Many people with his record of accomplishment would long ago have stopped listening in favor of pontificating.
Leaders with good judgment tend to be good listeners and readers.
Buchanan was more than a good listener—he was adept at eliciting information that people might not otherwise volunteer. His questions were designed to draw out interesting responses. He told me that when deciding whether to accept a directorship, for example, he would ask questions such as “Where would you place this company on a spectrum of white to gray?” “At first this sounds like a classic piece of managementese that is clever but meaningless,” he said. “Yet it is sufficiently open-ended to draw out replies on a wide range of subjects and sufficiently pointed to produce a meaningful response.”
Information overload, particularly with written material, is another problem. It’s not surprising that CEOs with huge demands on their time and attention struggle to get through the volume of emails and briefing papers they receive. As a director of a large listed company, I would get up to a million words to read ahead of a big meeting. Confronted with such a deluge, it’s tempting to skim and to remember only the material that confirms our beliefs. That’s why smart leaders demand quality rather than quantity in what gets to them. Three hundred pages for the next big meeting? It’s six pages maximum for agenda items at Amazon and the Bank of England.
Overload is not the only challenge when it comes to reading. A more subtle risk is taking the written word at face value. When we listen to people speak, we look (consciously or unconsciously) for nonverbal clues about the quality of what we’re hearing. While reading, we lack that context; and in an era when the term “fake news” is common, decision makers need to pay extra attention to the quality of the information they see and hear, especially material filtered by colleagues or obtained through search engines and social media exchanges. Are you really as careful in assessing and filtering as you should be, knowing how variable the quality is? If you believe that you never unconsciously screen out information, consider whether you choose a newspaper that agrees with what you already think.
People with good judgment are skeptical of information that doesn’t make sense. We might none of us be alive today if it weren’t for a Soviet lieutenant colonel by the name of Stanislav Petrov. It came to light only after the fall of communism that one day in 1983, as the duty officer at the USSR’s missile tracking center, Petrov was advised that Soviet satellites had detected a U.S. missile attack on the Soviet Union. He decided that the 100% probability reading was implausibly high and did not report the information upward, as were his instructions. Instead he reported a system malfunction. “I had all the data [to suggest a missile attack was ongoing],” he told the BBC’s Russian service in 2013. “If I had sent my report up the chain of command, nobody would have said a word against it.” It turned out that the satellites had mistaken sunlight reflected from clouds for missile engines.
To improve:
Active listening, including picking up on what’s not said and interpreting body language, is a valuable skill to be honed, and plenty of advice exists. Beware of your own filters and of defensiveness or aggression that may discourage alternative arguments. If you get bored and impatient when listening, ask questions and check conclusions. If you’re overwhelmed by written briefing material, focus on the parts that discuss questions and issues rather than those that summarize the presentations you’ll hear at the meeting. (Far too many board packs are stuffed with advance copies of presentations.) Look for gaps or discrepancies in what’s being said or written. Think carefully about where the underlying data is coming from and the likely interests of the people supplying it. If you can, get input and data from people on more than one side of an argument—especially people you don’t usually agree with. Finally, make sure the yardsticks and proxies for data you rely on are sound; look for discrepancies in the metrics and try to understand them.
Trust: Seek Diversity, Not Validation
Leadership shouldn’t be a solitary endeavor. Leaders can draw on the skills and experiences of others as well as their own when they approach a decision. Who these advisers are and how much trust the leader places in them are critical to the quality of that leader’s judgment.
Unfortunately, many CEOs and entrepreneurs bring people on board who simply echo and validate them. The disgraced executives Elizabeth Holmes and Sunny Balwani of the start-up Theranos regarded anyone who raised a concern or an objection as a cynic and a naysayer. “Employees who persisted in doing so were usually marginalized or fired, while sycophants were promoted,” according to the Financial Times. Recently jailed for 18 years, Wu Xiaohui, the founder and leading light of China’s Anbang Insurance Group, had built up a diverse international empire, buying major assets that included New York’s Waldorf Astoria hotel. He also surrounded himself with “unimpressive people who would just follow his orders and not question them,” one employee told FT.
The historian Doris Kearns Goodwin, in her book Team of Rivals, noted that Abraham Lincoln assembled a cabinet of experts he respected but who didn’t always agree with one another. McKinsey has long included the obligation (not a suggestion) to dissent as a central part of the way it does business. Amazon’s Leadership Principles specify that leaders should “seek diverse perspectives and work to disconfirm their beliefs.”
Alibaba’s Jack Ma thinks along the same lines. Recognizing his own ignorance of technology (he was 33 when he got his first computer), Ma hired John Wu of Yahoo as his chief technology officer, commenting, “For a first-class company we need first-class technology. When John comes, I can sleep soundly.” Ma isn’t the only mega-entrepreneur who has looked for advisers with organizational and personal qualities and experience to fill a void in himself. Facebook’s Mark Zuckerberg hired Sheryl Sandberg for a similar reason. And Natalie Massenet, founder of the online fashion retailer Net-a-Porter, hired the much older Mark Sebba, the “understated chief executive of Net-a-Porter who brought order to the ecommerce start-up in the manner of Robert De Niro in The Intern,” according to the Times of London. My brother Michael told me that one reason his company’s chain of opticians, under the brand GrandOptical, became the largest in France is that he partnered with Daniel Abittan, whose operational excellence complemented Michael’s entrepreneurial vision and strategic skills.
To improve:
Cultivate sources of trusted advice: people who will tell you what you need to know rather than what you want to hear. When you are recruiting people on whose advice you will rely, don’t take outcomes as a proxy for their good judgment. Make judgment an explicit factor in appraisals and promotion decisions. Usha Prashar, who chaired the body that makes the UK’s most-senior judicial appointments, pointed to the need to probe how a candidate did things, not just what he or she had done. Dominic Barton of McKinsey told me that he looked for what was not being said: Did people fail to mention any “real” difficulties or setbacks or failures in their careers to date? One CEO said he asked people about situations in which they’d had insufficient information or conflicting advice. Don’t be put off by assessments that a candidate is “different.” Someone who disagrees with you could provide the challenge you need.
Experience: Make It Relevant but Not Narrow
Beyond the data and evidence pertinent to a decision, leaders bring their experience to bear when making judgment calls. Experience gives context and helps us identify potential solutions and anticipate challenges. If they have previously encountered something like a current challenge, leaders can scope out areas in which to focus their energy and resources.
Mohamed Alabbar, the chairman of Dubai’s Emaar Properties and one of the Middle East’s most successful entrepreneurs, gave me an example. His first major property crisis, in Singapore in 1991, had taught him about the vulnerability that comes with being highly geared in a downturn—and in real estate, only those who learn the lessons of overgearing in their first crash survive in the long term. Alabbar has since navigated Dubai’s often dramatic economic cycles and today owns a portfolio that includes the Burj Khalifa, the world’s tallest building, and the Dubai Mall, one of the world’s largest shopping malls.
Success Is Not a Reliable Proxy for Judgment
It’s tempting to assume that past successes are a sign of good judgment, and in some cases they may be. The multigenerational success of some German midsize companies and the sheer longevity …
But—and it’s a big but—if the experience is narrowly based, familiarity can be dangerous. If my company is planning to enter the Indian market, I might not trust the judgment of a person whose only product launches have been in the United States. I would probably be less worried about someone who had also launched new products in, say, China and South Africa, because such a person would be less likely to ignore important signals.
In addition, leaders with deep experience in a particular domain may fall into a rut, making judgments out of habit, complacency, or overconfidence. It usually takes an external crisis to expose this failure, for which the lack of lifeboats for the Titanic is the enduring symbol and the 2008 financial crisis the moment of truth for many apparently unassailable titans. The equivalent today are those leaders who have underestimated the speed with which environmental issues would move center stage and require a tangible response.
To improve:
First, assess how well you draw on your own experience to make decisions. Start by going through your important judgment calls to identify what went well and what went badly, including whether you drew on the right experience and whether the analogies you made were appropriate. Record both the wrong and the right. This is tough, and it’s tempting to rewrite history, which is why it can be helpful to share your conclusions with a coach or colleagues, who might take a different view of the same experience. Try also to recruit a smart friend who can be a neutral critic.
Leaders with deep experience in a particular domain may fall into a rut.
Second, especially if you’re a young leader, work to expand your experience. Try to get postings abroad or in key corporate functions such as finance, sales, and manufacturing. Get yourself on an acquisition team for a major deal. And as a CEO, a crucial support you can give high-potential managers is more-varied exposure, so get involved in career planning. That will not just do the young managers a favor; it will help the company and very possibly you, because it will broaden the experience into which you can tap.
Detachment: Identify, and Then Challenge, Biases
As you process information and draw on the diversity of your own and other people’s knowledge, it’s critical that you understand and address your own biases. Although passion about objectives and values is a wonderful leadership quality that can inspire followers to greater efforts, it can also affect how you process information, learn from experience, and select advisers.
The ability to detach, both intellectually and emotionally, is therefore a vital component of good judgment. But it’s a difficult skill to master. As research in behavioral economics, psychology, and decision sciences has shown in recent years, cognitive biases such as anchoring, confirmation, and risk aversion or excessive risk appetite are pervasive influences in the choices people make.
The German utility RWE provides a cautionary example. In a 2017 interview its chief financial officer revealed that the company had invested $10 billion in constructing conventional power-generation facilities over a five-year period, most of which had to be written off. RWE conducted a postmortem to understand why an investment in conventional power technology had been chosen at a time when the energy industry was switching to renewables. It determined that decision makers had displayed status quo and confirmation biases in evaluating the investment context. It also found a number of cases in which hierarchical biases had been in play: Subordinates who doubted the judgment of their bosses had kept quiet rather than disagree with them. Finally, the CFO said, RWE had suffered from “a good dose of action-oriented biases like overconfidence and excessive optimism.”
It is precisely for their ability to resist cognitive biases and preserve detachment in decision-making that we often see CFOs and lawyers rise to the CEO position, especially when an organization is in a period of crisis and people’s jobs are under threat. This quality was widely praised after the International Monetary Fund chose Christine Lagarde as its director following the dramatic exit in 2011 of her predecessor, Dominique Strauss-Kahn, in the wake of a lurid scandal. Although Lagarde was not an economist—unusual for an IMF chief—she had demonstrated her abilities as France’s finance minister despite little political experience. And, undoubtedly, having been a partner in a major international law firm equipped her to approach negotiation with detachment—a critical capability at a time when the global financial system was under severe stress.
To improve:
Understand, clarify, and accept different viewpoints. Encourage people to engage in role-playing and simulations, which forces them to consider agendas other than their own and can provide a safe space for dissent. If employees are encouraged to play the role of a competitor, for example, they can experiment with an idea that they might be reluctant to suggest to the boss.
Leadership development programs are a great forum in which to challenge assumptions by exposing people to colleagues from different cultures and geographies, who come to the discussion with different views.
Finally, people with good judgment make sure they have processes in place that keep them aware of biases. After discovering how much value had been destroyed, RWE established new practices: Major decisions now require that biases be on the table before a discussion and, when necessary, that a devil’s advocate participate. Acknowledge that mistakes will occur—and doubt the judgment of anyone who assumes they won’t.
Options: Question the Solution Set Offered
In making a decision, a leader is often expected to choose between at least two options, formulated and presented by their advocates. But smart leaders don’t accept that those choices are all there is. During the 2008–2009 financial crisis, President Obama pressed Treasury Secretary Timothy Geithner to explain why he wasn’t considering nationalizing the banks. Geithner recalls, “We had one of those really tough conversations. Are you confident this is going to work? Can you reassure me? Why are you confident? What are our choices? I told him that my judgment at the time was that we had no option but to play out the thing we’d set in motion.”
Obama was doing what all good leaders should do when told “We have no other option” or “We have two options and one is really bad” or “We have three options but only one is acceptable.” Other options almost always exist, such as doing nothing, delaying a decision until more information is available, or conducting a time-limited trial or a pilot implementation. Tim Breedon, formerly the CEO of the UK financial services company Legal & General, described it to me as “not being boxed in by the way things are presented.”
When You Have to Move Fast
In most cases, good judgment requires reflection before action. A pause for reflection may well make you less likely to be swept along by anger or fear and more likely to ask for additional evidence, consider …
In hindsight, many bad judgment calls were inevitable simply because important options—and the risk of unintended consequences—were never even considered. This happens for a variety of reasons, including risk aversion on the part of people supplying potential answers. That’s why thoroughly exploring the solution set is key to a leader’s exercise of judgment. It’s not the CEO’s job to come up with all the options. But he or she can ensure that the management team delivers the full range of possibilities, counteracting fears and biases that cause the team to self-edit. When all the options can be debated, the judgment is more likely to be right.
To improve:
Press for clarification on poorly presented information, and challenge your people if you think important facts are missing. Question their weighting of the variables on which their arguments depend. If timing appears to be a key consideration, determine that it’s legitimate. Factor in the risks associated with novel solutions—stress and overconfidence—and look for opportunities to mitigate them through piloting. Use modeling, triangulation, and the opportunities afforded by artificial intelligence. Follow King Solomon (a popular nominee in answer to my question “Who do you think has/had good judgment?”) and dig out people’s stakes in the final decision. A telltale sign is being oversold on a particular outcome. What are the personal consequences to them (and to you) if their solution works or fails? Consult those you trust. If there isn’t anyone, or enough time, try to imagine what someone you trust would do. Get clear about rules and ethical issues, because they will help you filter your choices. Finally, don’t be afraid to consider radical options. Discussing them could make you and others aware of some that are less radical but well worth considering and may encourage other people to speak up.
Delivery: Factor in the Feasibility of Execution
You can make all the right strategic choices but still end up losing out if you don’t exercise judgment in how and by whom those choices will be executed. In 1880 the French diplomat and entrepreneur Ferdinand de Lesseps persuaded investors to support digging a canal in Panama to link the Atlantic and Pacific Oceans. Because de Lesseps had just completed the Suez Canal, investors and politicians—failing to understand that building a canal through sand does not qualify you to build one through jungle—did not give his plans the scrutiny they deserved. His approach proved disastrously unsuitable, and it was left to the U.S. government to complete the canal by taking a very different approach.
When reviewing projects, smart leaders think carefully about the risks of implementation and press for clarification from a project’s advocates. This is as important for small decisions as it is for big ones.
A leader with good judgment anticipates risks after a course has been determined and knows by whom those risks are best managed. That may not be the person who came up with the idea—particularly if the proposer is wedded to a particular vision, as was the case with de Lesseps. More generally, flair, creativity, and imagination aren’t always accompanied by a capability to deliver—which is why small tech firms often struggle to capitalize on their inspiration and are bought out by less-inventive but better-organized giants.
To improve:
In assessing a proposal, make sure that the experience of the people recommending the investment closely matches its context. If they point to their prior work, ask them to explain why that work is relevant to the current situation. Get the advocates to question their assumptions by engaging in “premortem” discussions, in which participants try to surface what might cause a proposal to fail. RWE now does this as part of its project-evaluation process.
CONCLUSION
Leaders need many qualities, but underlying them all is good judgment. Those with ambition but no judgment run out of money. Those with charisma but no judgment lead their followers in the wrong direction. Those with passion but no judgment hurl themselves down the wrong paths. Those with drive but no judgment get up very early to do the wrong things. Sheer luck and factors beyond your control may determine your eventual success, but good judgment will stack the cards in your favor.
Article link: The Elements of Good Judgment (hbr.org)
Editor’s Note: When the members of the class of 2010 entered business school, the economy was strong and their post-graduation ambitions could be limitless. Just a few weeks later, the economy went into a tailspin. They’ve spent the past two years recalibrating their worldview and their definition of success.
The students seem highly aware of how the world has changed (as the sampling of views in this article shows). In the spring, Harvard Business School’s graduating class asked HBS professor Clay Christensen to address them—but not on how to apply his principles and thinking to their post-HBS careers. The students wanted to know how to apply them to their personal lives. He shared with them a set of guidelines that have helped him find meaning in his own life. Though Christensen’s thinking comes from his deep religious faith, we believe that these are strategies anyone can use. And so we asked him to share them with the readers of HBR.
Before I published The Innovator’s Dilemma, I got a call from Andrew Grove, then the chairman of Intel. He had read one of my early papers about disruptive technology, and he asked if I could talk to his direct reports and explain my research and what it implied for Intel. Excited, I flew to Silicon Valley and showed up at the appointed time, only to have Grove say, “Look, stuff has happened. We have only 10 minutes for you. Tell us what your model of disruption means for Intel.” I said that I couldn’t—that I needed a full 30 minutes to explain the model, because only with it as context would any comments about Intel make sense. Ten minutes into my explanation, Grove interrupted: “Look, I’ve got your model. Just tell us what it means for Intel.”
I insisted that I needed 10 more minutes to describe how the process of disruption had worked its way through a very different industry, steel, so that he and his team could understand how disruption worked. I told the story of how Nucor and other steel minimills had begun by attacking the lowest end of the market—steel reinforcing bars, or rebar—and later moved up toward the high end, undercutting the traditional steel mills.
When I finished the minimill story, Grove said, “OK, I get it. What it means for Intel is…,” and then went on to articulate what would become the company’s strategy for going to the bottom of the market to launch the Celeron processor.
I’ve thought about that a million times since. If I had been suckered into telling Andy Grove what he should think about the microprocessor business, I’d have been killed. But instead of telling him what to think, I taught him how to think—and then he reached what I felt was the correct decision on his own.
That experience had a profound influence on me. When people ask what I think they should do, I rarely answer their question directly. Instead, I run the question aloud through one of my models. I’ll describe how the process in the model worked its way through an industry quite different from their own. And then, more often than not, they’ll say, “OK, I get it.” And they’ll answer their own question more insightfully than I could have.
My class at HBS is structured to help my students understand what good management theory is and how it is built. To that backbone I attach different models or theories that help students think about the various dimensions of a general manager’s job in stimulating innovation and growth. In each session we look at one company through the lenses of those theories—using them to explain how the company got into its situation and to examine what managerial actions will yield the needed results.
On the last day of class, I ask my students to turn those theoretical lenses on themselves, to find cogent answers to three questions: First, how can I be sure that I’ll be happy in my career? Second, how can I be sure that my relationships with my spouse and my family become an enduring source of happiness? Third, how can I be sure I’ll stay out of jail? Though the last question sounds lighthearted, it’s not. Two of the 32 people in my Rhodes scholar class spent time in jail. Jeff Skilling of Enron fame was a classmate of mine at HBS. These were good guys—but something in their lives sent them off in the wrong direction.
As the students discuss the answers to these questions, I open my own life to them as a case study of sorts, to illustrate how they can use the theories from our course to guide their life decisions.
One of the theories that gives great insight on the first question—how to be sure we find happiness in our careers—is from Frederick Herzberg, who asserts that the powerful motivator in our lives isn’t money; it’s the opportunity to learn, grow in responsibilities, contribute to others, and be recognized for achievements. I tell the students about a vision of sorts I had while I was running the company I founded before becoming an academic. In my mind’s eye I saw one of my managers leave for work one morning with a relatively strong level of self-esteem. Then I pictured her driving home to her family 10 hours later, feeling unappreciated, frustrated, underutilized, and demeaned. I imagined how profoundly her lowered self-esteem affected the way she interacted with her children. The vision in my mind then fast-forwarded to another day, when she drove home with greater self-esteem—feeling that she had learned a lot, been recognized for achieving valuable things, and played a significant role in the success of some important initiatives. I then imagined how positively that affected her as a spouse and a parent. My conclusion: Management is the most noble of professions if it’s practiced well. No other occupation offers as many ways to help others learn and grow, take responsibility and be recognized for achievement, and contribute to the success of a team. More and more MBA students come to school thinking that a career in business means buying, selling, and investing in companies. That’s unfortunate. Doing deals doesn’t yield the deep rewards that come from building up people.
I want students to leave my classroom knowing that.
Create a Strategy for Your Life
A theory that is helpful in answering the second question—How can I ensure that my relationship with my family proves to be an enduring source of happiness?—concerns how strategy is defined and implemented. Its primary insight is that a company’s strategy is determined by the types of initiatives that management invests in. If a company’s resource allocation process is not managed masterfully, what emerges from it can be very different from what management intended. Because companies’ decision-making systems are designed to steer investments to initiatives that offer the most tangible and immediate returns, companies shortchange investments in initiatives that are crucial to their long-term strategies.
Over the years I’ve watched the fates of my HBS classmates from 1979 unfold; I’ve seen more and more of them come to reunions unhappy, divorced, and alienated from their children. I can guarantee you that not a single one of them graduated with the deliberate strategy of getting divorced and raising children who would become estranged from them. And yet a shocking number of them implemented that strategy. The reason? They didn’t keep the purpose of their lives front and center as they decided how to spend their time, talents, and energy.
It’s quite startling that a significant fraction of the 900 students that HBS draws each year from the world’s best have given little thought to the purpose of their lives. I tell the students that HBS might be one of their last chances to reflect deeply on that question. If they think that they’ll have more time and energy to reflect later, they’re nuts, because life only gets more demanding: You take on a mortgage; you’re working 70 hours a week; you have a spouse and children.
For me, having a clear purpose in my life has been essential. But it was something I had to think long and hard about before I understood it. When I was a Rhodes scholar, I was in a very demanding academic program, trying to cram an extra year’s worth of work into my time at Oxford. I decided to spend an hour every night reading, thinking, and praying about why God put me on this earth. That was a very challenging commitment to keep, because every hour I spent doing that, I wasn’t studying applied econometrics. I was conflicted about whether I could really afford to take that time away from my studies, but I stuck with it—and ultimately figured out the purpose of my life.
Doing deals doesn’t yield the deep rewards that come from building up people.
Had I instead spent that hour each day learning the latest techniques for mastering the problems of autocorrelation in regression analysis, I would have badly misspent my life. I apply the tools of econometrics a few times a year, but I apply my knowledge of the purpose of my life every day. It’s the single most useful thing I’ve ever learned. I promise my students that if they take the time to figure out their life purpose, they’ll look back on it as the most important thing they discovered at HBS. If they don’t figure it out, they will just sail off without a rudder and get buffeted in the very rough seas of life. Clarity about their purpose will trump knowledge of activity-based costing, balanced scorecards, core competence, disruptive innovation, the four Ps, and the five forces.
My purpose grew out of my religious faith, but faith isn’t the only thing that gives people direction. For example, one of my former students decided that his purpose was to bring honesty and economic prosperity to his country and to raise children who were as capably committed to this cause, and to each other, as he was. His purpose is focused on family and others—as mine is.
The choice and successful pursuit of a profession is but one tool for achieving your purpose. But without a purpose, life can become hollow.
Allocate Your Resources
Your decisions about allocating your personal time, energy, and talent ultimately shape your life’s strategy.
I have a bunch of “businesses” that compete for these resources: I’m trying to have a rewarding relationship with my wife, raise great kids, contribute to my community, succeed in my career, contribute to my church, and so on. And I have exactly the same problem that a corporation does. I have a limited amount of time and energy and talent. How much do I devote to each of these pursuits?
Allocation choices can make your life turn out to be very different from what you intended. Sometimes that’s good: Opportunities that you never planned for emerge. But if you misinvest your resources, the outcome can be bad. As I think about my former classmates who inadvertently invested for lives of hollow unhappiness, I can’t help believing that their troubles relate right back to a short-term perspective.
When people who have a high need for achievement—and that includes all Harvard Business School graduates—have an extra half hour of time or an extra ounce of energy, they’ll unconsciously allocate it to activities that yield the most tangible accomplishments. And our careers provide the most concrete evidence that we’re moving forward. You ship a product, finish a design, complete a presentation, close a sale, teach a class, publish a paper, get paid, get promoted. In contrast, investing time and energy in your relationship with your spouse and children typically doesn’t offer that same immediate sense of achievement. Kids misbehave every day. It’s really not until 20 years down the road that you can put your hands on your hips and say, “I raised a good son or a good daughter.” You can neglect your relationship with your spouse, and on a day-to-day basis, it doesn’t seem as if things are deteriorating. People who are driven to excel have this unconscious propensity to underinvest in their families and overinvest in their careers—even though intimate and loving relationships with their families are the most powerful and enduring source of happiness.
If you study the root causes of business disasters, over and over you’ll find this predisposition toward endeavors that offer immediate gratification. If you look at personal lives through that lens, you’ll see the same stunning and sobering pattern: people allocating fewer and fewer resources to the things they would have once said mattered most.
Create a Culture
There’s an important model in our class called the Tools of Cooperation, which basically says that being a visionary manager isn’t all it’s cracked up to be. It’s one thing to see into the foggy future with acuity and chart the course corrections that the company must make. But it’s quite another to persuade employees who might not see the changes ahead to line up and work cooperatively to take the company in that new direction. Knowing what tools to wield to elicit the needed cooperation is a critical managerial skill.
The theory arrays these tools along two dimensions—the extent to which members of the organization agree on what they want from their participation in the enterprise, and the extent to which they agree on what actions will produce the desired results. When there is little agreement on both axes, you have to use “power tools”—coercion, threats, punishment, and so on—to secure cooperation. Many companies start in this quadrant, which is why the founding executive team must play such an assertive role in defining what must be done and how. If employees’ ways of working together to address those tasks succeed over and over, consensus begins to form. MIT’s Edgar Schein has described this process as the mechanism by which a culture is built. Ultimately, people don’t even think about whether their way of doing things yields success. They embrace priorities and follow procedures by instinct and assumption rather than by explicit decision—which means that they’ve created a culture. Culture, in compelling but unspoken ways, dictates the proven, acceptable methods by which members of the group address recurrent problems. And culture defines the priority given to different types of problems. It can be a powerful management tool.
In using this model to address the question, How can I be sure that my family becomes an enduring source of happiness?, my students quickly see that the simplest tools that parents can wield to elicit cooperation from children are power tools. But there comes a point during the teen years when power tools no longer work. At that point parents start wishing that they had begun working with their children at a very young age to build a culture at home in which children instinctively behave respectfully toward one another, obey their parents, and choose the right thing to do. Families have cultures, just as companies do. Those cultures can be built consciously or evolve inadvertently.
If you want your kids to have strong self-esteem and confidence that they can solve hard problems, those qualities won’t magically materialize in high school. You have to design them into your family’s culture—and you have to think about this very early on. Like employees, children build self-esteem by doing things that are hard and learning what works.
Avoid the “Marginal Costs” Mistake
We’re taught in finance and economics that in evaluating alternative investments, we should ignore sunk and fixed costs, and instead base decisions on the marginal costs and marginal revenues that each alternative entails. We learn in our course that this doctrine biases companies to leverage what they have put in place to succeed in the past, instead of guiding them to create the capabilities they’ll need in the future. If we knew the future would be exactly the same as the past, that approach would be fine. But if the future’s different—and it almost always is—then it’s the wrong thing to do.
This theory addresses the third question I discuss with my students—how to live a life of integrity (stay out of jail). Unconsciously, we often employ the marginal cost doctrine in our personal lives when we choose between right and wrong. A voice in our head says, “Look, I know that as a general rule, most people shouldn’t do this. But in this particular extenuating circumstance, just this once, it’s OK.” The marginal cost of doing something wrong “just this once” always seems alluringly low. It suckers you in, and you don’t ever look at where that path ultimately is headed and at the full costs that the choice entails. Justification for infidelity and dishonesty in all their manifestations lies in the marginal cost economics of “just this once.”
I’d like to share a story about how I came to understand the potential damage of “just this once” in my own life. I played on the Oxford University varsity basketball team. We worked our tails off and finished the season undefeated. The guys on the team were the best friends I’ve ever had in my life. We got to the British equivalent of the NCAA tournament—and made it to the final four. It turned out the championship game was scheduled to be played on a Sunday. I had made a personal commitment to God at age 16 that I would never play ball on Sunday. So I went to the coach and explained my problem. He was incredulous. My teammates were, too, because I was the starting center. Every one of the guys on the team came to me and said, “You’ve got to play. Can’t you break the rule just this one time?”
I’m a deeply religious man, so I went away and prayed about what I should do. I got a very clear feeling that I shouldn’t break my commitment—so I didn’t play in the championship game.
In many ways that was a small decision—involving one of several thousand Sundays in my life. In theory, surely I could have crossed over the line just that one time and then not done it again. But looking back on it, resisting the temptation whose logic was “In this extenuating circumstance, just this once, it’s OK” has proven to be one of the most important decisions of my life. Why? My life has been one unending stream of extenuating circumstances. Had I crossed the line that one time, I would have done it over and over in the years that followed.
The lesson I learned from this is that it’s easier to hold to your principles 100% of the time than it is to hold to them 98% of the time. If you give in to “just this once,” based on a marginal cost analysis, as some of my former classmates have done, you’ll regret where you end up. You’ve got to define for yourself what you stand for and draw the line in a safe place.
Remember the Importance of Humility
I got this insight when I was asked to teach a class on humility at Harvard College. I asked all the students to describe the most humble person they knew. One characteristic of these humble people stood out: They had a high level of self-esteem. They knew who they were, and they felt good about who they were. We also decided that humility was defined not by self-deprecating behavior or attitudes but by the esteem with which you regard others. Good behavior flows naturally from that kind of humility. For example, you would never steal from someone, because you respect that person too much. You’d never lie to someone, either.
It’s crucial to take a sense of humility into the world. By the time you make it to a top graduate school, almost all your learning has come from people who are smarter and more experienced than you: parents, teachers, bosses. But once you’ve finished at Harvard Business School or any other top academic institution, the vast majority of people you’ll interact with on a day-to-day basis may not be smarter than you. And if your attitude is that only smarter people have something to teach you, your learning opportunities will be very limited. But if you have a humble eagerness to learn something from everybody, your learning opportunities will be unlimited. Generally, you can be humble only if you feel really good about yourself—and you want to help those around you feel really good about themselves, too. When we see people acting in an abusive, arrogant, or demeaning manner toward others, their behavior almost always is a symptom of their lack of self-esteem. They need to put someone else down to feel good about themselves.
Choose the Right Yardstick
This past year I was diagnosed with cancer and faced the possibility that my life would end sooner than I’d planned. Thankfully, it now looks as if I’ll be spared. But the experience has given me important insight into my life.
I have a pretty clear idea of how my ideas have generated enormous revenue for companies that have used my research; I know I’ve had a substantial impact. But as I’ve confronted this disease, it’s been interesting to see how unimportant that impact is to me now. I’ve concluded that the metric by which God will assess my life isn’t dollars but the individual people whose lives I’ve touched.
I think that’s the way it will work for us all. Don’t worry about the level of individual prominence you have achieved; worry about the individuals you have helped become better people. This is my final recommendation: Think about the metric by which your life will be judged, and make a resolution to live every day so that in the end, your life will be judged a success.
Article link: How Will You Measure Your Life? (ampproject.org)
by Aneel Chima and Ron Gutman
October 29, 2020
To say that 2020 is a year of disruption and change is to understate the obvious. Our daily lives, from educating our kids, managing our health, and working from home, to simple social rituals like dinner with friends, underwent rapid multi-dimensional change. Nascent trends — virtualization of the workspace, online learning, virtual health, and e-commerce — accelerated exponentially. Changes anticipated to take years occurred in months and, in some cases, weeks and even days. Understandably, leaders have struggled mightily to address these overlapping changes simultaneously, dealing with economic, health, and logistical crises that have unfolded at top speed.
Much as we might like to think of 2020 as an anomaly, it may not be. Conditions for accelerating change have been building for years. Advancements in information technology, automation, human interconnectivity, Artificial Intelligence, and the network effects among them, created a new reality where change is much more rapid, continual, and ubiquitous. Covid-19 and its derivatives laid bare a “new normal” of change, marked by three dimensions:
- It’s perpetual — occurring all the time in an ongoing way.
- It’s pervasive — unfolding in multiple areas of life at once.
- It’s exponential — accelerating at an increasingly rapid rate.
This three-dimensional (3-D) change is defining our emerging future and, as a consequence, effective leadership will be defined by the ability to navigate this new reality.
The problem is, our models for leadership weren’t built for this kind of 3-D change. Human minds evolved for thinking linearly and locally in the face of challenge, not exponentially and systemically. Noted futurist Ray Kurzweil asserted, “The future is widely misunderstood. Our forebears expected it to be pretty much like their present, which had been pretty much like their past.” But, projecting our pasts onto our futures exposes a fundamental error: Linear thinking can never catch-up and adapt to the perpetual, pervasive, and exponential change occurring around us — it’s simply too fast and too complex.
We need a new form of leadership, better equipped to navigate this unprecedented kind of change. For this purpose, we gathered, under the Stanford University umbrella, world-class luminaries — leaders who generate impact and change at a global scale — for conversations on the future of leadership and change-making. What emerged was a new vision of leadership, which we call Sapient Leadership. A Sapient Leader is characterized by being wise, sagacious, and discerning in navigating change while also being humane in the face of change that can often feel alien. This kind of leadership emphasizes — counterintuitively — an anti-heroic leader. Sapient Leaders exhibit authenticity, humility, and vulnerability, inspiring the necessary trust and psychological safety that drives shared learning and intelligence, resulting in enhanced collective performance and leading to a better future for all.
Limits of Linear Thinking in an Era of 3-D Change
In a world that’s relatively stable and mostly predictable, where change is incremental, punctuated by relatively few bursts of large change — what’s often called “disruption” — a model of leadership that relies on linear, local thinking can be useful. Much of the leadership literature focuses on the qualities, skills, abilities of the leader as an individual, and the linear and local maps they use to navigate the world. However, 3-D change presents a “high seas” environment where the leader navigates multiple domains — the waves and ever-evolving weather — of change simultaneously. In this environment, linear and local thinking can never adapt fast enough, leaving us increasingly ill-equipped to manage our rapidly changing business and work environments, our physical and mental health and well-being, and the major trends that shape our societies and cultures.
Change, by its nature, leaves people and organizations feeling confused, vulnerable, and fractured at a time when resilience, cohesion, and collaboration are necessary to perform at the highest levels. An emerging body of literature points to psychological safety, shared purpose, and distributed cognition as powerful drivers of leadership, team, and organizational performance, particularly in rapidly changing environments. The days of “leader as hero” — the solo, individualistic leader who inspires certainty in a deterministic way forward — are over. This evolution in how we think about change and leadership has only accelerated in the past year.
Fortuitously, our spring course at Stanford University, LEAD 111 “Luminaries: Life Lessons from Leaders and Change-makers” became a study of how top tier leaders embodied this emerging approach to leadership. Finalized one week before the Covid-19 pandemic struck the west coast, our original plan was to create a new framework of leadership suitable for a time of disruption, accelerating change, and a highly polarized political and social environment, and we designed the course to engage leaders and change-makers in conversations across sectors, generations, and the political spectrum. We wanted to know how change-oriented leaders operate. As the pandemic unfolded, however, we expanded the course to create a new model of leadership. And recognizing that these questions were of immediate and broad interest, we invited more leaders within and beyond the Stanford community to weigh in on how they were navigating this 3-D change.
We engaged leaders across sectors to analyze — in real time — how they adapted: captains of industry, such as Doug McMillon, president and CEO of Walmart and chairman of the Business Roundtable; innovators in health care such as Toby Cosgrove, former CEO of the Cleveland Clinic, heart surgeon, and White House advisor; global social change-makers such as Halla Tómasdóttir, CEO of The B Team, investor, co-founder of Reykjavik University, and runner-up in Iceland’s 2016 presidential elections; leading-edge technologists and innovators such as Bret Taylor, president and COO of Salesforce, co-creator of Google Maps and the “Like” button, and board member of Twitter.
The essential question we had was this: If leadership is significantly defined by the ability to skillfully navigate 3-D change, what type of leadership is most effective for our emerging future, one defined by perpetual, pervasive, and exponential change? The answers that emerged formed the basis for Sapient Leadership.
How to Practice Sapient Leadership
The four pillars of Sapient Leadership emerged out of the discussions with our luminaries as they were navigating 3-D change in real-time — each leader, in some capacity, articulated a version of these ideas. Leader humility, authenticity, and openness instills trust and psychological safety. In turn, trust and psychological safety empower individuals and teams to perform at their highest capabilities. Additionally, continuously learning teams are essential for keeping pace with and effectively navigating 3-D change. Finally, shared purpose and common values enhance focus, cohesion, and resilience in the midst of 3-D change.
1. Leader humility, authenticity, and openness instills trust and psychological safety.
In times of uncertainty, leaders often posture themselves, maximizing perception of power and control. In contrast, Halla Tómasdóttir modeled authenticity and humility when she reflected on her challenges as a candidate during the Icelandic presidential election. She, along with many of our luminaries, openly questioned the traditional paradigm of a leader as an individualistic hero. Instead, she highlighted the need to build trust through openness, saying, “what this crisis has shown us is that the leadership style of ‘I know it all’ is not a good leadership style for this moment or any other challenge we are going to continue to face and need to deal with collectively, collaboratively, with compassion, and with care.”
In a world of 3-D change, leaders need to continuously evolve themselves in order for their organization to evolve and grow. Rather than bending the organization to the will of the leader, a leader must be willing to instead exhibit humility and flexibility and change according to what the organization and circumstances require. Tómasdóttir exemplified this notion in her personal philosophy: “leadership is not given to the few — it’s inside of all of us, and life is all about unleashing that leadership.” This leadership style, which engenders trust and psychological safety within teams and organizations, animates much of her work with the B Team members that she’s leading — Sir Richard Branson, Arianna Huffington, Ajay Banga, Mary Robinson, and Marc Benioff, among others.
Our other luminaries echoed Tómasdóttir’s message about Sapient Leadership in the context of 3-D change. Doug McMillon said: “I don’t run Walmart, I help lead Walmart” asserting that leadership of this sort needs to go beyond words. Leaders, he said, “have to live it. It has to be authentic. It has to be habitual.”
2. Trust and psychological safety empower individuals and teams.
3-D change amplifies our innate and evolved human tendencies to skew towards threat perception, anxiety, and divisiveness when experiencing stress and encountering ambiguity. As such, psychological safety is even more important during these times change. Individuals and the teams they comprise thrive in environments where trust and psychological safety are present. In a recent extensive study at Google, code-named Project Aristotle — for the maxim frequently attributed to him, “The whole is greater than the sum of the parts” — researchers found that the most important factor associated with the highest performing teams was psychological safety. When team members feel safe to be vulnerable in front of one and to take risks, they perform at their best.
A consistent theme running throughout conversations with all of our luminaries was the essential nature of empowering teams and individuals to perform at their highest capabilities, especially now. “Change is not a solo sport,” said Bret Taylor, president and COO of Salesforce. “All great change has been done by great teams, great communities, and great networks.” When recalling times of rapid change throughout his career — from the creation of Google Maps to inventing the “like” button to scaling rapidly worldwide during the early days of Facebook — Bret asserted the importance of leadership that motivates strong relationships, fluid communication, and a foundation of trust to driving exceptional team performance.
3. Continuously learning teams enable effective navigation of 3-D change.
In a world where change is perpetual, pervasive, and exponential, Sapient Leaders, their teams, and their organizations must continually learn, update mental-maps, deploy new tools, and course-correct based on the best ideas and practices. “If you want to make a change in something you have to get into it deep,” said Toby Cosgrove, describing his openness to learning transformative ideas from anywhere he could. When he was the CEO of the Cleveland Clinic he regularly immersed himself in contexts where he could learn a better way. “If I heard somebody was doing something someplace in the world, I would pick up my pencil and paper and I would go and watch them do it,” he said. “I traveled someplace, learned something, and tried to bring it back and incorporate it.” What he was doing as a leader was both modeling leadership as a process of continual learning so others would replicate in their way, as well as disseminating what he learned throughout the organization in order to improve on existing processes and innovate new ones.
In a world of 3-D change, no one person or organization can master all knowledge across all domains, no single person or organization can master enough skills in breadth, depth, or pace, to keep up. Instead, learning must be inspired by leadership, reinforced by culture, occur across a variety of domains, coordinated through the whole and shared openly and actionably to create the broader picture. The analogy here is to mosaic vision, or the compound eye, where thousands of specific receptor units, oriented in different directions, work in coordination to create a composite perspective with a very wide angle of view, continually updating in real time as the organism moves through time and space. Without data and input to synthesize into understanding and action, a team or organization will be perpetually impoverished. To keep pace with 3-D change, Sapient Leaders need to enhance the breadth, depth, and pace of learning in their organizations to meet the extent and velocity of change.
4. Shared purpose and values enhance focus, cohesion, and resilience during 3-D change.
Professor Bill Damon, our esteemed colleague at Stanford University and one of the world’s leading purpose researchers, defines purpose as a stable intention to accomplish something that is both personally meaningful and serves the world larger than the self. Purpose, necessarily informed by our values and arising from a sense of personal meaning, unites our inner world with our actions in the world around us in a unique and powerful way in service of a vision larger than ourselves.
In times of 3-D change, which by its nature amplifies uncertainty and ambiguity, shared purpose and values increase organizational focus, enhance team cohesion, and amplify personal and collective resilience. They can also powerfully mobilize large numbers of people to solve complex problems together.
Doug McMillon, CEO of Walmart and chairman of the Business Roundtable, recounted Walmart’s Five Guiding Principles, which provided the organization focus, resilience, and a basis for cohesive action during the early challenging stages of the pandemic.
- Start with the people: “Support our associates financial health, physical health, and emotional health and well-being. They are on the front line.”
- Focus on the fundamentals and first principles: “Serve our customers — we had to keep the food supply chain going to avoid chaos.”
- Make sure our own home is in order: “Managing the business through the crisis — making sure inventory is under control, making sure we have cash flow, etc.”
- Keep building for the future, not for the past: “Continue assertively into online e-commerce, grocery delivery, leverage what’s already been put into play that customers want.”
- We’re all in this together: “What can we do to help other people through this crisis that does good for this company and society?”
Doug recounted how these principles guided Walmart’s actions during the early turbulence of the Covid-19 pandemic. “We received a call from the White House with a request to open drive-through testing stations throughout the nation in Walmart parking lots,” he recalled. “Although we didn’t know exactly how to do it and didn’t have a way to charge for it, Walmart’s response was fully committed, rapid, at scale, and across distributed geographies. Walmart’s ethos during this time: ‘Don’t worry about the short-term financials. Go do what’s right and it will all eventually work out.’”
The shared purpose and values articulated in Walmart’s Five Guiding Principles allowed collective action that was focused, cohesive, and resilient by many people across multiple geographies in the early times of 3-D change. Focus and cohesion allowed rapid learning of new skills, it allowed decisiveness during uncertainty, and it promoted working together towards a shared goal bigger than any individual or the company. Further, resilience allowed the courage to try something new and execute quickly, without giving up, in the face of ongoing change and challenge.
The Future of Leadership
Along with the myriad challenges it brought, the singular realization of 2020 is that 3-D change is the new normal. Navigating perpetual, pervasive, and exponential change is the quintessential test of effective leadership in this era. Leaders, teams, and organizations that don’t skillfully navigate change will fail. Mastering this new reality requires fundamental enhancements to our collective capabilities. Sapient Leadership enables the creation of perpetual, pervasive, and exponential capacity building necessary for handling 3-D change effectively. In addition, our recent conversations with Sapient Leaders have uncovered new ways in which exponential and transformative technologies can further enhance and amplify human capabilities. This topic is the basis for a future article we are preparing.
The key of Sapient Leadership is that it fits into the long history of the evolution of our species. Sapient, in its definition, refers to the nature of humans — it is in our nature to adapt or risk perishing. The challenge of 3-D change is that it amplifies the pressures on leaders, teams, and organizations to evolve and adapt faster, or become irrelevant. Change that used to take place over years and decades is now taking place in weeks or days. We, as a species, have never confronted change of this magnitude or at this pace. Sapient Leadership is a framework that enables accelerated adaptation in a wise and humane way. It builds into its structure the imperative for leaders, teams, and organizations to continuously evolve in order to overcome the challenges of 3-D change. Sapient Leaders and their successful organizations change with change itself.
Article link: https://hbr.org/2020/10/what-it-takes-to-lead-through-an-era-of-exponential-change?
Aneel Chima, PhD, is the Director of Health and Human Performance and of the Stanford Flourishing Project. Outside of academia he is co-founder and managing partner of AT THE CORE, a consulting boutique specializing in facilitating transformative change through enhancing the emotional, social, and neurophysiological drivers of team and leadership thriving.
Ron Gutman is an inventor (HOPES Health Operating System, Dr. AI), a serial technology/healthcare entrepreneur (his companies have served more than 500 million users worldwide), an investor, an author (of the popular TED Book and talk on the Powers of Smiling, and other publications on innovation, technology, and leadership), and a Stanford lecturer.
by Donald Martin, Jr. and Andrew Moore
October 28, 2020
Artificial Intelligence (AI) has become one of the biggest drivers of technological change, impacting industries and creating entirely new opportunities. From an engineering standpoint, AI is just a more advanced form of data engineering. Most good AI projects function more like muddy pickup trucks than spotless race cars — they are a workhorse technology that humbly makes a production line 5% safer or movie recommendations a little more on point. However, more so than many other technologies, it is very, very easy for a well-intentioned AI practitioner to inadvertently do harm when they set out to do good. AI has the power to amplify unfair biases, making innate biases exponentially more harmful.
As Google AI practitioners, we understand that how AI technology is developed and used will have a significant impact on society for many years to come. As such, it’s crucial to formulate best practices. This starts with the responsible development of the technology and mitigating any potential unfair bias which may exist, both of which require technologists to look more than one step ahead: not “Will this delivery automation save 15% on the delivery cost?” but “How will this change affect the cities where we operate and the people — at-risk populations in particular — who live there?”
This has to be done the old-fashioned way: by human data scientists understanding the process that generates the variables that end up in datasets and models. What’s more, that understanding can only be achieved in partnership with the people represented by and impacted by these variables — community members and stakeholders, such as experts who understand the complex systems that AI will ultimately interact with.
Faulty causal assumptions can lead to unfair bias.
How do we actually implement this goal of building fairness into these new technologies — especially when they often work in ways we might not expect? As a first step, computer scientists need to do more to understand the contexts in which their technologies are being developed and deployed.
Despite our advances in measuring and detecting unfair bias, causation mistakes can still lead to harmful outcomes for marginalized communities. What’s a causation mistake? Take, for example, the observation during the Middle Ages that sick people attracted fewer lice, which led to an assumption that lice were good for you. In actual fact, lice don’t like living on people with fevers. Causation mistakes like this, where a correlation is wrongly thought to signal a cause and effect, can be extremely harmful in high-stakes domains such as health care and criminal justice. AI system developers — who usually do not have social science backgrounds — typically do not understand the underlying societal systems and structures that generate the problems their systems are intended to solve. This lack of understanding can lead to designs based on oversimplified, incorrect causal assumptions that exclude critical societal factors and can lead to unintended and harmful outcomes.
For instance, the researchers who discovered that a medical algorithm widely used in the U.S. health care was racially biased against Black patients identified that the root cause was the mistaken causal assumption, made by the algorithm designers, that people with more complex health needs will have spent more money on health care. This assumption ignores critical factors — such as lack of trust in the health care system and lack of access to affordable health care — that tend to decrease spending on health care by Black patients regardless of the complexity of their health care needs.
Researchers make this kind of causation/correlation mistake all the time. But things are worse for a deep learning computer, which searches billions of possible correlations in order to find the most accurate way to predict data, and thus has billions of opportunities to make causal mistakes. Complicating the issue further, it is very hard, even with modern tools, such as Shapely analysis, to understand why such a mistake was made — a human data scientist sitting in a lab with their supercomputer can never deduce from the data itself what the causation mistakes may be. This is why, among scientists, it is never acceptable to claim to have found a causal relationship in nature just by passively looking at data. You must formulate the hypothesis and then conduct an experiment in order to tease out the causation.
Addressing these causal mistakes requires taking a step back. Computer scientists need to do more to understand and account for the underlying societal contexts in which these technologies are developed and deployed.
Here at Google, we started to lay the foundations for what this approach might look like. In a recent paper co-written by DeepMind, Google AI, and our Trust & Safety team, we argue that considering these societal contexts requires embracing the fact that they are dynamic, complex, non-linear, adaptive systems governed by hard-to-see feedback mechanisms. We all participate in these systems, but no individual person or algorithm can see them in their entirety or fully understand them. So, to account for these inevitable blindspots and innovate responsibly, technologists must collaborate with stakeholders — representatives from sociology, behavioral science, and the humanities, as well as from vulnerable communities — to form a shared hypothesis of how they work. This process should happen at the earliest stages of product development — even before product design starts — and be done in full partnership with communities most vulnerable to algorithmic bias.
This participatory approach to understanding complex social systems — called community-based system dynamics (CBSD) — requires building new networks to bring these stakeholders into the process. CBSD is grounded in systems thinking and incorporates rigorous qualitative and quantitative methods for collaboratively describing and understanding complex problem domains, and we’ve identified it as a promising practice in our research. Building the capacity to partner with communities in fair and ethical ways that provide benefits to all participants needs to be a top priority. It won’t be easy. But the societal insights gained from a deep understanding of the problems that matter most to the most vulnerable in society can lead to technological innovations that are safer and more beneficial for everyone.
Shifting from a mindset of “building because we can” to “building what we should.”
When communities are underrepresented in the product development design process, they are underserved by the products that result. Right now, we’re designing what the future of AI will look like. Will it be inclusive and equitable? Or will it reflect the most unfair and unjust elements of our society? The more just option isn’t a foregone conclusion — we have to work towards it. Our vision for the technology is one where a full range of perspectives, experiences and structural inequities are accounted for. We work to seek out and include these perspectives in a range of ways, including human rights diligence processes, research sprints, direct input from vulnerable communities and organizations focused on inclusion, diversity, and equity such as WiML (Women in ML) and Latinx in AI; many of these organizations are also co-founded and co-led by Googler researchers, such as Black in AI and Queer in AI.
If we, as a field, want this technology to live up to our ideals, then we need to change how we think about what we’re building — to shift to our mindset from “building because we can” to “building what we should.” This means fundamentally shifting our focus to understanding deep problems and working to ethically partner and collaborate with marginalized communities. This will give us a more reliable view of both the data that fuels our algorithms and the problems we seek to solve. This deeper understanding could allow organizations in every sector to unlock new possibilities of what they have to offer while being inclusive, equitable and socially beneficial.
Article link: https://hbr.org/2020/10/ai-engineers-need-to-think-beyond-engineering?
Donald Martin, Jr. is Sr. Staff Technical Program Manager and Social Impact Technology Strategist at Google.
Andrew Moore is Head of Google Cloud AI & Industry Solutions.
The Future of Jobs report maps the jobs and skills of the future, tracking the pace of change. It aims to shed light on the pandemic-related disruptions in 2020, contextualized within a longer history of economic cycles and the expected outlook for technology adoption, jobs and skills in the next five years.
Online Report link: https://www.weforum.org/reports/the-future-of-jobs-report-2020#report-nav
by Eric Siegel
October 23, 2020
Machine learning can ascertain a lot about you — including some of your most sensitive information. For instance, it can predict your sexual orientation, whether you’re pregnant, whether you’ll quit your job, and whether you’re likely to die soon. Researchers can predict race based on Facebook likes, and officials in China use facial recognition to identify and track the Uighurs, a minority ethnic group.
Now, do the machines actually “know” these things about you, or are they only making informed guesses? And, if they’re making an inference about you, just the same as any human you know might do, is there really anything wrong with them being so astute?
Let’s look at a few cases:
In the U.S., the story of Target predicting who’s pregnant is probably the most famous example of an algorithm making sensitive inferences about people. In 2012, a New York Times story about how companies can leverage their data included an anecdote about a father learning that his teenage daughter was pregnant due to Target sending her coupons for baby items in an apparent act of premonition. Although the story about the teenager may be apocryphal — even if it did happen, it would most likely have been coincidence, not predictive analytics that was responsible for the coupons, according to Target’s process detailed by The New York Times story — there is a real risk to privacy in light of this predictive project. After all, if a company’s marketing department predicts who’s pregnant, they’ve ascertained medically sensitive, unvolunteered data that only healthcare staff are normally trained to appropriately handle and safeguard.
INSIGHT CENTER
- AI and Equality Designing systems that are fair for all.
Mismanaged access to this kind of information can have huge implications on someone’s life. As one concerned citizen posted online, imagine that a pregnant woman’s “job is shaky, and [her] state disability isn’t set up right yet…to have disclosure could risk the retail cost of a birth (approximately $20,000), disability payments during time off (approximately $10,000 to $50,000), and even her job.”
This isn’t a case of mishandling, leaking, or stealing data. Rather, it is the generation of new data — the indirect discovery of unvolunteered truths about people. Organizations can predict these powerful insights from existing innocuous data, as if creating them out of thin air.
So are we ironically facing a downside when predictive models perform too well? We know there’s a cost when models predict incorrectly, but is there also a cost when they predict correctly?
Even if the model isn’t highly accurate, per se, it may still be confident in its predictions for a certain group of pregnant individuals. Let’s say that 2% of the female customers between age 18 and 40 are pregnant. If the model identifies customers, say, three times more likely than average to be pregnant, only 6% of those identified will actually be pregnant. That’s a lift of three. But if you look at a much smaller, focused group, say the top 0.1% likely to be pregnant, you may have a much higher lift of, say, 46, which would make women in that group 92% likely to be pregnant. In that case, the system would be capable of revealing those women as very likely to be pregnant.
The same concept applies when predicting sexual orientation, race, health status, location, and your intentions to leave your job. Even if a model isn’t highly accurate in general, it can still reveal with high confidence — for a limited group — things like sexual orientation, race, or ethnicity. This is because, typically, there is a small portion of the population for whom it is easier to predict. Now, it may only be able to predict confidently for a relatively small group, but even just the top 0.1% of a population of a million would mean 1,000 individuals have been confidently identified.
It’s easy to think of reasons why people wouldn’t want someone to know these things. As of 2013, Hewlett-Packard was predictively scoring its more than 300,000 workers with the probability of whether they’d quit their job — HP called this the Flight Risk score, and it was delivered to managers. If you’re planning to leave, your boss would probably be the last person you’d want to find out before it’s official.
As another example, facial recognition technologies can serve as a way to track location, decreasing the fundamental freedom to move about without disclosure, since, for example, publicly-positioned security cameras can identify people at specific times and places. I certainly don’t sweepingly condemn face recognition, but know that CEO’s at both Microsoft and Google have come down on it for this reason.
In yet another example, a consulting firm was modeling employee loss for an HR department, and noticed that they could actually model employee deaths, since that’s one way you lose an employee. The HR folks responded with, “Don’t show us!” They didn’t want the liability of potentially knowing which employees were at risk of dying soon.
Research has shown that predictive models can also discern other personal attributes — such as race and ethnicity — based on, for example, Facebook likes. A concern here is the ways in which marketers may be making use of these sorts of predictions. As Harvard professor of government and technology Latanya Sweeney put it, “At the end of the day, online advertising is about discrimination. You don’t want mothers with newborns getting ads for fishing rods, and you don’t want fishermen getting ads for diapers. The question is when does that discrimination cross the line from targeting customers to negatively impacting an entire group of people?” Indeed, a study by Sweeney showed that Google searches for “black-sounding” names were 25% more likely to show an ad suggesting that the person had an arrest record, even if the advertiser had nobody with that name in their database of arrest records.
“If you make a technology that can classify people by an ethnicity, someone will use it to repress that ethnicity,” says Clare Garvie, senior associate at the Center on Privacy and Technology at Georgetown Law.
Which brings us to China, where the government applies facial recognition to identify and track members of the Uighurs, an ethnic group systematically oppressed by the government. This is the first known case of a government using machine learning to profile by ethnicity. This flagging of individuals by ethnic group is designed specifically to be used as a factor in discriminatory decisions — that is, decisions based at least in part on a protected class. In this case, members of this group, once identified, will be treated or considered differently on the basis of their ethnicity. One Chinese start-up valued at more than $1 billion said its software could recognize “sensitive groups of people.” Its website said, “If originally one Uighur lives in a neighborhood, and within 20 days six Uighurs appear, it immediately sends alarms” to law enforcement.
Implementing the differential treatment of an ethic group based on predictive technology takes the risks to a whole new level. Jonathan Frankle, a deep learning researcher at MIT, warns that this potential extends beyond China. “I don’t think it’s overblown to treat this as an existential threat to democracy. Once a country adopts a model in this heavy authoritarian mode, it’s using data to enforce thought and rules in a much more deep-seated fashion… To that extent, this is an urgent crisis we are slowly sleepwalking our way into.”
It’s a real challenge to draw the line as to which predictive objectives pursued with machine learning are unethical, let alone which should be legislated against, if any. But, at the very least, it’s important to stay vigilant for when machine learning serves to empower a preexisting unethical practice, and also for when it generates data that must be handled with care.
Article link: https://hbr.org/2020/10/when-does-predictive-technology-become-unethical?
healthcarereimagined 