Copyright © 2023 by Kleiner Powell International (KPI)

All rights reserved. No part of this publication may be reproduced, distributed, or transmitted in any form or by any means, including photocopying, recording, or other electronic or mechanical methods, without the prior written permission of the publisher, except in the case of brief quotations embodied in critical reviews and certain other noncommercial uses permitted by copyright law.

Imagine you have to make a life-or-death choice in a matter of seconds. You’re responsible for a self-driving car with sudden brake failure. It is careening forward with two possible paths. You have to decide, under pressure, who lives and who dies in a succession of scenarios: Three homeless people, or a doctor and an executive? Children or elderly people? Humans or pets? Jay-walkers or law-abiding street-crossers? Pregnant or nonpregnant women? Hit a barrier and kill the passenger, or hit a pedestrian in the crosswalk?

What’s the best choice?

More than 2.3 million people from 233 countries have volunteered to answer these questions since the MIT Media Lab first posted the Moral Machine experiment in 2016. It is the largest online experiment in moral psychology ever created—an experience that invites people to choose the right ethical path for a powerful vehicle enabled by artificial intelligence.

The Moral Machine is built on the trolley problem, the well-known thought experiment introduced by philosopher Philippa Foot in 1967. In all of its many variations, some people must live, others must die, and there is limited time to choose. Media Lab faculty member Iyad Rahwan chose that problem as a way to test people’s attitudes about self-driving cars. Rahwan and his fellow researchers wanted to explore the psychological roadblocks that might keep people from using these vehicles or other AI systems. To realize their potential value in reducing traffic congestion, augmenting safety, and cutting greenhouse gas emissions, the public must accept, purchase, and use these vehicles. This experiment would illuminate the barriers to acceptance.

The MIT researchers would have been happy to attract 500 participants: enough to make the results statistically significant. But the thought experiment struck a nerve. The preeminent journal Science and the New York Times published articles on the Moral Machine and included links to the site.3 On the day the Science article appeared, two MIT graduate students behind the simulation, Edmond Awad and Sohan Dsouza, had to fly from Boston to Chicago for a conference. By the time their two-hour flight landed, Rahwan was already calling them frantically. About 100,000 people had visited the website at the same time and the unexpected traffic crashed the server. Awad and Dsouza had to relaunch the site during the taxi ride to their hotel, using a smartphone as a Wi-Fi hotspot.

The experiment continued to go viral, off and on, during the next few years. Popular gaming commentators like PewDiePie and jacksepticeye posted YouTube videos of themselves playing this moral dilemma game, with 5 million and 1.5 million views, respectively. People discussed it on the front page of Reddit. One reason for the experiment’s growing popularity was undoubtedly the ongoing news coverage of fatal accidents with self-driving cars. A Tesla Model S killed a passenger in February 2016 when it collided with a tractor-trailer truck in Williston, Florida. An Uber autonomous vehicle (AV) struck a woman walking her bicycle across a road in Tempe, Arizona in March 2018. There have been more such fatal crashes—11 just in the United States between May and September 2022.

The Moral Machine results show that as artificial intelligence and automated systems become part of everyday life, they are forcing people to think about risk and responsibility more generally. In the experiment, millions of people expressed deeply held opinions about who should be sacrificed: children or adults, women or men, rich or poor? We rarely ask these questions of human drivers, but people want to think them through when AI is at the wheel.

As the authors of this book, we decided to do the experiment ourselves, responding to 13 horrific scenarios. As a former coder working on amphibious cars, Juliette took it very seriously, as if the responses really did mean life or death. Art felt more detached. To him, it was like playing a 1980s-era computer game with its simple graphics—but there was an unexpected gut punch. The site asked three questions at the end: Do you believe that your decisions on Moral Machine will be used to program actual self-driving cars? (Probably not, he thought. He doubted that the automakers would listen.) To what extent do you feel you can trust machines in the future? (After doing the experiment, he trusted them less.) To what extent do you fear that machines will become out of control? (The answer seemed much more complicated to him now.)

 Taking the Trolley Problem to Scale

“Never in the history of humanity have we allowed a machine to autonomously decide who should live and who should die, in a fraction of a second, outside of real-time supervision,” wrote Awad, Rahwan, and colleagues in their 2018 Nature article looking back at the experiment. “We are going to cross that bridge any time now, and it will not happen in a distant theater of military operations; it will happen in that most mundane aspect of our lives: everyday transportation. Before we allow our cars to make ethical decisions, we need to have a global conversation to express our preferences to the companies that will design moral algorithms, and to the policymakers that will regulate them. The Moral Machine was deployed to initiate such a conversation, and millions of people weighed in from around the world.”

The results of that global conversation were sobering. Among all of those respondents, making 40 million decisions in 10 languages, there were only three universal preferences. Nearly everyone wanted to spare more lives rather than fewer lives. Almost all respondents favored humans over pets. They also preferred children over adults. That was it, except for a weak but still-evident general preference for saving the lives of pregnant women.

Beyond that, there is no consensus and there are strong contradictions. Respondents in many Asian countries say the elderly should be spared before the young. Residents of French-heritage countries and Latin America say the opposite. People in countries like the United States, with high gross domestic product (GDP) per capita, prefer to save law-abiding pedestrians over jaywalkers. The opposite is true for people in places with lower GDP. A high Gini coefficient, which indicates a major gap between rich and poor, correlates with wanting to save wealthy pedestrians over unhoused people. The opposite is true for countries with a high social safety net. There are strong disagreements about saving groups that look like families with two adults and children, versus saving doctors and business leaders. The Moral Machine is still online, and it still taps a nerve.

People are attracted to it, in part, because they realize that self-driving vehicles—and other automated systems—are increasingly in widespread use. They are powerful, accessible, easy to use, and appear to give us what we want.

The Moral Machine, however, shows that people—especially those in different countries, cultures, and contexts—don’t agree on what we want. We all control for different priorities. Our values often don’t align. How, then, can we expect AI to know what priorities to control for and whose interests to look after—not just in life-and-death situations, but everywhere?