The Alignment Problem by Brian Christian

“In the real world, it doesn't matter if something is a cyclist or a pedestrian because you want to avoid them either way. The second factor is our own ability to mathematically define what a system such as an autonomous car should do when it encounters a problem that requires a response. Does the data fundamentally represent reality? And it turns out, no – there’s this key concept called jaywalking that was not present.” “That’s a useful way for thinking about how real-world systems can go wrong,” says Christian, “It’s a function of two things – the first is the quality of the training data. A human being pushing a bicycle did not fit any of those categories and did not behave in a way that would be expected of any of them. On top of this, the system was trained to rigidly segment objects in the road into a number of categories – such as other cars, trucks, cyclists, and pedestrians. It was totally unprepared to deal with a person being in the middle of the road, where they should not have been. When the National Transportation Safety Board investigated what had caused the collision between the Uber test vehicle and Herzberg, who was pushing a bicycle across a road, they found that the AI controlling the car had no awareness of the concept of jaywalking. This was the death of Elaine Herzberg in Arizona, US, in 2018. One incident that Christian uses to illustrate how this misalignment can play out in the real world is the first recorded killing of a pedestrian in a collision involving an autonomous car.

This is the sort of thing that’s waiting for us if we develop these systems that are sufficiently general and powerful … I think we are at a moment in the real world where we are filling the world with these brooms, and this is going to become a real issue.” “Wiener argued that this isn’t the stuff of fairytales. The story serves as a good example of the dangers of these qualities when they aren't accompanied by human values like common sense and judgment. Most people reading this will probably be familiar with the Disney cartoon in which Mickey Mouse attempts to save himself the effort of doing his master’s chores by using a magic spell to imbue a broom with intelligence and autonomy. “There is this fundamental problem … that has a history that goes back to the 1960s, and MIT cyberneticist Norbert Wiener, who likened these systems to the story of the Sorcerer’s Apprentice,” Christian tells me. Christian’s work is particularly concerned with the encroachment of computer-aided decision-making into fields such as healthcare, criminal justice, and lending, where there is clearly potential for them to cause problems that could end up affecting people’s lives in very real ways.

I recently sat down with Christian to discuss some of the topics.

The Alignment Problem by Brian Christian