How Self-Driving Cars Get Help From Humans Hundreds of Miles Away

This is a command center in Foster City, Calif., operated by Zoox, a self-driving car company owned by Amazon. Like other robot taxis, the company’s self-driving cars sometimes struggle to drive themselves, so they get help from human technicians sitting in a room about 500 miles away.

Inside companies like Zoox, this kind of human assistance is taken for granted. Outside such companies, few realize that autonomous vehicles are not completely autonomous.

For years, companies avoided mentioning the remote assistance provided to their self-driving cars. The illusion of complete autonomy helped to draw attention to their technology and encourage venture capitalists to invest the billions of dollars needed to build increasingly effective autonomous vehicles.

“There is a ‘Wizard of Oz’ flavor to this,” said Gary Marcus, an entrepreneur and a professor emeritus of psychology and neural science at New York University who specializes in A.I. and autonomous machines.

If a Zoox robot taxi encounters a construction zone it has not seen before, for instance, a technician in the command center will receive an alert — a short message in a small, colored window on the side of the technician’s computer screen. Then, using the computer mouse to draw a line across the screen, the technician can send the car a new route to follow around the construction zone.

“We are not in full control of the vehicle,” said Marc Jennings, 35, a Zoox remote technician. “We are providing guidance.”

As companies like Waymo, owned by Google’s parent company, Alphabet, and Cruise, owned by General Motors, have begun to remove drivers from their cars, scrutiny of their operations has increased. After a series of high-profile accidents, they have started to acknowledge that the cars require human assistance.

While Zoox and other companies have started to reveal how humans intervene to help driverless cars, none of the companies have disclosed how many remote-assistance technicians they employ or how much it all costs. Zoox’s command center holds about three dozen people who oversee what appears to be a small number of driverless cars — two in Foster City and several more in Las Vegas — as well as a fleet of about 200 test cars that each still have a driver behind the steering wheel.

When regulators last year ordered Cruise to shut down its fleet of 400 robot taxis in San Francisco after a woman was dragged under one of its driverless vehicles, the cars were supported by about 1.5 workers per vehicle, including remote assistance staff, according to two people familiar with the company’s operations. Those workers intervened to assist the vehicles every two and a half to five miles, the people said.

The expenses associated with remote assistance are one reason robot taxis will struggle to replace traditional ride-hailing fleets operated by Uber and Lyft. Though companies like Zoox are beginning to replace drivers, they still pay people to work behind the scenes.

“It may be cheaper just to pay a driver to sit in the car and drive it,” said Thomas W. Malone, a professor at the Massachusetts Institute of Technology Center for Collective Intelligence.

Waymo and Cruise declined to comment for this story.

While those companies use traditional cars retrofitted for self-driving, Zoox is testing a new kind of vehicle in Foster City, just south of San Francisco, and in Las Vegas, not far from the Strip.

After testing the vehicles with Zoox employees, their family members and friends, the company plans to make the service available to the public this year. But this robot taxi, like all others, will lean on human assistance.

In Foster City, the company operates what it calls a “fusion center,” where employees monitor robot taxis operating both locally and in Las Vegas, several hundred miles away. From their computer screens, these workers can track live feeds of the road from cameras installed on the cars as well as a detailed overhead view of each car and its surroundings, which is stitched together using data streaming from an array of sensors on the vehicle.

The workers can provide verbal assistance to riders via speakers and microphones inside the cars. They can also assist a car if it encounters a scenario it cannot handle on its own.

Self-driving cars can reliably handle familiar situations, like an ordinary right turn or a lane change. They are designed to brake on their own when a pedestrian runs in front of them. But they are less adept in unusual or unexpected situations. That’s why they still need the humans in the fusion center.

But even though self-driving cars have remote assistance, they still make mistakes on the road.

After reviewing the incident, Zoox indicated that its car had struggled to recognize the fire trucks because they were yellow, not red. “We continue to test and refine our driving software,” Whitney Jencks, a company spokeswoman, said.

Zoox will also continue to lean on human assistance.

“We think that computers should be able to replicate humans and replace humans in all ways,” Dr. Malone, the M.I.T. professor, said. “It is possible that might happen. But it hasn’t yet.”