Tesla's AI head details Autopilot coding challenges

The company claims its fleet of cars represents the largest deployment of robots in the world.

Tesla's director of artificial intelligence, Andrej Karpathy, has detailed some of the coding challenges that may have stalled Autopilot development.

An expert in machine vision and deep learning, Karpathy joined Tesla around a year ago to lead the Autopilot development team through a transition from classical programming to AI algorithms.

Karpathy recently described his work at the Train AI 2018 conference, captured on video.

The executive says the Autopilot team is mostly focused on managing large data sets and labeling features. The 'Software 2.0' approach uses a basic architecture, such as a neural net, and essentially optimizes its own algorithms with help from Tesla's coders. A significant amount of time is spent training the system to properly deal with rare encounters such as unusual lane markings, signals or vehicle types.

"For example, Tesla famously tries to save money all over the place, so instead of having a dedicated sensor for sensing whether or not it's raining, Elon looked at this and he's like, 'well you see raindrops so [machine] vision can do it,'" Karpathy recalls. "And now it's my problem."

Deploying a machine-vision solution for automatic wipers required extensive training to handle windshields that are dirty or covered in ice, or obscured by glare from the sun. The first iterations misinterpreted tunnels and smudges illuminated by sun glare.

"This ended up being pretty difficult," Karpathy says. "I thought this was straightforward. It's not straightforward."

Elon Musk recently announced that Autopilot is nearing its next big update, version 9.0, that will begin to roll out the first features promised in the "full self-driving" upgrade package. The update is expected to be ready by August.