Elon Musk claims the carmaker will someday be the most valuable AI company in the world thanks to the reams of video data collected by its cars. Does all that data really give the company a competitive advantage?
Tesla shouldn’t be seen as an electric car manufacturer anymore. It’s an AI company—if you believe CEO Elon Musk. His confidence is tied to a unique dataset: petabytes of video harvested from the company’s cars as Tesla customers log millions of driving miles worldwide.
In theory, all that real-world data is exactly what Tesla needs to train its cars to operate without any human assistance, a goal that’s core to Musk’s vision for the future of Tesla. But there’s a problem: That data isn’t necessarily as helpful as Musk claims. Some of it isn’t useful at all.
Building AI that can drive a car as well as a human is a challenge dramatically different than building a natural language processing chatbot like ChatGPT, which was trained on billions of words scraped from the internet. While the goal with ChatGPT and competing systems like xAI’s Grok is to use pattern recognition to provide reliable information and answers to questions, the results often fall short in embarrassing ways. But if the AI controlling a vehicle screws up, people can die.
“Any 17-year-old can learn to drive in about 20 hours of practice”
Yann Lecunn, Meta’s chief AI scientist
Driving a car is a very different proposition with many more variables — driving conditions, weather, construction, changing traffic patterns, how other vehicles are moving. Successfully wrangling all those variables, and being ready to react to unexpected developments, is at the crux of autonomous driving AI. Training it on endless videos of people driving down highways doesn’t do much to help the AI learn how to handle what it most needs to: the edge cases that cause collisions or other dangerous scenarios.
“It can make you drive really smoothly in normal situations, but when things get a little weird, you’ve got nothing,” said a computer scientist and executive at an autonomous tech company, who asked not to be named because he didn’t want to openly criticize Tesla. “And you’ve learned only bad habits. Nine out of 10 people roll through a stop sign. If all you’re doing is learning what people do, you’ll roll through stop signs.”
That’s also why Tesla’s robotaxi rivals use laser lidar, for 3D images, and radar to detect solid objects in a vehicle’s path–to get richer, more detailed images of the world. And while it’s possible to rely on camera data alone, you need “the best camera systems to really handle it,” Drago Anguelov, Waymo’s head of research, said at Google’s developer conference a few years ago. “That’s a very big bet that you can achieve it. It’s very, very risky, and it’s not necessary.”
Yann LeCun, Meta’s chief AI scientist and a professor of computer science at New York University, also isn’t convinced that Tesla’s data gives it a competitive advantage.
“The impact of data is generally overstated: as you get more data, performance improves, but there are diminishing returns,” he said. “A doubling of data volume brings marginal improvements that are still far from human reliability.” Even with massive amounts of data, no company has developed so-called Level-5 autonomy, the point at which a vehicle can drive itself in all the circumstances a human can.
“Yet any 17-year-old can learn to drive in about 20 hours of practice,” said LeCun. “This tells you that current AI architectures are missing something big in their ability to understand the world and to learn from limited amounts of data or trials.”
“If somebody doesn’t believe Tesla is going to solve autonomy, I think they should not be an investor in the company”
Elon Musk
None of this has stopped Tesla bulls from betting on Musk’s AI vision, even as EV sales — and the company’s stock — continue to crater and protestors demonstrate outside Tesla stores over Musk’s role as budget cutter-in-chief for President Trump’s controversial DOGE initiative. Some equity analysts remain convinced Musk knows something others don’t. “We believe autonomous is worth $1 trillion alone and this thesis will be proven out over the coming years,” Dan Ives of Wedbush Securities told Forbes.
Musk and Tesla didn’t respond to requests for comment. Neither did Ashok Elluswamy, the head of Tesla’s AV program.
‘Garbage In, Garbage Out’
Musk has pinned Tesla’s future to artificial intelligence applications, including humanoid robots and smart factories, abandoning a long-held goal for the company to sell 20 million EVs a year by 2030. One reason is likely intensifying EV competition, especially from companies like China’s BYD. Another is that if Tesla can solve automated driving, it’s cheaper and more lucrative to deploy hundreds of thousands of fare-generating electric robotaxis worldwide than adding more plants to build and sell millions of personal vehicles.
It’s so core to the company that Musk doesn’t want doubters buying the stock. “If somebody doesn’t believe Tesla is going to solve autonomy, I think they should not be an investor in the company,” Musk said in a 2024 earnings call.
In January, he announced that Tesla’s vast data reservoir is being tapped at its new “Cortex” data center in Austin to improve its Full Self-Driving software–which despite the name, requires human supervision at all times. That AI-enabled feature, along with Tesla’s original Autopilot system, certainly needs improvement: over the years FSD and Autopilot have been linked to 52 fatal accidents worldwide.
Huge amounts of camera data is helpful, but it doesn’t instantly make Tesla an AI market leader. “Having access to unique data feeds is certainly some kind of advantage,” said computer scientist Alex Ratner, CEO of Snorkel AI, which makes software to help automate labeling of raw data.
“There are no guarantees all the edge cases that cars need to learn will be in the data at sufficient numbers to generate learned behavior”
Missy Cummings
“But the old saying, ‘garbage in, garbage out,’ applies just as much as ever here,” Ratner, who has a family member working for Waymo, told Forbes. “In the curation of data, what video feed is coming from a good driver versus a bad driver? That’s nontrivial and super important because these models … learn from the most common thing they see.”
Companies that have spent years refining AI to drive cars and trucks safely, including Waymo, Zoox, Aurora and Waabi, have focused on creating good data that focuses on enough edge cases, mastering extreme or dangerous types of road situations using advanced computer simulation and structured real-world tests. Tesla’s data isn’t necessarily representative of those much rarer events.
“There are no guarantees all the edge cases that cars need to learn will be in the data at sufficient numbers to generate learned behavior,” said AI expert Missy Cummings, a George Mason University professor who’s advised federal and California regulators on autonomous vehicle tech. That makes it hard to solve problems all AV developers have faced, like unexpected “phantom” braking events when the AI misunderstands road circumstances it detects as hazards.
Even identifying the most meaningful bits of driving data for training purposes from endless miles of on-road video is extremely hard to do, said an AV researcher and computer scientist briefed on Tesla’s approach who asked not to be named.
“So you have bazillions of miles of data,” the person said. “How are they making sure to pick all the stuff that matters to train on?”
It’s hard to say since Tesla hasn’t been open about its process. Nor is it an active member of the AI research community, where engineers from all the largest tech companies regularly publish papers detailing their latest research.
“Tesla has pretty much zero presence on the AI R&D circuit–conferences, publications, etc.,” said LeCun. “It’s like they don’t exist.”
Spotty Track Record
Tesla’s accomplishments in autonomous driving have fallen short of Musk’s targets over and over again. His 2016 promise that a Tesla would be able to drive across the U.S. without human intervention still hasn’t happened. His 2019 goal of having a million robotaxis in operation by 2020? Not even close.
“Elon has been wildly over-claiming and under-delivering on ‘full self-driving’ consistently for almost a decade,” said LeCun. “It was obvious for many of us that all those claims were either lies or signs of self-delusion. I don’t understand how anyone could still believe anything he claims on the topic.”
But that hasn’t stopped Musk from making more and more promises — or his most ardent fans from continuing to invest in him. The prototypes he’s shown off so far, though, seem pretty far off. Last October, he hosted a staged demonstration of the company’s CyberCab that ferried event participants around Universal Studios’ Los Angeles film lot. But even on a closed studio lot, Tesla technicians could be seen monitoring, if not remotely controlling, the low-speed prototypes. Likewise, versions of Tesla’s “Optimus” humanoid robot on hand to serve drinks to attendees were controlled remotely.
“I think long-term, Optimus has the potential to [generate] north of $10 trillion in revenue–like it’s really bananas,” Musk said on the results call.
The real test comes in June with Tesla’s robotaxi pilot service in Austin — assuming it debuts on time. “We’ll be scrutinizing it very carefully to make sure there’s not something we missed,” Musk said on the results call. “It will be autonomous ride-hailing for money in Austin in June, and then as shortly as possible in other cities in America.” (Tesla has also applied for a permit to run a taxi-like service in California, with vehicles it owns and operates, though not for robotaxis, the California Public Utilities Commission told Forbes.)
Mastering autonomy “won’t come from Tesla,” LeCun said. “They simply don’t have a research organization with a long leash and enough talented scientists to do that.”
Musk has some catching up to do with Alphabet’s Waymo, far and away the country’s leader in robotaxis. It operates its automated ride service in Phoenix, San Francisco, Los Angeles and Austin, as of last week. Last month it said the company is booking more than 200,000 paid trips per week with a fleet of only about 700 vehicles. Later this year, it expands to Atlanta and plans to launch in Miami next year. Alphabet hasn’t disclosed Waymo’s revenue, though Forbes estimates it was over $100 million from 4 million booked rides in 2024.
Waymo has had minor accidents, though so far its robotic fleet hasn’t been linked to fatal ones. Meanwhile, Tesla owners routinely upload video of their vehicles doing dangerous maneuvers while operating in FSD mode, like nearly colliding with other vehicles on a highway exit ramp in New Jersey or running red lights in China.
Ultimately, Musk’s AI pitch for Tesla comes down to how financially valuable it will be for his company, seeing trillions of dollars of new revenue in the coming years.
Meta’s LeCun thinks a “paradigm shift” is needed to enable machines to learn how the world works from video, which could take another decade of research.
“My hunch is we won’t get to human-like, full autonomy–and practical humanoid robots–until we figure out how to get AI systems to learn how the world works like animals and humans do.”
Look back on the week that was with hand-picked articles from Australia and around the world. Sign up to the Forbes Australia newsletter here or become a member here.
