This is how Tesla's FSD v12 learns.
As humans with driver's licenses, we know what to do at the intersection pictured below.
We've been taught that a red sign with the word "STOP" means that once we arrive at the sign, we need to stop and check for cross traffic. We've also been taught that the white lines on either side of the car are "barriers" that tell us where the car should be as we approach the perpendicular line.
We've also learned about cross walks, right of way rules, speed bumps, rain, snow, cyclists, not to smash onto oncoming traffic, navigating through crap (or missing) lane markings, etc.
Some of this we've learned by reading a book and then practicing said things on the road, and other things we've learned by driving around and experiencing the environments on our own.
With Tesla's latest v12 FSD update, Tesla's vehicles learn in a similar way, but it's actually much broader than you think.
Tesla isn't telling the AI that trains the FSD system what a stop sign is. It isn't telling it what white lines are. What pedestrian sidewalks are. What other cars look like. What red brake lights mean. etc.
Instead, for the specific example below, Tesla is feeding the AI a ton of video depicting what proper driving looks like at a stop sign, with drivers coming to a stop while slowing down at a reasonable speed while centered between the lines.
With this footage, the AI says to itself "OK - one thing I'm noticing is that every time the car comes to a stop, the surrounding areas have these "STOP" sign things on either side every single time, and the car is always centered between white lines on the road when it approaches these signs".
The AI then "writes code" for the car to behave correctly at every stop sign it encounters.
This is how Tesla's system learns.
This means that for Tesla to reach self-driving under any condition, it needs to collect all driving conditions that a human encounters with many examples of each. It needs to see stop signs that are just on one side. Stop signs that are partially covered by a tree. Stop signs that have been vandalized. Etc. Etc. Etc.
Luckily, Tesla is able to do this because it has a fleet of ~4 million cars driving around the world today, and this fleet is growing exponentially. This means that every condition a Tesla finds itself in, that footage can be used by the AI system to learn the proper behavior based on how Tesla drivers navigate that scenario.
This means that the Tesla @elonmusk was driving on FSD v12 learned from all other Tesla drivers in the world driving their own cars.
With this data, the AI system was able to generate commands for the steering wheel, accelerator, and pedal to navigate around its own environment as good as a human could, and possibly significantly better than a human as Tesla collects and processes more data.
Imagine having a car with this AI system that never gets tired, never makes a mistake, is always paying attention, is constantly monitoring every angle around the car, etc.
This is what Tesla has achieved with v12.
As we finish out the decade, Tesla has plans to reach an annual goal of 20 million car sold per year by 2030. All these cars will be outfitted with the camera systems used to collect the data that the AI system uses to train itself.
Tesla is also investing billions of dollars in training compute to dramatically increase how much data the AI can process at once, which will allow the company to make improvements quicker and be able to process every conceivable scenario that a driver could face on the road.
If Tesla's AI successfully learns how to drive under any condition, this will mark one of the greatest technological achievements of our time.
The age of the self-driving car is finally here.
Tesla's FSD v12 is a REALLY big deal.
Last night @elonmusk livestreamed a ~45-minute video of a Model S driving itself around using Tesla's latest self-driving software, FSD v12.
Self-driving is something the company has been trying to solve since Tesla Autopilot was released in 2015. Since then, Tesla has made steady progress improving the code to handle all kinds of road situations, has added cameras around the car, and removed sensors, all the while rewriting the code multiple times to solve for things the car couldn’t handle.
And even though Tesla cars can drive themselves in many situations, the biggest challenge with self-driving cars is the thousands (or millions) of situations drivers face on a daily basis that are completely unexpected or difficult to solve for, like other drivers acting irrationally, inclement weather, debris on the road, weird (or lack of) lane markings, etc.
Up to this point, Tesla and other companies have to spend a large amount of time running through simulated scenarios to generate code that would teach the system how to handle these situations. This code is oftentimes written by a human and needs to account for every variation of something happening on the road. And even then, there are thousands (or millions) of situations that a simulation won’t come with, since real life is so damn complicated and complex.
The approach many have taken to try and solve this problem is by overfitting their self-driving cars with a ton of different sensors like LIDAR, radar, ultrasonics, cameras, and other sensors in addition to generating High-Definition maps of the areas where the cars are meant to be driven in. They’ve done this in hopes that there’s a combination of sensors and map data that would allow them to solve for the most unexpected situations. Here’s a picture of a GM Cruise vehicle that uses this approach.
However, yesterday’s video has demonstrated a breakthrough in how self-driving cars can operate.
Instead of using many sensors and hardware on their cars to process the world, Tesla is using 8 cameras and a computer that’s specifically built to process video data. That’s 9 total parts vs everyone else’s 30, 40, 50+ parts to process the world.
Using only vision to process the world and not needing things like LIDAR, radar, and other sensors to interpret the physical objects around the car is impressive enough, but HOW the car learns to do this is what the real breakthrough is.
With FSD v12, Tesla takes the video data that is collected by its fleet of ~4 million cars and runs it through an AI that is built using Neural-Nets (like ChatGPT but for the real world), and then the AI figures out what the car should do based on what it sees.
What’s important to highlight here is that Tesla has done 0 work in telling the car how it should interpret the world. That means that the AI doesn’t explicitly know what a lane is, what a traffic light is, what a stop sign is, what a cone is, what a pothole is, what rain is, etc.
What Tesla does is it shows the AI a metric-ton of video of a car driving around using the 8 cameras that are outfitted around the car, and the AI learns how to do the same. The more video Tesla feeds it, the better the system gets. The more unique situations that are collected from the fleet of Teslas, the better the system gets at accounting for those situations.
This ��AI code” will then be beamed to every Tesla in the world, and the on-board computer will be able to process its surroundings without the need to connect to Tesla’s AI server. It’s no different than you or I getting into a car and driving it. Instead of our 2 eyes collecting video around us and using our brain to process the info, the 8 camera system will collect the video and Tesla’s on-board computer will process it using the “AI code”.
In other words, Tesla has moved away from humans figuring out how to write code that tells the car what to do, and instead feeds video to an AI, and the AI figures out the best way to account for every situation.
The only thing Tesla needs moving forward is more video and more compute power (chips) to process videos.
That’s it.
This is profound. Everything that is captured on video with the 8 camera system is something the AI will be able to figure out how to navigate through. Snow. Potholes. Deer. Cyclists. Aliens. You name it. And this “code” that is generated by the AI will get better and better as Tesla’s compute capabilities grow with their purchases of NVIDIA’s H100 chips and the build-out of their in-house DOJO compute system. Not to mention the growing fleet of Teslas that will capture more and more video data around the world. Every Model S, 3, X, and Y that is sold today is a video-capturing robot that feeds the AI. And every Cybertruck and Compact Car will be the same.
And believe it or not, it gets even crazier.
Now that Tesla has come up with a real-world data collection and processing system with its cameras and on-board computer, this same system can be used on other physical products that can learn how to move around its surroundings.
This is where Tesla’s Optimus Bot comes into play. Tesla will be able to use its 8 camera system on the Tesla Bot to collect video of its surroundings, beam it back to the AI, the AI figures out how to best do that thing that it’s collecting video for, and then beam back the “AI code” for the Bot to process with its on-board compute.
We are not far away from a world where a humanoid robot watches you do the dishes, sends back the data to the mothership to process it, and then the next morning the Bot has learned how to do the dishes - not only using the video it gathered from you doing it, but from every other person in the world washing dishes.
Do this process with literally anything.
ChatGPT showed the masses what the potential of AI can be. NVIDIA showed the masses just how much demand there is for hardware that is used by AI systems.
And now, Tesla just showed the masses what AI means for real-world physical applications.
The world has changed - again.
40,000 followers on @X! 🎉 We're thankful for everyone who helped us reach this milestone!
To show our thanks to the worldwide community, we're giving away four (4) Lifetime Premium memberships to our vehicle companion service.
To participate, repost this post.
(Details below)
@wholemars You mean 11.4.7? That should be going to wide release within a week.
Version 12, which an almost total rewrite (~99% AI) is being tested by our QA drivers.
Too much new code to go to the public yet, but it is mind-blowing imo.
Voisiko nämä turve-, hiili- ja öljyihmiset mennä perustamaan oman sähköyhtiön joka tuottaa ja myy turve-, hiili- tai öljysähköä. Mitä ne täällä Twitterissä meuhkaa. Ei kun perustamaan jos se on hyvä bisnes. Tekemään tekemään!
After wide release of FSD Beta 10.69.2, price of FSD will rise to $15k in North America on September 5th.
Current price will be honored for orders made before Sept 5th, but delivered later.