isaac

human@bracketbot
ml-man napoleon-stan prev-racing🏎️-physics

[isaac @isaacbau]

DIFFUSION MATH FROM ABSOLUTE SCRATCH 🔥

Diffusion math is infamous for being very complex but in this deep dive I try to convince u that it's not

We go thru Statistical Mechanics, Stochastic Processes, DDPMs, -LogLikelihood, etc all derived by hand w my own software!

[Aleph @alephneuro]

We recently obtained the highest-resolution 3D images of the human brain ever taken from outside the skull. This is the first look.

Introducing Aleph, a research lab building brain interfaces for the telepathic future. (1/n)

[isaac @isaacbau]

@oldestasian can I buy

[kelly @kellyhongsn]

we trained a 20b model specialized in agentic search

along with our tech report, we release our model weights and the full data generation pipeline used for training

[isaac @isaacbau]

@Takeraparterer Yeah, 100%. This is honestly a toy model, the real world model from comma is on the original tweet I quoted. Def recommend giving it a read

[isaac @isaacbau]

I trained (overfit ~1hr) their small indoors agent with the official dataset using the same nanogpt + mlp architecture (confirmed by quickly reverse eng the published ONNX). It's really cool!

It's like a tiny FSD,
takes frame latents, wheel speeds, current action (wasd keys) and returns the next frame logits (next 'image', speeds, action) and a plan (following 16 actions)

[isaac @isaacbau]

When I read comma's paper on training a driving AI with a world model I was talking with some friends about how this could totally be domain swapped (outdoor sidewalk naviagtion, indoors, drones, etc)

Just watched the comma con videos and it’s really cool to see that complete autonomy is actually the goal

[isaac @isaacbau]

@Takeraparterer Yeah, you can totally control it and predict frames! You just need better data, I tried with this model and it outputs noise very quick

[isaac @isaacbau]

@esther_confused Hahahh “neutral” just means is going forward, you have basically every combination of awsd. You could train a hostile mode though (bracket bot), that’d be fun!

[isaac @isaacbau]

@yassineyousfi_ I didn't know that was the goal, I think it's the right path! I watched the comma con talks, it was really cool to see the entire stack explained. You guys are doing amazing work, sincerely

[isaac @isaacbau]

When I read comma's paper on training a driving AI with a world model I was talking with some friends about how this could totally be domain swapped (outdoor sidewalk naviagtion, indoors, drones, etc)

Just watched the comma con videos and it’s really cool to see that complete autonomy is actually the goal

[comma @comma_ai]

We wrote a paper about training driving agents end-to-end in a worldmodel simulator.

Blog post: https://t.co/WvEG5Cekc2

[isaac @isaacbau]

Finally, I generally really like comma as a company. No "AI will replace drivers in 3 months", no multi M rounds; revenue and a product you can buy. It's funny but from the comma con videos you can literally learn their entire stack end to end, it's just good engineering

[isaac @isaacbau]

When I read comma's paper on training a driving AI with a world model I was talking with some friends about how this could totally be domain swapped (outdoor sidewalk naviagtion, indoors, drones, etc)

Just watched the comma con videos and it’s really cool to see that complete autonomy is actually the goal

[isaac @isaacbau]

The magic from the world model (from comma's paper) is that it's decoupled from the actual low level implementation (tho representative of the training data). Meaning that for example comma can train both a Toyota Prius and a Ram 1500 with the same world model, just a different 'vehicle model'

Doing this for other domains is ofc quite hard (driving is arguably the easiest) but what's exciting is that it successfully launched in the real world! Today you can buy a comma 4 and go self drive multiple cars all trained with the same model simulator (generally, I think they have different training distributions depending on countries and so on)

I haven't explored the 'high policy' models enough but I bet getting a drone to follow the same motion-target of the world model (under car constraints: camera at the same height, same velocity, turning speed, ...) is totally doable: you'll get a drone that behaves like a car

Eventually I think this will happen:
https://t.co/IZyDYuG6XA

[isaac @isaacbau]

We should have enough data from the Miami delivery robots to build a sidewalk navigation "world model" (avoid crashes, be careful with strollers and chihuahuas, don't fall off the curb, ...)

Which can then be mapped into a robo-agnostic motion target (trajectory), then each platform gets its own "Vehicle Model" + low level controller to realize that motion

With a lot of work (camera viewpoints, low lv locomotion, human reaction, ...) we could get a humanoid and a robodog to learn to walk around a city, similar to how they manage to drive both a Ram 1500 and a Toyota Prius

[isaac @isaacbau]

All credit to @comma_ai, I literally took their model, dataset and bodyjim script to recreated it

https://t.co/HTZCKBPt9p

[isaac @isaacbau]

We can get a tiny FSD for bracket bot @sincethestudy, since it works with a modified nanogpt it's a great follow up to the Karpathy lecture

[isaac @isaacbau]

Of course I originally intended to recreate the exact same model and compare them but I looked at the data and it didn't seem to be nearly enough

[isaac @isaacbau]

We should have enough data from the Miami delivery robots to build a sidewalk navigation "world model" (avoid crashes, be careful with strollers and chihuahuas, don't fall off the curb, ...)

Which can then be mapped into a robo-agnostic motion target (trajectory), then each platform gets its own "Vehicle Model" + low level controller to realize that motion

With a lot of work (camera viewpoints, low lv locomotion, human reaction, ...) we could get a humanoid and a robodog to learn to walk around a city, similar to how they manage to drive both a Ram 1500 and a Toyota Prius

[isaac @isaacbau]

@chesterzelaya although in Spanish he does say it was made by kids and he seems to be in a high school science festival, he prob just said Nvidia because that’s the only company he knows

[isaac @isaacbau]

@chesterzelaya not the ultra sonic sensor 💀

[isaac @isaacbau]

@esther_confused @dcxStep Hahah sounds good, I’m down

[isaac @isaacbau]

this is honestly so fun

[isaac @isaacbau]

~2y ago I re imagined user interfaces with diffusion models [🖼️]: segmentation, resizing as generation, text+selection inpainting, collaborative creation, interconnected canvases, pencil as a primitive

At the time I taped together a bunch of SD models, different controlnets and a rough agent like LLM for prompts. I spent the past weekend refactoring to use @GoogleDeepMind's nano-[🍌🍌] and it's incredibly cool!!

Technical breakdown of my favorite features: