Gabriel Garrett

10,000x AI Engineer. Chief Singularity Officer

member of the permanent overclass

Writing on AGI Futurism / Mindfulness / Panpsychism @

[Gabriel Garrett @GabGarrett]

The secret to success: they WON'T tell you:
> sit down
> become very still
> try to do absolutely nothing for as long as possible
> keep doing that until nirvana is reached
> escape the endless cycle of death and rebirth

[hrishik @saibojnal]

kay neurons are interpretable enough that models can modify their own weights toward some desiderata.

blogpost incoming

[Shuibai Zhang @ShuibaiZ69721]

Your MacBook can now host a coding agent at 300+ tps!

I gave DeepSeek V4.1 Flash a job: write a specialized inference engine for LiquidAI's LFM2.5-24B-A2B on Apple's M5 Max. For $21 of API, it built Molten: 300+ tps, about 1.5x faster decode than mlx-lm and llama.cpp.

Why specialized? General frameworks like mlx-lm and llama.cpp have to run any model on any hardware. A specialized engine serves one model on one chip at batch size 1, so there is far more room to optimize.

Molten is native Rust + Metal: one ~6 MB binary, no Python, PyTorch or MLX. It comes with a chat CLI and an OpenAI-compatible server with streaming and tool calls, so you can point your own coding agent at it. It needs an M5-series Mac with 24 GB+ (sorry, M1-M4 friends).

The numbers (M5 Max, 128 GB)
• 305 tps at 256 tokens of context, vs 190 for mlx-lm and 199 for llama.cpp
• Still 248 tps at 16K, where coding agents actually live (mlx-lm: 165)
• Faster time to first token: 1.76 s vs 2.17 s for a 16K prompt
• Same weights and math as mlx-lm

How it was built
The agent (DeepSeek V4.1 Flash in the Pi harness) ran in a recursive self-improvement (RSI) loop. Each session starts from the best engine so far, plus the feedback from every earlier round. A fixed evaluator rebuilds each candidate from source, checks its tokens against mlx-lm, times and profiles it. Only correct and faster engines survive.

• Hour 3: first correct engine, 0.56x mlx-lm
• Day 2.5: passes mlx-lm
• Day 5.7: 1.28x end to end, 1.53x decode

78 agent sessions, about $21 (1.36B tokens, 98.6% of them cache hits).

What the RSI loop found useful for speeding up:

• Fuse chains of small ops into single kernels: norm + router + top-4 expert pick, expert FFN + combine, Q/K/V projections + KV-cache write, Q/K norm + RoPE + attention, LM head + argmax.
• Keep the decode loop on the GPU. Tokens never round-trip to the CPU, and the GPU stays 99% busy.
• Hard-code every shape, so the 4-bit kernels stream weights at ~75% of the memory bus.
• Target the hardware: prefill runs on the M5 GPU's Neural Accelerators through Metal 4 tensor ops, with MoE routing and expert GEMMs fused and kept on the GPU.

(It's not Rust, by the way. llama.cpp is native C++ too. It's the specialization.)

TBH, I don't fully understand all of it yet lol. The agents wrote this in under 6 days, and it's going to take me longer than that to read it. As the human researcher, I'm now officially the slowest-improving component of my own RSI loop🤣, so I'll spend the next few weeks learning from my agents.

Where I think this goes: local inference will matter more and more, and a specialized engine can beat a general one. As RSI matures, the cost of building one keeps falling toward zero. I think every model, every chip, and maybe every workload will eventually get its own inference engine.

The idea is inspired by @perplexity_ai 's Lily. Thanks to @liquidai for the great model, and Apple's MLX, whose kernels Molten adapted from.

Repo here:
https://t.co/E0iayak9C4

[Gabriel Garrett @GabGarrett]

@ashxhart these people can’t stand the world leaving them behind

[Ash Hart @ashxhart]

We’re adding new neurons to running local AI models 🤯

Teach it a new fact, save it into the weights, then restart it. Open a fresh chat and it can still remember.

That’s Living Weights in TensorFold. It learns between replies, on your own hardware.

Still experimental, but this is pretty wild.

Myself an @petruspennanen have been working on it and will be bringing a breakthrough soon 🤯🚀

[Gabriel Garrett @GabGarrett]

the race is really on when American labs start timing their model releases to be on the same day as Chinese model releases

[Gabriel Garrett @GabGarrett]

the simulation is running on a lower poly count lately

[Murph @ninjasniperpon]

Hexagonal growth in a black olive tree.

[Gabriel Garrett @GabGarrett]

@DeepDishEnjoyer Elon’s also publicly fudding memory to try to get better prices

[Gabriel Garrett @GabGarrett]

make a plan for if this happens for you. you don’t want to allow yourself to be economically neutered on a whim

[log4j @log4j]

banned for reverse engineering work? that’s wild

oh well, -$500, and a bunch of banked resets) but gained motivation to get 3 more machines and finish my cluster.

i have never used openai to exploit or find vulnerabilities, only local reverse engineering which tripped daybreak.

[brainstormity @brainstormity]

System One Computer Use is now open source 👇
https://t.co/GkhLAsHHUG

- Screenshot → decide → click/type loop
- Dashboard with screenshots and decisions for every step
- Cost tracking per step (~$0.002 a task)
- Stops when it isn’t sure instead of guessing

It’s a prototype and a base for users to build on...

Fork it and let's see what you build on it!

[Gabriel Garrett @GabGarrett]

seeing everyone screeching about banning guardrail-removed models is really making me want to go download all of them.

[Gabriel Garrett @GabGarrett]

you can’t actually automate customer support.

sure, you can delegate some parts to bots, but this is the experience you’re going to lead people to have by default

[Isaac King 🔎 @IsaacKing314]

Currently arguing with a company over whether their blog post advertising a feature implies that their platform should include that feature.

Their support emails are AI-generated, even after I asked for a human. The entire website is AI-generated. Their API docs tell me not to bother reading them, just paste their "integration prompt" into my AI. The founder posted an AI-generated article on Twitter talking about how AI runs the company.

I'm growing increasingly confident that their AI made up a feature that does not exist and wrote dozens of articles advertising this feature in various countries that they do not support.

When I asked their AI assistant to enable this feature on my account, it told me the feature doesn't exist. When I pointed it to the company blog post announcing the feature, it began responding in Chinese.

This company received $6 million from Y Combinator.

[Gabriel Garrett @GabGarrett]

@ashxhart @Alibaba_Qwen @QwenDevs @petruspennanen @volatilemarkts @MiaAI_lab @plotarmordev Will that same kernel apply on the M5U? 👀

[Gabriel Garrett @GabGarrett]

@ashxhart wow this is really fast

[Gabriel Garrett @GabGarrett]

this seems like something that should’ve been standardized a decade ago

[xevrion @xevrion_the1]

you finally finish a 180 GB download
hit extract
"not enough disk space"

unzip needs room for the archive AND the files. hatchzip doesn't: it unpacks from the end and eats the archive as it goes

free and open source. windows, mac, linux

[Gabriel Garrett @GabGarrett]

@ghimibip The best case scenario is it turns into the pitbull problem. Bred for fighting and then people start keeping them as pets that attack people

[Gabriel Garrett @GabGarrett]

I feel like this might go very wrong

[MTS @MTSlive]

SITUATION DETECTED: Warhound launched from stealth today with a $200M seed at a $1.2B valuation. Ben Lamm and Palmer Luckey are cloning gene-edited bio-enhanced dogs and equipping them with custom Anduril vests, goggles, and sensors for the military and law enforcement.

[Gabriel Garrett @GabGarrett]

@yimingdothan And yet, look what they already accomplish with a fraction of a fraction

[Choblin @choblin29]

HOLY MOLY. Something genuinely strange just happened in AI training.

Q Labs says it can now pretrain a transformer WITHOUT backpropagation, cutting the time to reach a fixed validation-loss target from 1.76 hours to just 1.6 seconds in only 10 days. A nearly 4,000x speedup.

The latest run also beats backprop + Adam on validation loss.

Its Dust algorithm was already estimated to be up to 10,000x more compute-efficient than the previous state-of-the-art zeroth-order method.

The biggest question now is scaling. Some of the tricks won't scale, so the team is moving from 10M to 30M training tokens.

[Gabriel Garrett @GabGarrett]

@cryptosibbers @BobClawblaw @jayleaton @ashxhart @MiaAI_lab thanks!

[Gabriel Garrett @GabGarrett]

@cryptosibbers @BobClawblaw @jayleaton @ashxhart @MiaAI_lab awesome, what prefill speed are you seeing?