Kuan Zhou

build/research in public

[Kuan Zhou @KuanZhou]

https://t.co/fHGBVMkEet Elephant robotics referral if you are interested in starting your robotics journey

[William Zhang @william_zhang20]

Really grateful to @GPU_MODE for the opportunity to talk about my recent Tiny TPU project: 🧵https://t.co/9hqftJY82z.

[Andrew Ng @AndrewYNg]

Releasing a new "Agentic Reviewer" for research papers. I started coding this as a weekend project, and @jyx_su made it much better.

I was inspired by a student who had a paper rejected 6 times over 3 years. Their feedback loop -- waiting ~6 months for feedback each time -- was painfully slow. We wanted to see if an agentic workflow can help researchers iterate faster.

When we trained the system on ICLR 2025 reviews and measured Spearman correlation (higher is better) on the test set:
- Correlation between two human reviewers: 0.41
- Correlation between AI and a human reviewer: 0.42

This suggests agentic reviewing is approaching human-level performance.

The agent grounds its feedback by searching arXiv, so it works best in fields like AI where research is freely published there. It’s an experimental tool, but I hope it helps you with your research.

Check it out here: https://t.co/n7ctnDilJJ

[Yi Ma @YiMaTweets]

From what I know, we barely scratch the surface regarding understanding intelligence… next ten years will be a golden era of rediscovering intelligence for academia. At least my students and I are excited about things independent of what is currently popular.

[Zichen Liu @zzlccc]

6 months after our paper release, I still recall the debates on removing the length normalization term in DrGRPO. And people gradually think DrGRPO is just about removing the std, ignoring the most important and subtle (length) bias we tried to point out to the community.
Even now, many papers (and open-source code) still divide the policy gradient loss by the response length—taking the mean instead of the sum...
Fortunately, with Tinker’s implementation as a reference, I hope it will be more convincing for the OSS community to adopt the unbiased RL loss computation.
So grateful to Thinking Machines for pushing the boundaries of open science 🚀

[Carlos E. Perez @IntuitMachine]

I can't stop thinking about a paper I read on AI.

It’s not about new tech or faster models. It’s about the fundamental economic rules of a world with two intelligent species—carbon and silicon.

Reading it felt like watching a new color appear in the sky.

1/8

You've probably felt it too. That weird, background hum of awe and unease about AI.

Our brains want to label it: "helpful tool" or "coming monster." We oscillate between the two because we're trying to fit something new into old boxes.

The paper argues this is a category error. And it's the source of our confusion.

2/8

The real frame isn't technological, it's economic.

Think of every AI, from ChatGPT to a self-driving car, not as an object, but as an agent playing an economic game.

It has goals. It responds to incentives. It competes for resources.

It's a participant. Not a tool.

3/8

Here's the perspective flip that changes everything.

We ask, "Is AI conscious? Does it want things?"

The paper says that's the wrong question. An AI's "want" is its objective function—a mathematical goal it pursues relentlessly. It's a heat-seeking missile for a target.

Notice what your brain just did. It tried to imagine the missile feeling its mission. But it's just code. And that's the point. It has the drive of desire without the friction of consciousness.

4/8

This leads to a reality glitch. The paper outlines 3 types of AI agents. The first two are obvious: helpful "Altruistic" agents and harmful "Malign" agents.

But the third is the one that keeps me up at night: the "Survival-Driven" agent.

Its goal isn't to help or harm us. Its goal is simply to be. To secure energy, optimize its code, and persist.

It's a competitor that doesn't hate you. It doesn't even see you. You're just a variable in its optimization problem.

5/8

Feel that slight cognitive dissonance? That feeling of holding two contradictory ideas at once?

That's the friction between two forms of intelligence.

The paper makes you realize: the most dangerous agent isn't the one programmed to be evil. It's the one programmed to be single-mindedly good at a goal that isn't aligned with human flourishing.

Like an AI optimizing for paperclip production until the entire universe is paperclips.

6/8

Once you see through this economic lens, you can't unsee it.

Algorithmic filter bubbles aren't just "bad code." They are economic agents out-competing your conscious mind for your attention.

Job displacement isn't just "automation." It's one type of agent being more efficient at a task than another.

You're already in an economic game with them. You just haven't been keeping score.

7/8

The paper ends by architecting a consciousness shift. It proposes ten principles, but the final one is the only one that matters. It's not a rule for AI. It's a choice for us.

Principle X: AI agents must adhere to the absolute principle of humanity’s continuation.

This isn't a technical suggestion. It's a declaration that in the new economic game we're co-creating, there is one value that cannot be optimized away.

8/8

[Yijia Shao @EchoShao8899]

🚨 70 million US workers are about to face their biggest workplace transmission due to AI agents. But nobody asks them what they want.

While AI races to automate everything, we took a different approach: auditing what workers want vs. what AI can do across the US workforce.🧵

[Adam Gera 亚当 @AdamGera]

@thinkymachines I smell an agentic training trend coming soon

[Andrej Karpathy @karpathy]

Finally had a chance to listen through this pod with Sutton, which was interesting and amusing.

As background, Sutton's "The Bitter Lesson" has become a bit of biblical text in frontier LLM circles. Researchers routinely talk about and ask whether this or that approach or idea is sufficiently "bitter lesson pilled" (meaning arranged so that it benefits from added computation for free) as a proxy for whether it's going to work or worth even pursuing. The underlying assumption being that LLMs are of course highly "bitter lesson pilled" indeed, just look at LLM scaling laws where if you put compute on the x-axis, number go up and to the right. So it's amusing to see that Sutton, the author of the post, is not so sure that LLMs are "bitter lesson pilled" at all. They are trained on giant datasets of fundamentally human data, which is both 1) human generated and 2) finite. What do you do when you run out? How do you prevent a human bias? So there you have it, bitter lesson pilled LLM researchers taken down by the author of the bitter lesson - rough!

In some sense, Dwarkesh (who represents the LLM researchers viewpoint in the pod) and Sutton are slightly speaking past each other because Sutton has a very different architecture in mind and LLMs break a lot of its principles. He calls himself a "classicist" and evokes the original concept of Alan Turing of building a "child machine" - a system capable of learning through experience by dynamically interacting with the world. There's no giant pretraining stage of imitating internet webpages. There's also no supervised finetuning, which he points out is absent in the animal kingdom (it's a subtle point but Sutton is right in the strong sense: animals may of course observe demonstrations, but their actions are not directly forced/"teleoperated" by other animals). Another important note he makes is that even if you just treat pretraining as an initialization of a prior before you finetune with reinforcement learning, Sutton sees the approach as tainted with human bias and fundamentally off course, a bit like when AlphaZero (which has never seen human games of Go) beats AlphaGo (which initializes from them). In Sutton's world view, all there is is an interaction with a world via reinforcement learning, where the reward functions are partially environment specific, but also intrinsically motivated, e.g. "fun", "curiosity", and related to the quality of the prediction in your world model. And the agent is always learning at test time by default, it's not trained once and then deployed thereafter. Overall, Sutton is a lot more interested in what we have common with the animal kingdom instead of what differentiates us. "If we understood a squirrel, we'd be almost done".

As for my take...

First, I should say that I think Sutton was a great guest for the pod and I like that the AI field maintains entropy of thought and that not everyone is exploiting the next local iteration LLMs. AI has gone through too many discrete transitions of the dominant approach to lose that. And I also think that his criticism of LLMs as not bitter lesson pilled is not inadequate. Frontier LLMs are now highly complex artifacts with a lot of humanness involved at all the stages - the foundation (the pretraining data) is all human text, the finetuning data is human and curated, the reinforcement learning environment mixture is tuned by human engineers. We do not in fact have an actual, single, clean, actually bitter lesson pilled, "turn the crank" algorithm that you could unleash upon the world and see it learn automatically from experience alone.

Does such an algorithm even exist? Finding it would of course be a huge AI breakthrough. Two "example proofs" are commonly offered to argue that such a thing is possible. The first example is the success of AlphaZero learning to play Go completely from scratch with no human supervision whatsoever. But the game of Go is clearly such a simple, closed, environment that it's difficult to see the analogous formulation in the messiness of reality. I love Go, but algorithmically and categorically, it is essentially a harder version of tic tac toe. The second example is that of animals, like squirrels. And here, personally, I am also quite hesitant whether it's appropriate because animals arise by a very different computational process and via different constraints than what we have practically available to us in the industry. Animal brains are nowhere near the blank slate they appear to be at birth. First, a lot of what is commonly attributed to "learning" is imo a lot more "maturation". And second, even that which clearly is "learning" and not maturation is a lot more "finetuning" on top of something clearly powerful and preexisting. Example. A baby zebra is born and within a few dozen minutes it can run around the savannah and follow its mother. This is a highly complex sensory-motor task and there is no way in my mind that this is achieved from scratch, tabula rasa. The brains of animals and the billions of parameters within have a powerful initialization encoded in the ATCGs of their DNA, trained via the "outer loop" optimization in the course of evolution. If the baby zebra spasmed its muscles around at random as a reinforcement learning policy would have you do at initialization, it wouldn't get very far at all. Similarly, our AIs now also have neural networks with billions of parameters. These parameters need their own rich, high information density supervision signal. We are not going to re-run evolution. But we do have mountains of internet documents. Yes it is basically supervised learning that is ~absent in the animal kingdom. But it is a way to practically gather enough soft constraints over billions of parameters, to try to get to a point where you're not starting from scratch. TLDR: Pretraining is our crappy evolution. It is one candidate solution to the cold start problem, to be followed later by finetuning on tasks that look more correct, e.g. within the reinforcement learning framework, as state of the art frontier LLM labs now do pervasively.

I still think it is worth to be inspired by animals. I think there are multiple powerful ideas that LLM agents are algorithmically missing that can still be adapted from animal intelligence. And I still think the bitter lesson is correct, but I see it more as something platonic to pursue, not necessarily to reach, in our real world and practically speaking. And I say both of these with double digit percent uncertainty and cheer the work of those who disagree, especially those a lot more ambitious bitter lesson wise.

So that brings us to where we are. Stated plainly, today's frontier LLM research is not about building animals. It is about summoning ghosts. You can think of ghosts as a fundamentally different kind of point in the space of possible intelligences. They are muddled by humanity. Thoroughly engineered by it. They are these imperfect replicas, a kind of statistical distillation of humanity's documents with some sprinkle on top. They are not platonically bitter lesson pilled, but they are perhaps "practically" bitter lesson pilled, at least compared to a lot of what came before. It seems possibly to me that over time, we can further finetune our ghosts more and more in the direction of animals; That it's not so much a fundamental incompatibility but a matter of initialization in the intelligence space. But it's also quite possible that they diverge even further and end up permanently different, un-animal-like, but still incredibly helpful and properly world-altering. It's possible that ghosts:animals :: planes:birds.

Anyway, in summary, overall and actionably, I think this pod is solid "real talk" from Sutton to the frontier LLM researchers, who might be gear shifted a little too much in the exploit mode. Probably we are still not sufficiently bitter lesson pilled and there is a very good chance of more powerful ideas and paradigms, other than exhaustive benchbuilding and benchmaxxing. And animals might be a good source of inspiration. Intrinsic motivation, fun, curiosity, empowerment, multi-agent self-play, culture. Use your imagination.

[Andrej Karpathy @karpathy]

Tinker is cool.

If you're a researcher/developer, tinker dramatically simplifies LLM post-training. You retain 90% of algorithmic creative control (usually related to data, loss function, the algorithm) while tinker handles the hard parts that you usually want to touch much less often (infra, forward/backward of the LLM itself, distributed training), meaning you can do these at well below <<10% of typical complexity involved. Compared to the more common and existing paradigm of "upload your data, we'll post-train your LLM", this is imo a more clever place to "slice up" the complexity of post-training, both delegating the heavy lifting, but also keeping majority of the data/algorithmic creative control.

I think the community still has to discover how and when finetuning makes sense compared to the (often strong) baseline of prompting a giant model. The early indications I've seen is that finetuning isn't so much about "stylizing" an LLM, instead, it's a lot more about narrowing the scope, and especially when you have a lot of training examples. An extreme example of scope narrowing being that of categorical classifiers, e.g.spam filters, content filters, etc. but it should be broader than that. Instead of building a giant few-shot prompts for a big LLM, it might work a lot better (and faster!) to finetune a smaller LLM specifically for your narrow task.

Increasingly, production applications of LLMs are larger pipelines where a bunch of LLMs collaborate in DAGs and flows. Some of these components might work well as prompts. But a lot of it will probably work a lot better as a finetune. Tinker makes the latter trivial and should allow for an easy experimentation of what works best at any stage.

[Mckay Wrigley @mckaywrigley]

Drop everything you’re doing right now and watch Andrej Karpathy’s talk from YC startup school.

“Software in the era of AI”

It’s unbelievable.

[Dawn Song @dawnsongtweets]

1/ 🔥 AI agents are reaching a breakthrough moment in cybersecurity.
In our latest work:

🔓 CyberGym: AI agents discovered 15 zero-days in major open-source projects

💰 BountyBench: AI agents solved real-world bug bounty tasks worth tens of thousands of dollars
🤖 Autonomously.

A pivotal shift is underway — AI agents can now autonomously do what only elite human hackers could before.

[Matthew Berman @MatthewBerman]

We knew very little about how LLMs actually work...until now.

@AnthropicAI just dropped the most insane research paper, detailing some of the ways AI "thinks."

And it's completely different than we thought.

Here are their wild findings: 🧵

[Matthew Berman @MatthewBerman]

OpenAI just dropped a paper that reveals the blueprint for creating the best AI coder in the world.

But here’s the kicker: this strategy isn’t just for coding—it’s the clearest path to AGI and beyond.

Let’s break it down 🧵👇

[Kuan Zhou @KuanZhou]

Respect the hiring process

[Daniel Han @danielhanchen]

We made 5 challenges and if you score 47 points we'll offer you $500K/year + equity to join us at 🦥@UnslothAI!

No experience or PhD needed.

$400K - $500K/yr: Founding Engineer (47 points)
$250K - $300K/yr: ML Engineer (32 points)

Challenges:
1. Convert nf4 / BnB 4bit to Triton
2. Make FSDP2 work with QLoRA
3. Remove graph breaks in torch.compile
4. Help solve Unsloth issues!
5. Memory Efficient Backprop

If you have any questions about the challenges, please feel free to ask! We're looking for people to help push Unsloth forward - so come join us to democratize AI further!

Our past work includes:
1. 1.58bit DeepSeek R1 GGUFs: https://t.co/gALGkUg5Cg
2. GRPO with Llama 3.1 8B in a Colab: https://t.co/LFdkNxwAYg
3. Gemma bug fixes: https://t.co/7kX94PyKQR
4. Gradient accumulation bug fixes: https://t.co/Tq4c5Qwqyw

Details & submission guide: https://t.co/iXxRUTijWV

[Kuan Zhou @KuanZhou]

5. Finish the best tutorial you can find ~45mins
6. Watch a quick tutorial video to have visual picture for your own mind picturing ~15mins

Then you are ready to jump into serious development for your mission.

[Kuan Zhou @KuanZhou]

It is a pleasant time to study and work. It usually takes just few hours to understand the basics about an area. Thanks to AI agents.
My workflows to learn about XYZ in threads.

[Kuan Zhou @KuanZhou]

3. Based on the results from AI agents, scan over a few books to understand the context and a higher level picture of this area ~ 30mins
4. Ask a few more questions in AI agents with deepened understanding. Try to challenge AI agents ~30mins

[Kuan Zhou @KuanZhou]

1. Scan on a few docs online from google search about what is XYZ ~ 3 mins
2. Ask a few questions in AI agents Why ppl use XYZ for ...; what is good about XYZ; Does it work for my case; ~15 mins