jonny

curious about intelligence and learning

[Jerry Liu @jerryjliu0]

Parenting is crazy because it's like building rl envs for children

[Jason Fried @jasonfried]

The speed at which a product is developed doesn't inherently make the product better or worse. The number of commits doesn't make the product better or worse. The number of people or agents working on it doesn't make it better or worse. The number of hours you’re pouring into it doesn’t make it better or worse. Working on a weekend, or late into the evening, doesn’t make it better or worse. Talking about these things sounds like it’s talking about the product, but it’s not talking about the product.

Those are all development metrics and styles of work. They don't speak to the product itself. They're all back of the house. Important to those making the work, but not inherently important to the people using the work.

Development metrics and style say nothing about fit. They say nothing about quality. They say nothing about clarity. They say nothing about usability. They say nothing about utility. They say nothing about performance. They say nothing about how customers feel when they use your product.

It's like reviewing a restaurant by talking about how many cooks were in the kitchen, how long the shift lasted, how many orders they fired, or how late they stayed cleaning up. None of that tells you about the meal, the product, the customer’s experience.

Mistaking making for the thing you've made is like confusing the kitchen for the meal or the blueprint for the building. They're connected, but they're not the same thing. One of them matters a whole lot more than the other.

[jonny @frogeaterfirst1]

Most people reading this will already be fairly tolerant. But there is a step beyond thinking of yourself as x but tolerating y: not even to consider yourself an x. The more labels you have for yourself, the dumber they make you.

https://t.co/1whWbxlQIs

[TensorTonic @TensorTonic]

If you're serious about AI engineering, spend a weekend implementing these 8 reports:

1. DeepSeek-V3: MoE routing with auxiliary-loss-free load balancing and multi-token prediction

2. DeepSeek-R1: Reasoning from pure RL with no SFT warm start, then distilled into 1.5B to 70B models

3. Mistral 7B: Sliding window attention and a rolling KV cache for efficient long-context inference

4. Mixtral 8x7B: Sparse MoE with top-2 routing at roughly the inference cost of a 13B dense model

5. Llama 3: 15T token pretraining, continued pretraining for longer context, and a production-scale RLHF pipeline

6. Qwen2.5: Shared tokenizer across dense and MoE models with strong multilingual performance

7. Gemma 2: Teacher-student distillation and logit softcapping for more stable training

8. LoRA: Low-rank adapters that made fine-tuning billion-parameter models practical on a single GPU

[Ash Tilawat @ashtilawat]

This week, I gave 63 interns from Stanford, MIT, and UT a hard challenge:

**Train your own small language model.**

Not prompt one.

Not wrap GPT-5.5 in a nicer UI.

Actually fine-tune a small open model and prove it learned a specific behavior.

The catch?
It cannot be something a frontier model already does well.
That is the whole game.

Frontier models are insanely broad. They are 80–90% good at almost everything. But “pretty good” breaks down when the task is niche, domain-specific, or reliability-sensitive.

That gap is where fine-tuning gets interesting.

A small model is not going to beat Claude or GPT on general intelligence.

But it can beat them in a narrow sliver.
A weird edge case.
A specific teaching behavior.
A company-specific review style.
A failure mode the frontier model keeps missing.

That is what the interns are hunting for this week.

1. Find the gap.
2. Generate the data.
3. Filter it hard.
4. Fine-tune the model.
5. Build the eval.
6. Prove the behavior moved from the prompt into the weights.

The biggest lesson:
The model is not the deliverable.
The dataset is.

[Blake Scholl 🛫 @bscholl]

Critics: I’ll believe it when I see it
Builders: I believe it so I’ll make it

[Sophia Xing @xingsthatmatter]

I'm a founder who travels between US and China frequently. A couple days ago I posted about the things China quietly does better than the US. It clearly hit a nerve. Here's the honest flip side - things about the US that makes a difference.

Not about politics. Just daily life: 🧵

[X @X]

250 looks good on you, America 🇺🇸

[sarah guo @saranormous]

Very proud to be an American!

There is no better system: individual rights, popular sovereignty, limited but capable government, rule of law, pluralism, free speech and markets, earned status, self-correction, invention — and ambition.

We will be better for the next 250 by leaning again into who we are, and as an ongoing nation-project, fixing the pieces of that which need work (housing, capacity to build physical things, governance capacity, trust collapse, defense against propaganda). 🇺🇸

[François Chollet @fchollet]

One way to live 400 years is to do (and experience) 5x more things per day

[Arthur MacWaters @ArthurMacwaters]

It’s crazy how we found a way to purify literal rocks into 99.9999999% pure silicon, then have magicians pull it into crystals, then etch a trillion little runes on it, put some electricity into it, and now it has 130 IQ

and the most common thing to do with this miracle is email

[Samuel Spitz @samuel_spitz]

There are only 3 businesses left in software:

1. lab
2. vendor to labs (data, compute)
3. affiliate for labs (reselling tokens)

[Peter H. Diamandis, MD @PeterDiamandis]

Andrew Carnegie spent a fortune building 2,500 free libraries so a poor kid could read what a rich one read. We are finishing the job he started. The only difference is the library now talks back, in every language, and it never closes.

[Fernando @Franc0Fernand0]

@rauchg When execution was expensive, half of the bad choices were weeded out by the price. Nobody built the wrong thing for six months by accident. Now nothing stands in your way. Judgment is the only filter left. Cheap execution doesn't lower the value of being right; it raises it.

[Guillermo Rauch @rauchg]

Human judgement in engineering is ironically even more crucial now. Deciding what to build. Deciding on the right architectures. Deciding whether you regenerate from scratch $$$ or reuse existing legos. Managing tech debt. You can do anything now, but you can't do everything.

[Aravind Srinivas @AravSrinivas]

Moving fast is essentially an expression of humility as you make frequent contact with reality

[François Chollet @fchollet]

If your benchmark relies on a static dataset or sampling from a static distribution densely known at training time, then it is fundamentally measuring memorization/retrieval. Which might be fine if you're looking for a retrieval benchmark! But don't confuse it with intelligence.

[jason @jxnlco]

@leodido /tasks/<task>/{plan, worklog, goal}.md

[jonny @frogeaterfirst1]

Courage sits somewhere in between cowardice and recklessness.

[Nathan Lambert @natolambert]

Another quick lecture -- I've been asked many times for prereq's to my book and what you should know, so built a little lecture (with GLM 5.2) to cover some more basics.

Topics include:

00:00 Introduction & Course Prerequisites
01:37 Language Models Overview
02:47 The LM Head
04:29 Softmax & Log-Probabilities
06:13 Anatomy of an LM Training Example
06:37 Computing LLM Probabilities (+Phoebe the Dog)
09:52 Three Common Masks in Post-Training
11:03 A Small Decoding Review
12:14 Training an LM: Cross-Entropy
13:23 Optimization & Fine-Tuning
13:55 Pretraining to Midtraining to SFT Pipeline
15:25 Probability Essentials: KL Divergence & Entropy
19:36 Sigmoid & Pairwise Likelihood
20:29 Reinforcement Learning Framing (MDP)
22:28 Transitioning Tools into Post-Training
23:12 Recommended Resources & Wrap-Up

Happy learning and I'm still taking questions from during the course for Q&A videos.