Huy B

Building Moji a minimal Japanese flashcard app

[jietang @jietang]

Thoughts About Scaling Law

Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter count is only meaningful alongside three others — how much data you have, where you intend to spend your compute, and who will run the model, under what conditions.

The field learned this the hard way. Kaplan et al. (2020) fit an exponent that told everyone to grow parameters faster than data — roughly 2.7:1 — and the industry complied: GPT-3, Gopher, MT-NLG. Hoffmann et al. (2022) redid the experiment across four hundred models and found the compute-optimal split is closer to 20 tokens per parameter, and that with sufficient compute the two should grow at the same rate rather than drifting apart. The error in the earlier fit compounded with every order of magnitude of compute, which is why the largest models of that generation were the most misallocated. The trillion-parameter round was, in retrospect, a detour the whole field took together and then reversed.

Chinchilla wasn't the end either. It optimized training compute for models that would be trained once and evaluated. Today a model is called billions of times a day and inference dominates lifetime cost. Put inference into the objective and the optimum moves toward smaller models trained far longer — deliberate over-training, which is what Llama-2-7B and Gemma-2-9B were doing at roughly 290 and 889 tokens per parameter.

Sparsity moved the target again. In a MoE model two quantities have to be kept apart: total parameters govern roughly how much the model can hold — knowledge, facts, the long tail — while activated parameters and effective depth govern roughly how far it can think, how many steps of a causal chain it can carry before it comes apart. A dense 20:1 ratio does not transfer. And the ratio isn't a single number at all: Roberts et al. (2025) find the optimal tokens-per-parameter is task-dependent, with memorization favoring more parameters and reasoning favoring more data. Follow-up work on MoE observes that at fixed TPP, pushing total parameters higher actually degrades reasoning, while activating more experts reliably helps it.

This matters for what we are building toward. Finding a vulnerability is not a retrieval problem. It doesn't come from having memorized more CVEs; it comes from carrying a twenty-step chain of inference to the end without losing the thread. That capability does not live in total parameter count.

Which brings us to this release. Total parameters appear to matter up to a threshold — enough to hold the world — after which additional capability comes from scaling elsewhere: effective depth per forward pass, and above all post-training. GLM-5.3 is our controlled experiment on that claim. Same base, same architecture, same total and activated parameters as GLM-5.2. One month of scaling long-horizon environments and RL. The gains are not marginal. Well, scaling has more than one dial. We turned the post-training one this time because it had the most slack left in it — not because the others are finished. Base model size, pretraining data, compute spent per forward pass: all of them are still on the table, and we will come back to each. What this experiment taught us is that the dials do not have to be turned together, and that the one worth turning next is rarely the one that was worth turning last. We are not done scaling. Next time, maybe mid-training, pre-training, and even more.

[Aaron Francis @aarondfrancis]

don't worry bro, it's byte-identical

[alec @alec_dilanchian]

We added a hand drawn camera to One Year 📸

[Huy B @bit5andpiece5]

@LinoLeighton Why did active trial drop all of sudden?

[Huy B @bit5andpiece5]

Codex controlling my Mac feels like the future: browser open, tabs handled, tiny tasks done. Created by Codex btw.

[Huy B @bit5andpiece5]

You can create nice launch video with Appstore connect

[Huy B @bit5andpiece5]

Kanji mnemonics can be really beautiful

[Huy B @bit5andpiece5]

@everton_dev Nice flow!

[Anthropic @AnthropicAI]

New on the Engineering Blog:

Building Managed Agents—our hosted service for long-running agents—meant solving an old problem in computing: how to design a system for “programs as yet unthought of.”

Read more: https://t.co/YYaEub2QGV

[Chris Lattner @clattner_llvm]

@Zyyon_ Please don’t tell anyone: we aren’t just open sourcing all the models. We are doing the unspeakable: open sourcing all the gpu kernels too. Making them run on multivendor consumer hardware, and opening the door to folks who can beat our work.

Plz keep it quiet, ok? 😉

[Josh Kale @JoshKale]

Everyone’s saying OpenAI got the “same deal” Anthropic was banned for.
Read the fine print. They’re not the same:

On weapons:
Anthropic asked for “no fully autonomous weapons without human oversight” = a human involved in the decision.

OpenAI’s deal says “human responsibility for the use of force” = someone accountable, which can happen after the fact.

Oversight ≠ Responsibility. One requires a human before the trigger. The other requires a name on the paperwork after.

On surveillance:
Dario said explicitly: current law hasn’t caught up with AI. The government can already buy your movement data, browsing history, etc without a warrant. AI can assemble that into a complete picture of your life, at scale. That’s mass surveillance without breaking a single law.

Anthropic wanted protections beyond current law.

OpenAI’s deal says the Pentagon “reflects them in law and policy.” That’s existing law as the safeguard, the exact law Anthropic said is insufficient.

Same words. Different agreements. Read them carefully

[hardmaru @hardmaru]

Instead of forcing models to hold everything in an active context window, we can use hypernetworks to instantly compile documents and tasks directly into the model's weights. A step towards giving language models durable memory and fast adaptation.

Blog: https://t.co/iHoifpsLMu

[Adam @adamdotdev]

I find it very funny when anyone feels confident that they've figured out agentic programming, even funnier when they're trying to teach others how to do it. I've been working on OpenCode since May of last year and I still have days (like yesterday) where I'm not even sure any of this is a good idea lol

I end up landing on "yes, these models are an incredible tool" but it's still all very confusing, lots of tangled thoughts and emotions and realities.

I badly miss the mundane coding tasks that broke up my days/weeks, the ones where you put on the headphones and just bang out 600 lines of code. But, no question, replacing those hours of my time with a few minutes of waiting on an agent is a boost and worth being excited about, despite the mixed emotions.

Then there's the distance that can creep in between you and the codebase if you start getting apathetic. I think it's pretty common at this point to make even small changes by prompting the models. It's less friction than finding the relevant code and making the change yourself. And less friction seems to win, must be some law of the universe or some shit. When most or all of your interactions with a codebase start flowing through the models, you start to lose track of where things live, which abstractions/components are carrying the weight, etc. It's a scary feeling to wake up and realizing you can't even reliably @<mention> a precise file for a change you want to make, and you have to get more vague, leaning harder on the model.

It all creeps up on you, there's an undeniable dopamine hit from using these things, and the resulting come down is predictable, like coming off a sugar high. On the positive side, it's really nice seeing other devs go through the same cycles, knowing we're all in this together and we'll ultimately figure it out.

[Thariq @trq212]

one of the biggest realizations I've had working on Claude Code is that you fundamentally have to design agents for prompt caching first, almost every feature touches on it somehow

I wrote this in a day but it's the culmination of months of learnings, hope you enjoy it

[Huy B @bit5andpiece5]

@cedric_chee What do you mean its not competing with Suno?

[Peter Steinberger 🦞 @steipete]

@iwantlambo yah they only sent love letters from legal

[NASA Mars @NASAMars]

By studying samples analyzed by the Curiosity rover, scientists have taken another step toward understanding whether life could have ever existed on Mars.

A new study suggests that non-biological sources cannot fully account for the abundance of organic compounds found in a sample collected by the rover.

Dig into the details: https://t.co/zZKCgKEROr

[dax @thdxr]

everyone's talking about their teams like they were at the peak of efficiency and bottlenecked by ability to produce code

here's what things actually look like

- your org rarely has good ideas. ideas being expensive to implement was actually helping

- majority of workers have no reason to be super motivated, they want to do their 9-5 and get back to their life

- they're not using AI to be 10x more effective they're using it to churn out their tasks with less energy spend

- the 2 people on your team that actually tried are now flattened by the slop code everyone is producing, they will quit soon

- even when you produce work faster you're still bottlenecked by bureaucracy and the dozen other realities of shipping something real

- your CFO is like what do you mean each engineer now costs $2000 extra per month in LLM bills

[Huy B @bit5andpiece5]

Great read

[Armin Ronacher ⇌ @mitsuhiko]

Today @thorstenball said he's the bottleneck. I too am the bottleneck and it made me think. https://t.co/FIwf1zAMU7

[Huy B @bit5andpiece5]

@clattner_llvm Yes please!