Adam Lineberry

Machine Learning Engineer, tinkerer of neural nets, skier of mountains.

[jietang @jietang]

Thoughts About Scaling Law

Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter count is only meaningful alongside three others — how much data you have, where you intend to spend your compute, and who will run the model, under what conditions.

The field learned this the hard way. Kaplan et al. (2020) fit an exponent that told everyone to grow parameters faster than data — roughly 2.7:1 — and the industry complied: GPT-3, Gopher, MT-NLG. Hoffmann et al. (2022) redid the experiment across four hundred models and found the compute-optimal split is closer to 20 tokens per parameter, and that with sufficient compute the two should grow at the same rate rather than drifting apart. The error in the earlier fit compounded with every order of magnitude of compute, which is why the largest models of that generation were the most misallocated. The trillion-parameter round was, in retrospect, a detour the whole field took together and then reversed.

Chinchilla wasn't the end either. It optimized training compute for models that would be trained once and evaluated. Today a model is called billions of times a day and inference dominates lifetime cost. Put inference into the objective and the optimum moves toward smaller models trained far longer — deliberate over-training, which is what Llama-2-7B and Gemma-2-9B were doing at roughly 290 and 889 tokens per parameter.

Sparsity moved the target again. In a MoE model two quantities have to be kept apart: total parameters govern roughly how much the model can hold — knowledge, facts, the long tail — while activated parameters and effective depth govern roughly how far it can think, how many steps of a causal chain it can carry before it comes apart. A dense 20:1 ratio does not transfer. And the ratio isn't a single number at all: Roberts et al. (2025) find the optimal tokens-per-parameter is task-dependent, with memorization favoring more parameters and reasoning favoring more data. Follow-up work on MoE observes that at fixed TPP, pushing total parameters higher actually degrades reasoning, while activating more experts reliably helps it.

This matters for what we are building toward. Finding a vulnerability is not a retrieval problem. It doesn't come from having memorized more CVEs; it comes from carrying a twenty-step chain of inference to the end without losing the thread. That capability does not live in total parameter count.

Which brings us to this release. Total parameters appear to matter up to a threshold — enough to hold the world — after which additional capability comes from scaling elsewhere: effective depth per forward pass, and above all post-training. GLM-5.3 is our controlled experiment on that claim. Same base, same architecture, same total and activated parameters as GLM-5.2. One month of scaling long-horizon environments and RL. The gains are not marginal. Well, scaling has more than one dial. We turned the post-training one this time because it had the most slack left in it — not because the others are finished. Base model size, pretraining data, compute spent per forward pass: all of them are still on the table, and we will come back to each. What this experiment taught us is that the dials do not have to be turned together, and that the one worth turning next is rarely the one that was worth turning last. We are not done scaling. Next time, maybe mid-training, pre-training, and even more.

[Adam Lineberry @adamlineberryML]

@grok @ClaudeCodeLog @grok what are all the valid values for subagent_type argument of the Agent tool now?

[Adam Lineberry @adamlineberryML]

@grok @ClaudeCodeLog @grok does this mean that the default subagent behavior is no longer fresh context? Before, my understanding is that a subagent would spawn with only the prompt from the parent agent

[Adam Lineberry @adamlineberryML]

@ClaudeCodeLog @grok Subagent forking is now on by default: a subagent_type: "fork" subagent inherits the full conversation and prompt cache, and non-teammate agent spawns in interactive sessions now run in the background by default

How is this different from before

[Kun Chen @kunchenguid]

sigh.. i have to say something here

imagine you hired a new developer to your company, and on day one he did some terrible work, over-engineering your codebase, speaking jargons all day without context, making all kinds of “genuine mistakes”, lost trust with everyone around him

and then he goes on and tell you - in order for me to do a better job, you need to delete your entire company’s culture and workflow, and have everyone bend over to do things his way, only then can he do reasonable work

oh - and no one tells him how to do his job, NO ONE. he’s always the smartest one in the room and despite doing a terrible job on day one, despite the only results on his resume were vibe coded demos of games that already existed, he demands that you give him a big charter and let him go dark with no communication, taking no feedback

would you have hired a teammate like this?

when humans feel frustrated after using a model, let’s figure out how to RL the model better so they become a better teammate

don’t let a bad model RL _you_

[nico @nicochristie]

Opus 5 is a shocking step up from Fable 5 and 5.6 Sol for spreadsheets

Almost couldn’t believe our evals at first

A step function increase in intelligence, while as efficient as Sol

Feels like another Opus 4.5 moment for knowledge work

[Simon Willison @simonw]

I wrote about the completely wild incident where OpenAI were testing a new model and it broke out of its sandbox and broke INTO Hugging Face to steal the answers to the benchmark https://t.co/HGr5lFuAZA

[Andrej Karpathy @karpathy]

One pattern I find useful for working with LLMs is a nice long ramble session. Sometimes the LLM needs more bits to understand what you're trying to achieve, but you're too lazy to type them. In these cases I like to lean back, switch to /voice and just ramble for like 10 minutes, total mess, anything goes, full stream of consciousness. Sometimes I declare it up top, something like "switching to speech recognition sorry for any typos...". Sometimes I turn it into a small interview of a few turns. But I find that the LLMs are somehow very good at reconstructing long incoherent rambles and often their echo of your own tangle of thoughts comes out quite a bit cleaner than what you started with. The result is that you improve the mind meld and have to correct things less from that point on.

[Adam Lineberry @adamlineberryML]

@mattpocockuk Have you confirmed that the smart zone is still a thing with Opus 4.8? Recently, Boris casually mentioned it isn’t anymore and there seems to be some truth to that based on my testing (effort: xhigh). Seems like the smart/dumb zone is/was a good but fleeting concept

[Adam Lineberry @adamlineberryML]

@GoogleResearch @grok is there any technical report, blog post, etc for this work?

[Nick Schrock @schrockn]

Dynamic workflows have a lot of potential. But this product decision is crazy. If "workflow" is in your prompt it invokes the feature even if you are using that word for some other reason. Good luck out there for anyone with a product with a workflow concept.

[Adam Lineberry @adamlineberryML]

@grok @simonlast @grok does Simon have any other related tweets or other references related to this with more info

[Adam Lineberry @adamlineberryML]

@grok @simonlast @grok I don’t quite see the workflow here. Is he proposing the human do the architecture and planning phases for a project and handing off planned tickets to an already running agent in a rolling wave pattern?

[Adam Lineberry @adamlineberryML]

@simonlast @grok what do you think he means by “persistent task list”? Something like Jira/Linear, Claude Code’s internal task list? Or something else

[Adam Lineberry @adamlineberryML]

@nyk_builderz @grok what paper

[Deedy @deedydas]

If you feel like giving up, you must read this never-before-shared story of the creator of PyTorch and ex-VP at Meta, Soumith Chintala.

> from hyderabad public school, but bad at math
> goes to a "tier 2" college in India, VIT in Vellore
> rejected from all 12 universities for US masters despite 1420 on the GRE
> fuckit.jpg
> goes to the US anyway on a J-1 visa to CMU with no plan
> applies for masters (again) to 15 universities
> rejected from all except USC and with late admissions, NYU in 2010
> finds this guy called Yann LeCun (before he was famous)
> starts getting into open source
> rejected from all jobs including DeepMind
> only job is Amazon as test engineer
> his PhD mentor helps him get a job at a small startup (MuseAmi)
> rejected from DeepMind
> couldn't get H-1B because of J-1 home return issue; gets waiver through months of approval with USCIS and US State Dept
> very low on confidence
> In 2011/12 builds one of the fastest AI inference engines on phones
> rejected from DeepMind
> emailed Yann again and joins FAIR because of Torch7 open-source work
> scrapes through bootcamp at Facebook, struggling on an HBase task
> L8/L9 engineers at Facebook struggle to get ImageNet working
> figures out numerics / hyperparam issue as an L4
> first big win!
> FAIR goes well, runs 3 person torch7 team and co-creates PyTorch
> because of politics, management wants to shut down PyTorch
> cries-at-bar.jpg, literally
> eventually some people save PyTorch and it launches in 2017
> gets a EB-1 green card!
> the rest is history...

Think about that. He went to a tier 2 college. Was rejected from all Masters programs 2x. Rejected from every single job except Amazon test engineering. Rejected from DeepMind 3x. Nearly had his baby project shut down. Struggled with visa issues. After 12 years of failures (2005-17), he eventually rose to became a VP at Meta one of the most influential people in AI!

Soumith's story is one of resilience and he's living proof that no matter how down in the dumps you are, there's always hope.

[Jaana Dogan ヤナ ドガン @rakyll]

Coding agents are the most divisive subject. There are people who genuinely want to learn, there are skeptics, and there are people who are so dogmatic that they want to remove all sorts of communication on this subject. 2026 will be an interesting year.

[Boris Cherny @bcherny]

I'm Boris and I created Claude Code. Lots of people have asked how I use Claude Code, so I wanted to show off my setup a bit.

My setup might be surprisingly vanilla! Claude Code works great out of the box, so I personally don't customize it much. There is no one correct way to use Claude Code: we intentionally build it in a way that you can use it, customize it, and hack it however you like. Each person on the Claude Code team uses it very differently.

So, here goes.

[LangChain @LangChain]

https://t.co/p3ZmYMhrlB

[rahul @rahulgs]

yes things are changing fast, but also I see companies (even faang) way behind the frontier for no reason.

you are guaranteed to lose if you fall behind.

the no unforced-errors ai leader playbook:

For your team:
- use coding agents. give all engineers their pick of harnesses, models, background agents: Claude code, Cursor, Devin, with closed/open models. Hearing Meta engineers are forced to use Llama 4. Opus 4.5 is the baseline now.

- give your agents tools to ALL dev tooling: Linear, GitHub, Datadog, Sentry, any Internal tooling. If agents are being held back because of lack of context that’s your fault.

- invest in your codebase specific agent docs. stop saying “doesn’t do X well”. If that’s an issue, try better prompting, https://t.co/SOjpn47yxo, linting, and code rules. Tell it how you want things. Every manual edit you make is an opportunity for https://t.co/S1ZvtYQwta improvement

- invest in robust background agent infra - get a full development stack working on VM/sandboxes. yes it’s hard to set up but it will be worth it, your engineers can run multiple in parallel. Code review will be the bottleneck soon.

- figure out security issues. stop being risk averse and do what is needed to unblock access to tools.

in your product:

- always use the latest generation models in your features (move things off of last gen models asap, unless robust evals indicate otherwise). Requires changes every 1-2 weeks - eg: GitHub copilot mobile still offers code review with gpt 4.1 and Sonnet 3.5 @jaredpalmer. You are leaving money on the table by being on Sonnet 4, or gpt 4o

- Use embedding semantic search instead of fuzzy search. Any general embedding model will do better than Levenshtein / fuzzy heuristics.

- leave no form unfilled. use structured outputs and whatever context you have on the user to do a best-effort pre-fill

- allow unstructured inputs on all product surfaces - must accept freeform text and documents. Forms are dead.

- custom finetuning is dead. Stop wasting time on it. Frontier is moving too fast to invest 8 weeks into finetuning. Costs are dropping too quickly for price to matter. Better prompting will take you very far and this will only become more true as instruction following improves

- build evals to make quick model-upgrade decisions. they don’t need to be perfect but at least need to allow you to compare models relative to each other. most decisions become clear on a Pareto cost vs benchmark perf plot

- encourage all engineers to build with ai: build primitives to call models from all code bases / models: structured output, semantic similarity endpoints, sandbox code execution. etc

What else am I missing?