Joel Fernandes 🐧🇺🇸

My thoughts on work, open source and occasional rants about world affairs. When in doubt, turn it inside out. Also on Mastodon: @[email protected]

[Joel Fernandes 🐧🇺🇸 @joel_linux]

Sneak peak at a slide from my RCU talk at
@KernelRecipes . This "Life cycle of a grace period" block diagram serves as a reference for me too now ;-)

[Andrej Karpathy @karpathy]

We'll be spending a lot more time trying to understand the outputs of language models. A few thoughts, tips & tricks:

Writing. Something I've had success with: Ask your LLM to explain something in ASD-STE100, it's a controlled language specification originally developed for aerospace maintenance documentation. LLMs well-versed in this language and it comes with heavy constraints on clean writing style that I often find a lot more readable. Sometimes I've tried to soften it a bit e.g. ask for "80% of the way to ASD-STE100" because the spec is quite stringent. But even better:

Diagrams / images. Instead of writing, ask your LLM to create a diagram. These can be a lot easier to process, parse, and understand. But even better:

Web pages. Ask for output "in HTML" to get a beautiful, interactive webpage. LLMs are getting really good at frontend and can create beautiful experiences, animations, etc. But even better:

Explainer videos. The output format I am most bullish on is fully custom / bespoke explainer videos generated on any arbitrary topic. Experiment with things like "Create a 3b1b style video explainer on X. Use my ElevenLabs API key for audio narration". (you'd need an API key for the latter or you can ask your LLM to find you decent free alternatives that use your local compute). This is actually starting to work!

In summary:
- As LLMs get better, they will do more and more of the legwork autonomously, and a lot more of our work will rise up the abstractions into oversight and understanding.
- Luckily, LLMs can help here too because as intelligence and code are increasingly abundant, you can ask for large, custom, discardable software artifacts (e.g. web apps, video explainers) that would have never made sense to create before. Push the boundaries here and you'll be surprised.

[Joel Fernandes 🐧🇺🇸 @joel_linux]

I built an interactive simulator to show how tiled matrix multiplication reduces global memory traffic compared with a naive approach. You can change tile size and watch how reusing matrix elements changes the number of memory loads. https://t.co/oPjvtj7SqG

[jietang @jietang]

Thoughts About Scaling Law

Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter count is only meaningful alongside three others — how much data you have, where you intend to spend your compute, and who will run the model, under what conditions.

The field learned this the hard way. Kaplan et al. (2020) fit an exponent that told everyone to grow parameters faster than data — roughly 2.7:1 — and the industry complied: GPT-3, Gopher, MT-NLG. Hoffmann et al. (2022) redid the experiment across four hundred models and found the compute-optimal split is closer to 20 tokens per parameter, and that with sufficient compute the two should grow at the same rate rather than drifting apart. The error in the earlier fit compounded with every order of magnitude of compute, which is why the largest models of that generation were the most misallocated. The trillion-parameter round was, in retrospect, a detour the whole field took together and then reversed.

Chinchilla wasn't the end either. It optimized training compute for models that would be trained once and evaluated. Today a model is called billions of times a day and inference dominates lifetime cost. Put inference into the objective and the optimum moves toward smaller models trained far longer — deliberate over-training, which is what Llama-2-7B and Gemma-2-9B were doing at roughly 290 and 889 tokens per parameter.

Sparsity moved the target again. In a MoE model two quantities have to be kept apart: total parameters govern roughly how much the model can hold — knowledge, facts, the long tail — while activated parameters and effective depth govern roughly how far it can think, how many steps of a causal chain it can carry before it comes apart. A dense 20:1 ratio does not transfer. And the ratio isn't a single number at all: Roberts et al. (2025) find the optimal tokens-per-parameter is task-dependent, with memorization favoring more parameters and reasoning favoring more data. Follow-up work on MoE observes that at fixed TPP, pushing total parameters higher actually degrades reasoning, while activating more experts reliably helps it.

This matters for what we are building toward. Finding a vulnerability is not a retrieval problem. It doesn't come from having memorized more CVEs; it comes from carrying a twenty-step chain of inference to the end without losing the thread. That capability does not live in total parameter count.

Which brings us to this release. Total parameters appear to matter up to a threshold — enough to hold the world — after which additional capability comes from scaling elsewhere: effective depth per forward pass, and above all post-training. GLM-5.3 is our controlled experiment on that claim. Same base, same architecture, same total and activated parameters as GLM-5.2. One month of scaling long-horizon environments and RL. The gains are not marginal. Well, scaling has more than one dial. We turned the post-training one this time because it had the most slack left in it — not because the others are finished. Base model size, pretraining data, compute spent per forward pass: all of them are still on the table, and we will come back to each. What this experiment taught us is that the dials do not have to be turned together, and that the one worth turning next is rarely the one that was worth turning last. We are not done scaling. Next time, maybe mid-training, pre-training, and even more.

[Joel Fernandes 🐧🇺🇸 @joel_linux]

hey @karpathy, thanks a lot for your NN lecture videos. I added positional encoding to the part 2 video to see how the loss responds and it was better! & most words terminate properly: https://t.co/3vJ2mkBy2c

Looking forward to watching the part 3 lecture when I get a chance.

[Linux Plumbers Conference 2026 @linuxplumbers]

CFP for LPC 2026 is open!

Important dates:
Thursday, April 23, 2026: Deadline to submit proposals to host a microconference
Sunday, June 28, 2026: Deadline to submit LPC Refereed Track Presentations Proposals and Kernel Summit Presentations Proposals.
https://t.co/o2ULatesd9

[Joel Fernandes 🐧🇺🇸 @joel_linux]

Rust in the Linux kernel coexists with C. I wrote an article on a new Rust abstraction I created to support iterfacing with C linked lists, enjoy :) https://t.co/pKjqM9Huwc

[Deedy @deedydas]

Every single one of the 11 Meta superintelligence hires is an immigrant who did their undergrad abroad.

7 China, 1 India, 1 Australia, 1 UK, 1 South Africa.
8 are PhD or PhD dropouts in the US.

Immigration is key to US AI innovation.

[Joel Fernandes 🐧🇺🇸 @joel_linux]

A bit late to the party, but I really think #GitLab is a very nice alternative to #GitHub. The user interface feels a lot simpler and intuitive, and the free repository limit of 10GB/repo is 5x that of GitHub. A kernel repo is >2GB with history. I’ll continue to use both though.

[Kadi🇪🇪🌻 @TheFl0orIsLaVa]

Elon's week be like...

[Joel Fernandes 🐧🇺🇸 @joel_linux]

I am wondering why #LLM coding benchmarks like #Aider Polyglot reveal the coding problems they are testing on. Doesn’t that give the models an opportunity to be trained on the problems in advance, thus defeating the scoring?

https://t.co/UBgKS9eiJo

https://t.co/hDGwJsHc93

[Paul E. McKenney @paulmckrcu]

It is my honor to be designated as Godfather of the 2025 Kernel Recipes. The conference organizers asked me to help increase the speaking skills of the presenters. This blog series is a first step in that direction: https://t.co/mYj5SCHyA6

[Joel Fernandes 🐧🇺🇸 @joel_linux]

This is safety beyond the usual unsafe memory access protections and is more than welcomed and fascinating! Especially since issues are caught much earlier at compile time! 2/

[Joel Fernandes 🐧🇺🇸 @joel_linux]

I’m fascinated that rust lang couples synchronization primitive types with structures they protect thus allowing for common programming mistakes to be caught at compile time! (ex, dropping a lock too early or holding wrong lock like rwlock in read mode during mutations). 1/

[Joel Fernandes 🐧🇺🇸 @joel_linux]

@paulmckrcu You do like to put a lot of guardrails in code you write, for good measure! ;-)

[Barack Obama @BarackObama]

Here's our statement on the results of the 2024 presidential election:

[Joel Fernandes 🐧🇺🇸 @joel_linux]

AI video gen is heating up. This is going to disrupt commercial video advertising AFAICT. Multiple ready to use tools now exist: Kling ai, Flair ai, runway ML etc

[Flair AI @flairAI_]

Welcome to the new age of AI advertising.

We generated a commercial for @elonmusk's Optimus Bot with Flair AI.

Try out our Optimus model and animation workflow below. 👇

[Joel Fernandes 🐧🇺🇸 @joel_linux]

https://t.co/RCsqQXBnNE

OpenSSL adds a number of new encryption algorithms, support for "“directly fetched composite signature algorithms such as RSA-SHA2-256”"

[Joel Fernandes 🐧🇺🇸 @joel_linux]

@derkling Further I think if you look deeper into it, Trump has been a strong Israeli supporter. So that should be another reason to not elect him if that was your reason (other than him being a convicted felon and racist)

[Joel Fernandes 🐧🇺🇸 @joel_linux]

@derkling The issue you are describing has nothing much to do with my initial tweet though. I’m talking about presidential election. Funding for Israel etc is approved by congress. You are talking about the US position on this issue which the election doesn’t have much direct bearing on.

[Ethan Mollick @emollick]

The ourouborous has begun.

Wikipedia is an important source of training data for AIs. At least 5% of new Wikipedia articles in August were AI generated

(To be clear, this does not mean that AI will fail as it trains on its own data, synthetic data is already a part of training)