Eric Auld

Performance and kernels at Together AI.

What's next?

[Eric Auld @ericauld]

"The easiest person to fool is yourself."

[Eric Auld @ericauld]

@thsottiaux Performance.

For me the features are great already now that connections is out. And it’s been quite reliable, a couple bugs aside

[Eric Auld @ericauld]

@thsottiaux (Probably in development stuff):

- Seamless interaction with threads via iPhone
- Interact with remote dev instances, ability to migrate workflows there if I want to close my laptop (connections is not released for me yet)

Love Codex!

[Sebastian Raschka @rasbt]

Added lots of improvements to the LLM Architecture Gallery in the last 2 weeks.

Imho the coolest one yet: A diff tool many of you were asking for!

https://t.co/NO7z6XSRHS

[Adam Aaronson @aaaronson]

Word with the lowest standard deviation of letter position in the alphabet, for each length

[Eric Auld @ericauld]

@yaroslavvb What will they use?

[Eric Auld @ericauld]

It seems you’re trying to have it both ways by calling it “a pattern” but avoiding its having physical causal effects…aren’t such effects part of how we get to call it “a pattern”?

So suppose we have a brain-scanning machine with a light bulb that lights up when pattern X is detected and not otherwise. Is that not causal enough?

[Eric Auld @ericauld]

@yaroslavvb What browser / extension is this?

[Séb Krier @sebkrier]

Yes, I've been saying this for a while now. See for example https://t.co/4dkPks2zls and Danzig's work here: https://t.co/bQrsrIwK0k

I don't think the predominant narrative of AI as a singular entity, a Sand God, a discrete moment in time, or a 'separate species' (as Tegmark puts it) is correct or helpful. As Danzig argues, AI is indeed "alien," but only in the same way a stock market or the DMV is alien: they are all reductionist, correlative intelligences.

They strip the world of context, reducing reality to standardized inputs like prices or tokens to process information at scales humans cannot. To me at least, this shared "alien" nature normalizes AI as the latest evolution in a lineage of artificial processors we’ve lived with for centuries.

So instead of a unitary being or species, AGI should be understood as a collection of complex systems, models, and products that functions similarly to (and integrates with) existing human macro-systems. An amplifier for the bureaucracies and markets that already govern us, not a discrete 'biological-style' agent. Its governance is a continuous sociopolitical struggle (insert always has been meme) that is shaped by many different forces, not a one-time mathematical proof of safety before a launch.

Relatedly, I feel like the current discourse also has a blind spot for the 'demand' side. We obsess over the supply (R&D, model scaling, 'the AGI') as if these systems are created in a vacuum. I think this is how people end up with scenarios where AGIs are just doing things for their own sake, completely detached from human preferences (who are usually described as 'disempowered').

But they aren't; they are pulled and shaped by downstream demand, cost constraints, and efficiency needs. This economic reality has implications for how the technology develops. See also Drexler's CAIS model (https://t.co/aKP11EgjCS) - Drexler anticipated much of this and the core intuitions remain true, even if slightly out of date. You won’t see one omniscient agent, but a proliferation of specialized systems, models of varying sizes, and distinct products rising in parallel because that is what is economically viable.

This is why the AGI governance conversation often feels so confused. If you view AGI as a singular biological entity, you make two mistakes: safetyists project human-like 'intent' where they should be looking at incentives, and policymakers reach for a singular 'FDA' when instead they need to look into different different markets, sectors, products etc.

You can’t have a single regulator or discrete safety rules for 'The Economy' or 'The Bureaucracy,' and you won't be able to have one for 'Intelligence' either. Models still matter of course - none of this means you shouldn't test, evaluate, and understand them better - but I think we overindex on this frame a bit. And as Dean says, none of this is to downplay concerns and risks: but I do think it has implications for how to understand and address them.

[Eric Auld @ericauld]

Totally, just instruct the agent that this is what you want. Codex 5-high is by far the most capable of conceptual discussion in my observation.

With Codex you can also switch in midstream to just talking to ordinary GPT-5-Pro (ie not specifically a coding model) which can be good if you’re discussing concepts

[Eric Auld @ericauld]

@NoahEpstein_ @srush_nlp Composer

[Aleksa Gordić (水平问题) @gordic_aleksa]

New in-depth blog post time: "Inside NVIDIA GPUs: Anatomy of high performance matmul kernels". If you want to deeply understand how one writes state of the art matmul kernels in CUDA read along.

(Remember matmul is the single most important operation that transformers execute both during training and inference. Most of NVIDIA compute is spent on it. Gaining 1% in efficiency translates to massive savings in the order of many nuclear reactors :P)

I, yet again, realized i underestimated the effort. 😅 Here is one more booklet (lol). 47 figures!

I covered:

* The fundamentals of the GPU architecture with an emphasis on the memory hierarchy, building mental models for GMEM, SMEM, and L1/L2, and then connecting them to the CUDA programming model. Along the way we also looked at the "speed of light," how it's bounded by power, with hardware reality leaking into our model.

* PTX/SASS, and how to steer the compiler into generating what we actually want (is that loop being unrolled, are we using vectorized loads like LDG.128, etc.). I've annotated one PTX/SASS example for a simple matmul kernel in excruciating detail. Even if you're new to compilers you should find this useful.

(i actually found various inefficiencies in both compilers - fun!)

* Many core concepts such as tile/wave quantization, occupancy, ILP (instruction-level parallelism), roofline model, etc. Also building intuition around fundamental equivalences: dot product as a sum of partial outer products, why square tiles are the right shape for high arithmetic intensity, etc.

* The warp tiling method - which is near SOTA assuming you can't use tensor cores, TMA, async mem instructions, and bf16. Just maximizing GPU's performance using nothing but CUDA cores, registers and shared memory.

* Finally, we step into Hopper (H100): TMA, swizzling, tensor cores and the wgmma instruction, async load/store pipelines, scheduling policies like Hilbert curves, clusters with TMA multicast, faster PTX barriers, and more.

As always lots of examples, lots of visuals. This is the first time i could see warp tiling kernel and be like "oh i get it completely". I just needed my mental image transformed into an actual image.

A few years ago I was really inspired by @Si_Boehm's excellent blog post on how matmul works, but I also found it had several errors, some unclear explanations, and it was quite outdated. Building on @pranjalssh amazing work (who did a great job building sota kernels for H100) and my own research, this is the final result.

---

Again a huge thank you to @Hyperstackcloud (GPU cloud) for giving me an H100 (PCIe) node to run some of the experiments and analysis that i needed to write this up.

Also a big thank you to my friends Aroun (who did a very thorough review of the post; Aroun's doing cool GPU/AI stuff at Magic and was previously GPU architect at Apple and Imagine, he's one of the best GPU people i know and we worked together on llm.c w/ @karpathy) and the amazing @marksaroufim! (PyTorch) for taking the time during weekend when they didn't have to. :)

[Noah Smith 🐇🇺🇸🇺🇦🇹🇼 @Noahpinion]

According to the IMF, world GDP per capita (PPP) is now $25,591.

That's pretty incredible. That's about where the United States was in 1972.

What a victory for humanity.

[Eric Auld @ericauld]

@_divyansh1910 Seems right. It’s not a good mainstream strategy, it’s a niche path. True of academic math also btw

[Eric Auld @ericauld]

A post about my transition from academic math to GPU programming for ML, and an offer to mentor people on a similar path.

https://t.co/p2vzstUZle

[Eric Auld @ericauld]

@BasidMLE 0

[Eric Auld @ericauld]

Good episode https://t.co/rdCUFjvpIH

[Eric Auld @ericauld]

Excellent from ⁦@jenniferdoleac⁩ https://t.co/kDzhSkyFle

[Eric Auld @ericauld]

@mayankjha1089 What’s the complaint?