Joe Malone

ML Frameworks @tenstorrent | Previously compilers @Apple and @MathWorks

[Joe Malone @averagemalone]

time to see what all the hype is about

[Joe Malone @averagemalone]

Really enjoying Inference Engineering by @philipkiely so far!
The disaggregated inference section says you should use lower TP for prefill than decode though, which seems backwards

[Joe Malone @averagemalone]

One of the easiest on-ramps to test programming TT HW - can’t beat free!

[Artem Y @artem_aero]

Run @tenstorrent kernels on your CPU.

ttsim is a bit-exact, full-system simulator of Wormhole and Blackhole. The whole tt-metal stack on Linux, Mac (UTM/QEMU) or Windows (WSL2) — numerics match silicon bit-for-bit.

Bring up kernels of explore the architecture. No silicon required. First of its kind. Now Open Source
https://t.co/AtcZ9L9e6T

[Joe Malone @averagemalone]

one more DSL will fix me

[Joe Malone @averagemalone]

Duality of man:
“It’s not bit-accurate because varied float op ordering so it’s unusable!”

vs

“We use DiLoCo with partial layer updates and it more or less converges, great!”

[Joe Malone @averagemalone]

worlds fastest -> worlds most fun to play with

[Kevin Mi @kevinmi920]

Introducing Infinite Studio ♾.

Last week, @tenstorrent x @prodia announced the fastest Wan 2.2 video generation in the world.

We built a demo to show what that speed unlocks: directing an infinite movie in real time.

Demo 👇

[Artem Y @artem_aero]

Tenstorrent can run AI magically fast.
Try it yourself.
https://t.co/IVyaTcsjCe

- Super fast large LLMs. DS R1 685B model flying at 350t/s/u on 16 Blackhole galaxies
- Super fast video generation with Wan 2.2 Lighting (Prodia) finishing 5 seconds clip within 3 seconds on 4 galaxies.

I know you want more. You’ll get it fast. The team is cooking more.

[Joe Malone @averagemalone]

just keep scaling
just keep scaling

[Jim Keller @jimkxa]

SuperCluster 36 up and running. 4 Galaxy all to all in a torus. 9 Quads all to all connected. Looks like one computer to software.
More silicon, faster computer

[Mikhail Avady @mavady]

Just became fastest in the world with video by 10x, faster than realtime

[Joe Malone @averagemalone]

@mavady faster than realtime 👀

[Monty Anderson @monty10x]

3.5s wan 2.2 @prodialabs x @tenstorrent

[Igor @igorchovpan]

[1/5] Got a working TRL pipeline that makes a tinyllama model solve math questions.

gsm8k test accuracy: 34.65%

Pipeline: tinyllama-v1.1b-math-code + NuminaMath CPT (2 epochs) + GSM8K Format Priming + GRPO (1 epoch).

[Joe Malone @averagemalone]

Flaws in something you buy are frustrating
Flaws in something homemade are just part of the story

[Mikhail Avady @mavady]

News is out @tenstorrent + @prodialabs made the worlds fastest video generation

[Joe Malone @averagemalone]

Pretty pumped for these demos to go live

[Sally Ward-Foxton @sallywf]

.@tenstorrent will launch its new gen cluster-scale systems next week, but in the mean time, Jim Keller @jimkxa and Jasmina Vasiljevic @JasminaVas gave me a sneak preview of their video generation demo running on 256 BlackHole chips (spoiler - it's fast):
https://t.co/7oVYauf0VE

[Joe Malone @averagemalone]

Maybe I’ve been reading too much @simonsarris, but I’m thinking it will be a Year of Beauty

[Tenstorrent @tenstorrent]

Run anything, anywhere. On May 1st, see Tenstorrent’s solutions for yourself, deployed at scale.

Watch TT-Deploy live at https://t.co/T64mcrcRhU

[Joe Malone @averagemalone]

Imagine the high of leaving a nit on Zuck’s PR

[ludwig @ludwigABAP]

thanks man

[Andrej Karpathy @karpathy]

Three days ago I left autoresearch tuning nanochat for ~2 days on depth=12 model. It found ~20 changes that improved the validation loss. I tested these changes yesterday and all of them were additive and transferred to larger (depth=24) models. Stacking up all of these changes, today I measured that the leaderboard's "Time to GPT-2" drops from 2.02 hours to 1.80 hours (~11% improvement), this will be the new leaderboard entry. So yes, these are real improvements and they make an actual difference. I am mildly surprised that my very first naive attempt already worked this well on top of what I thought was already a fairly manually well-tuned project.

This is a first for me because I am very used to doing the iterative optimization of neural network training manually. You come up with ideas, you implement them, you check if they work (better validation loss), you come up with new ideas based on that, you read some papers for inspiration, etc etc. This is the bread and butter of what I do daily for 2 decades. Seeing the agent do this entire workflow end-to-end and all by itself as it worked through approx. 700 changes autonomously is wild. It really looked at the sequence of results of experiments and used that to plan the next ones. It's not novel, ground-breaking "research" (yet), but all the adjustments are "real", I didn't find them manually previously, and they stack up and actually improved nanochat. Among the bigger things e.g.:

- It noticed an oversight that my parameterless QKnorm didn't have a scaler multiplier attached, so my attention was too diffuse. The agent found multipliers to sharpen it, pointing to future work.
- It found that the Value Embeddings really like regularization and I wasn't applying any (oops).
- It found that my banded attention was too conservative (i forgot to tune it).
- It found that AdamW betas were all messed up.
- It tuned the weight decay schedule.
- It tuned the network initialization.

This is on top of all the tuning I've already done over a good amount of time. The exact commit is here, from this "round 1" of autoresearch. I am going to kick off "round 2", and in parallel I am looking at how multiple agents can collaborate to unlock parallelism.
https://t.co/WAz8aIztKT

All LLM frontier labs will do this. It's the final boss battle. It's a lot more complex at scale of course - you don't just have a single train. py file to tune. But doing it is "just engineering" and it's going to work. You spin up a swarm of agents, you have them collaborate to tune smaller models, you promote the most promising ideas to increasingly larger scales, and humans (optionally) contribute on the edges.

And more generally, *any* metric you care about that is reasonably efficient to evaluate (or that has more efficient proxy metrics such as training a smaller network) can be autoresearched by an agent swarm. It's worth thinking about whether your problem falls into this bucket too.