Shashwat

ML Systems & Compilers | GPUs | Grad Research @ ASU

[Shashwat @shashwat023]

Grateful to be a contributor, thanks @willccbb for the opportunity.

Checkout my ifbench environment, modes (train, test , multi turn test)

@PrimeIntellect Environments Hub.

Let me know what you think !

[Prime Intellect @PrimeIntellect]

Introducing the Environments Hub

RL environments are the key bottleneck to the next wave of AI progress, but big labs are locking them down

We built a community platform for crowdsourcing open environments, so anyone can contribute to open-source AGI

[Div Garg @divgarg]

Most models run on whatever inference engine they ship with. That's rarely the fastest option.

We built our own, AGI-RUN.

On a phone it beat every model's own native engine, and ran up to 9.9x faster than the best SOTA framework out there. Same models, we just use the chip better.

[AGI, Inc. @agi_inc]

A full camera pipeline in under 5 million parameters, running entirely on-device.

That's what @NotACyborgYet built. Tomorrow night, he presents it at AGI HQ for Edge Cases #003, our research event for on-device inference, small models, and efficient training.

Tom is CTO at @GlassImaging, did his PhD at Edinburgh, and spent years on the iPhone camera at @Apple including Portrait Mode.

His talk: neural image signal processing, where a single network replaces the entire pipeline of separate algorithms and outperforms them on-device.

Fri Jun 26, 5:30 PM PT. 45-min talk, off-the-record Q&A, dinner after. Researchers and students. A few seats left.

[Shashwat @shashwat023]

Those Leap motion days

[th0rgal @Th0rgal_]

Just reverse engineered my Oura Ring 5 so I can control my computer like a wizzard. @ouraring please send my love to whoever buried a feature to stream live accelerometer data

[Div Garg @divgarg]

Up to 96% lower AI inference latency on @Samsung's and @SKhynix's chips, with zero hardware changes. All of it came from the compiler.

That's new research from @AdamA7741, who built it during his PhD with Prof. Aviral Shrivastava at MPS Lab at @ASU and is now on our on-device team.

AI chips spend a surprising amount of their time and power moving data around, not doing compute. The industry's most promising fix is processing-near-memory (PNM): compute right where the data sits. @Qualcomm, @Samsung, @AnkerOfficial and @SKhynix are all building these chips. First ones ships this year.

But hardware is only half the story. You also need a compiler, the software that translates an AI model into instructions the chip can run. Today every PNM compiler is tuned by hand for one chip and one task. Change either, and performance collapses.

Adam's compiler, DPC, goes after exactly that: one compiler that works across any of these chips. It beat both vendors' own hand-tuned compilers on the architectures they were built for.

We’re building off this approach at @agi_inc. Our agents run on phones, cars, and wearables, and on small devices memory and moving data are the main constraints.

So we optimize the model's workload and memory use first, because those gains transfer from chip to chip. Cost models decide how the model maps to each device. Hardware-specific tuning comes last.

Same playbook as DPC, running on today's silicon. It's how our models run well across very different chips, and they can run on a phone locally handling tasks most people assume would need a desktop.

The silicon is coming. The software is the race.

[Shashwat @shashwat023]

good read

[Vlad Feinberg @FeinbergVlad]

How to land a job at a frontier lab

https://t.co/oHIqLgBMbC

[Shashwat @shashwat023]

@eliebakouch @kylelostat 😂

[Andrej Karpathy @karpathy]

New post: nanochat miniseries v1

The correct way to think about LLMs is that you are not optimizing for a single specific model but for a family models controlled by a single dial (the compute you wish to spend) to achieve monotonically better results. This allows you to do careful science of scaling laws and ultimately this is what gives you the confidence that when you pay for "the big run", the extrapolation will work and your money will be well spent. For the first public release of nanochat my focus was on end-to-end pipeline that runs the whole LLM pipeline with all of its stages. Now after YOLOing a few runs earlier, I'm coming back around to flesh out some of the parts that I sped through, starting of course with pretraining, which is both computationally heavy and critical as the foundation of intelligence and knowledge in these models.

After locally tuning some of the hyperparameters, I swept out a number of models fixing the FLOPs budget. (For every FLOPs target you can train a small model a long time, or a big model for a short time.) It turns out that nanochat obeys very nice scaling laws, basically reproducing the Chinchilla paper plots:

Which is just a baby version of this plot from Chinchilla:
Very importantly and encouragingly, the exponent on N (parameters) and D (tokens) is equal at ~=0.5, so just like Chinchilla we get a single (compute-independent) constant that relates the model size to token training horizons. In Chinchilla, this was measured to be 20. In nanochat it seems to be 8!

Once we can train compute optimal models, I swept out a miniseries from d10 to d20, which are nanochat sizes that can do 2**19 ~= 0.5M batch sizes on 8XH100 node without gradient accumulation. We get pretty, non-itersecting training plots for each model size.

Then the fun part is relating this miniseries v1 to the GPT-2 and GPT-3 miniseries so that we know we're on the right track. Validation loss has many issues and is not comparable, so instead I use the CORE score (from DCLM paper). I calculated it for GPT-2 and estimated it for GPT-3, which allows us to finally put nanochat nicely and on the same scale:
The total cost of this miniseries is only ~$100 (~4 hours on 8XH100). These experiments give us confidence that everything is working fairly nicely and that if we pay more (turn the dial), we get increasingly better models.

TLDR: we can train compute optimal miniseries and relate them to GPT-2/3 via objective CORE scores, but further improvements are desirable and needed. E.g., matching GPT-2 currently needs ~$500, but imo should be possible to do <$100 with more work.

Full post with a lot more detail is here:
https://t.co/na8zVLqWLf
And all of the tuning and code is pushed to master and people can reproduce these with scaling_laws .sh and miniseries .sh bash scripts.

[Edward Z. Yang @ezyang]

New blog: The gap between a Helpful Assistant and a Senior Engineer https://t.co/JJA6VyR2ni

[Jeff Dean @JeffDean]

Performance Hints

Over the years, my colleague Sanjay Ghemawat and I have done a fair bit of diving into performance tuning of various pieces of code. We wrote an internal Performance Hints document a couple of years ago as a way of identifying some general principles and we've recently published a version of it externally.

We'd love any feedback you might have!

Read the full doc at: https://t.co/jej95g236P

[Shashwat @shashwat023]

Normal reddit comments

[Shashwat @shashwat023]

🤣🤣🤣🤣🤣

[Pedro Domingos @pmddomingos]

Announcing the Schmidhuber Award, which recognizes the most impactful research in AI and will be given every year to Juergen Schmidhuber.

[Shashwat @shashwat023]

Heading to LLVM Dev Meeting tomorrow. Would love meetup with people working on heterogeneous compilation, and working on AI workloads compilation.

[Shashwat @shashwat023]

@karpathy @nathanrs Conway's game of life

[Shashwat @shashwat023]

@excalidraw has set the beauty standard for diagrams

[Shashwat @shashwat023]

I used to think compiler research was too “niche” like, how many jobs even exist for compiler engineers?

But after diving into papers and projects, I realized it’s more than that, it’s a mindset.

You start thinking beyond algorithmic optimality : cache behavior, op counts, hardware-specific tradeoffs.

Even if you don’t work on compilers directly, you gain a deeper way of writing and reasoning about code. This perspective is invaluable across almost every tech domain. As compute grows and workloads diversify, compiler challenges only become more critical.
It’s less niche than it seems, it’s foundational.

[george hotz archive @geohotarchive]

AI Coding https://t.co/gZIH163F48

[Shashwat @shashwat023]

After reading @gordic_aleksa’s blog on vLLM (with lovely visuals, much like @Si_Boehm’s), the OS parallels really stand out. PagedAttention from virtual memory paging, and scheduling from process management.

And of course, @karpathy’s idea of an LLM OS lingers in the back of my mind. All I can think is this space has to evolve far beyond just the chat interface.