A very interesting blog post by @EliaIsPosting about applying autoresearch to a @GPU_MODE competition, winning 3rd place!
Setup includes an orchestrator + subagents: idea generation, workers to implement the ideas, redteamers to audit the conclusions.
Measuring actual signal and not regular variance is obviously a major challenge, which is highlighted in this blog post. The solution was an escalating evaluation protocol, which brings the comparison floor to ~0.15%.
Link to blog post: https://t.co/l6FiwNpj0s
Today, our CTO @ChrisKitching17 joins @GPU_MODE - one of the sharper corners of the accelerated HPC community - for a live talk on YouTube.
📅 7-8:30PM London / 11AM-12:30PM SF
Streamed via GPU MODE's Discord. If you spend your day in kernels, compilers, or wrestling GPUs from more than one vendor, come watch.
(1/9) I'm thrilled to share the open-source release of Mixture-of-Kittens (MoK), our MoE megakernel for NVL72s! MoK fuses all mixture-of-experts communication and computation into a single, fully deterministic kernel, and powers Composer training across tens of thousands of GPUs.
Joint work with @nash_c_brown, @hmwildermuth, @tmwilliamlin168, and @ellev3n11
Kinda wild that our community got together and collectively made Kimi inference on MI355X faster than B200 across all settings. Congrats to the winners team RadeonFlow and thanks again to AMD for working so closely with our community
I stand on the shoulders of giants who stood on the shoulders of giants. So much of the progress we’ve made has been because of the openness of our great predecessors. Proud to support https://t.co/BjHIUIERtq
Can a model run deeper at test time than it was ever trained to? And if depth becomes a loop instead of a stack, do we need better optimizers to keep training stable?
@coreauto is collaborating with @tilderesearch on an optimizer x architecture competition: "One Layer Deeper"
We picked a cursed problem y = x^(2^T) mod N where each squaring depends on the last. A vanilla transformer has a fixed depth so it falls flat on its face the moment T exceeds the serial compute it can do in one forward pass. A fun reminder that architectures have inductive biases.
This isn't quite a nano-gpt speedrun experience and it's not a typical @GPU_MODE kernel competition either, it's closer in spirit to the LLM efficiency competition I worked on many years ago with @weiwei_msr and @Jisaacso in that it's open ended in a specific way.
The UX is kernelbot like, your submission is a single python file that defines your model, optimizer and loss. You don't need a GPU to participate at all since we're reusing a queue based job system powered by @modal and @northflank
There's tons of unexplored ideas and I'm really not sure what will win out but I'd be particularly excited to see people playing with new optimizers, adaptive depth and depth extrapolation.
Since this is a new format, we'll work with the community to refine the rules for another week before we start for real. Submissions are open now!!
Great results thus far from the 9 participants in the @GPU_MODE competition. 11B tokens and $500 in @modal credits thus far.
Thanks to @damngamerz and everyone else for participating!
Can a model run deeper at test time than it was ever trained to? And if depth becomes a loop instead of a stack, do we need better optimizers to keep training stable?
@coreauto is collaborating with @tilderesearch on an optimizer x architecture competition: "One Layer Deeper"
We picked a cursed problem y = x^(2^T) mod N where each squaring depends on the last. A vanilla transformer has a fixed depth so it falls flat on its face the moment T exceeds the serial compute it can do in one forward pass. A fun reminder that architectures have inductive biases.
This isn't quite a nano-gpt speedrun experience and it's not a typical @GPU_MODE kernel competition either, it's closer in spirit to the LLM efficiency competition I worked on many years ago with @weiwei_msr and @Jisaacso in that it's open ended in a specific way.
The UX is kernelbot like, your submission is a single python file that defines your model, optimizer and loss. You don't need a GPU to participate at all since we're reusing a queue based job system powered by @modal and @northflank
There's tons of unexplored ideas and I'm really not sure what will win out but I'd be particularly excited to see people playing with new optimizers, adaptive depth and depth extrapolation.
Since this is a new format, we'll work with the community to refine the rules for another week before we start for real. Submissions are open now!!
Second problem is now out: dense symmetric eigenproblem A=QΛQT. Solution due on July 15!
We've also enabled ncu profiling for your agents on a @verdacloud cloud box sponsored by our good friends at Brev at @NVIDIAAI
We taught a brand-new mini-series this year at @SCSatCMU on Modern GPU Programming for ML Systems, as part of the ML Systems course, touching on fun questions like what data layout swizzling is, how to use 3D TMA, and state-of-the-art Blackwell programming. We released a curated online book based on the materials: https://t.co/5ZJg2lySNO check it out