for almost a couple of decades, this large software engineering industry put food on the table for millions of people.
sadly with the advent of claude code and codex this large monolithic machinery has hyper optimized itself to remove the most unoptimized cogs in it
Claude opus and fable are getting insanely great at software engineering to the point where its actually scary.
literally anybody can write a major component in IT that would have taken millions in engineering costs.
> it took us 4 days and 500$ in tokens
> we beat SGLang by 15%
> it took 6 prompts
> 3 out of those 6 were:
(a) "carry on"
(b) "proceed"
(c) "push to github"
and... we did it from SCRATCH (git init)
context: it has recently become a very clear industry trend to build a custom inference engine for each open source model that comes out... a paradigm made popular by a WAVE of papers that came out earlier this year, like hazy research's megakernels and MetaInfer
the usual argument against traditional inference engines (like vllm / sglang) is that they, like all conventional swe products, use abstractions. the use of abstractions is synonymous with performance overhead (triton vs cuda, python vs c++, etc). combine this with increasing hardware complexity, increasing parallelization strategies, increasing kernel specialization techniques... and you can make a very clear (but very FLAWED) argument for building an engine customized purposely for one model.
why flawed? it is true that traditional inference engines waste cpu time running generic abstractions. sure. but the gpu kernels (the functions run on the gpu) are almost always longer, and so the wasted cpu time is not the bottleneck. if you know your shapes, and run cuda graphs ahead of time (and you almost always do), this is NEVER a problem.
so why would you build your own engine? why are people RAVING about this?
so we tested it out. we built one for DeepSeek V4.1 Flash. The goal was two-fold:
1) is it feasible to do so (spoiler: yes)
2) is it useful to do so (spoiler: also yes...but for different reasons)
we built it in rust (in retrospect, this did not really matter), and it took us 6 prompts and cost 500$ end to end.
unshackled from working within a pre-existing codebase or inheriting code debt, we:
- implemented custom kernels (GEMV, MoE, and AllReduces) specifically for DSV4.1 at our batch size
- we were able to put an entire spec dec step in one graph, and verify across 7 tokens at that
- we were able to mix deepseek's 4 residual streams in 1 fused kernel instead of 80
- fused the routing so that picking and running the expert happened in a single step (saving one write and one read i/o from each expert)
iterating on the model was cheap to do in a custom sandbox-like engine, and we were able to go much faster than a traditional dev cycle. we were then able to port every single thing into sglang, and matched performance
imo, it is this easy iteration that makes this paradigm useful. little else matters.
and, ofc, we've open-sourced the engine and upstream(ed/ing) the patches to @sgl_project
Those who have viewed Lex's vlog featuring Jensen in Taiwan would undoubtedly recognize the profound humility, gratitude, and generosity Jensen consistently shows toward others. It is deeply painful and disheartening for me to see him depart.
NVIDIA letting Jensen go is a huge mistake, for America and for humanity. It's Apple firing Steve Jobs all over again, but much bigger, given the role of AI in defining the future of our world.
I've been keeping quiet on this out of respect for Jensen's family and team, who I've met and who are amazing humans, and because I personally hate infidelity. So my heart goes out to them in this difficult personal family moment.
But I can also say that Jensen is human, and makes mistakes like all of us do. Not that I matter at all, but I've known Jensen for a long time and have gotten the chance to spend a lot of time with him, behind the scenes, away from mics and cameras. I have seen him show kindness, humility, care to everyone around him, including strangers, including from the poorest most struggling parts of society, treating all the same with the same thoughtfulness and attention, because they are fellow humans and not because he has something to gain from them. That shows rare character, to maintain humility and humanity through all of it.
Another data point is what people who know him well have said privately to me about Jensen over the years. I've never had anyone tell me anything but how awesome he is or how much they love him. Probably over a thousand people at this point. This includes hundreds of engineers, CEOs, colleagues, friends, etc.
Also, since there was some spy talk. Jensen is as proud an American as it gets. I'm in shock frankly at all of this, including the mass hysteria around it. I love America, and I think NVIDIA letting Jensen go is terrible for America.
I stand with Jensen.
Love you all โค๏ธ
I used Gemini 4 Carbon to get Windows 11 running on my iPhone Duo! ๐ป
Unfold the screen and it's a pocket PC, including Antigravity running coding agents on my phone.
This is the actual desktop ARM OS running locally with touch controls.
Demo + repo below!