Introducing SID-2, a new search model built on compute-scaling swarms.
Our swarms find better results than Opus and Astra, while being 8x faster and 20x lower cost.
By 2030, AI will write an internet's worth of text every 24 hours. Current search pipelines were built on hand-designed rules, adding more won't scale to meet this incoming volume of data.
SID-2 is available on our API today. Try it on your most demanding searches.
Why do we divide by n-1 instead of n when we want to estimate the variance? This is Bessel's correction term, and it made no sense to me back in university when I first learned it.
But I finally understood why!
Got 2 minutes to spare? Here's the explanation!!
We’ve published an update on our recent security incident. See a detailed overview of what happened and our remediation steps here: https://t.co/2lf6lRJPlL
> it took us 4 days and 500$ in tokens
> we beat SGLang by 15%
> it took 6 prompts
> 3 out of those 6 were:
(a) "carry on"
(b) "proceed"
(c) "push to github"
and... we did it from SCRATCH (git init)
context: it has recently become a very clear industry trend to build a custom inference engine for each open source model that comes out... a paradigm made popular by a WAVE of papers that came out earlier this year, like hazy research's megakernels and MetaInfer
the usual argument against traditional inference engines (like vllm / sglang) is that they, like all conventional swe products, use abstractions. the use of abstractions is synonymous with performance overhead (triton vs cuda, python vs c++, etc). combine this with increasing hardware complexity, increasing parallelization strategies, increasing kernel specialization techniques... and you can make a very clear (but very FLAWED) argument for building an engine customized purposely for one model.
why flawed? it is true that traditional inference engines waste cpu time running generic abstractions. sure. but the gpu kernels (the functions run on the gpu) are almost always longer, and so the wasted cpu time is not the bottleneck. if you know your shapes, and run cuda graphs ahead of time (and you almost always do), this is NEVER a problem.
so why would you build your own engine? why are people RAVING about this?
so we tested it out. we built one for DeepSeek V4.1 Flash. The goal was two-fold:
1) is it feasible to do so (spoiler: yes)
2) is it useful to do so (spoiler: also yes...but for different reasons)
we built it in rust (in retrospect, this did not really matter), and it took us 6 prompts and cost 500$ end to end.
unshackled from working within a pre-existing codebase or inheriting code debt, we:
- implemented custom kernels (GEMV, MoE, and AllReduces) specifically for DSV4.1 at our batch size
- we were able to put an entire spec dec step in one graph, and verify across 7 tokens at that
- we were able to mix deepseek's 4 residual streams in 1 fused kernel instead of 80
- fused the routing so that picking and running the expert happened in a single step (saving one write and one read i/o from each expert)
iterating on the model was cheap to do in a custom sandbox-like engine, and we were able to go much faster than a traditional dev cycle. we were then able to port every single thing into sglang, and matched performance
imo, it is this easy iteration that makes this paradigm useful. little else matters.
and, ofc, we've open-sourced the engine and upstream(ed/ing) the patches to @sgl_project
I have a particularly nasty case of cancer. When I was hospitalized again at my lowest point last year, I told Jensen. He immediately reached out to contacts at Stanford, who were able to cut through red tape, and get me into a clinical trial for which I've turned out to be a perfect fit. I think everyone can have their own views about recent events. But any claims that Jensen did not really care about his staff is simply inaccurate. I, personally, owe him more than I can express.
I think the best tribute we can pay to Jensen's legacy is to drive NVIDIA's continued success and push the whole industry forward, by showing we've learned the lessons from years of working here. We will compare ourselves against Speed of Light. The mission will be the boss. And we will not be scared to take smart, bold risks and fail fast with ruthless intellectual honesty.
But if there is one management story I remember the most, it's a particular Jensen grilling from my time in autonomous vehicles that stayed with me. Luckily I was not in the hot seat for this one. Team A didn't want to use Team B's framework, because they thought it was too slow. Jensen was surprised - it's just applying a bunch of optimized CUDA kernels in sequence, how could it be slow?
So he asked the executive from team B to draw the entire data flow sequence from their framework on the whiteboard and highlight every memcpy. Sure enough, there were a bunch of unnecessary memcpy's surrounding each pipeline stage. Everyone in the room could see why it wasn't getting to SoL and a path to improve things. No arguing about fiefdoms or appeals to authority - the data got us all on the same page.
The lesson was not (just) that unnecessary memory copying is bad. It's that executives should not be afraid to probe into details if something doesn't add up, and leaders need to be technical enough and honest enough to engage in real time and not turn every meeting into "I'll get back to you in a week." If that means counting memcpy's with a billionaire CEO, so be it.
My best executive meetings at NVIDIA have been when I'm fully prepared to present every detail of my team's work, and my worst failures have come when I realized I couldn't go deep enough to really solve the problem.
But the thing that gives me confidence is that Jensen was never the only one to live this philosophy. Anyone who has presented to an SVP at NVIDIA knows they are at risk for getting grilled on fundamental constraints and technical details. I have been part of amazing conversations with both recent and old timer leaders where we were able to get to the core of a problem in an hour rather than a month, because we had the right people in the room and were not afraid to go deeper.
The company is filled with supremely talented people who have been trained for 3 decades to work at the highest level, and you can see it in what we produce every day. I'm so proud to be an NVIDIAn, and I can't wait to show off what we ship in the coming months and years.
I have a major result to share.
I’ve found conclusive forensic evidence that the top 4 Chinese AI labs—DeepSeek, Moonshot, MiniMax and Ant Group—have built their newest models on NVIDIA’s most advanced Blackwell chips, which are barred from export to China.
This sounds insane, but I discovered that almost every AI model carries a hidden signature of the GPU that trained it. When a model is first set up, the starting values of its weights are laid out in a pattern that depends on the exact GPU. Each combination of GPU and seed number produces a unique pattern—a sort of fingerprint of the hardware used to create the model. Traces of that pattern survive training into the published model.
Again, this evidence isn’t based on any insider information or leaking. It’s based on a forensic analysis of the model weights released by the companies themselves.
I searched more than six trillion combinations of chip layout and seed number and matched them to open-weights models. In doing so, I found conclusive evidence that six Chinese AI models—including DeepSeek-V4-Pro, Kimi K3, and MiniMax-M3—were initialized on NVIDIA Blackwell systems. Four of the models match Grace Blackwell, the GB200 and GB300 parts in NVIDIA’s flagship liquid-cooled rack-scale systems.
I then verified these results on real hardware. Running unmodified PyTorch with the seeds I’d recovered, a Blackwell B300 reproduced DeepSeek-V4-Flash’s untouched starting weights bit for bit, and a rented GB200 reproduced Kimi K3’s to a correlation of 0.999995. A Hopper H100 reproduced none of DeepSeek-V4-Flash’s, and changing the seed by one broke the match. (DeepSeek’s seed is 953049583 by the way.) The method also was verified to match 24 known open-weights models to the hardware that developers said they used.
It is exceedingly unlikely that these results are a coincidence. Across six trillion combinations, the best accidental match should reach only about 8 sigma (i.e., eight standard deviations above what random chance would produce). The real matches range from 49 to 1,385 sigma, and even the weakest single test hits 20.
These were data-center-scale training runs. With some analysis of release schedules and model details, I estimate that the largest, Kimi K3 and DeepSeek-V4-Pro, used 2,000 to 3,000 Blackwell GPUs each. Clusters of this size cost on the order of $150 to $200 million each to buy, and across these labs alone would add up to more than half a billion dollars of export-controlled GPUs, not counting experimental compute or otherwise. It’s quite likely that we see multi-billion dollar expenditure on export-controlled GPUs.
I am highly confident in this result, and it has been independently verified by senior researchers at OpenAI and Anthropic. You can read the full report at https://t.co/YOtzND0Jqk.
There is much more I’ve identified about these models through other forms of checkpoint forensics like this, and will have more to share soon.