Seph Mard

Gen AI Products @NVIDIA ⚡️ Previously @weights_biases, @DataRobot, @Ally.

Aspiring Cyclist 🚴 | @DukeMBB fan 🏀 😈 | 🗣️ My thoughts ≠ my employer’s.

[Seph Mard @SephMard]

Jensen’s CES keynote highlighted evaluation as foundational to frontier AI.

NVIDIA Nemotron is a leading open frontier model, and open models require open evaluation. Open evaluation should be the standard - not the exception.
My team at @nvidia built NeMo Evaluator for Nemotron development and is now releasing it to the community.

[Seph Mard @SephMard]

🔗 Technical Blog: https://t.co/jqqM0ryC8H

[Seph Mard @SephMard]

Nemotron 3 Super is live! 🚀

As part of the Super 3 release, we used NVIDIA NeMo Evaluator to run end-to-end evaluation across the entire model development phases, from long-context reasoning to multi-step agent benchmarks. Every reported result is backed by a reproducible evaluation flow.

What this means in practice:
1️⃣ End-to-end evaluation with NeMo Evaluator across agentic and reasoning benchmarks
2️⃣ A fully released evaluation recipe so anyone can rerun, inspect, and validate the pipeline
3️⃣ Open weights, datasets, and training + eval infrastructure for complete transparency

Reproducibility is not an afterthought. It's central to Nemotron's commitment to openness! The full evaluation recipe for Nemotron 3 Super is available with the release, so developers can replicate results or adapt the pipeline to their own models.

Links in thread 👇

[Seph Mard @SephMard]

🔗 NeMo Evaluator Recipe: https://t.co/mE1czuEuVw

[Seph Mard @SephMard]

@llm_wizard Let the bots rip! 🦞

[Seph Mard @SephMard]

Time to let the bots rip 🤖 🦞

[Seph Mard @SephMard]

@xeophon @lexfridman @rasbt @natolambert Tangentially related is our groups recent work on data contamination. @MiZawalski has some interesting takeaways.

https://t.co/lVWE1vMlLd

[Seph Mard @SephMard]

Proud of this team and excited to see this work accepted to ICLR 2026 🎉

CoDeC is a clean, practical approach to detecting data contamination. No training, no reference models, just two forward passes with impressive accuracy.

Great example of impactful research on evaluation through NeMo Evaluator.

Huge congrats to the authors 👏

#ICLR2026 #Evaluation #AIResearch #NVIDIA

[Seph Mard @SephMard]

The "New" Frontier of AI Evaluation 🚀

I’ve been diving into the latest discussion between @lexfridman, @rasbt, and @natolambert. It's exciting to hear how critical evaluation is to expanded the frontier of AI. For anyone building in the evaluation space, the shift is clear: we are moving from "vibes" to verifiable truth.

My key take aways 👇

🔹 RLVR is the New RLHF: The biggest shift in 2025 is Reinforcement Learning with Verifiable Rewards (RLVR). Instead of subjective human preferences, we are scaling models against binary, verifiable truths—math answers, code execution, and logic chains.

🔹 The Contamination Crisis: Static benchmarks are breaking. With models potentially "training on the test," the gold standard is shifting toward dynamic, private evaluations and data created after training cutoffs to ensure true reasoning, not memorization.

🔹 Eval-as-a-Moat: In a world of massive compute, the highest leverage point for researchers isn't more GPUs—it's building better rubrics. Defining a representative problem set that frontier models fail on is now a "career rocket ship."

🔹 Beyond Code Snippets: The next hurdle for evaluation is moving past standalone functions and into "distributed systems" and complex architectures where ground truth is harder to define.

[Seph Mard @SephMard]

@xeophon @lexfridman @rasbt @natolambert Great point @xeophon . "Breaking" may be too harsh. But they are continuing to saturate quickly and its clear data contamination is a growing issue.

[Seph Mard @SephMard]

State of AI in 2026: LLMs, Coding, Scaling Laws, China, Agents, GPUs, AGI | Lex Fridman Podcast #490

https://t.co/iU02R7VpPh

[Seph Mard @SephMard]

The future of LLMs isn't just about more data; it's about more verifiable feedback loops. We built NeMo Evaluator to deliver frontier scale to AI evaluation, and we open-sourced it to share our approach to the community.

Try it now! https://t.co/di1e2JOviv.

[Seph Mard @SephMard]

Proud of this team and excited to see this work accepted to ICLR 2026 🎉

CoDeC is a clean, practical approach to detecting data contamination. No training, no reference models, just two forward passes with impressive accuracy.

Great example of impactful research on evaluation through NeMo Evaluator.

Huge congrats to the authors 👏

#ICLR2026 #Evaluation #AIResearch #NVIDIA

[Michał Zawalski @MiZawalski]

🎉 We are thrilled to announce that our paper on CoDeC (Contamination Detection via Context) has been accepted to #ICLR2026!

[Seph Mard @SephMard]

Great to see NeMo Evaluator being adopted by the community!

@ZechenZhang5 from Æthos integrated NeMo Evaluator into AI-Research-SKILLs, an open-source repository he built to plug together key components of modern agentic AI research workflows. With NeMo Evaluator, it now exposes hundreds more benchmarks out of the box.

NeMo Evaluator was built for the community to make standardized, reproducible, and scalable model evaluation easy to embed into existing research tooling. Seeing it integrated directly into projects like AI-Research-SKILLs is a strong signal the approach is resonating.

Check it out 🔗

🔗 Repo: https://t.co/bfQc6SsHEr

🔗 Integration: https://t.co/QJ0FiKN9Pg

🔗Nemo Evaluator:
https://t.co/TNgjfTQqsh

[Nathan Lambert @natolambert]

Excited to announce the Relative Adoption Metric a new way of studying model downloads that contextualizes it across time and model sizes.

While building The ATOM Project and other tools to measure the open ecosystem at @interconnectsai, we are often frustrated with using downloads as a primary metric. We, and the community, know that small models are downloaded much more, so it makes some adoption metrics favor organizations releasing small models. Over the 1,100+ leading LLMs we track carefully, more than 1.4 billion of ~2 billion total downloads come from models in the 1-9B range.

This small model dominance happens to be partially caused by far more models *being released* at that size. Among the top 10 downloaded models at each size category, the median models from 1-9B parameters are only downloaded about 4X the count of models of 100B+ parameters. Still, this difference combined with the potential of small models to be outliers in downloads—by being loaded in the continuous integration (CI) tests of ML developers checking their code and other at-scale automated systems—makes small models dominate plots.

We created the **Relative Adoption Metric**, reported as a RAM Score, to be able to tell within 30-90 days if a new model is on track to be ecosystem defining. We can already see that some models, such as GPT-OSS, are truly exceptional. In releasing only 2 models, OpenAI is well on the map as a top 5-10 open model lab in adoption metrics—this is hard to see when comparing organizations versus each other, when OpenAI's competitors may have many models.

We're also excited to see that some recent larger models from newer AI labs on the scene, such as MiniMax or Moonshot AI, are outperforming the metric, indicating competition in the large MoE space dominated by DeepSeek earlier in the year.

We're excited to support the ecosystem with this new tool!

Some early observations include:
1. GPT-OSS is extremely off the charts. It's on track to be one of the top downloaded models of all time, despite implementation complexity and fairly large sizes (20 & 120B)
2. @MiniMax_AI M2.1 is outperforming most of the recent large Chinese models: @Kimi_Moonshot K2 Thinking is on-track to be in the top 10, DeepSeek v3.2 is underperforming past DeepSeek releases, and @Zai_org GLM 4.7 isn't breaking through as much as I originally thought.
3. @NVIDIAAI's Nemotron Nano 3 from Nvidia (30B total, 3B active) is off to a strong start, maybe the best of any Nemotron model to date.
4. OCR models like DeepSeek OCR and @allen_ai's OlmoCR 2 are overperforming. Quietly one of Ai2's most impactful releases yet.

Here's an abbreviated list of the top 10 models per size category (full list on the website):
- <1B - Mostly Qwen small models + SmolLM2, gemma-3-1b-it
- 1-5B - Qwen 1.5B/3B, Llama 3.2 1B/3B, Phi-3-mini, gemma-2-2b
- 7-9B - Llama 3.1/3/2 8B variants, Qwen 7B, Mistral 7B
- 10-50B - gpt-oss-20b, Qwen 14B/32B, DeepSeek-R1-Distill-32B, Mixtral 8x7B, Llama Vision 11B
- 50-100B - Llama 70B variants, Qwen3-Next-80B, Qwen 72B, DeepSeek-R1-Distill-70B
- 100-250B - gpt-oss-120b, Mixtral 8x22B, Mistral Large, Llama-4-Scout, Qwen3-235B, MiniMax-M2
- 250B+ - Llama 405B, DeepSeek R1/V3 variants, Kimi-K2, GLM-4.6

We use median instead of mean because outliers can skew averages dramatically. For example, the 1-5B bucket at 30d has a single outlier (Qwen2.5-1.5B-Instruct at 50.5M) that skews the mean from 0.57M to 5.51M — nearly 10x. Using median + IQR (interquartile range, 25th-75th percentile) gives more representative reference values, which gives a simple metric to understand if a model is on track to be a top model in the ecosystem.

[No Priors @NoPriorsPod]

The 2026 Jensen Huang No Priors interview:

00:17 2025 surprises
04:12 AI is job creation
12:31 Robotics for labor shortages
15:14 Open source, layer cake of making money in AI
21:52 The myth of "God AI"
23:54 Doomers and regulation
29:25 Jensen teaches Tokenomics
35:09 The return to research
37:49 Future of engineering
43:20 Next ChatGPT moment
46:00 Self driving
54:06 We need energy
58:49 US-China relations in '26
1:04:43 Is there an AI bubble?

[Nathan Lambert @natolambert]

For more, see the post on interconnects: https://t.co/By95PDmgCo
or visit https://t.co/juOA99twjy

[Seph Mard @SephMard]

Evaluation recipe for Nemotron 3 Nano
🔗 https://t.co/XsHMfgUOJQ

[Seph Mard @SephMard]

Jensen’s CES keynote highlighted evaluation as foundational to frontier AI.

NVIDIA Nemotron is a leading open frontier model, and open models require open evaluation. Open evaluation should be the standard - not the exception.
My team at @nvidia built NeMo Evaluator for Nemotron development and is now releasing it to the community.

[Seph Mard @SephMard]

🔗Try NeMo Evaluator today https://t.co/di1e2JOviv

🔗

[NVIDIA AI @NVIDIAAI]

Why are the world’s leading models built on mixture of experts?

Ian Buck, Vice President of Hyperscale and HPC at NVIDIA, unpacks the architecture powering today’s frontier AI and how extreme co-design is driving smarter models at lower cost.

🎧 Listen to the NVIDIA AI Podcast → https://t.co/UR0hjz2T30

[Seph Mard @SephMard]

@viviennezhangx @llm_wizard @UsIsabel26234

[Seph Mard @SephMard]

Just read “Why Benchmarking is Hard” by Florian Brand & @js_denain from @EpochAIResearch — a great deep dive into the messy realities of AI evaluation. 📊 Benchmarks are full of hidden variables & inconsistencies, making apples-to-apples comparison near impossible right now.

https://t.co/tLpwrbBtDZ

[Seph Mard @SephMard]

https://t.co/di1e2JP383