Ethan Hersch

[Ethan Hersch @EthanHersch]

Was great working on this project with @stai_research,
@StanfordAILab, @BrandoHablando, @ObbadElyas,
@srivatsavad, @kirillacharya, Jolene Wang, @zhankezhou, @StevenDillmann, @yegordb, and @sanmikoyejo. Congrats!

[Ethan Hersch @EthanHersch]

🚨🚨Announcing CertJudge🚨🚨

But who judges the judge? …

New work from Stanford University, Harvard University, Hong Kong Baptist University

[Ethan Hersch @EthanHersch]

8/8: Open source 🤝

Certify the judge — don't re-label it.

📰Paper [arXiv soon]: https://t.co/h4vxzowEBy

💻 Code: https://t.co/8tkN5QWFV4

Poster: https://t.co/Bxw90Wtg6b

ICML 2026 Math-AI workshop. Feedback welcome!

[Ethan Hersch @EthanHersch]

🚨🚨Announcing CertJudge🚨🚨

But who judges the judge? …

New work from Stanford University, Harvard University, Hong Kong Baptist University

[Ethan Hersch @EthanHersch]

7/8: Downstream + scope 📈

The certified judge ranks theorem-generation agents on VeriBench: DSPy ReAct 0.615 > Trace++ 0.588 > Trace+ 0.477 > baseline 0.470. Scope, stated honestly: validation-set calibration evidence — not universal reliability. A locked held-out test is next.

[Ethan Hersch @EthanHersch]

6/8: Catches what agreement misses 🛑

A deliberately broken judge still hits ≈0.45 human Spearman — yet fails weak monotonicity (12.5%). Known failure mode: aggressive α-renaming scores 0.399 vs ≥0.849 for milder rewrites. Judges key on surface strings the kernel ignores.

[Ethan Hersch @EthanHersch]

5/8: Results 📊

TI_var predicts judge-level human alignment: validation-set Spearman 0.833 (avg labels), 0.905 (pass1), 0.738 (pass2) — across GPT-5.3-Codex, Claude Opus 4.7, and Qwen3-Coder judge families.

[Ethan Hersch @EthanHersch]

I wanted to understand post-training by actually doing it‼️

So I took a Qwen2.5-0.5B base model and walked through SFT, DPO, and GRPO, with notes on how these methods fit into the post-training pipeline.

Hopefully I can teach you something: https://t.co/UIverijNBw

[Ethan Hersch @EthanHersch]

Reproduced CLIP from scratch for ~$5 💸

Two pretrained encoders (ResNet and DistilBERT), symmetric InfoNCE loss, 40 epochs on Flickr30k. It was a cool experience

Blog post: https://t.co/J2F2WjMjy1
Check it out!