Hangoo Kang

MSCS @StanfordAILab | AI research @ Scaling Intelligence Lab with @Azaliamirh
Previously @siebelschool @uiuc_focal_lab

[Hangoo Kang @hangoo_kang]

“TRACE: Capability-Targeted Agentic Training” got Spotlight @ ICML AIWILD 🎉

Beats direct RL, GEPA, & synthetic-agent data on SWE-Bench Verified and τ²-Bench.

TRACE-Qwen3.6-27B tops GPT-5.2-Codex, GLM 5, & Claude 4.5 Sonnet on SWE-Bench.

Co-led with @TarunSures41845. Thanks to @JonSaadFalcon and our advisor @Azaliamirh. Details below 👇

[Alex Dimakis @AlexGDimakis]

We are releasing AutoResearchExam, a benchmark on open-ended machine learning and engineering tasks. Our benchmark covers seven research areas including model training, data curation, AI safety and interpretability.

https://t.co/Rho35ULgS5

In each task, we give agents 24 hours with a CPU or GPU machine to develop and improve their solutions through experiments and feedback. We measure both speed and quality with a combined score. Our benchmark has a unique feature: testing if agents create improvements that hold up on data they never see. We find that AI research agents often overfit as they try to improve.

We see an interesting head-to-head comparison at the frontier: Astra starts the strongest and holds the lead for up to 19 hours but Fable 5.1 catches up and gets the top performing spot in the final hours.

Qwen3.8 Max, Gemini 3.8 Flash and Grok 4.6 all sit on the cost-performance Pareto frontier, giving strong options at lower API budgets.

Anthropic's Opus and Fable retain nearly all their validation performance on hidden tests, with gaps of 1.1% and 2.9%. Astra's improvement over Sol extends to generalization too, with that gap falling from 6.9% to 1.7%. (1/n)

[Hangoo Kang @hangoo_kang]

@AlexGDimakis Congrats on the release! What a benchmark 🥳

[Alex Dimakis @AlexGDimakis]

How to post-train a model to personalize it on your code repo? In our latest research in Bespoke Labs, we post-trained a model to improve its performance on a given Github repository.

Starting from Inkling base, we use supervised fine-tuning (SFT) with trajectories coming from a strong teacher model, and reinforcement learning (GRPO) on repository-specialized environments that we curated. SFT gave a 52pp improvement in performance on the held-out fontTools evaluation set. Further RL training lifts the total improvement to 57pp compared to the base Inkling model.

In addition to the in-distribution evaluation our post-trained Inkling shows good performance on Terminal-Bench 2.1 and SWE-Bench Lite while becoming 40% more token efficient due to post-training.

Read our full research blog post here:
https://t.co/i9Xj0siSRb

Many thanks to Thinking Machines Lab for their credit contribution that helped support this research.

[Hangoo Kang @hangoo_kang]

@Azaliamirh @TIME @annadgoldie @RicursiveAI Congrats!!

[Azalia Mirhoseini @Azaliamirh]

Honored to be named to @TIME's TIME100 List of the World's Most Influential People in AI, even more so to share it with my co-founder @annadgoldie!

Anna and I started working on AI for Chip Design almost a decade ago. Last year, we started @RicursiveAI to transform end-to-end chip design from years to days! Watching that vision become reality piece by piece has been the most thrilling / fulfilling experience ever!

https://t.co/fsSMaFhuRg

[Azalia Mirhoseini @Azaliamirh]

LLM-as-a-Verifier keeps pushing the frontier of cost vs. capability!

On Terminal-Bench 2.1, it made DeepSeek V4 Flash accuracy go from 79% → 88%, while being 4-11x cheaper than competitors!

Try it here: https://t.co/HaDYEYesQR

@jackyk02

[Jacky Kwok @jackyk02]

Scaling self-verification with DeepSeek V4 Flash beats Claude Fable 5 on Terminal-Bench 2.1, while being 11x cheaper 💰

As open-source models become more capable, they can now generate large numbers of high-quality candidate solutions and verify their own outputs at very low cost.

For example, we find that sampling just 5 solutions with DeepSeek V4 Flash and ranking them using the same model with LLM-as-a-Verifier can lead to a significant boost in accuracy (79% → 88%), outperforming closed frontier models on Terminal-Bench.

Try it out today: https://t.co/UhudjKDaYI

More on verification scaling in my previous post.

[Azalia Mirhoseini @Azaliamirh]

Intelligence-per-joule is increasing quickly because models and chips are improving and the gains compound. Though demand for inference is growing even faster.

https://t.co/1VM0T9fQQy
@Avanika15 @JonSaadFalcon @HazyResearch

[Alex Dimakis @AlexGDimakis]

A collection of 15k open math problems, made easy for agents to explore.

[Hangoo Kang @hangoo_kang]

@simonguozirui @keramakr Awesome work Simon!

[Ricursive Intelligence @RicursiveAI]

Our co-founder @Azaliamirh met with @TIME at last week's @RaiseSummit to discuss Ricursive. We're using AI to revolutionize chip design. And instead of years, it will take days.

[ICML Conference @icmlconf]

That's a wrap! We hope you had a fun, productive, and idea-filled Gangnam-styled #ICML2026! It's been a pleasure posting with you -- stay tuned for more content. We wish you a safe trip home, and see you next year!

[Hangoo Kang @hangoo_kang]

@AlexGDimakis Congrats!! 🎉

[Jacky Kwok @jackyk02]

How can we extract richer signals from AI Feedback?

Introducing LLM-as-a-Verifier✨— a simple verification scaling framework that achieves SOTA on agentic benchmarks 🚀

The key idea:
- Use fine-grained scoring granularity (e.g., 1-20 instead of the standard 1-5 scale)
- Take the expectation over the full logprob distribution of score tokens
- Scale repeated evaluation and criteria decomposition

You can use these fine-grained signals for more effective test-time scaling, RL, and agent monitoring! It achieves SOTA across Terminal-Bench V2, SWE-Bench Verified, RoboRewardBench, and MedAgentBench 👑

Advised by @Azaliamirh @istoica05 @drmapavone @chelseabfinn

🧵👇

[Azalia Mirhoseini @Azaliamirh]

Introducing Distill to Detect (D2D), an auditing method that surfaces hidden biases in fine-tuned LLMs, even when the auditor has no idea which topic the bias is on!

D2D distills the difference between the suspected and base models into a tiny 4M-parameter Cartridge (a learned, compressed prefix). Because the Cartridge's capacity is too small, it keeps only the most coherent part of the difference between the two models, surfacing the bias!

Great work led by @talaei_shayan and @AbhinavChinta10!

[Hangoo Kang @hangoo_kang]

We thank Debangshu Banerjee, Tanvir Bhathal, Alex Bloom, Andy Dimnaku, Simon Guo, Sid Jha, Hermann Kumbong, Jacky Kwok, Andrew Shi, and Shayan Talaei for their feedback.

Thanks to Prime Intellect, Lambda Labs, Google, Modal, and IBM for compute that made this possible!

[Hangoo Kang @hangoo_kang]

“TRACE: Capability-Targeted Agentic Training” got Spotlight @ ICML AIWILD 🎉

Beats direct RL, GEPA, & synthetic-agent data on SWE-Bench Verified and τ²-Bench.

TRACE-Qwen3.6-27B tops GPT-5.2-Codex, GLM 5, & Claude 4.5 Sonnet on SWE-Bench.

Co-led with @TarunSures41845. Thanks to @JonSaadFalcon and our advisor @Azaliamirh. Details below 👇

[Hangoo Kang @hangoo_kang]

We’ll be presenting our work at ICML on Friday, July 10 (FAGEN), and Saturday, July 11 (AIWILD Spotlight). Please stop by, we’d love to chat if you’re interested!