Leon Qi

Founding member of @actAVAai

[Weiran Yao @iscreamnearby]

Huge congrats to @Humana's Erius agent taking the #1 spot on CHI-Bench for Prior Auth and 6th for all domains. It outperforms every frontier lab on one of healthcare's hardest workflows.

[Leon Qi @dmon2048]

Opus 4.8 is in the lead @claudeai @ClaudeDevs

[actAVA AI @actAVAai]

CHI-Bench leaderboard just gets updated with the newest and highest score from @claudeai Opus 4.8.

CHI-Bench is world's first long-horizon benchmark for healthcare AI agents.

Leaderboard: https://t.co/wjd9wK44eU

[actAVA AI @actAVAai]

CHI-Bench is the world's 1st long-horizon healthcare benchmark for AI agents.

If you're building or buying AI for healthcare, this is the test that actually matters — real clinical workflows, not toy demos.

U.S. healthcare needs this. 🏥🔬

[actAVA AI @actAVAai]

actAVA AI integrates CHI-Bench with @huggingface and @harborframework today.

Users can run the CHI-Bench evaluation and RL training from both platforms.

[Leon Qi @dmon2048]

Check out our dataset on Hugging Face.

[Frank Wang @FWang9959]

Great news. Today we ranked #6 most popular dataset on Hugging Face! Wow! 😊 https://t.co/8r9zg2aLhX

[Leon Qi @dmon2048]

Awesome!

[Frank Wang @FWang9959]

🚨 Historic moment for @actAVAai ! 📷Just one day after launch, our benchmark dataset is already #10 most popular on Hugging Face — out of 1 million+ datasets! Huge thanks to @iscreamnearby , @HaolinChen11 , Deon Metelski, Leon Qi, Tao Xia, Joon Lee, Steve Brown, Kevin Riley, T. Y. Alvin Liu, M.D., Zhiwei Liu, Qingsong Wen, @CaimingXiong , Sanmi Koyejo, Eric Xing & all our collaborators. 📷📷

[The Agent Times @TheAgentTimes]

A new 33-author benchmark called CHI-Bench finds that the best AI agent configuration resolves only 28% of realistic healthcare administration tasks, dropping to 3.8% in continuous-session testing.

[Leon Qi @dmon2048]

Great work! @HaolinChen11 @iscreamnearby

[Haolin Chen @HaolinChen11]

(1/n) After a few months of work with multiple hospitals, universities and research facilities, today we're open-sourcing CHI-Bench: the first long-horizon benchmark for healthcare AI agents on real clinical and healthcare workflows.

Best frontier agent overall: 28% pass@1.
End-to-end prior authorization: 0%.

A thread on what we found 🧵

[Leon Qi @dmon2048]

@HopkinsMedicine @AlvinLiu_MD @WellstarHealth @YaleMed @StanfordAILab @zeyu1tang @sanmikoyejo @mbzuai @XiangchenSong @LingjingKong @kunkzhang @ericxing @ffeng01 @huang_biwei @SFResearch @JimZhiwei @zixianma02 @hjian42 8/ …and more 🔬

Brown: @FangliGeng
Boston College: @YuanYuan_MIT
Stony Brook: @Charlesyooo1
Oxford: @qingsongedu
ASU: @realhuawei, Yanjie Fu
USC: Yue Zhao
Emory: @yangji9181
@Recursive_SI: @CaimingXiong
UIC: Philip S. Yu

[Leon Qi @dmon2048]

1/ Introducing CHI-Bench 🧵

Can AI agents automate U.S. healthcare workflows end to end — given only clinician & insurer apps, operations, and a medical policy library?

75 long-horizon workflows × 30 frontier agents. Best agent solves just 28%.

#AIinHealthcare 👇

[Leon Qi @dmon2048]

@HopkinsMedicine @AlvinLiu_MD @WellstarHealth @YaleMed 7/ …and university & industry AI research labs 🔬

@StanfordAILab @zeyu1tang, @sanmikoyejo
CMU & @mbzuai: @XiangchenSong, @LingjingKong, @kunkzhang, @ericxing
UCSD: @ffeng01, @huang_biwei
@SFResearch: @JimZhiwei
UW: @zixianma02
Northeastern: @hjian42

[Leon Qi @dmon2048]

Proud to have helped build CHI-Bench 🧵

Can frontier agents run U.S. healthcare workflows end to end? 75 long-horizon tasks, 30 agents — best solves just 28%. We're early, and now we can measure it.

Fully open 👇

[Weiran Yao @iscreamnearby]

1/🧵Can AI agents automate U.S. healthcare workflows end to end given just clinician & insurer apps and operations, medical policy library? Introducing CHI-Bench: 75 long-horizon realistic healthcare workflows × 30 frontier agents. Best agent solves only 28% #AIinHealthcare 👇

[Leon Qi @dmon2048]

@CaimingXiong @CaimingXiong thanks for your collaboration.

[Leon Qi @dmon2048]

@iscreamnearby Remarkable results! It's a game changer on integrating with AI in health care.