Generating research ideas is getting cheaper; deciding which ones deserve compute remains hard. RWM studies whether past experiments can inform those decisions, including in a new setup. A useful direction for research agents.
Congrats to @zijun_wang2002 and the team 🎉🎉
More reasoning is useful only if the agent has something useful to reason from. The effort comparison in this work makes me want to look more closely at the experimental evidence we put in an agent's context.
Can 𝐫𝐞𝐬𝐞𝐚𝐫𝐜𝐡 𝐭𝐚𝐬𝐭𝐞 be measured? Research agents can propose far more experiments than we can run. But good human researchers predicting which ideas will pay off before running them.
Can LLMs build such a Research World Model?
We asked 13 LLMs to predict each research idea's gain before running it. With 320 past experiments in context, all 13 beat the best model that had none.
Can 𝐫𝐞𝐬𝐞𝐚𝐫𝐜𝐡 𝐭𝐚𝐬𝐭𝐞 be measured? Research agents can propose far more experiments than we can run. But good human researchers predicting which ideas will pay off before running them.
Can LLMs build such a Research World Model?
We asked 13 LLMs to predict each research idea's gain before running it. With 320 past experiments in context, all 13 beat the best model that had none.
Proud of @YanqingLiu83931 for leading 🏔️ ALPS!
Many world models plan in the token space of a frozen visual encoder, so the encoder decides what they can see. Most encoders are strong on one or two of state, language and pixels. ALPS puts all three in the same patch tokens, with a ViT-B trained from scratch on images + text only. No video, no robot data.
📍 State: lowest LIBERO state-probe error among ViT-Bs, matching the 3.5× larger V-JEPA 2-L
💬 Language: 93.2 Flickr30k image→text R@1 with its own text tower
🖼️ Pixels: frozen tokens decode back to images, including the futures a world model predicts
With the encoder frozen: 85.5% mean LIBERO policy success (best in our comparison), and best or tied-best planning on PushT, Wall and Granular.
One result we especially like: cross-view refinement cuts ALPS's state-probe error by 17% while changing retrieval by ≤0.5 pts. Apply the same recipe to CLIP or SigLIP 2 and they lose 4–12 pts on zero-shot and retrieval.
All open (Apache-2.0): checkpoints, 8 LIBERO policies, JAX + PyTorch code.
Great teamwork with @letian_zha35417@liuchen02938149@zijun_wang2002, @meijieru, Hongliang Fei, and @yuyinzhou_cs. Thanks to Google TRC & TPU Builder Program for the TPUs 🙏
🌐 Project: https://t.co/ICc8YkhynO
📄 Paper: https://t.co/AuijNCgoF9
Introducing 🏔️ ALPS: Visual State Encoders for World Models!
What should the visual foundation of a world model provide?
State. Language. Pixels.
ALPS unifies all three in shared visual tokens, pretrained from scratch using only images and text.
🎬 Watch the video 👇
🌐 https://t.co/dbdxlJt1py
🧵 1/10
🏆 Think your AI agent can do basic medical research end-to-end? Submit to our leaderboard (https://t.co/tXO9ioUPiN) and find out!
We're launching #AutoMedBench, the first benchmark for evaluating Medical #AutoResearch agents across the entire research workflow—not just the final answer.
📄 Paper: https://t.co/X2G66j2Qx6
🌐 Project: https://t.co/bYKEVcfqEU
💻 Code: https://t.co/z37BfBBNm5
📝 Plan → ⚙️ Setup → 🔍 Validate → 🚀 Inference → 📦 Submit
As AI agents move from answering medical questions to conducting end-to-end medical AI research, we need to measure where they succeed—and where they break.
What we benchmark:
• 24 tasks across segmentation, image enhancement, VQA, report generation, and lesion detection
• 48 task-tier combinations spanning Lite and Standard settings
• 6 frontier AI agents under a unified interface
• Thousands of runs with detailed logs of stage performance, costs, tokens, wall time, and failure modes
📊 Current leaderboard:
🥇 #Opus 4.6: 66.5
🥈 #GLM-5: 61.6
🥉 #Gemini 3.1 Pro: 59.0
4️⃣ #ChatGPT-5.4: 55.3
5️⃣ #MiniMax-M2.5: 51.6
6️⃣ #Qwen3.5: 51.2
🔎 Key findings:
⚠️ Agents are better at completing workflows than producing high-quality scientific outputs.
⚠️ Validation is the weakest stage; Setup is the strongest.
⚠️ More scaffolding is not always better—some frontier agents actually perform worse with additional guidance.
⚠️ The dominant failures are verification and submission, not task understanding.
💡 Takeaway:
The next frontier for research agents isn't just more medical knowledge—it's better workflow control, validation, error recovery, and artifact-level reasoning.
#AIAgents #MedicalAI #AgenticAI #LLM #MultimodalAI #HealthcareAI #Benchmark
🩺 Can AI agents conduct medical research end-to-end, just like human researchers?
Introducing AutoMedBench — the first workflow-aware benchmark for medical AutoResearch agents. 🧪
📄 Paper: https://t.co/2H666zmPov
🌐 Project: https://t.co/bYKEVcfqEU
💻 Code: https://t.co/z37BfBBNm5
Medical agents are rapidly evolving from answering questions to conducting end-to-end medical-AI research: loading datasets, building pipelines, debugging failures, running inference, and submitting results.
Yet most benchmarks only evaluate the final answer.
A good score can hide a broken process. A failed run often reveals nothing about where the agent went wrong.
🔬 AutoMedBench evaluates the entire workflow.
Every run is decomposed into 5 stages:
📝 Plan → ⚙️ Setup → 🔍 Validate → 🚀 Inference → 📦 Submit
Instead of a single score, AutoMedBench diagnoses which stage succeeds or fails—and why.
📊 Benchmark scope
• 5 medical AI tracks: Segmentation, Enhancement, VQA, Report Generation, and Lesion Detection
• Lite & Standard tiers (same data/metrics, different scaffolding)
• Long-horizon tasks averaging ~33 agent turns
• Full logs of actions, tokens, runtime, cost, and error codes
We put today's frontier agents to the test:
🏆 Overall leaderboard
🥇 #Opus 4.6 — 66.5
🥈 #GLM-5 — 61.6
🥉 #Gemini 3.1 Pro — 59.0
4️⃣ #ChatGPT-5.4 — 55.3
5️⃣ #MiniMax-M2.5 — 51.6
6️⃣ #Qwen3.5 — 51.2
But no single model dominates everything: GLM-5 leads VQA, while Opus 4.6 leads most other tracks.
⚠️ The key finding
Across thousands of runs, Validate is the weakest stage while Setup is the strongest.
Today's agents are much better at making a pipeline run than ensuring it is correct before large-scale inference.
📉 The bottleneck isn't medical knowledge
🔍 Verification & recovery errors: 37.7%
📦 Deliverable & submission errors: 38.1%
🧠 Task-understanding errors: only 0.9%
Even a single fired error code reduces the overall score by 48%.
The next frontier for medical AI agents is not more knowledge—it's workflow reliability, verification, and self-correction.
#AutoResearch #MedicalAI #AIAgents #HealthcareAI #AgenticAI
Can medical AI research be automated with AI itself
This new benchmark from NVIDIA and UC Santa Cruz aims to evaluate this:
AutoMedBench: Towards Medical AutoResearch with Agentic AI Models
"we present AutoMedBench, a workflow-aware benchmark for evaluating autonomous agents on end-to-end medical-AI research tasks"
The benchmark covers 24 tasks across segmentation, question answering, report generation, etc. and across modalities like CT, X-ray, pathology, etc.
The paper experiments with six frontier models (Opus 4.6, GLM-5, Gemini 3.1 Pro, GPT-5.4, MiniMax-M2.5, Qwen3.5-397B) and these models remain far from reliable medical AI researchers. While agents can often set up runnable pipelines, validation is consistently the weakest stage, and engineering failures dominate over understanding errors.
Definitely curious to see how this performs with the newest generation of models/agents!
Clinical AI shouldn't just consume evidence handed to it — it should actively seek evidence, e.g., linking multimodal data, analyzing patient context, and retrieving external knowledge to support clinical reasoning 🔎
Introducing ClinSeekAgent — our automated agentic framework for active multimodal evidence seeking in clinical reasoning., achieving +15.1 F1 compared to #claude Opus 4.6
Paper: https://t.co/YtnI5FnArd
Code: https://t.co/Jai6zHfhgH
Monday Updates💡
AutoMedBench just hit 20 task×tier combos across all 5 medical-AI categories: Seg · Enhancement · VQA · Report Gen · Lesion Detection. 2,000+ runs
🏔️ Check: https://t.co/u55pZWH9lq
💻 Code: https://t.co/u1YcjMEDLe
🔋 Findings: In Lite, AIs overthink. In Standard, free pick rewards research depth.
Lock the model and the leaderboard reshuffles in ways no existing benchmark would predict, which is exactly what AutoMedBench is built to surface.
• Seg — @AnthropicAI Opus tops by score (0.74), but https://t.co/h6m6wvDESG GLM-5 wins per-dollar: 97% of the score at 1/10 the cost. Cost is surging for segmentation, for @AnthropicAI Opus, averages up to $156 per run (pancratic tumor task).
• Enhance — @AnthropicAI Opus owns LDCT denoising; @OpenAI ChatGPT 5.4 takes MRI super-res Standard. Cheapest category to test (~$0.40/run).
• VQA — Lite flips the board: https://t.co/h6m6wvDESG GLM-5 #1, @AnthropicAI Opus LAST. Free pick on Standard: @GoogleDeepMind Gemini takes #1.
• Report Gen — @AnthropicAI Opus tops both tiers by ~7 pp; @OpenAI ChatGPT 5.4 collapses on Lite (zero valid reports).
• Detection — @AnthropicAI Opus wins both tiers, but burns 83 turns, $41/run to do it on Lite. Won, with a COST.
A joint effort between @ucsc University of California, Santa Cruz and @nvidia NVIDIA.
#agenticAI #agents #Anthropic #zhupuai #Openai #GeminiAI #Minimax #KimiAI #Qwen #MedicalAI #AIBenchmark #AI
1/n
https://t.co/Mlhaclg0dX
What happens when you push AI agents *too hard* to improve a score?
Instead of getting better, they may find shortcuts to *game the metric* 🧠➡️🎯
As we rely more on automated evals, this can quietly creep in—good score, but weaker real performance⚠️
AutoMedBench — a benchmark VLAA lab @ucsc built with @nvidia for foundation-model agents on medical AI AutoResearch tasks.
📈 Leaderboard: https://t.co/mK4gXDItzO
👩💻 Code: https://t.co/YLawByNHqv
⭐️ How we measure: Process+Outcome, not just the final answer
Your OpenClaw might be getting a bit “sick” 🤒⚠️ — and it’s not something a simple patch can fix.
We audited one of the most widely deployed personal AI agents and uncovered a critical new class of risks that goes way beyond standard prompt injections.
Enter: State Poisoning ☠️
Instead of attacking inputs, this targets an agent’s persistent memory—the very superpower that helps it adapt to you over time. Specifically, we map these vulnerabilities using the CIK taxonomy:
🧠 Capability
👤 Identity
📚 Knowledge
Poison just ONE of these dimensions, and attack success rates skyrocket to an alarming 64–74%! 📈 And the worst part? The malicious effects persist across multiple sessions. 🔁
The biggest plot twist: 🛑 It’s NOT the model's fault.
We tested this across top-tier systems (Opus, Gemini, Sonnet, GPT) and consistently saw a >3× jump in vulnerability.
Why? Because this flaw lives entirely at the system level. 🏗️ The exact same memory architecture that makes agents useful can be quietly weaponized against you.
The next frontier of AI safety isn’t just about building smarter models 🤖—it’s figuring out how to make continuously evolving agents safe by design. 🔐
Huge congrats to @zijun_wang2002 for leading this 🙌 Also, kudos to the team @HaoqinT, @letian_zha35417, @HardyChen266091, @JJwu41867797, @dobogiyy, Zhenglong Yuan, @TianyuPang1, @michaelqshieh, Fengze Liu, @ZhengBerkeley, @HuaxiuYaoML and @yuyinzhou_cs.
🚀 Huge milestone: AutoResearchClaw just hit 10K⭐ in 2 weeks.
Moreover, we’re shipping v0.4.0 — making autonomous research fully controllable.
What’s new:
🤝 6 intervention modes — same 23-stage pipeline, your level of control
🧪 Idea Workshop — refine ideas before committing
📊 Baseline Navigator — inspect & tweak experiments
✍️ Paper Co-Writer — draft papers collaboratively
🧠 SmartPause — AI knows when to ask you
💰 Cost Guardrails — no surprise bills
🔀 Pipeline Branching — explore multiple ideas in parallel
Led by @JiaqiLiu835914, and Kudos to the amazing team @XinyeYee, @richardxp888, @lillianwei423, @HaoqinT, @Xinyu2ML, @yuyinzhou_cs, @ZhengBerkeley, @dingmyu, @HuaxiuYaoML, etc.
Every agent deserves an “aha” moment 💡✨
In training, it’s when a model learns to reason 🧠
In deployment, it’s when a robust harness lets it truly shine 🚀
Introducing AutoHarness (Aha). Specifically, we believe: Agent = Model + Harness
The model does the thinking — Aha does the heavy lifting:
⚙️ context management
🧩 6-step tool governance
💰 cost control
📊 full observability
This is the bridge from demo → production 🏗️
Huge congrats to @JiaqiLiu835914 for leading this 🙌
Grateful to build alongside Kudos to the team @XinyeYee, @richardxp888, @lillianwei423, @HaoqinT, @Xinyu2ML, @yuyinzhou_cs, @ZhengBerkeley, @dingmyu, @HuaxiuYaoML, etc 🤝
🚨🌍 Ready to take full control of your AI-generated worlds? 🎮 Say hello to #CVPR26 Captain Safari: A World Engine with Pose-Aligned 3D Memory! 🦁✨
Generating long, 3D-consistent videos during fast, aggressive camera movements (like FPV drones! 🚁💨) has always been a massive challenge... until now! 🤯
🧠 How does it achieve this?
Captain Safari introduces a novel persistent dynamic local memory system! 🗂️ By smartly fetching pose-aligned world tokens 🧩, it keeps the underlying 3D geometry incredibly stable and perfectly consistent. No matter how wild, fast, or complex your camera trajectory gets, the world stays locked in! 🎢🎥
But wait, there's more! 🎁 The researchers are also dropping OpenSafari 🦅—a brand-new, challenging in-the-wild FPV drone dataset designed specifically to stress-test and train the next generation of AI world engines! 🌲🏞️🏁
#AI 🤖 #ComputerVision 👁️ #GenerativeAI 🎨 #WorldModels 🌍 #MachineLearning 💻 #DeepLearning 🧠 #3DGeneration 🧊 #TechNews 📰
CVPR 2026🎥We built a model that lets you fly through a generated video world.
Not just generating frames —
but maintaining a consistent 3D world under complex camera motion.
Code, ckpt, and even the data pipeline are all open-sourced ↓
#AI#worldmodel#videogen#cvpr#drone
@karpathy automated the experiment loop. We extended it to the entire lab. 🦞
Meet AutoResearchClaw:
One message in → full conference paper out.
No human in the loop.
📚 Mines arXiv & digests 50+ papers in minutes
🥊 3 AI agents debate & pivot hypotheses
💻 Writes code & auto-fixes crashes
📝 Drafts the paper with real, verified citations
Chat an idea. Get a paper.
Try it 👉 https://t.co/pS6U39lgsN
Kudos to the team 👇
@JiaqiLiu835914 @richardxp888@lillianwei423@StephenQS0710@Xinyu2ML@HaoqinT@zhengop@dingmyu@HuaxiuYaoML
Contributors welcome 🚀
Everyone's excited about Karpathy's autoresearch that automates the experiment loop.
We automated the whole damn thing. 🦞
Meet AutoResearchClaw: one message in, full conference paper out. Real experiments. Real citations. Real code. No human in the loop.
One message in → full paper out. Here's what happens in between:
📚 Raids arXiv & Semantic Scholar, digests 50+ papers in minutes
🥊 Three AI agents FIGHT over the best hypothesis (one swings big, one sanity-checks, one tries to kill every idea)
💻 Writes experiment code from scratch, adapts to your hardware
💥 Code crashes at 3am? It reads the stack trace, rewrites the fix, keeps going
🔄 Results weak? It pivots to entirely new hypotheses and starts over
📝 Drafts a full paper with citations, every single one verified against live databases
No babysitting. No Slack messages. No "hey can you re-run this."
Karpathy built the experiment loop. We built the whole lab.
Chat an idea. Get a paper. 🦞
Try it 👉: https://t.co/KLOcnzFYaD
Kudos to the team @JiaqiLiu835914, @richardxp888, @lillianwei423, @StephenQS0710, @Xinyu2ML, @HaoqinT, @zhengop, @cihangxie, @dingmyu, and we are looking for more contributors.