Today, we’re releasing the Preview version of RSI-Infra Bench. We give AI agents a 10-hour budget to optimize infrastructure across real training workloads.
Full details in our blog: https://t.co/4kuiC9ISAh
Our design principles:
1. A fixed wall-clock budget. Each model gets 10 hours per task. This measures both capability and inference speed—how much useful optimization it can deliver in the time available.
2. Real training workloads. Eight tasks cover LLMs, multimodal understanding and generation, and stages from pretraining to SFT, RL, and OPD—including architectures such as GDN and YOCO.
3. Feedback throughout the run. Agents can check correctness and measure speedup after each iteration. We then run a detailed final evaluation using the Harbor protocol.
For the first time, we see clear signals that codec beats uniform frame sampling on general video tasks.
Today we're releasing LLaVA-OneVision-2.5, with major efforts from @GlintLabAI.
In OV-2, we aim to prove that codec can work for video understanding. We trained on video using both ways, frame sampling and codec, and each did better on different kinds of tasks. We observed codec can work but mostly good at temporal grounding tasks, not on all video tasks.
But we think it can work in a more unified way. Even a single image can be stored in a compressed view, like JPEG. Video is even less a pile of independent images.
In OV-2.5, we show that codec input comes out ahead of frame sampling on most of the general video benchmarks.
More deltas on OV-2.5:
1/ One model, two modes. The same model and the same chat template serve both offline and streaming video. In offline mode,
2/ In offline mode, the model is good at understanding extremely long video. 9× spatial compression and a 262k context let it read 1,000+ frames in a single pass. And it beats Qwen3.5-4B on 16 of 17 video benchmarks.
3/ In streaming mode, the model is proactive and knows when to speak. It watches video streams at 24 fps, and the 4B model ties an 11B streaming model on OVO-Bench and more than doubles the next best score on StreamArena proactive interaction.
Weights and training code are open today, and the technical report is coming soon.
Github: https://t.co/9NJ04buXdY
HF: https://t.co/RrrGbvo4Gb
@DanielLenton1 Good idea. We actually tried telling each model how much time it had left, but found that some would stop optimizing early, so we removed that information. We’ll give it another try.
GPT-6.2 Astra is back, but it still trails Claude Opus 5.5 and Sonnet. (GPT-6 Astra seems has been downgraded)
Github Repo: https://t.co/cqY9usomc3
Blog: https://t.co/jr7fuRyBqe
Today, we’re releasing the Preview version of RSI-Infra Bench. We give AI agents a 10-hour budget to optimize infrastructure across real training workloads.
Full details in our blog: https://t.co/4kuiC9ISAh
Our design principles:
1. A fixed wall-clock budget. Each model gets 10 hours per task. This measures both capability and inference speed—how much useful optimization it can deliver in the time available.
2. Real training workloads. Eight tasks cover LLMs, multimodal understanding and generation, and stages from pretraining to SFT, RL, and OPD—including architectures such as GDN and YOCO.
3. Feedback throughout the run. Agents can check correctness and measure speedup after each iteration. We then run a detailed final evaluation using the Harbor protocol.
Today, we’re releasing the Preview version of RSI-Infra Bench. We give AI agents a 10-hour budget to optimize infrastructure across real training workloads.
Full details in our blog: https://t.co/4kuiC9ISAh
Our design principles:
1. A fixed wall-clock budget. Each model gets 10 hours per task. This measures both capability and inference speed—how much useful optimization it can deliver in the time available.
2. Real training workloads. Eight tasks cover LLMs, multimodal understanding and generation, and stages from pretraining to SFT, RL, and OPD—including architectures such as GDN and YOCO.
3. Feedback throughout the run. Agents can check correctness and measure speedup after each iteration. We then run a detailed final evaluation using the Harbor protocol.
Today, we’re releasing the Preview version of RSI-Infra Bench. We give AI agents a 10-hour budget to optimize infrastructure across real training workloads.
Full details in our blog: https://t.co/4kuiC9ISAh
Our design principles:
1. A fixed wall-clock budget. Each model gets 10 hours per task. This measures both capability and inference speed—how much useful optimization it can deliver in the time available.
2. Real training workloads. Eight tasks cover LLMs, multimodal understanding and generation, and stages from pretraining to SFT, RL, and OPD—including architectures such as GDN and YOCO.
3. Feedback throughout the run. Agents can check correctness and measure speedup after each iteration. We then run a detailed final evaluation using the Harbor protocol.
@snapolino We ran the optimized opus implementation end-to-end, with no shortcuts. Its correctness is well-validated(welcome to find other problem we missed), and You can check our case study about 3dgs—there are indeed some rather unconventional optimizations here.
(8/8) RSI-Infra Bench is a Preview, and we welcome feedback to improve it.
Blog Link: https://t.co/K5GPTckLnn
Github Link: https://t.co/zBSCAPzt4b
We’re especially interested in working with infrastructure experts on new workloads, correctness checks, timing methodology, and deeper analysis of the traces.
Read the full methodology, results, model diagnostics, and case studies in our blog, and explore the code on GitHub.
Help us make the next version a better measure of what AI agents can accomplish on real infrastructure.
(7/8) Three case studies span algorithm–systems co-design, kernel engineering, and distributed execution:
- 3DGS × Opus: 5.82×. Visibility-aware Lazy Adam and a fused training pipeline, passing N1 checks.
- R3 × Opus: 2.41×. Structural changes and handwritten kernels under bitwise N0 correctness requirements.
- Slime GRPO × Sonnet: 2.40×. Redistributing GDN computation across GPUs while preserving bitwise forward outputs; the complete implementation passes N1 checks.
These are final evaluated speedups for the complete implementations.
Read the full case studies: https://t.co/tdzIV9DUOR