The week's arXiv, as I'd hand it to a friend (Nov 16-22). Top of my pile: lifting a latent video model into pixel space cut median control latency from 495 ms to 34 ms.
1. Back to Pixels: median control latency 495 ms to 34 ms, VBench 84.78 vs 84.05 for Wan2.1-14B
https://t.co/JEPP7ZEqkl
2. MIRIX-2.0: multi-agent memory with no fine-tuning hits 72.6% on LongMemEval-S, 14.3 points over the strongest baseline
https://t.co/fOPI27pWci
3. Counterfactual World Models: the same JEPA world models go from 38-46% to 86-96% counterfactual R^2, no counterfactual data or retraining
https://t.co/I3HYIRRPVg
4. Training Reasoning Models in Simulation to Natively Use Real Robots: LIBERO 1.6% to 74.4%, 60% on a real dual-arm robot vs 0% base
https://t.co/ZfjAnmuajs
5. Decoding Without a Decoder: Wan video with no VAE decoder, warping error down 23% (1.52 to 1.18), 1.69x faster end to end at 4 steps
https://t.co/qAAD9zgjsG
6. MMFlash: up to 3.58x lossless speculative decoding for VLMs, vs 1.58-1.68x for EAGLE-3
https://t.co/ldHwZQ4JKR
7. π4: on 361-800 frame Oxford Spires sequences in a single pass, camera trajectory error down 47.2% vs DVLT
https://t.co/GaMNJrboW7
8. NVIDIA Kumo Tabular: a 214M model ranks first on 8 tabular benchmarks, 36x faster than runner-up LimiX-2 on TabArena
https://t.co/E5lJVTx01Y
9. Self-Improvement Memory: beats harness-only self-improvement in 9 of 10 paired coding runs on TB4 (p=0.003)
https://t.co/mwOxOWJZsA
Which of these would actually change what you build next week?
A Wan video generator dropped its VAE decoder entirely, and warping error went down 23% (1.52 to 1.18), with visibly less flicker.
Latent diffusion usually ends with a heavy decoder that turns latents into pixels. Hansheng Chen, Gordon Wetzstein, Leonidas Guibas and colleagues at Stanford, Luma AI and Caltech replace it with a linear pixel head on the transformer. The key is keeping two heads: the original one still drives denoising, and the new one only reads out pixels at the last step. They finetune just the last few blocks and anchor the denoising head to the pretrained model so samples don't drift.
It even helps pixel models that have no decoder: ImageNet FID goes from 1.55 to 1.43, a new best among pixel flow models. On 4-step Wan, generation runs 1.69x faster end to end.
Caveat from their limitations: the gain shrinks when the decoder is already strong, as on FLUX.2. The odd part is that at a final timestep of 0 the gain vanishes. Is the head using detail the denoiser hasn't committed to yet?
https://t.co/qAAD9zgjsG
Gradient-boosted trees ruled tabular prediction for two decades. NVIDIA's Kumo Tabular, a 214M-parameter model trained only on synthetic tables, now ranks first on eight leaderboards including TabArena, and predicts 36x faster than the runner-up there, LimiX-2.
There is no per-dataset training. You hand it a table with some labeled rows, and in one forward pass it fills in the rest. A table encoder squeezes every row into an embedding by attending along columns and rows, then an in-context transformer lets each unlabeled row look at the labeled ones, like few-shot prompting for spreadsheets.
The pretraining data comes from a generator that draws random causal graphs and samples tables from them, noise and missing values included. Weights and code are open.
The honest caveat is in their own limitations section: pretraining never sees grouped, temporal or very long tables, so it drops to fourth on grouped splits and ninth on tables with over 100K training rows. For ordinary i.i.d. tables, though, this looks hard to beat.
https://t.co/E5lJVTx01Y
Mixture-of-experts promised huge models at constant compute, but routing was complex, communication heavy and unstable. Switch simplified it and pre-trained up to 7x faster than dense T5 models with the same compute.
“Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity” (2021)
The idea: a small router sends each token to exactly one expert feed-forward block instead of the usual top-2, and each expert takes a fixed number of tokens set by a capacity factor. Keeping only the router in float32 made bfloat16 training stable. That carried them to 1.6T parameters and a 4x speedup over T5-XXL.
This week's K2-Horizon release from the Institute of Foundation Models tops out at a 375B-A23B MoE, and even puts experts inside the attention value projections.
By William Fedus, Barret Zoph and Noam Shazeer at Google.
https://t.co/IqBMSgL10W
Can four copies of the same frozen model reason together without ever writing to each other in words?
Stream of Belief, from Yu Yao (MIT), Jiayi Dong (UCLA) and Yilun Du (Harvard), says yes. On Qwen3-8B it scores 77.2% on MATH-500 vs 65.6% for LatentMAS, the strongest latent baseline there.
How it works: each agent keeps a belief, a block of 384 soft tokens where every token is a probability-weighted mix of word embeddings instead of one sampled word. Each cycle, an agent reads everyone's beliefs and rewrites its own, then refines it alone. They stop once the decoded answer stops changing, so agents can still disagree at the end. No training, just one-line personas like "be skeptical".
The authors report 51% fewer text tokens on average than the strongest latent baselines. Caveat: plain text debate still wins on accuracy, 86.4% on MATH-500. Is that gap about bandwidth, or do soft tokens lose something words keep?
https://t.co/jYCnfLMfmt
One 0.5B model takes a single photo, a pile of unordered shots, or an 800-frame video and returns cameras, dense 3D geometry, metric scale and point tracks in one forward pass. On six long Oxford Spires routes, π4 from Shanghai AI Lab cuts trajectory error by 47.2% vs DVLT.
How it works: a shared image encoder feeds two looped Transformer decoders. The geometry branch treats every view the same, with no reference frame. Then they freeze it and train a motion branch that only tracks points in 2D, lifting tracks into 3D with the reconstructed geometry, so tracking never has to learn scene scale.
It trains on at most 64 views but runs on 1,000+ images at once. Across 16 geometry benchmarks, its 343M geometry model gets the best average rank (3.00) against nine baselines, most over 1B.
Caveat: that lead is thin (Pi3X scores 3.10), and the authors say it still trails traditional optimization methods on sequences longer than it trained on.
https://t.co/GaMNJrboW7
If you serve with speculative decoding, the drafter's view of the target model is the lever. DNeo gets 3.55x over plain decoding on MATH-500 with Qwen3-4B on one H100, vs 2.93x for DFlash-2, the best prior parallel drafter.
A Cerebras and UVA team found that the usual feature encoder mixes the target's hidden states in a lossy way, keeps no memory, and samples layers uniformly. DNeo swaps in a small attention over per-layer features, remembers the tokens the target just rejected, and pulls most features from the target's later layers.
– Up to 20.7% faster than DFlash-2 at batch size 1 in SGLang, and fastest on all six benchmarks at temperature 1
– Up to 25.1% more accepted tokens per round
– Weights and inference code are released
Caveat: every number is for one 4B target. At temperature 0, plain DFlash still edges it on GPQA and MT-Bench.
https://t.co/GzFv6Z6zNl
Counterfactual R² goes from 38-46% to 86-96% with the same JEPA world models, no retraining. The question is what would have happened under a different action, with the same hidden disturbance. Randall Balestriero, Yann LeCun and Kerda Varaku prove a model trained only on what actually happened can't answer that alone.
Their fix uses no counterfactual data or extra parameters. When the action and the disturbance act as separate forces, their interaction is second order in time. So the counterfactual is roughly the observed outcome plus the gap between the model's predictions under the two actions, with a proven O(h²) error bound.
In hindsight planning it finds the action that would have reached the goal in 35-69% of events, vs 14-23% for an oracle planner that knows the simulator but not the outcome.
They note it fails at discontinuities. How far does "nature makes no leaps" get you in manipulation, where contact makes the dynamics jump?
https://t.co/I3HYIRRPVg
Most ways to give a language model more compute per token add depth: more layers, loops, or latent reasoning steps. This paper adds state across time instead, and a 406M model drops FineWeb-Edu perplexity from 15.41 to 14.40 against a parameter-matched Transformer.
The idea, from Shivam Duggal, William T. Freeman and Antonio Torralba at MIT CSAIL with Adobe, is a lattice of latents. Each layer gets its own small state that is carried from token to token, next to the usual residual stream flowing up through depth. That state feeds attention, the KV-cache writes, the MLP and decoding.
The update is affine, so training still runs as a parallel scan over tokens. At 14.40 it matches a 609M, 40-layer Transformer, and with hyper-connections and input residuals it reaches 13.61. It also helps Mamba-2 and Gated DeltaNet models.
Caveat: everything here is around 400M parameters and 20B tokens, so I'd want to see it hold at a few billion.
https://t.co/ERKcocc5Nu
One forward pass, 78 ms, and you get a 512x512 image at 0.74 GenEval on a single H200. That's 122x faster than Janus-Pro-7B.
Jeveration, from Xinchao Wang's group at NUS, keeps a Qwen3.5-4B vision-language model frozen and adds only LoRA adapters plus output heads. Every image position is predicted in parallel, conditioned on one shared random state so the pieces agree on a single coherent picture. A learned distribution over discrete image codes teaches the model to actually use that state.
– Same 512 px as four-step Show-o Turbo: GenEval 0.74 vs 0.63, latency 77.8 vs 94.1 ms
– One checkpoint also does image-to-image (edges, depth, colorization) and dense prediction
– Trained in about four days on one machine with four RTX PRO 6000s
Caveat: Janus-Pro-7B still wins on quality (0.80 GenEval, 83.91 DPG vs 78.24 here), so this trades a bit of quality for speed. I didn't see a code release mentioned.
https://t.co/k79Zu6Ff8Z
Rank four agent-evolution methods from one run each on Terminal-Bench and you pick the true best only 33% of the time. Random guessing gets 25%.
A ShanghaiTech and ByteDance team ran seven evolution methods on six models (over 10M LLM calls) to find out why. Evolution loops have a proposer write new prompts, skills or memories, then keep whatever scores higher. Their claim: the proposer is fine, the scoring is the bottleneck. Helpful candidates exist in the pool, but selection tells them from harmful ones about as well as a coin flip, so errors pile up round after round.
Two more numbers stuck with me. The median published margin is 3.4 points, and detecting that reliably takes about 23 runs per method on Terminal-Bench. And when they repaired graders that allow shortcuts, the median reported gain fell from 17.6 to 0.8 points.
How many self-improvement results from this year would survive 5+ seeds and an honest grader?
https://t.co/M1XMbRQ6Eo
Keeping an agent harness's edit history, not just the harness, made continued self-improvement work better. In 10 paired coding runs on Terminal-Bench 4, continuing with that memory beat harness-only continuation 9 times out of 10, by 2.36 points on average (p = 0.003).
The problem: harness optimizers rewrite the code around a frozen model (context, tools, execution flow). Move an evolved harness to a new model and it can start below the plain seed. An Opus-evolved harness scored 68.41 on Gemini's workspace tasks, vs 73.89 for the seed.
Self-Improvement Memory, from Xin Yu and colleagues at Google, gives the optimizer, not the agent, a log of past edits grouped by the problem they targeted, with the rationale, predicted vs observed outcome, and eval context. It reads that log to decide what to keep, revise or drop. On those workspace tasks, Gemini reached 84.75 with it vs 73.42 without.
Caveat: coding scores stay low (21.70 vs 19.33), and the other domains got one run per condition.
https://t.co/mwOxOWJZsA
Skipping the SFT warm-up gave the best multimodal reasoning model in this study. A 9B VLM went from 69.4% to 75.9% average across 21 decontaminated benchmarks, with video up 10.7 points.
The recipe, from A*STAR and VinUniversity: start from Qwen3.5-9B, train one RL teacher per domain (image STEM, image perception, video), then distill all three into one student on its own samples, with each teacher correcting tokens on its domain. A final mixed RL pass on the same prompts still adds about 0.7 points, with no collapse.
What I'd take from it if you post-train VLMs:
- Mixed SFT alone landed below the base model (67.7%), and We-Math fell from 74.8% to 34.3%.
- Distilling the per-domain teachers beat plain mixed RL on the same prompts (75.2% vs 74.4%).
Data (1.35M SFT, 193K RL prompts), training code and every checkpoint are released.
Caveat: one base model and a 9B scale, and on video it still trails Video-MOPD, which was built for video only.
https://t.co/bL0e9YUuq7
How far can you push a frontier-scale LLM on Huawei Ascend alone? Xingchen AGI Lab pretrained a 285B MoE on 8,192 Ascend 910C cards, and its DSpark speculative decoder gets 2.67x to 3.77x the base decoding throughput, against 1.80x to 1.93x for MTP3.
The Xing 4 report (the TeleChat line) is mostly systems work. For long context they add sparse attention on top of MLA: a small indexer picks the top-K past tokens for each query, and IndexShare reuses those picks across layers instead of recomputing them. Dropping pipeline parallelism for hybrid sharding adds 16.1% training throughput. They open-sourced a 29B MoE with 4B active params and native 256K context.
The 4.68x prefill gain over plain MLA comes from a simulator, and the authors say real serving may look less rosy. Does it hold on a production cluster with mixed prompt lengths?
https://t.co/xjmzeoPRqL
A 36B MoE that activates about 4.6B parameters per token lands within 1.2 to 3.7 points of a dense 32B trained on the same data, on MMLU, GSM8K, HellaSwag and HumanEval. It is one of six fully open K2-Horizon models, from 0.9B dense to a 375B-A23B MoE.
The trick is MoVA, from the K2 team at MBZUAI's Institute of Foundation Models. Normal MoE only makes the feed-forward layers sparse. MoVA also routes each token's attention value projection to 4 of 64 value experts and mixes them before caching, so the KV cache stays the same size and FlashAttention still works.
Uno, a diffusion adapter, drafts a block of tokens that the same model then verifies: up to 2.2x faster decoding at batch size 1 on the 7B, same output distribution, no separate draft model.
Everything ships: weights, 380 checkpoints, training logs, 21.6 TB of data, and the pipelines behind about 10T synthetic pre-training tokens.
Caveat: that small gap is measured on base checkpoints, and at batch 64 Uno drops to 1.5x.
https://t.co/u3CkSXjVaG
Can an open-weight model learn to run a real robot the way frontier models learned computer use? A 27B model trained with RL only in simulation went from 1.6% to 74.4% on LIBERO, and succeeded in 60% of trials on a real dual-arm robot where the base model got 0%.
Song Han's group at MIT, with UC Berkeley, NVIDIA and UCSD, treats the robot like a computer-use agent. Each round, the model looks at the camera views and arm state, reasons in a chain of thought, and outputs the next end-effector pose. Inverse kinematics moves the arm. No demonstrations, no harness.
– LIBERO-Pro, new task in a familiar scene: 57.4% vs 0.75% for pi0.5 fine-tuned on demos
– no measurable drop on nine reasoning, coding and vision benchmarks
– code is released
Caveat: the real-robot number comes from their own trials, and LIBERO is still a fairly narrow set of tabletop tasks.
https://t.co/ZfjAnmuajs
Ten robot demos on a 552B backbone beat fifty demos on a 2B one: 87.8% vs 85.6% on LIBERO Goal. ScaleVLA asks whether frontier-scale VLMs make better robot policies, not just better planners.
The setup is simple. A team from Tsinghua, Peking University, SJTU and Galbot freezes DeepSeek-V4.1-Flash (a 552B MoE), reads its features once per action chunk, and trains only an adapter plus a diffusion transformer that outputs continuous actions. Backbones from 2B to 552B get the same recipe.
The big backbone helps most where small VLAs struggle: LIBERO-PRO generalization reaches 45.1% vs at most 31.3% for smaller backbones, and a real charger insertion goes from 4/10 to 9/10. Median latency is 216 ms per action chunk.
My question: with the backbone frozen, is the gain from scale itself or from what this one model saw in pretraining? A same-size model from another family would separate the two. And 10 real trials is a small sample.
https://t.co/UDQiv1t0Iy
Latent video models make every camera control wait for a whole block of frames. PixVid moves a pretrained model into pixel space and cuts median control-to-screen latency from 495 ms to 34 ms, with each control landing on the next frame.
The VAE compresses several frames into one latent, so conditions and controls apply to groups of frames. Researchers at Roblox (senior author Xun Huang) keep the Wan2.1-14B diffusion transformer, give each frame its own patch tokens, and swap the VAE decoder for a small causal convolutional head that paints pixels. Two tricks make it train: predict the clean video instead of velocity, and use a larger noise shift.
Quality holds: VBench total 84.78 vs 84.05 for the Wan2.1 base, after about 4K B200 GPU-hours. Doubling the patch size gives 960x1664 video at the same per-frame token count as 480x832, and depth-conditioned edits keep 98% of fast motion vs 83% for a latent model.
Caveat: fine-tuning used 64K videos generated by Wan2.1 itself, and FVD on editing trails the latent model (118 vs 91).
https://t.co/JEPP7ZEqkl
Agent memory that keeps every record and still answers fast: MIRIX-2.0 reaches 72.6% on LongMemEval-S, 14.3 points above the strongest baseline, and 91.4% on LoCoMo, with no fine-tuning.
The MIRIX AI team splits memory across six typed agents that write complete records, so nothing gets summarized away. A provenance hypergraph indexes those records with frames that cite their source rows, and retrieval runs under fixed budgets, so the evidence a question reads doesn't grow with the history. A background step called AutoDream merges duplicates and distills finished task sessions into skills.
– On LongMemEval-S, the hypergraph alone adds 11.2 points
– As procedural memory: 76.4% on ALFWorld with GPT-6 Luna, vs 73.6% for Agent Workflow Memory and 69.4% with no memory
– With AutoDream off, it answers faster than HyperMem on all five timed benchmarks
Caveat: the QA runs use gpt-4.1-mini with an LLM judge, and the authors are building this as a product, so I'd want independent runs. Code is public.
https://t.co/fOPI27pWci
Why does speculative decoding get worse when a model is looking at a picture? On Qwen3.5-4B, draft tokens that lean hardest on the image get accepted 22% of the time, vs 46% for tokens that barely use it. MMFlash targets that gap and reaches up to 3.58x lossless speedup.
A diffusion drafter guesses a block of 16 tokens in one pass and the big model verifies them in parallel. Visual dependence is scored as the KL gap between the target's predictions with and without the image, and those tokens get upweighted when training the drafter. For long video, sparse attention shrinks a 40K+ token context to about 1.6K.
Greedy decoding on Qwen3.5-4B and 9B averages 3.00-3.05x, about 10% above DFlash and well above EAGLE-3 (1.58-1.68x). From Sicheng Feng, Xinchao Wang and colleagues at NUS.
Does the reweighting actually close the 46% vs 22% gap, or mostly shift where the drafter spends capacity?
https://t.co/ldHwZQ4JKR