Jessy asked where my paper was. I said, "My agent deleted it, but I don’t believe that.” She was confused, as I sound self-conflicting. Then she got angry as she figured the agent had deleted it, and I was blindly trusting my agent. LLMs, unlike Jessy, don't reason like that.
Heading to #EMNLP2026 in Budapest 🇭🇺? Check out our Oral paper, UReason: Benchmarking Reasoning-to-Generation Alignment in Unified Multimodal Models!
🤔 UMMs unify understanding and generation. But are their text and images truly aligned?
💡 Our finding: UMMs usually get the reasoning right, but the image often doesn't follow it. Even more surprising: if we drop the reasoning and give the model only the final prompt it wrote itself, every model we tested does better, by a large margin.
📍 Oral Session 2 · Oct 25, 11:00–12:30 · Room G13-14
👇
LZ77, a 1977 breakthrough, still powers most compressed files you download. I found a way to improve it: give each match length its own window.
The result: two open-source compressors, now in lzbench and TurboBench, one beating Zstandard's top level on Silesia. How it works ↓
Every back-reference ("copy 12 bytes from 3,000 bytes ago") costs bits, and the farther back it points, the more it costs. Yet LZ4 and Zstandard give every match the same reach: a 4-byte match, which saves almost nothing, may point as far back as a 300-byte one, and the distance code must allow for that.
So in my compressors, short matches may only point to nearby data, where distances are cheap to write, and long matches may reach up to 128 MB back. The decoder reads the length first, so it knows the window and no extra bits are sent.
The rest of the design builds on that rule. Each group of lengths gets its own distance code; one symbol carries the length, the literal run before the match and any reuse of a recent distance; and the parser prices every candidate under these codes to find the cheapest parse.
The numbers:
• WLZ4 (LZ4 class): files 14% smaller than LZ4HC at its highest level, at 78% of its decompression speed. It reaches Zstandard level 3's ratio while decompressing 2.5× faster, but compresses far more slowly: it's built for compress-once, read-many data.
• WZIP (Zstandard class): on the Silesia corpus, a 3.5% better ratio than Zstandard at its maximum level (1.2% without WZIP's filters for executables and tables), at 88% of its decompression speed.
Beyond lzbench and TurboBench, Matt Mahoney's Large Text Compression Benchmark lists WZIP: no compressor ranked above it there decompresses faster.
Its theorical framework is submitted to IEEE Trans. Information Theory (arXiv 2610.06530). It proves that the windows keep LZ77's long-run optimality while cutting search time and memory. It also isolates why the length code matters: on purely random data, the original 1977 format's overhead stays on the order of log(B)/B whatever longest match it allows, where B is the window size in bits, while a match cap that grows with the window, plus a cheap code for lengths, brings it down to log(log B)/B.
Code, benchmarks and papers in the first reply ↓
Can a unified multimodal model (UMM) teach itself, with its text and visual abilities making training data for each other?
Yes: over 4 rounds, a 7B model climbs 45.7% → 60.2%, vs. 46.3% for continued training.
🧵 1/10
🔁 𝐖𝐡𝐚𝐭 𝐢𝐟 𝐥𝐨𝐨𝐩𝐞𝐝 𝐋𝐌𝐬 𝐜𝐨𝐮𝐥𝐝 𝐮𝐬𝐞 𝟑× 𝐟𝐞𝐰𝐞𝐫 𝐩𝐚𝐫𝐚𝐦𝐞𝐭𝐞𝐫𝐬 𝐚𝐧𝐝 𝐚 𝟑× 𝐬𝐦𝐚𝐥𝐥𝐞𝐫 𝐊𝐕 𝐜𝐚𝐜𝐡𝐞 𝐚𝐧𝐝 𝐬𝐭𝐢𝐥𝐥 𝐛𝐞 𝐜𝐨𝐦𝐩𝐚𝐫𝐚𝐛𝐥𝐞 𝐭𝐨 𝐚 𝐬𝐭𝐚𝐧𝐝𝐚𝐫𝐝 𝐓𝐫𝐚𝐧𝐬𝐟𝐨𝐫𝐦𝐞𝐫?
Introducing 𝐑𝐞𝐭𝐡𝐢𝐧𝐤𝐢𝐧𝐠 𝐚𝐭 𝐅𝐢𝐱𝐞𝐝 𝐏𝐨𝐢𝐧𝐭𝐬. We argue that looped LMs don't have to pay for every loop. They can scale by adding FLOPs at constant memory.
The key is the fixed point of the loop. We introduce:
– 𝐅𝐢𝐱𝐞𝐝-𝐏𝐨𝐢𝐧𝐭 𝐒𝐡𝐨𝐫𝐭𝐜𝐮𝐭𝐬: what fixed points buy a looped model in training (pre-training, post-training) and inference (prefill, decoding)
– A 𝐥𝐞𝐚𝐫𝐧𝐞𝐝 𝐝𝐞𝐩𝐭𝐡 𝐩𝐫𝐢𝐨𝐫 and 𝐨𝐫𝐭����𝐨𝐠𝐨𝐧𝐚𝐥 𝐢𝐧𝐣𝐞𝐜𝐭𝐢𝐨𝐧: two ways to shape those fixed points
🧵 Long thread ahead (0/n).
Definitely agree with every word about @MaxMa1987. I still remember how blown away I was the first time I heard that Max was building xLLM completely from scratch two years ago. I was genuinely like, “wait, wow!” But then I thought, yeah… of course. This is Max. Deep respect🫡 !
And having @chufan_shi in the lab is just unfairly good and he somehow making the whole lab 10x more fun🤣.
This is a very late comment, and a long one.
At EMNLP 2024 in Miami, I got to talk with Max. I found it hard to believe that someone as senior as him was still on the front line, writing infra code himself. That was when I made up my mind: I wanted to do my PhD at USC, with Max.
After I started in 2025, I got to build xLLM(https://t.co/o0RRafZZo9), our pre-training framework, with Max(@MaxMa1987) and many great collaborators. More and more code today is written by models, and vibe coding is everywhere. Max still reviews every single xLLM PR and goes through every detail of the library with me.
I honestly don't know if this way of working fits this era. But I deeply respect it. Tools will keep changing. The care behind good work, I believe, will not.
Thank you to all the xLLM collaborators, and to everyone in Max Lab(@jin_lynn808@lemonnoel52@tao_muzi@Shicheng_Wen@MengxiWu567@Wendy_xwz@nuan_wen@sunjiao123sun_, NancyXu and Ming) for so much support along the way. I'm also very grateful to Hector Liu(@waterluffy), Eric Xing(@ericxing), and IFM(@IFM_AI) for supporting this full open-source release: the training infra, the attention backend, and the checkpoint toolkit, together.
I know there are still many things we're not satisfied with. But I believe going fully open, even when it's not perfect, is the right direction. The field moves forward when people can see, question, and improve the whole stack, not just the final model.
If you're at COLM 2026, go meet Max and talk with him about xLLM.
Today I finished my PhD qualifying exam, with Max as my committee chair. Maybe a PhD degree really means nothing these days. But I didn't come for the degree. I came to learn from someone who still builds things with his own hands, and I'm still glad I made that choice in Miami.
P.S. Glad I got X Premium, or I couldn't have written this much.
This is a very late comment, and a long one.
At EMNLP 2024 in Miami, I got to talk with Max. I found it hard to believe that someone as senior as him was still on the front line, writing infra code himself. That was when I made up my mind: I wanted to do my PhD at USC, with Max.
After I started in 2025, I got to build xLLM(https://t.co/o0RRafZZo9), our pre-training framework, with Max(@MaxMa1987) and many great collaborators. More and more code today is written by models, and vibe coding is everywhere. Max still reviews every single xLLM PR and goes through every detail of the library with me.
I honestly don't know if this way of working fits this era. But I deeply respect it. Tools will keep changing. The care behind good work, I believe, will not.
Thank you to all the xLLM collaborators, and to everyone in Max Lab(@jin_lynn808@lemonnoel52@tao_muzi@Shicheng_Wen@MengxiWu567@Wendy_xwz@nuan_wen@sunjiao123sun_, NancyXu and Ming) for so much support along the way. I'm also very grateful to Hector Liu(@waterluffy), Eric Xing(@ericxing), and IFM(@IFM_AI) for supporting this full open-source release: the training infra, the attention backend, and the checkpoint toolkit, together.
I know there are still many things we're not satisfied with. But I believe going fully open, even when it's not perfect, is the right direction. The field moves forward when people can see, question, and improve the whole stack, not just the final model.
If you're at COLM 2026, go meet Max and talk with him about xLLM.
Today I finished my PhD qualifying exam, with Max as my committee chair. Maybe a PhD degree really means nothing these days. But I didn't come for the degree. I came to learn from someone who still builds things with his own hands, and I'm still glad I made that choice in Miami.
P.S. Glad I got X Premium, or I couldn't have written this much.
🧩 Generating structured visuals is one thing. Editing them while keeping everything coherent is another.
Excited to share InfoEdit with @ChengYANG_yc, @Chufan_Shi, and the team! #EMNLP2026
🥳 xLLM intro video, made entirely by Claude Opus 5.5.
Best viewed on desktop; on mobile, rotate to landscape.
(Sorry, the voice is robotic; better one next time.)
As promised, we open-sourced our training infra, xLLM: https://t.co/NR5ArqOfCU
together with an efficient attn backend, xattn: https://t.co/U1uh9Be8Mz
and a ckpt conversion & serving toolkit, xBridges: https://t.co/0OuLjOrljv
🎉 Our paper, “Asymmetric Idiosyncrasies in Multimodal Models,” is accepted to @COLM_conf 2026!
You can tell WHICH model wrote a caption with 99.53% accuracy. Render it into an image, and attribution collapses to 49.85%.
Where does the fingerprint go? 🧵
1/5
SoftDelta Attention, an efficient approximation to Delta Attention by avoiding the compute of attention matrix inverse. It can also be regarded as an improved/corrected version of Differential Attention v2 to support stable pre-training.
In a controlled head-to-head comparison with Softmax Attention and Differential Attention v2 in the scale of 780M parameters and 420B tokens, SoftDelta Attention significantly outperforms Softmax Attention on both training loss and downstream accuracies, while Differential Attention suffers training loss collapse.
Blog: https://t.co/BytdMQFmR9
Paper is in progress
Great work led by @Shicheng_Wen@Chufan_Shi@jin_lynn808 and wonderful collaborations with @IFM_MBZUAI @waterluffy
Excited to introduce StereoPolicy, led by @EvansXuHan.
📷📷🤖StereoPolicy is an effective way to add geometric cues to modern robot policy models while keeping the strengths of pretrained 2D encoders.
⁉️Why stereo for robot manipulation?
Monocular RGB often lacks the depth cues needed for precise manipulation, while RGB-D and point clouds can be noisy or brittle, especially on reflective and transparent objects in real-world deployment.
Instead of explicitly reconstructing disparity, depth, or point clouds, StereoPolicy directly fuses synchronized left/right RGB views to learn implicit stereo cues, avoiding extra reconstruction latency that can make real-time manipulation difficult.
Project Page: https://t.co/e07jsbKJg5
Reasoning VLMs often say "let me check the figure again." But do they actually look? Or just say they will, without re-attending to the image?
Introducing VisualSwap — #ICML2026 Spotlight 🌟 — diagnosing this silent failure.
🔍 How should new experts be initialized?
We compare different initialization strategies when expanding the expert pool. Interestingly, we find expert learning is fast: the newly added experts quickly become active and adapt to the routing distribution.
What if MoE capacity could grow like memory, expanding only when needed?
🚀 Excited to introduce EMO: Frustratingly Easy Progressive Training of Extendable MoE
EMO saves MoE training costs. As models scale, the wall-clock savings become even larger.
📊 Performance
We empirically expand a 1.1B dense model into a 9.6B-A1.1B MoE, achieving competitive downstream performance with ~10% GPU cost savings over directly training a fixed large-expert-pool MoE. We also find that EMO maintains strong expert utilization after expansion.