Enxin Song

Video Game Entertainment

[Jiatao Gu @thoma_gu]

We’re teaching a new course at Penn: CIS 6280 · World Models 🌍
https://t.co/3mCdWdOVsg

Every day, LLMs amaze me with one more thing they can solve. But I’ve always felt there’s more to learn than what humans have written down—from observing the world, interacting with it, and experiencing what happens next. To me, that’s what world models are about—and why I see them as one of AI’s next big bets.

However, I struggled to find a systematic course on them—so we are building one!

Topics will include: representation learning, generative models, simulation, model-based RL, video and 3D generation, world models for robotics, reasoning, and code-based world models.

Hands-on work will include:
- Building an environment
- Training a world model
- Learning a policy
and a final research project with leaderboard.

We’re already six lectures in. Slides, demos, and readings are public, and we’ll keep adding materials throughout the fall.

Big thanks to our TA team— @TongMutianTMT @hagsaeng_bag @EnxinSong @KeelyAi04 —for helping bring this course to life!

[Weili Xu @_weilix]

It’s been fun to see how research interacts organically with production this summer at @togethercompute

[Mihir Prabhudesai @mihirp98]

Images, video, audio, actions — generative modeling has converged on one recipe: compress into continuous latents, generate in latent space, decode.

Language is the lone exception: still generated token by token, as a long discrete stream.

Latent Thought Flows: We compress 256 text tokens into 8 continuous latents, generate them with a one-step flow model, and read them out as text using an autoregressive decoder.

Result: a better inference-compute vs. generation-quality Pareto frontier than a tuned autoregressive baseline — thanks to compression and one-step generation.

Co-led w/ @ZhengyangGeng 🧵

[Wenhao Chai @wenhaocha1]

DeepSeek V4 (https://t.co/dSUi2tSkGq): their long-context recipe shares a lot with directions the video understanding community has been exploring for years. I had doubted these aggressive compression strategies would carry over to language tokens; they do.

MovieChat's (https://t.co/1sLhGB2m97) long/short-term memory ↔ V4's compressed and local-window KV. AuroraCap's (https://t.co/B8CP7ieCNc) token merging ↔ V4's KV compression. We also tried bringing NSA into video with VideoNSA (https://t.co/38cEKTF5pD).

Language tokens may not be as dense as we assumed. Keeping all of them isn't strictly necessary. Congrats to DeepSeek on the release. please deliver more!

[Shusheng Yang @shushengyang]

I’m excited to present VideoNSA at #ICLR2026 tomorrow (Pavilion 4, Poster #3307, 3:15–5:45 PM). Feel free to stop by and chat about sparse attention and long-video understanding!

[Scott Lowe @scottclowe]

New paper: "Self-Distillation of Hidden Layers for Self-Supervised Representation Learning"

We introduce Bootleg — a simple twist on I-JEPA/MAE that dramatically improves self-supervised representations.

The idea: MAE predicts pixels (stable but low-level). I-JEPA predicts final-layer embeddings (high-level but unstable). Bootleg bridges the two by predicting representations from multiple hidden layers of the teacher network — early, middle, and late — simultaneously.

Why it works: early layers provide stimulus-driven grounding that prevents collapse; deep layers provide semantic targets; and the information bottleneck of compressing all abstraction levels through masked patches forces the encoder to build richer representations.

The method is quite simple on top of I-JEPA: extract targets from evenly-spaced blocks, z-score and concatenate, widen the predictor's final layer. That's it.

Frozen probe results (no fine-tuning):

ImageNet-1K: 76.7% with ViT-B (+10pp over both I-JEPA and MAE)
iNaturalist-21: 58.3% with ViT-B (+17pp over I-JEPA, +15pp over MAE)
ADE20K segmentation: 30.9% mIoU with ViT-B (+11pp over I-JEPA, +6pp over MAE)
Cityscapes segmentation: 35.9% mIoU with ViT-B (+11pp over I-JEPA, +5pp over MAE)

Gains hold across ViT-S, ViT-B, and ViT-L.

Single-view, batch-size independent — no augmentation stack, no multi-crop, no contrastive loss, no large compute requirements.

Our study is just on images, but this change can be readily deployed to MAE and JEPA models across all domains.

https://t.co/PXJlRV4I6w

[Chongjie Ye @ychngji6]

https://t.co/k4DHsjmXJi

[Jia-Bin Huang @jbhuang0604]

Beyond softmax attention

Linear attention and its variants enable faster inference without growing the KV cache.

Let’s learn the core ideas behind efficient sequence modeling. 👇
https://t.co/geNiBXKdlI

[Wenhao Chai @wenhaocha1]

I believe that long-context multimodal modeling is a key path toward making advanced AI (whether AGI or ASI) truly useful for everyone. Today we are closer than ever, but major challenges remain. Broadly, I see two core problems: encoding and decoding.

Encoding. An AI system must be able to perceive and understand long-context multimodal content—for example, an entire day of human activity or the full history of a project. Such contexts naturally mix text, video, images, audio, code, actions, and more. My previous work pushes models toward deeper video understanding. Yet current systems still struggle with this goal; many even fail on long-context text-only tasks. This raises a central question: are we limited mainly by data, or also by model architectures and training strategies?

A further challenge is the tension between context and weights. A model’s understanding of the current context can be distorted by its pretraining. For example, a model trained heavily on PyTorch 1.0 documentation may mis-handle PyTorch 2.0 codebases. If an AI system always trusts its pretraining, it becomes less adaptable. If it always follows the given context, it becomes easy to manipulate and unsafe. This leads to an important question: Can an AI system continuously and selectively update itself from long-context signals at deployment time? I believe this is a promising and still underexplored direction.

Decoding. An AI system must also be able to generate long-context multimodal content—ideally within a single end-to-end model. Current LLM-based systems are strong at long-form text generation, but for multimodal outputs (e.g., images and video), only a few systems such as Nano Banana have reached practical usability.

This reveals many open questions in today’s dominant paradigms. Is it enough to use diffusion for visual generation while staying autoregressive for text? Can we bring multimodal reasoning strategies into visual generation? Is there a better visual tokenization method than patch-based representations? Can fully end-to-end training outperform diffusion? And can reinforcement learning be equally powerful for multimodal generation?

Efficiency is also critical. Modern architectures such as sparse attention, linear attention, and hybrid models are still underexplored, yet are likely essential for scaling.

Benchmarking long-context multimodal models is itself difficult. Even evaluating generated visual content alone is challenging—we still lack reliable alternatives to human preference for video evaluation.

By the year of 2025, long-context multimodal modeling has reached a stage where its potential is clear and within reach. However, significant work is still needed to fully realize this vision—and to bring powerful, reliable AI into the hands of everyone.

[Lambda @LambdaAPI]

https://t.co/aE4FpTbpBc

[Saining Xie @sainingxie]

Introducing Cambrian-S

it’s a position, a dataset, a benchmark, and a model

but above all, it represents our first steps toward exploring spatial supersensing in video. 🧶

[Wenhao Chai @wenhaocha1]

Our paper Video-MMLU has been awarded Outstanding Paper at the ICCV Workshop!
I happened to receive this wonderful news while soaking in the water couldn’t be happier!
Huge thanks to the Knowledge-Intensive Multimodal Reasoning Workshop Committee for the honor.

[Enxin Song @EnxinSong]

For more experiments & findings, check out:
📄 Paper: https://t.co/ZmJm6ZUbxc
🌐 Project: https://t.co/9P2DBkhsuu
💻 Code: https://t.co/aTru16jVFn
🤗 Model: https://t.co/l0FjibC1ew

[Enxin Song @EnxinSong]

Token compression causes irreversible information loss in video understanding. 🤔 What can we do with sparse attention? We introduce VideoNSA, a hardware-aware and learnable hybrid sparse attention mechanism that scales to 128K context length.

[Enxin Song @EnxinSong]

4️⃣ While attention sinks have been widely studied in recent LLMs such as GPT-OSS, we observe that learnable sparse mechanisms in VideoNSA can induce dynamic attention sinks, and the effect is branch-specific.

[Wenhao Chai @wenhaocha1]

🎉 We’re excited to host two challenges at LOVE: Multimodal Video Agent Workshop at CVPR 2025, advancing the frontier of video-language understanding! @CVPR #CVPR2025

📌 Track 1A: [VDC] Video Detailed Captioning Challenge
Generate rich and structured captions that cover multiple aspects of each video.
This challenge is based on the VDC benchmark (proposed by AuroraCap, ICLR 2025) and evaluated using VDCScore.
📊 The leaderboard has been updated with results from 40 models — check it out and benchmark your own!

🔗 Track 1A Overview: https://t.co/waTDLctn9Z
📈 VDC Leaderboard: https://t.co/gEKmgr15W3

📌 Track 1B: [Video-MMLU] Multi-Discipline Lecture Understanding Challenge
Evaluate your model’s ability to understand and reason over lecture videos in math, physics, and chemistry.
This dual-task benchmark combines video captioning (Review Notes) and question answering (Take Quiz), jointly testing perception, memory, and reasoning.
📊 We’ve evaluated 96 open-source and proprietary models on Video-MMLU — how well can your model do?

🔗 Track 1B Overview: https://t.co/tWUq8dGp9T
📈 Video-MMLU Leaderboard: https://t.co/bIH30YzgSq

🏆 Top 3 teams in each track will be formally recognized at the CVPR workshop and awarded official certificates.
🗓️ Submission Deadline: June 6, 2025

I’m so glad to be co-hosting the workshop with @MikeShou1 @EnxinSong @joyachen_ again.

[Enxin Song @EnxinSong]

For more details, please check below：

🌐 Website: https://t.co/qQArFsTg2P
📄 Paper: https://t.co/FZ9Liqfcs9
💻 Code: https://t.co/FiseKbC91L
📚 Data: https://t.co/AXKFd6UaVr

We thank the support of Lambda, Inc. for providing compute resources for this project. @LambdaAPI

[Enxin Song @EnxinSong]

🎉 Introducing Video-MMLU, a new benchmark for evaluating large multimodal models on classroom-style lectures in math, physics, and chemistry!

🧑‍🏫📚Video-MMLU requires strong reasoning capabilities and world knowledge compared to the previous benchmarks for video LMMs.

[Enxin Song @EnxinSong]

We also explore how the number of visual tokens and the base LLMs influence performance, offering insights into the interplay between multimodal perception and reasoning in lecture comprehension.