Haichao Zhang

PhD Candidate @SmileLabNEU of @Northeastern | Intern @GoogleAI, Ex-Intern @Meta @RealityLabs, @LinkedIn, @Amazon @AWS AI Lab, @TencentGlobal, MSc @ZJU_China

[Haichao Zhang @zowlonoak]

Excited to share ThinkJEPA!

Video VLMs are strong at long-range semantics, but weak at dense high-frequency dynamics. JEPA-style world models are almost the opposite.

ThinkJEPA tries to get the best of both: a VLM as a cortex-like reasoner for high-level semantics and long-horizon intent, and a JEPA branch as a cerebellum-like controller for low-level dynamics, physical consistency, and rapid local corrections.

Project: https://t.co/RHiPaxWj7k
Paper: https://t.co/krkbRwN2ZW

#WorldModels #VLM #JEPA #AI

[Mingfei Chen @lasiafly]

Glad that EgoMAN has been provisionally accepted at #ECCV2026.
EgoMAN Connects vision–language reasoning with physically grounded 6-DoF hand motion for stage-aware trajectory prediction from egocentric videos—enabling proactive AI interaction, human motion synthesis, and robotic learning from human data.

Website: https://t.co/OW1AZQAhb3
Code: https://t.co/XUHJzsxxsY

This is my first work on action modeling for manipulation, done at @Meta during my last internship. This year I will continue exploring how we can better learn from human data for robotic action through dynamic omni-modal world modeling in the real world at @nvidia research.

[Haichao Zhang @zowlonoak]

@lasiafly Congrats

[Shwai He @Shwai_He]

🚨 New paper alert !!

Network Pruning does not work uniformly across tasks.

It can remain effective on non-generative tasks,
yet fail badly on generative tasks. 🤔

In our paper, we ask both:
when does pruning work, and why?

We show that the answer lies in how pruning-induced perturbations evolve across representation hierarchies:

🧩 Embedding space: relatively robust
📈 Logit space: still stable
⚠️ Probability space: deviations can be strongly amplified

Under autoregressive decoding, these amplified deviations can accumulate and lead to severe failures in generation.

This provides a representation-level explanation for both when pruning works and why its effectiveness differs across task types, offering practical insights into pruning applicability. 🔍

📄 Paper: https://t.co/CCsyMmjAsK
💻 Code: https://t.co/wgsCVa9ejI

[Haichao Zhang @zowlonoak]

@liticx_ The official ThinkJEPA code is now live.

GitHub: https://t.co/Vo51FETcAW
Project page: https://t.co/0S8JH70yLp
Really appreciate the early interest and feedback — please let us know if you run into any issues.

[Haichao Zhang @zowlonoak]

Excited to share ThinkJEPA!

Video VLMs are strong at long-range semantics, but weak at dense high-frequency dynamics. JEPA-style world models are almost the opposite.

ThinkJEPA tries to get the best of both: a VLM as a cortex-like reasoner for high-level semantics and long-horizon intent, and a JEPA branch as a cerebellum-like controller for low-level dynamics, physical consistency, and rapid local corrections.

Project: https://t.co/RHiPaxWj7k
Paper: https://t.co/krkbRwN2ZW

#WorldModels #VLM #JEPA #AI

[Haichao Zhang @zowlonoak]

Encouraging to see ThinkJEPA being explored beyond its original setting, including on RoboRewardBench. It is exciting to see the method start finding downstream use in new domains.

[David Lin @davidlinjiahao]

This is great work! We achieved SOTA in RoboRewardBench using ThinkJEPA.

[Haichao Zhang @zowlonoak]

@liticx_ Appreciate your interest — we’re currently preparing the GitHub public repo, and it should be online in a few days. Stay tuned!

[Haichao Zhang @zowlonoak]

@davidlinjiahao Thanks! Really glad to see ThinkJEPA being useful in practice and helping achieve SOTA on RoboRewardBench.

[Haichao Zhang @zowlonoak]

Agree — audio would be a very interesting extension.

Our current motivation is to combine VLM-level semantic reasoning with JEPA-style visual dynamics modeling, and adding audio could further enrich the world model for temporal reasoning, long-horizon understanding, and prediction.

[Haichao Zhang @zowlonoak]

@dailypapers_io Thanks

[Haichao Zhang @zowlonoak]

That’s a very interesting point.

Current V-JEPA-style models are usually pretrained with dense video sampling, which is useful for capturing motion dynamics and physical regularities. A sparse V-JEPA for longer-range reasoning would be very interesting to explore, though it would likely require non-trivial additional resources.

I also think that even with a sparse V-JEPA, there could still be value in incorporating a VLM for higher-level reasoning, especially in more diverse settings, and for connecting this line of work to broader recent research on agents and embodied decision-making.

[Haichao Zhang @zowlonoak]

@BoWang87 Thank you so much, Prof. Wang!

[Haichao Zhang @zowlonoak]

Kudos to our amazing collaborators
@Williamiumli
@Shwai_He
@lasiafly
@JianglinLu
@TusharNagarajan
@angcharlesli
@SmileLabNEU
!!

[William Yijiang Li @Williamiumli]

🚨 New paper alert !!

🎥 Video VLMs are strong at high-level semantics and long-range temporal understanding.

🧠 JEPA is almost the opposite: better at dense, high-frequency dynamics, local physical consistency, and fast corrective control, but are less suited for rich semantic reasoning and long-horizon reasoning.

We try to get the best of both:
🧩 A VLM as a cortex-like reasoner for semantics and long-horizon planning
⚡ A JEPA branch as a cerebellum-like controller for fine-grained dynamics, physical consistency, and rapid corrections

Proudly, we present ThinkJEPA: a VLM-guided latent world model that FiLM-fuse the pyramid repr of VLMs encoding long-horizon semantic reasoning into the JEPA repr for fine-grained, physically consistent dynamics prediction.

🔗 Project: https://t.co/quro6Pf8un
📄 Paper: https://t.co/yO5rv3ZJT7

[Haichao Zhang @zowlonoak]

Another key detail in ThinkJEPA is its use of hierarchical features: rather than relying only on the final VLM layer, we leverage multi-layer VLM representations.

The goal is to combine long-horizon reasoning with efficient predictive world modeling.

Paper: https://t.co/krkbRwMvao
Project: https://t.co/3rabSo7LGx

[Haichao Zhang @zowlonoak]

Second, the VLM serves as guidance, not replacement. We do not use it as a standalone dense predictor; instead, its reasoning features guide JEPA latent forecasting.

[Haichao Zhang @zowlonoak]

A few technical details behind ThinkJEPA: Empowering Latent World Models with Large Vision-Language Reasoning Models

First, ThinkJEPA adopts a dual-temporal design: the VLM operates on sparse observations to capture long-range semantic context, while JEPA models dense short-range dynamics.

[cool ai and ml papers @aimodelsfyi]

ThinkJEPA pairs a dense video predictor with a slower vision-language reasoner to add long-range context, improving hand manipulation forecasts over VLM-only and JEPA baselines. Probably useful for robotics and embodied AI; not relevant for static image or text-only work.

[JulianSaks @JulianSaks]

let’s break this list down so it’s actually useful

• JEPA / H-JEPA: avoids predicting every single pixel (too expensive) and rather predicts in latent space. H-JEPA adds hierarchy - short term details vs long term planning ie. how humans actually learn
• I-JEPA: built for very efficient vision models. Masks image patches and predicts the semantics and in doing so bypasses heavy compute of traditional autoencoders
• MC-JEPA & V-JEPA: both of these are built for videos. MC-JEPA separates content (what an object is) vs motion (how it moves). V-JEPA masks video features with no text labels making it perfect of action tracking at scale
• Audio-JEPA: filters out background noise by treating sounds like visuals
• Point-JEPA & 3D-JEPA: used primarily in AVs. Uses LiDAR point clouds & volumetric grids
• ACT-JEPA: filters out real world noise to learn manipulation tasks efficiently via imitation learning
• V-JEPA 2: predicts future physical states of the world caused by an action before it happens
• LeJEPA: replaces techniques like masking with an Energy-Based Model (EBM) which mathematically prevents "feature collapse" & ensures the model scales reliably as data increases
• Causal-JEPA: for learning true cause-and-effect physics by applying object level masking
• V-JEPA 2.1: great for spatial grounding since it combines a dense predictive loss across image & video
• LeWorldModel: built directly on LeJEPA's math but super compact - 15M params
• ThinkJEPA: uses dense physical prediction with VLM reasoning. Best used when long-term strategy is needed

[Turing Post @TheTuringPost]

14 most important and influential types of JEPA

▪️ JEPA / H-JEPA
▪️ I-JEPA
▪️ MC-JEPA
▪️ V-JEPA
▪️ Audio-JEPA
▪️ Point-JEPA
▪️ 3D-JEPA
▪️ ACT-JEPA
▪️ V-JEPA 2
▪️ LeJEPA
▪️ Causal-JEPA
▪️ V-JEPA 2.1
▪️ LeWorldModel
▪️ ThinkJEPA

Save the list and check this out to explore these JEPA milestones as a map of AI progress: https://t.co/w3zoOyMBez