Lei Ke

Senior Research Scientist @TencentHunyuan; Previously at CMU @SCSatCMU @mldcmu,ETH and HKUST @eth_en @cvl_eth.

[Shunyu Yao @ShunyuYao12]

Hy2 -> Hy3 preview -> Hy3

Another massive leap forward, under half a year.

Not just a leap of reasoning or agentic capabilities.

Also a leap of anti-hallucination, reliability, and product experiences.

More on the way and so proud of the team! 🧑‍🍳🧑‍🍳🧑‍🍳

[Sonya Huang 🐥 @sonyatweetybird]

Today's Training Data episode takes us BTS on the infrastructure challenges required to do large RL runs at scale, featuring @ellev3n11 (Composer Lead at @cursor_ai) and @dzhulgakov (Co-Founder at @FireworksAI_HQ).

The Cursor team trained Composer 2 on Fireworks by starting with a strong base model (Kimi 2.5) and performing large-scale mid-training on code tokens and web data to learn common patterns and libraries, followed by a large-scale Reinforcement Learning run to learn how to navigate the Cursor harness, call tools, and write correct code.

Today's episode dives into the systems and infrastructure challenges of making that large RL run happening, and there were many (!!), from numerical mismatch to global distribution to synchronizing rollouts across asynchronous pipelines to keeping track of expert activation across runs and more.

Extremely nerdy in-the-weeds challenges that Federico and Dima were delighted to nerd out on together :)

Beyond RL infra, we also discussed Online vs Simulated rollouts, self-summarization for long-horizon agents, environment design ("the most powerful RL environment is the product itself"), and other technical nuggets.

PS: We filmed this episode before the SpaceX news, while the Cursor team was still compute-constrained. While Cursor now has *all* the flops, the takeaways and hurdles crossed ring true for any serious application-level company that is racing to post-train their own models.

I believe that more serious application companies will go the way of Cursor and post-train their own models.

00:00 Introduction
00:53 Why Cursor Trained Composer 2
04:55 Specialization vs Bitter Lesson
06:16 Composer 2 Training Recipe
16:32 Scaling RL Infrastructure Globally
23:32 Floating Point Drift
25:11 MoE Sensitivity Explained
26:25 Router Replay Fix
27:19 Real Time RL Loop
31:49 Long Horizon Agents
34:29 Why RL Everywhere
37:34 LLM as Judge Rewards
39:14 RL in Hard Domains
40:13 Build Your Own Environments
44:34 Closing Thoughts

[Tencent Hy @TencentHunyuan]

👋Hi /haɪ/, we're the Tencent Hy /haɪ/ team🐧

Today, we open source Hy3 preview (295B A21B), a leading reasoning and agent model in its size, with great cost efficiency.

Give us feedback to help improve Hy3 official!

🤗 https://t.co/jc10JODXJ8
📖 https://t.co/VIRoNnwng0

[DailyPapers @HuggingPapers]

Tencent just released Penguin Recap V on Hugging Face

5.8M multi-granularity video annotations spanning
dense timestamps, paragraphs and full summaries.

https://t.co/VQhvAzllXr

[DailyPapers @HuggingPapers]

Tencent just released the Penguin recap dataset on Hugging Face

68 million multimodal samples spanning DataComp, COYO, SA-1B and OpenImages
for training efficient Vision Language Models with LLM-based encoders.

[Tencent AI @TencentAI_News]

Penguin-VL is now open source 🐧

[Adina Yakup @AdinaYakup]

Penguin VL 🐧 Efficient Vision Language Model with LLM-based vision encoder from @TencentAI_News

https://t.co/Lley22hUGw

✨ 2B / 8B - Apache 2.0
✨ Long video reasoning
✨ LLM initialized vision encoder : tight V-L alignment
✨ Compact & strong: SOTA on image, document, OCR & video

[Machine Delusions @Machinedelusion]

I've implemented the new Penguin VL language model into Comfy

performs quite well for the size in both a 2B and 8B variant

You can find it in the link below

[AK @_akhaliq]

Penguin-VL

Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders

app: https://t.co/VZ8IvEdjN3

paper: https://t.co/XSM2GGVcCz

[DailyPapers @HuggingPapers]

Tencent released Penguin-VL

A compact vision-language model that replaces traditional CLIP/SigLIP pretraining with an LLM-initialized vision encoder, delivering strong multimodal reasoning with just 2B and 8B parameters.

[Lei Ke @leike_lk]

@ysu_ChatData @HuggingPapers hi Yongrui, thanks for your comment. We provide detailed encoder ablation exps using the same training data in the experiment part of our paper: https://t.co/ylVOAECZmq

[Lei Ke @leike_lk]

🔗 HF Daily Paper: https://t.co/FK0SqKACHV (Upvotes appreciated!)
🔗 Arxiv Paper: https://t.co/ylVOAECZmq
🔗 HF Models (2B/8B & Vision Encoder): https://t.co/04Z6SAKDoU
🔗 Project Page: https://t.co/1xwicEG3Mn

[Lei Ke @leike_lk]

🚀 Penguin-VL is out!

We ask a simple question: does a VLM vision encoder really need to start from CLIP/SigLIP-style contrastive pretraining?

Penguin-VL initializes the vision encoder directly from a text-only LLM, leading to competitive results at 2B/8B scales.

[DailyPapers @HuggingPapers]

Tencent just released Penguin-VL on Hugging Face

It swaps CLIP for an LLM-based vision encoder (Qwen3), achieving 86.8 InfoVQA, 90.5 ChartQA and 96.2 DocVQA.

https://t.co/7arOxvYUjd

[Lei Ke @leike_lk]

🔥 Tired of static, lifeless edits?

MotionEdit is an optical-flow–guided post-training recipe that consistently boosts FLUX and Qwen-Image-Edit, teaching models to understand action, motion, and interaction—not just appearance changes!

👉 https://t.co/SpaYEKoK6p

[Yixin Wan @ACL2026 @yixin_wan_]

🤔🤔Tired of static, lifeless image edits?
Not anymore! 🤗

🚀🚀We introduce MotionEdit, a framework supporting image editing that understands action, motion, interaction beyond static changes! 🤩🤩

🔗Full paper: https://t.co/lKKQX6DJPj
✨Project page: https://t.co/3XdkxVgM4b

[Yixin Wan @ACL2026 @yixin_wan_]

🤔🤔Tired of static, lifeless image edits?
Not anymore! 🤗

🚀🚀We introduce MotionEdit, a framework supporting image editing that understands action, motion, interaction beyond static changes! 🤩🤩

🔗Full paper: https://t.co/lKKQX6DJPj
✨Project page: https://t.co/3XdkxVgM4b

[Chen Geng @gengchen01]

Join us tomorrow at the #ICCV2025 workshop on generating digital twins from images and videos! Don’t miss amazing talks from Manolis Savva, @KaterinaFragiad, @mapo1, @jiajunwu_cs, @MattNiessner, @craigleili, @yanpei_cao, and @bgxc on cutting-edge progress!

@ICCVConference

[Frano Rajič @franorajic]

1/4 🚀 We’re excited to release MVTracker (ICCV 2025 Oral), the first data-driven multi-view 3D point tracker. MVTracker tracks arbitrary 3D points across multiple cameras, handling occlusions and varied camera setups without per-sequence optimization.

[Kwang Moo Yi @kwangmoo_yi]

Rajič et al., "Multi-View 3D Point Tracking"

Multi-view 3D points, iteratively updated by a Transformer with their neighbors used as context. Leads to significantly improved 3D tracks -- ICCV Oral

[Chen Geng @gengchen01]

📢 Call for Papers - We are organizing @ICCVConference Workshop on Generating Digital Twins from Images and Videos (gDT-IV) at #ICCV2025! We welcome submissions in two tracks:
📅 Deadline for Archival Paper Track: June 27
⏰ Deadline for Non-Archival Paper Track: July 31
🌐 Workshop website: https://t.co/Tyxg8nMPi3

[Dreaming Tulpa 🥓👑 @dreamingtulpa]

TAPIP3D can track 3D points in videos