Kevin Zhao

AI Research Engineer @tiktok_us, @BrownUniversity @nyuniversity alum, 👨🏻‍💻 on foundation models for video generation.

[Ceyuan Yang @CeyuanY]

Glad to share Seaweed-7B, a cost-effective foundation model for video generation. Our tech report highlights the key designs that significantly improve compute efficiency and performance given limited resources, achieving comparable quality against other industry-level models. To unleash the power of the foundation model, Seaweed-7B further enables a wide range of downstream applications including image-to-video generation, human video generation, subject-consistent video generation, video-audio joint generation, long video generation and storytelling, real-time generation, super-resolution generation, camera controlled generation.

Check out our webpage and report for more details:
Webpage: https://t.co/5s9Af4FQCb
Paper: https://t.co/GHVs4cvELt

It's a wonderful journey of the last year. Thanks to all teammates for their contributions, sincerely.

[Kevin Zhao @KevinZ8866]

(4/4) Many thanks to @_akhaliq for sharing our work. My amazing co-authors: xingyuni @LouisCorneone @fncheng2333 ziyanyang @BohanW123 @roadjiang To know more about our work, please check out:
Website: https://t.co/JpAB4FAX0N
Huggingface: https://t.co/ozV9L2e7tp

[Kevin Zhao @KevinZ8866]

(2/4) 💡With cutting-edge graphics engines like Blender and Unreal, we simulate 3D scenes and synthesis videos with targeted controls. By carefully curating, we close the data distribution gap between and target those synthetic videos would enhance physics respecting of videoGen.

[Kevin Zhao @KevinZ8866]

(3/4) 🌟To bridge the appearance gap of CGI and real-world videos, we innovate on various aspect of videoGen model training and inference. Our approach boosts physical fidelity across multiple challenging tasks while keeps the outputs photorealistic.

[Kevin Zhao @KevinZ8866]

(1/4) What happens when CGI meets AI video generation? 🚀🚀 Excited to share our latest research: how synthetic videos enhance physical fidelity of videoGen. We build and curate CGI videos and incorporate them into videoGen to boost physical realism📽️ #AI #VideoGeneration #CGI

[AK @_akhaliq]

Synthetic Video Enhances Physical Fidelity in Video Synthesis

A turtle swimming in a green background. + video matting illustration

[AK @_akhaliq]

discuss: https://t.co/94DNBVzIjC

[Ceyuan Yang @CeyuanY]

Check out our latest work CameraCtrl II. By carefully collecting and processing data and introducing as little inductive bias as we can, users are allowed to explore the generated world with appealing dynamics and consistency. Together with extension and distillation, CameraCtrl II can support ultra-fast interaction and long-term exploration.

Homepage: https://t.co/QeXSAwEa4w

[Kevin Zhao @KevinZ8866]

(5/5) Many thanks to all my wonderful collaborators at Brown University. Haotian Fu @haotiannnnnnnnn, Chen Sun @jesu9 and George Konidaris @BrownBigAI

[Kevin Zhao @KevinZ8866]

#EMNLP2024 (1/5) Long-horizon decision-making/planning is a formidable challenge for LLM agents. We introduce EPO: Hierarchical LLM Agents with Environment Preference Optimization, to tackle this issue via task decomposition and modeling different environment preference signals.

[Kevin Zhao @KevinZ8866]

(4/5) Our method achieves top results in ALFRED (1st on the public leaderboard! https://t.co/1wHCiWgwXA)
Code: https://t.co/rGerNWeQpe
Arxiv: https://t.co/WqyZJF6ZpV

[Kevin Zhao @KevinZ8866]

What an honor to have my poster next to professor Efros oral presentation!! #ICLR2024

[Kevin Zhao @KevinZ8866]

(7/7)If you are interested, please check out our
- website: https://t.co/KOBoEIVKF6
- arXiv: https://t.co/r10TJoZRpx
-code: https://t.co/86YJp2hFoV
Or visit our poster at #ICLR2024

[Kevin Zhao @KevinZ8866]

(0/7) #ICLR2024 How could LLM benefit video action forecasting? Excited to share our ICLR 2024 paper: AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?

[AK @_akhaliq]

AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?

paper page: https://t.co/XtcnI7TmZV

Can we better anticipate an actor's future actions (e.g. mix eggs) by knowing what commonly happens after his/her current action (e.g. crack eggs)? What if we also know the longer-term goal of the actor (e.g. making egg fried rice)? The long-term action anticipation (LTA) task aims to predict an actor's future behavior from video observations in the form of verb and noun sequences, and it is crucial for human-machine interaction. We propose to formulate the LTA task from two perspectives: a bottom-up approach that predicts the next actions autoregressively by modeling temporal dynamics; and a top-down approach that infers the goal of the actor and plans the needed procedure to accomplish the goal. We hypothesize that large language models (LLMs), which have been pretrained on procedure text data (e.g. recipes, how-tos), have the potential to help LTA from both perspectives. It can help provide the prior knowledge on the possible next actions, and infer the goal given the observed part of a procedure, respectively. To leverage the LLMs, we propose a two-stage framework, AntGPT. It first recognizes the actions already performed in the observed videos and then asks an LLM to predict the future actions via conditioned generation, or to infer the goal and plan the whole procedure by chain-of-thought prompting. Empirical results on the Ego4D LTA v1 and v2 benchmarks, EPIC-Kitchens-55, as well as EGTEA GAZE+ demonstrate the effectiveness of our proposed approach. AntGPT achieves state-of-the-art performance on all above benchmarks, and can successfully infer the goal and thus perform goal-conditioned "counterfactual" prediction via qualitative analysis.

[Kevin Zhao @KevinZ8866]

(6/7)I am fortunate to collaborate with many amazing researchers @ShijieWang20, @cezhhh, Changcheng Fu, Minh Quan Do, Nakul Agarwal, @KwonjoonL, and most importantly @jesu9 on this work!