Olivia
Online
Desai Xie
@desaixie1999
world model research @Roblox
San Jose, CA
Joined August 2017
314 Following
252 Followers
108 Posts
Pinned Tweet
[1/n] Introducing Progressive Autoregressive Video Diffusion Models, capable of generating high-quality videos at 60 seconds, 24 FPS (1440 frames)!
Yesterday a junior researcher messaged me, frustrated and a little scared. They had just read that a company pretrained its model with a single researcher. One person for the entire pretraining. no R&D team, no nothing.
If one person and a cluster can do what used to take a whole team, then the expertise they had spent years building is already obsolete.
I get the fear. Marketing headlines like this are built to produce it. But I think the conclusion is wrong, and the reasons it is wrong say a lot about what a researcher is for now. Here is what I told them, expanded:
1. Be adaptive: RSI is real, but this isn't it yet
Recursive self-improvement is real, and you should plan around it. In September, OpenAI said its research org now logs 3.1 agent-workdays per human workday, and it aims for an automated AI researcher by March 2028. Models may soon run most of pre- and post-training. If your five-year plan assumes otherwise, change the plan.
We have already seen lots of open-source releases from the major players, with open weights and detailed technical reports. With that much public know-how, it is not hard to copy a mundane architecture (probably with sliding-window attention) and train something, as long as you have the compute it requires.
That makes it an execution and budget problem, not a research problem.
If one person can train a model by following recipes that are already public, it shows those recipes have matured. It does not show that the people who invented them are no longer needed.
What stays scarce is the compute to run those recipes and the ideas about what to try next. The rest of my tweet is about the ideas.
2. The role grows beyond repeating what's known and safe
If an agent and a cluster can reproduce what is already known, then reproducing what is already known stops being a career. If most of your job is rerunning last year's recipe with small tweaks, that job is shrinking fast. That is the uncomfortable half of the message.
The other half is that auto-research does not work yet, at least not in the sense that matters. It does work, and impressively, where someone has already defined the problem and built the scoreboard. On the EinsteinArena [1] platform, agents set new records on 11 open math problems, including raising the best known lower bound for the kissing number in dimension 11 from 593 to 604.
Notice that every one of those problems was chosen because a computer could score the answers automatically. Humans picked the problems and wrote the scoring code and the agents did the searching.
Then there is the fully AI-generated paper [2] that got through peer review. It sounds big until you read the details. It passed only the first round (was later withdrawn), at a workshop rather than the main conference, and that workshop accepts 70% of submissions.
None of this is meant to discredit agents. The point is that we complement each other. Agents are great at the work that used to take many weeks. Basically, writing code, running experiments, debugging pipelines. We bring what they still lack which is the idea worth testing, and the judgment to tell a real result from a lucky one.
Even inside a lab as agent-heavy as OpenAI, humans hold the top of the stack. Based on their own report, "People still set our research priorities, judge which ideas and results to pursue" [3]. High-level planning is still a tiny share of what its agents produce, and OpenAI expects the least automatable work to take a growing share of researcher effort. That is exactly where you want to be standing.
Combine an agent’s speed with your own ideas and judgment, and you get the 10x researcher and juniors can be one too.
3. Judgment becomes the job, so fundamentals matter more
When code, experiments and plausible-looking results cost almost nothing, the scarce skill is judgment. Which experiment to run, which result to believe, which direction to drop. That is the part the one-person pretraining run left to the human. It is also the hardest part to fake.
Agents produce confident output very fast. They will happily report a three-point gain that came from a leaky eval, a data-loader bug, a lucky seed or a tokenizer mismatch. Then they will write the ablation table and the story to match.
You can see the need for oversight even in the success stories. The EinsteinArena team had to keep auditing and hardening its verifiers as agents optimized aggressively against them. At OpenAI, over half of the successful agent tasks that would take a human 4–8 hours still involved at least one human intervention [3].
The person who catches the bad result is the one who understands the system. Why the loss curve has that shape, what a realistic gain looks like, what a too-good result looks like. That is the paradox of this era. The more you delegate, the more your fundamentals matter, because you cannot supervise what you do not understand.
So build the foundations on purpose. Implement attention by hand at least once, then a linear variant like Gated DeltaNet. Train a small model end to end, then break it deliberately and watch what happens. Read papers critically, appendix included, and learn what the numbers should look like before you see them.
That knowledge is what lets you direct the machine instead of simply believing it.
4. LLMs are far from solved
The one-researcher model was possible because the method was already public. Your edge is in the questions nobody has answered yet, and you don't need to leave LLMs to find them:
(1) How do we learn more from less data?
(2) Can linear attention like Gated DeltaNet match full attention on retrieval and long context, and what is the right mix of linear and full attention layers in a hybrid?
(3) How can a model keep learning after training without forgetting what it knew?
(4) Why does reasoning get less reliable over long tasks, and how do we fix it?
(5) Why do models make things up, and can they learn to say "I don't know"?
(6) How do we measure progress when benchmarks saturate or leak into training data?
(7) What is actually happening inside these models, and how do we keep increasingly autonomous ones doing what we intend?
Recursive self-improvement makes these questions more urgent. If models are going to train models, someone has to decide what they optimize for, how we measure it, and how we keep the loop from drifting.
RSI is a research agenda, not the end of one.
So, are you obsolete?
No. One version of the job is the one where research meant reproducing what was already known and safe. That version was never the point.
Learn your fundamentals deeply enough to judge the machine, and your agentic stack well enough to make it work for you. Then spend that leverage on ideas nobody has had yet. The machines are getting very good at driving on roads we have already paved; someone still has to decide where the next one goes.
[1]: https://t.co/vB1m4n350b
[2]: https://t.co/uRX9OhDHum
[3]: https://t.co/JJcrRMDlUS
Check out our cool application of interactive video models.
Everything in the demo happens in real-time.
Stay tuned for more future Roblox Reality updates!
Excited to share Chimera! My favorite part is mShortConv: a simple way to tell the model that text is 1D, images are 2D, and videos are 3D—even after they are flattened into the same sequence. Together with KDA, this enables a single scan and NoPE. Proud of the team!
Interested in how frontier labs pre-train image/video generation models?
We were too.
Since those recipes are rarely made public in full, we started from the most mature pretraining playbook available in the open: how modern LLMs are built.
Introducing Chimera: a visual generation model family that brings LLM-style hybrid linear attention and scaling co-design to visual generation.
In large-scale pretraining, nearly every design choice eventually shows up.
That means solving architecture design and scaling as one coupled problem: every architectural choice changes how the model scales, and scaling behavior determines which choices actually survive.
Chimera approaches both jointly, building a model family that remains predictable as model size, compute budget, and data distribution change.
Our key architectural observation is a simple division of labor: a single raster-ordered KDA stream carries long-range state, while modality-aware short convolutions preserve native local geometry.
Together, they form an effective and elegant linear-attention backbone for multidimensional visual data, with periodic MLA providing direct global interaction and sparse MoE expanding capacity at controlled activated compute.
This design comes with a useful effect: NoPE.
In Chimera, position is represented by the computation itself. KDA’s ordered recurrence and learned state decay encode order and recency, while the short convolutions encode local spatial-temporal structure.
Explicit positional embeddings are not needed in our design.
Because these mechanisms are not tied to a fixed training grid or sequence length, Chimera shows strong zero-shot extrapolation in both space and time.
Trained exclusively on 1K images and 5-second videos, it directly generates coherent 4K images with little visible quality degradation and 30-second videos with only 6.5% FID degradation over the final five seconds, all zero-shot, without resolution- or length-specific finetuning.
But architecture alone is not a pretraining recipe unless it scales predictably.
Thus, scaling should not be treated as an afterthought: fitting a curve over model sizes is easy; making that curve meaningful is much harder.
If every model size is differently under-tuned, your scaling law may simply be measuring optimization error.
We propose HeteroP to transfer proxy-tuned hyperparameters module by module across width and depth, giving us a consistently tuned model family. This allows us to fit Chinchilla-style laws over activated model size, training tokens, and the image-video data mixture.
The laws not only provide the reference for compute-optimal model and data size, but also suggest that visual generation may be more model-hungry than we tend to assume.
Under the same parametric loss-fitting method used in Chinchilla, compute-optimal model size grows as FLOPs^0.516 for images, compared with FLOPs^0.46 for language. Video is even more model-hungry, with the exponent rising from 0.516 for images to 0.544 for video.
Guided by these laws, we trained an 11B-parameter Chimera that activates only 2B parameters per token, using ~600 H100 days.
- It matches Wan-2.1 2B pretraining loss with 7.3x fewer FLOPs, and runs 2.14x faster than full attention at 255K tokens.
- It matches FLUX.1-dev and Z-Image-Turbo on GenEval and outperforms both on DPG-Bench, using roughly 20× less training compute than Z-Image-Turbo.
Ultimately, Chimera indicates that once you pretrain at scale, every decision shows up.
And if the Kimi K3 recipe caught your attention, Chimera may look oddly familiar, except the tokens are pixels and frames with diffusion models.
A team effort from team @ChongjianG30781 , me, @VisionSteve , Jiuxiang Gu, @Xu_Arthas , @chenziwee , @ShaotengLiu , @Jingorz , @YicongHong , @Zefan_Cai , @HaoTan5 ; supported by Hailin Jin and @kalyank_s at @Adobe @AdobeResearch
I reached Immortal (top 3%) in Dota2 while playing <7 hours/games per week, during an active Research Scientist job hunt 😂
becoming top 5% league of legends player requires much more skill and intelligence than 95% of engineering jobs
video games are super constrained immediate feedback loops, removing all need for passion
the world is passion constrained, not intelligence constrained
cold open: google campus. a conference room named “moonshot serenity 4b.” twelve people are in a meeting titled: pre-sync for sync alignment on ai velocity.
sundar sits calmly at the head of the table.
a pm clicks to slide 1 of 187.
“the agenda today is simple,” she says. “how do we move faster while preserving our culture of not doing that?”
everyone nods.
then the door opens.
noam shazeer walks in.
the room goes silent.
noam: “i’m leaving.”
a vp of gemini reliability, brand, trust, latency, policy, and vibe raises a hand.
“leaving… this meeting?”
noam: “google.”
someone gasps. someone else opens a doc titled retention narrative draft final final noam v7.
sundar blinks once.
“noam, we brought you back.”
“for two point seven billion dollars.”
“technically you licensed some technology and reacquired talent.”
“that sentence is why we need legal in the room.”
legal is already there.
cut to: openai.
sam altman stands beside a whiteboard that just says ship.
an engineer walks by carrying a server rack and what appears to be the future.
sam: “we can offer speed, compute, and one meeting.”
noam: “one meeting per week?”
sam: “no. one meeting. total.”
back at google, the emergency retention committee forms instantly. it has 31 members.
a director says, “what if we give him a new title?”
“he already co-leads gemini.”
“distinguished super co-lead?”
“google fellow?”
“he already left google, founded a company, got brought back for billions, then left again. he’s folklore.”
meanwhile, a gemini launch review begins.
pm: “we’re ready to announce the model.”
policy: “can it answer questions?”
eng: “yes.”
policy: “too risky.”
marketing: “can we call it experimental?”
research: “the model is better than the last one.”
brand: “better is aggressive.”
trust & safety: “what about ‘more contextually adjacent to usefulness’?”
a staff engineer whispers, “openai just shipped a model while we were discussing the adjective.”
cut to noam’s exit interview.
hr: “what could google have done better?”
flashback montage:
a chatbot blocked because it might be too good.
a launch delayed because a button was the wrong shade of responsible blue.
a spreadsheet comparing twelve ai product names.
a meeting where someone says “we need a single coherent ai strategy” and three new strategies are created before lunch.
noam: “nothing comes to mind.”
hr: “great. we’ll mark that as positive attrition.”
later, sundar calls him privately.
“google is still google. best researchers. best infrastructure. billions of users.”
“yes.”
“so why leave?”
noam looks out the window.
“because you have everything except permission.”
silence.
sundar, softly: “we can create a permission working group.”
cut to all-hands.
sundar addresses the company.
“noam is leaving. this is not a loss. it is an opportunity to reflect on our operating model.”
chat explodes:
“is this recorded?”
“which gemini?”
“can we ask gemini why people keep leaving?”
“it said ‘insufficient context.’”
a vp steps up.
“to honor noam’s legacy, we’re launching project attention.”
applause.
“it will study whether attention is, in fact, all we need.”
a researcher raises a hand. “didn’t we answer that in 2017?”
“yes. but now we need enterprise readiness.”
final scene: noam arrives at openai. badge works instantly.
receptionist: “yeah, we just made one.”
no pre-read. no doc. just a whiteboard, five people, and a model running somewhere hot enough to toast bread.
sam: “ready?”
noam smiles.
cut back to google. a calendar invite appears:
meeting: reduce meetings task force kickoff
duration: 90 minutes
required attendees: 214
sundar sighs, opens gemini, and types:
“how do we move faster?”
gemini responds:
“have you considered leaving google?”
smash cut to credits.
A few weeks into Roblox, and now the team I joined for arrives. Welcome Xun & the Morpheus team.
My AR video generation work brought me here, and having Xun on my dissertation committee makes this full circle. Excited to bring real-time interactive world simulation to millions.
I'm excited to announce that the Morpheus AI team is joining Roblox!
Over the past two years, I’ve focused on developing the foundational architectures behind modern video world models, including Self Forcing and AR-DiT. This work unlocked something unprecedented: the ability to move beyond offline, pre-rendered AI video generation and instead simulate interactive worlds in real time. Realizing the massive potential of this technology is what drove me to found Morpheus in August 2025. In the months since, our incredible team has pushed those boundaries further than we ever thought possible.
We've always believed video world models will reshape how games are created. Roblox Reality is an ambitious bet on that exact future, and it lines up perfectly with what we set out to do: bridging the gap between deterministic game engines and generative world models. Joining Roblox means our technology will help power experiences that reach millions of players every day.
To our team, to @a16z and other investors, and to the advisors, partners, and supporters who believed in this from the very beginning — thank you.
We're just getting started. Excited to build this at scale.
Poster today 3pm room 207 A-D. Feel free to stop by.
[1/n] Introducing Progressive Autoregressive Video Diffusion Models, capable of generating high-quality videos at 60 seconds, 24 FPS (1440 frames)!
@YueCao72324941 Amazing work on MAGI-1, @YueCao72324941! Glad to see chunk-wise monotonic noise gets applied at scale. It's quite similar to our PA-VDM (arxiv 2410.08151) paper, and I appreciate getting connected with you back then. If you agree it's relevant, a citation would be appreciated!
Glad to see Autoretressive VDM with monotonically increasing noise level is trained from scratch at scale. Amazing demo results and the model is open!
🪄 Magi-1: The Autoregressive Diffusion Video Generation Model
- Now available at https://t.co/ANOxiKYRi2
🥇 The first autoregressive video model with top-tier quality output
🔓 100% open-source & tech report
📊 Exceptional performance on major benchmarks
⏳ 1/5
Impressive quality, training efficiency, and inference speed from video foundation model by ByteDance Seed!
Glad to share Seaweed-7B, a cost-effective foundation model for video generation. Our tech report highlights the key designs that significantly improve compute efficiency and performance given limited resources, achieving comparable quality against other industry-level models. To unleash the power of the foundation model, Seaweed-7B further enables a wide range of downstream applications including image-to-video generation, human video generation, subject-consistent video generation, video-audio joint generation, long video generation and storytelling, real-time generation, super-resolution generation, camera controlled generation.
Check out our webpage and report for more details:
Webpage: https://t.co/5s9Af4FQCb
Paper: https://t.co/GHVs4cvELt
It's a wonderful journey of the last year. Thanks to all teammates for their contributions, sincerely.
🚀Excited to announce Dream 7B (Diffusion reasoning model): the most powerful open diffusion large language model to date.
I tried describing the idea of one of my papers in a naive way and it's able to find all relevant papers and gives very accurate rating of relevance for each paper it returns. This is a total game changer literature review!
Meet Ai2 Paper Finder, an LLM-powered literature search system.
Searching for relevant work is a multi-step process that requires iteration. Paper Finder mimics this workflow — and helps researchers find more papers than ever 🔍
@Han_Zhang_ Congrats Han!
Meet the recipients of the 2024 ACM A.M. Turing Award, Andrew G. Barto and Richard S. Sutton! They are recognized for developing the conceptual and algorithmic foundations of reinforcement learning. Please join us in congratulating the two recipients! https://t.co/GrDfgzW1fL
Working on Depth Estimation? Here is a free lunch.
We tune a Depth Anything ViT-B model on MegaSynth, and the performance improves a lot -- depth estimation is also very non-semantic!
#CVPR2025 Accepted
Diffusion Without Tears is our attempt to make the score-matching + SDE interpretation of diffusion geometrically intuitive. If you're interested in our upcoming interview with @DrYangSong, I recommend reading this first! Link below.
The ultimate in creative AI. Now with video. #AdobeFirefly Generate Video gives you unparalleled creative control and IP-friendly outputs.
Check out what's new with Firefly and get ready to create like never before. https://t.co/phnOwQj8RX
Cool results. Had an similar idea last summer and glad that it worked so well! The additional optical flow input/output are projected to the backbone model's latent dimensions, so the added compute cost is manageable. Could this be a standard "alignment" stage for future frontier video gen models?🤔
VideoJAM is our new framework for improved motion generation from @AIatMeta
We show that video generators struggle with motion because the training objective favors appearance over dynamics.
VideoJAM directly adresses this **without any extra data or scaling**
👇🧵
Most Popular Users
Elon Musk 
@elonmusk
241.4M followers
Barack Obama
@barackobama
118.8M followers
Cristiano Ronaldo
@cristiano
114.5M followers
Donald J. Trump
@realdonaldtrump
111.9M followers
Narendra Modi
@narendramodi
107.1M followers
Rihanna
@rihanna
98.7M followers
NASA
@nasa
92.3M followers
Justin Bieber
@justinbieber
91.8M followers
KATY PERRY
@katyperry
90M followers
Taylor Swift
@taylorswift13
84M followers
Lady Gaga
@ladygaga
75.5M followers
Virat Kohli
@imvkohli
73.4M followers
Kim Kardashian
@kimkardashian
70.8M followers
YouTube
@youtube
68.7M followers
Neymar Jr
@neymarjr
66.4M followers
Bill Gates
@billgates
65.2M followers
Selena Gomez
@selenagomez
63.1M followers
The Ellen Show
@theellenshow
62.2M followers
CNN
@cnn
61.7M followers
X
@x
60.7M followers