Juanca Pérez

AI Research Scientist @AIatMeta, working on Generative AI for videos. My opinions are provisional, and I reserve the right to change them. From 🇨🇴

[Juanca Pérez @juancaperezs]

📢 Hiring a PhD intern in Generative AI @Meta (London) to work with my team on video generation.

If you’ve worked on diffusion / flow matching (plus ideally LLMs/VLMs) and have strong engineering skills, apply here: https://t.co/HxYgEVBa7T
24-week internship preferred

[Juanca Pérez @juancaperezs]

@johnschulman2 i'd agree in the general case. but post-training showed a couple k good traces outrun any tonnage, and 2 experts left all their thinking inside a coding tool, including dead ends

if textbooks = hard labels, these notes = soft ones, aka Hinton’s “dark knowledge” in distillation

[Juanca Pérez @juancaperezs]

@RickyTQChen A pleasure to have collaborated with you, Ricky! Best of luck in what comes next!

[Juanca Pérez @juancaperezs]

If you are attending ICML in South Korea, please come see our poster on a super cool paper led by Won:
“Rays as Pixels: Learning a Joint Distribution of Videos and Camera Trajectories”

[Wonbong Jang / Won @wbjang11]

Presenting Rays as Pixels at #ICML2026 tomorrow!

Wed Jul 8, 2:30-4:15 PM KST, Hall A #2604

One diffusion model, joint distribution over videos + camera trajectories: camera-controlled generation, pose prediction, and joint generation of both.

[Wonbong Jang / Won @wbjang11]

Presenting Rays as Pixels at #ICML2026 tomorrow!

Wed Jul 8, 2:30-4:15 PM KST, Hall A #2604

One diffusion model, joint distribution over videos + camera trajectories: camera-controlled generation, pose prediction, and joint generation of both.

[will depue @willdepue]

A Stargate for Data

Labs are on a trajectory towards >$100B/year of data spend by 2030. As we begin the trillion-dollar compute project, we need to think about the equivalent civilizational-scale effort for the other core ingredient: data.

At the foundation of the scaling revolution is a simple empirical law: deep neural networks improve smoothly, near magically, as you scale two things in proportion — (1) the size of the model and (2) the amount of data you train on. And despite the scaling laws being brutally diminishing, we’ve successfully bitten the bullet of logarithmic scaling with exponentially larger clusters and datasets, and received incredible new capabilities in return.

But this exponential scaling is bound to hit some limits. Oddly enough, compute has compounded fairly smoothly without limit, with trillions flowing into hypercluster buildout. Instead, we’re starting to hit the limits of an exponential demand for data. Gone are the days of being purely in the compute-limited regime, where we had effectively infinite internet data but never enough GPUs, we’re now entering a data-limited regime.

Luckily, this limitation is coinciding with staggering improvements in AI capabilities. Incredibly, we seem to have a real line of sight towards automating a majority of knowledge work with the methods we have today. RL + pretraining, and the data for each, will be generally sufficient to achieve most economically valuable tasks, given some minimal algorithmic progress and continued compute scaling.

In a data-limited world, economic progress & scientific acceleration will be directly bottlenecked by our coverage in each domain. We need to see data collection as imperative, deserving the same civilizational ambition we’ve given compute.

The internet as a one-time subsidy

It’s underrated how much all progress in AI owes everything to the blessing of the internet, this one-time civilizational subsidy to deep learning, decades of unintentional accumulation of a perfect dataset: every book, blog post, image, video, paper, discussion, etc. all digitized and freely available. Without the internet, we’d likely see comparably minimal progress in AI today, and in fact, if you notice where systems currently underperform, it’s almost always a domain where web coverage is limited and data is private, expensive, non-digitized, or non-existent.

But we’re running out of it. There are only about 300 trillion tokens of useful public human text, and the internet doesn’t produce nearly enough new high-quality data to match what scaling demands — we’re soon to hit the limits of public data for pretraining. And though the advent of RL bought us reprieve — chain-of-thought RL needed a new form of untapped data, gradable math & coding tasks, also available online — we’re quickly running dry of hard tasks for RL as well.

Why do we need so much data anyways? Humans learn comparably in far less time, needing just one textbook where language models might need the equivalent of hundreds to learn a new topic. It’s possible we discover methods that are massively more data efficient — synthetic data, data efficient architectures, other exotic algorithms — but fundamental progress is slow and highly unpredictable, and the recipe we have just works today.

And, while I’m wary of getting too deep here, even arbitrary data efficiency can’t replace data that just doesn’t exist in the first place. There’s a massive amount of missing information on the web: the dark matter of the internet — tacit knowledge, undocumented processes, etc. — most of which was never published and lives only inside organizations, the physical world, or just in people’s heads. I’ll leave it here and say, for reasons far longer than I can fit in this post [1], it’s best to operate on the assumption that our insatiable desire for data will continue as it has for the last decade.

There will be >$100B/year in data spend by 2030

We’re not screwed yet, of course. Only a fraction of useful data in the world is on the public internet, the rest is stored inside private datasets, corporations, personal archives, universities, governments, and otherwise. Labs can and will continue to license these private datasets, or create them from scratch, like Anthropic’s book scanning project. And we’ll increasingly task human experts to manufacture new high-quality data, with a large fraction of hard RL training tasks already being sourced this way.

But collecting this data, unlike before, will be expensive. As the free internet dries up and demand for data rises, we should see labs investing equally in data as compute, likely spending a significant fraction of their compute budgets on data. As we see trillions spent on compute, we should also expect hundreds of billions spent on data (human data & collection budgets), given their equivalent importance. And, notably, data spend is already tracking this way: total data spend across vendors, not counting internal lab efforts, is already roughly $7 billion per year. It’s quite reasonable we’ll see >10x by 2030.

Data is the moat

Data becoming increasingly private will also majorly shift the competitive landscape. While compute is a commodity — everyone buys the same chips and builds the same clusters — data really isn’t. The big reason why frontier models have felt eerily similar to one another, until now, is they were trained on substantially the same internet (pretraining data variability across labs seems pretty low). As labs diverge onto more exclusive, manually collected corpora, I think models will begin to increasingly diverge.

OpenAI pulling ahead in mathematics and Anthropic in cybersecurity isn’t an accident. I really think laser-focused collection of high-quality midtraining tokens, custom RL tasks, environments, with dedicated research effort, has driven much of the visible progress in the last year. James Betker has an excellent blog about “the ‘it’ in a model is the dataset”: model architecture and compute buy you efficiency and order-of-magnitude performance, but ultimately, models, of any architecture, are such incredible approximators of their dataset that the core meat of a model boils down to just that, nothing else. Data is a major moat.

AGI long, ASI short

As I’ve tweeted before, I’m confident that, despite the narrative, the data labeling industry will continue to fuel great businesses and be an excellent AGI long, ASI short. The argument is just: By the time the AGI labs no longer need data, it’s probably over for everything else too [2]. In this frame, the last companies left should be the data companies, as the last speck of economically relevant data is sucked in. And these companies are already among some of the fastest-growing companies in history: Mercor, founded three years ago, is rumored to be doing $2 billion in revenue with something like a few million expert labelers under contract.

While these businesses are very non-stationary, what type of data is needed shifts constantly, I don’t think that diminishes their value. The long-tail of the economy is long, and the value isn’t diminishing as you extend farther into more obscure information: as models get more capable, the value of the marginal dataset goes up, not down. Automating a full job means covering its full distribution of tasks, tools, edge-cases, and long-horizon loops. There’s some O-ring logic to it: a dataset that buys a 1% bump can justify a previously unjustifiable collection cost when it’s the difference between a system that does 99% of a job and one that does all of it [3].

The competitive dynamics of the data industry are still evolving but as demand for data is increasingly niche, ultra high-quality, expert-generated, I think we’ll see real consolidation. Again, contra-narrative, we’ll probably see true competitive differentiation built on brand, quality control of data (which, from personal experience, can vary massively), as well as in network effects from the talent networks themselves over time. We’ve already seen rapidly shifting data type demand work in favor of incumbents, benefiting those with early knowledge of where the market is headed.

The binding constraint

It’s truly remarkable that we seem to have the recipe — pretraining + RL — to absorb most economically valuable work, despite being far from a lot of what we expected from “AGI”. The same way chess engines revealed we never needed general intelligence to solve chess, as we originally thought, we’ll soon realize that software, mathematics, and the vast majority of the economy (including physical, just running ~3 years behind!) are the same. If recursive self-improvement or some other algorithmic breakthrough arrives, that’s wonderful, but we really don’t have to wait for it. The binding constraint between here and an automated economy isn’t that, it’s data coverage: every app, workflow, edge case, process, etc. sitting in private stores or someone’s head.

Ultimately, while we make tremendous strides in more efficient model architectures, and clusters like Stargate equip us with zettaflop-scale compute, we really aren’t making rapid progress collecting the data we lack.

We’ll soon live in a world where we have the methods & compute to accelerate scientific progress or economic growth, but not the data. And we’re already there today: frontier models would surely be as good at accounting/many medical tasks/legal advice as they are at software engineering if we only had the same pretraining & RL coverage as we did for code.

I really want to drill this in: The speed at which we automate the economy is going to be directly rate-limited by our ability to collect data about it.

Worth noting that under this assumption, with data as defensible and directly proportional to economic & scientific progress, data should also be considered a national strategic asset like compute. Imagine what we’d do in a world where we had a Manhattan Project-effort for AI and needed to mobilize data collection as a limiting factor. We should be concerned about China, with greater state capacity and authoritarian economic control, being capable of mobilizing data collection at national scale, potentially compounding their economy and scientific output faster than us down the line.

A Stargate for data

I’m leaving my complete ideas for a future post, as this one is already far too long, so I’d really like to pose the question here. Stargate exists because we organized trillions of dollars, international strategy, gigawatts around compute as a fundamental ingredient. What would equivalent ambition look like for data?

Obviously, scaling data collection, a heterogeneous mass of information across the economy, isn’t going to be as clear as scaling compute, as a homogenous infrastructural effort. A core division will be first, coverage — all uncaptured knowledge sitting across the economy/science/physical world and all that simply isn’t recorded — and, secondly, sheer volume in the domains we already train on: more hard math tasks, more high-quality web text, way more coding data, more legal drafts, etc.

I have a post coming soon which breaks down my proposals. There’s a lot of room for creativity. Quickly, we’ll probably want to start with a deep census of what we have and what we’re missing, predict what the 2030 model will still be bad at and work backward to what we should be collecting today. You can probably license a large amount, leveraging high lab valuations to buy datasets or companies altogether. There’s an adversarial nature to a lot of this collection with firms, so there’s lots of engineering to do this correctly. We should go convince important companies to turn off deletion policies, even if we’re not buying from them yet. Data flywheels in consumer products will be massive. Confidential training, government legislation for grant-funded research, running companies at a loss for their data, etc.

We’re headed towards hundreds of billions in expenditure, national prioritization, and major data limitation on the horizon. We have a great opportunity to think creatively about what a megaproject for data would look like: How do we, deliberately this time, construct the next internet’s worth of data?

Footnotes:

[1]: I’ll probably soon publish my much longer post explaining my position on data efficiency and why the value of this data is still pretty high in most worlds regardless of new algorithms.

[2]: The “AGI freeroll” bet: heads you win, tails ASI flips the world upside down anyways.

[3]: We already see a glint of validation of this point, given the data market is strongly tilting towards ultra-high-quality agentic data, rather than unskilled labeling — niche expert workflows, live environments, and evaluations requiring increasingly obscure talent & knowledge — yet shows increasing, not decreasing, revenues.

[Wonbong Jang / Won @wbjang11]

New paper: Rays as Pixels
We represent camera rays as pixels ("raxels") and learn a joint distribution over video + trajectories. One model does pose estimation and novel view synthesis, and we show the two are self-consistent.
📄https://t.co/WCIONf0X4a
🌐 https://t.co/0CTAjCevei

[Soumith Chintala @soumithchintala]

Leaving Meta and PyTorch

I'm stepping down from PyTorch and leaving Meta on November 17th.

tl;dr: Didn't want to be doing PyTorch forever, seemed like the perfect time to transition right after I got back from a long leave and the project built itself around me.

Eleven years at Meta. Nearly all my professional life. Making many friends for life. Almost eight years leading PyTorch, taking it from nothing to 90%+ adoption in AI. Walking away from this was one of the hardest things I've ever done. But I'm leaving with a full heart.

PyTorch handles exascale training now. It powers foundation models that are redefining intelligence. It's in production at virtually every major AI company. It's taught in classrooms from MIT to rural India. The tools I dreamed about making accessible? They are. The barrier to entry I wanted to lower? It's almost gone.

To be clear, there’s so much more to do. As long as AI evolves at a breakneck pace, PyTorch will continue to play catch up. Obsessing over the yet-to-come sometimes makes us forget how much we’ve already done.

To everyone who built this with me—who believed research should be joyful, that tools should be elegant, that open source changes everything—thank you. This wasn't my journey. It was ours.

What's next for me? Something small. Something new. Something I don't fully understand yet. Something uncomfortable. I could have moved to something else inside Meta. But I needed to know what's out there. I needed to do something small again. I couldn't live with the counterfactual regret of never trying something outside Meta.

It's very hard to leave. I probably have one of the AI industry’s most leveraged seats, I lead the software layer that powers the entire AI industry. Every major AI company and hardware vendor are on a speed dial. This kind of power is really hard to give up. But curiosity ultimately won out in my head.

Keep making AI delicious and accessible. I'll be watching. Probably filing issues. Definitely staying involved.

Is PyTorch going to be okay?

I don't want to be doing PyTorch forever. I don't want to be like Guido or Linus— bound to a single thing for decades. Last November, coinciding with the birth of my daughter, I started planning my exit with Aparna. My goal was to leave PyTorch in a good and stable place.

By this August, during the second half of my parental leave, I knew: Edward, Suo, Alban, Greg, John, Joe and Jana were ready. The team faced hard people, product, technical and organizational problems and didn’t feel the need to lean back on me to solve these for them (unlike in the past). The product story they crafted for the PyTorch Conference was coherent—really coherent. The things I'd flagged red were turning healthy. The project didn't need me anymore. Unlike 2020-2022 (when I stepped down to go do robotics and came back when Lin, Dima and Dwarak left), I have strong confidence that this time PyTorch is truly resilient. The most aligned culture carriers of PyTorch – Greg, Alban, Ed, Jason and Joe are at the decision table now, and people with strong value alignment – Suo, John and Jana have joined them at the table. And there’s a long list of equally value-aligned people willing to sit at the table should any of these people leave. There are many little things that make up my confidence on the people – John worked on Julia and open-source for a very long time (in fact we hacked a Torch.jl in 2015), Suo has been the strongest systems builder and strategic partner I’ve had for the past two years, and Jana worked on resilient core systems for a very long time, I’ve had long technical and organizational discussions with her over the past few months that give me confidence. And the product lineup and execution in 2025 should be sufficient evidence for any remaining doubt.

I’m confident that this band of PyTorchers are going to do exceptionally well. PyTorch might change in flavor because I no longer impose my own taste from the top, but I’m confident that the values are going to stay intact and the product is going to be awesome.

My time at Meta

The early years of FAIR were absolutely magical. I was part of a small family of absolutely brilliant people building state-of-the-art AI out in the open. From working on GANs with Emily Denton, Rob Fergus, Leon Bottou, Martin Arjovsky and the (now legendary) Alec Radford to building Starcraft bots with Gabriel Synnaeve, to building the first FAIR Cluster with Howard Mansell, to working on object detection with Adam Lerer and Piotr Dollar, to building PyTorch. It was more fun than I can describe in words. 2015 and 2016 were probably the most productive and professionally enjoyable years of my life. I’ll probably romanticize this period of my life forever.

When I joined FAIR, I had massive impostor syndrome, and the first 3 months were very very difficult. I can’t credit Andrew Tulloch enough for being the most thoughtful, kind and welcoming mentor, without whom I wouldn’t have made it. I’m so damn bullish for Meta just from the fact that he’s back.

---

My time on PyTorch was special.

I loved every part of building it—designing it, managing it, being the PM, TL, comms lead, doc engineer, release engineer, squashing bugs, growth hacking, turning it into a coherent product with hundreds of people, transitioning it to industry stakeholdership – the whole nine yards.

To the core PyTorch team at Meta: the engineers, researchers, open-source maintainers, docs writers, CI infrastructure folks, hardware partners, the community builders. To the hundreds more inside and outside Meta—thank you. You turned a library into a movement.

There are too many people to credit and thank, but I can't not mention Adam Paszke, Sam Gross, Greg Chanan, Joe Spisak, Alban Desmaison, Edward Yang, Richard Zou, Tongzhou Wang, Francisco Massa, Luca Antiga, Andreas Köpf, Zach DeVito, Zeming Lin, Adam Lerer, Howard Mansell and Natalia Gimelshein. And Schrep. They made the launch happen. And so many more people became centrally important later: Lu Fang, Xiaodong Wang, Junjie Bai, Nikita Shulga, Horace He, Mark Saroufim, Jason Ansel, Dmytro Dzhulgakov, Yangqing Jia, Geeta Chauhan, Will Constable, Briah Hirsh, Jane Xu, Mario Lezcano, Piotr Balecki, Yinghai Lu, Less Wright, Andrew Tulloch, Bruce Lin, Woo Kim, Helen Suk, Chris Gottbrath, Peng Wu, Joe Isaacson, Eli Uriegas, Tristan Rice, Yanan Cao, Elias Ellison, Animesh Jain, Peter Noordhuis, Tianyu Liu, Yifu Wang, Lin Qiao and hundreds more. It’s criminal of me to not take the space to list out everyone else I should be mentioning here. PyTorch is nothing without its people ❤️.

The most joyful moments of building PyTorch was meeting users eager to share their happiness, love and feedback. I remember a grad student coming to me at Neurips 2017, in a slurring emotional voice he said he’d been trying to make progress on his research for 3 years but within 3 months of using PyTorch he made so much progress that he was ready to graduate. That moment made it tangible that what we do matters, a lot, to a lot of people, even if you don't constantly hear from them. I do miss the intimacy of the PyTorch community, with a 300 person conference that felt like an extended family gathering, but I feel that’s a small price to pay considering the scale of impact PyTorch is truly having today – yes the Conference is now 3,000 people where market-moving deals get brokered, but it’s helping orders of magnitude more people to do their best AI work. I miss the intimacy, but I'm proud of that growth.

---

To Mark Zuckerberg and Mike Schroepfer, who believed that open-sourcing is fundamentally important and is a sound business strategy. This is so hard to understand for most people within the course of business, but we’ve run lock-step on this strategy without ever having to discuss it. Without you two, neither FAIR nor PyTorch would’ve happened. And those mean so much to me.

To Yann LeCun and Rob Fergus, for building the magical early FAIR that I so revere.

To Aparna Ramani, a leader that I find so rare at Meta in her ability to hold a really high bar for the org, technically brilliant with the span to discuss deep infra systems and industry-strategy within the same conversation and for being an absolute execution-machine! I’ve learned so much from you.

To Santosh, Kaushik, Delia, Oldham and Ben for being so welcoming to Infra. For someone coming over from FAIR with a wildly different culture, you all made me feel at home and made me part of the family, and thank you for that.

To all my managers who've championed me through the PSC video game – Serkan, Howard, Jerome, Abhijit, Yoram, Joelle, Aparna and Damien – I owe you a lifetime of drinks.

---

Signing off for now.

—Soumith

[Zhenjun Zhao @zhenjun_zhao]

Scaling Sequence-to-Sequence Generative Neural Rendering

@liu_shikun, @kam_woh, @wbjang11, Jiadong Guo, @han_junlin, @HaoZhe65347, @dougian, @juancprzs, @ZijianZhou524, Chi Phung, Tao Xiang, Juan-Manuel Pérez-Rúa

tl;dr: no explicit 3D representations; masked autoregressive framework; decoder-only rectified flow transformer

https://t.co/IsVcN2nYoS

[NAVER LABS Europe @naverlabseurope]

Announcing 2 new members of the *St3R family for human-centric 3D vision tasks!
Meet HAMst3R & HOSt3R @ICCVConference #iccv2025
- HAMSt3R (Human-Aware Multi-view Stereo 3D Reconstruction) extends MASt3R to handle scenes involving people.
Paper: https://t.co/uEXodKO6Zv
1/2 🧵

[Wonbong Jang / Won @wbjang11]

Happy to introduce our new CVPR paper—Pow3R: Empowering Unconstrained 3D Reconstruction with Scene and Camera Priors : https://t.co/i9nvnhHxbE

Inspired by the amazing DUSt3R, which reconstructs 3D from just two unposed images, we explored:

(1/n)

[Juanca Pérez @juancaperezs]

Some exciting news to share:
- In November, I successfully defended my thesis and earned my PhD in Computer Science from KAUST.
- I started a new role as an AI Research Scientist at Meta in London, focusing on Generative AI.

I'm deeply grateful to everyone who supported me!

[Juanca Pérez @juancaperezs]

A beautiful effect that is difficult to describe in simple words.

I still find it hard to believe this can be achieved w/o fine tuning. A fantastic (and apparently inherent?) property of diffusion models to be further exploited.

Proud to have been part of this project!

[Alejandro Pardo @PardoAlejo]

Ever noticed how one scene seamlessly transitions into another in films? That’s a match-cut—a subtle yet powerful cinematic trick.
Our MatchDiffusion, generates two videos from text prompts, designed to form a seamless match-cut—effortlessly and training-free. 🎥✨ 1/n

[Alejandro Pardo @PardoAlejo]

Ever noticed how one scene seamlessly transitions into another in films? That’s a match-cut—a subtle yet powerful cinematic trick.
Our MatchDiffusion, generates two videos from text prompts, designed to form a seamless match-cut—effortlessly and training-free. 🎥✨ 1/n

[Juanca Pérez @juancaperezs]

@FaccioAI @GoogleDeepMind Congratulations, Francesco. Most well deserved!

[Juanca Pérez @juancaperezs]

@artsiom_s Amazing work, Artsiom!

[Juanca Pérez @juancaperezs]

Proud to be part of this list of Outstanding Reviewers at ECCV!

[European Conference on Computer Vision #ECCV2026 @eccvconf]

The list of #ECCV2024 Outstanding Reviewers! Thank you for your service 🫡

https://t.co/NffKW0Tkbn

[Juanca Pérez @juancaperezs]

@Maureendss @Apple Congratulations @Maureendss !! Apple is lucky to have you!

[Sara Rojas Martinez @SarisRo]

Exciting News! 🎉 My paper got accepted at #ECCV2024! Huge thanks to my Adobe and KAUST collaborators!

💌 DATENeRF: Depth-Aware Text-based Editing of NeRFs 💌
@SarisRo, @JulienPhilip2, Kai Zhang, Sai Bi, Fujun Luan,@BernardSGhanem , Kalyan Sunkavalli

https://t.co/15LgNMzVhY

[KAUST Computer Vision Lab (IVUL) @KaustVision]

🚀 Exciting News! 🚀

@KaustVision Paper accepted at #ECCV2024! 🎉📚

🌟DATENeRF: Depth-Aware Text-based Editing of NeRFs🌟

Authors: @SarisRo, Julien Philip, Kai Zhang, Sai Bi, Fujun Luan, @BernardSGhanem, Kalyan Sunkavalli.

Project page: https://t.co/FcNQT7FGcN
@eccvconf 🇮🇹