Ian Huang

Building | Fellow @southpkcommons | Prev AI PhD @StanfordAILab @GoogleDeepMind @RealityLabs

[Ian Huang @IanHuang3D]

🏡Building realistic 3D scenes just got smarter!

Introducing our #CVPR2025 work, 🔥FirePlace, a framework that enables Multimodal LLMs to automatically generate realistic and geometrically valid placements for objects into complex 3D scenes.

How does it work?🧵👇

[Evan Tana @evantana]

Space. Energy. Robotics. Agent swarms. Quantum. Biotech. The range of what @southpkcommons Founder Fellowship cohort is building is unreal.

But right now, we connect.

Founders shared their childhood obsessions - Rubriks cubes, rocketry, speedrunning Contra, Scrabble competitions, building crossbows...what a fun and talented group of founders.

The journey is hard. Better to embark on it amongst people who make you sharper, braver, and less alone.

[William Shen @shenbokui]

Excited to introduce Uni-1, our new multimodal model that *unifies* understanding and generation.

TLDR: a team of ~15 researchers is going pound-for-pound with nano banana and gpt image 🧵

[Ian Huang @IanHuang3D]

@that_vision_guy @HankCouture 🙄

[Elisabetta Fedele @efedele16]

Tired of conditioning 3D generative models only through images and text? ➡️ We just released code of SpaceControl, our method to control the generation directly in 3D ✨👩‍🎨

👾: https://t.co/uPz94Co4P0
🌍: https://t.co/4xF83r9Ehu

@FrancisEngelman @orlitany @IanHuang3D @mapo1

[Ian Huang @IanHuang3D]

👀 !!

[Fan-Yun Sun @sunfanyun]

Our mission — “vibe-coding interactive worlds” — was never just about code-generation. It’s about using AI to give people programmatic control without needing to code.

All these clips are *gameplay videos*; infinitely persistent, created by people with have no domain knowledge.

If you'd like to be the first to build on our research and product, sign up to our gaming vibeathon next month. Winners will have the interactive experiences they've built be featured on the Dome — an immersive, 100-foot destination experience being built in San Jose!

[Ian Huang @IanHuang3D]

Thanks for having me for demo night @southpkcommons! If you're curious what we're building, dm me!

[Gopal @gopalkraman]

given that 3D data is scarce, @IanHuang3D explores how scaling multimodal inference lets LLMs "think harder" with external tools.

he also shared another exploration around “inferring” someone’s aesthetic from minimal signal, without relying on prompts.

[Gopal @gopalkraman]

given that 3D data is scarce, @IanHuang3D explores how scaling multimodal inference lets LLMs "think harder" with external tools.

he also shared another exploration around “inferring” someone’s aesthetic from minimal signal, without relying on prompts.

[Gopal @gopalkraman]

demo nights are back.

come hang out with us "at the edge" @southpkcommons (SF) next Thursday.

we will have a *very* wide range of demos, including:
- space lasers
- emotionally expressive robots
- "wise" jewelry
- research around scaling multimodal inference

[Francis Engelmann ☀️ @CVPR'26 @FrancisEngelman]

Visit our Demo on Controllable 3D Object Generation @ICCVConference - today afternoon
Oct Tue 21, 3pm-5pm - Exhibit Hall 1
Come early, so we can catch some waves before sunset 🌊🏄🏝️
Project: https://t.co/S1NkJaeO6p

@mapo1 @efedele16 @IanHuang3D @orlitany @GuibasLeonidas

[Andrej Karpathy @karpathy]

Finally had a chance to listen through this pod with Sutton, which was interesting and amusing.

As background, Sutton's "The Bitter Lesson" has become a bit of biblical text in frontier LLM circles. Researchers routinely talk about and ask whether this or that approach or idea is sufficiently "bitter lesson pilled" (meaning arranged so that it benefits from added computation for free) as a proxy for whether it's going to work or worth even pursuing. The underlying assumption being that LLMs are of course highly "bitter lesson pilled" indeed, just look at LLM scaling laws where if you put compute on the x-axis, number go up and to the right. So it's amusing to see that Sutton, the author of the post, is not so sure that LLMs are "bitter lesson pilled" at all. They are trained on giant datasets of fundamentally human data, which is both 1) human generated and 2) finite. What do you do when you run out? How do you prevent a human bias? So there you have it, bitter lesson pilled LLM researchers taken down by the author of the bitter lesson - rough!

In some sense, Dwarkesh (who represents the LLM researchers viewpoint in the pod) and Sutton are slightly speaking past each other because Sutton has a very different architecture in mind and LLMs break a lot of its principles. He calls himself a "classicist" and evokes the original concept of Alan Turing of building a "child machine" - a system capable of learning through experience by dynamically interacting with the world. There's no giant pretraining stage of imitating internet webpages. There's also no supervised finetuning, which he points out is absent in the animal kingdom (it's a subtle point but Sutton is right in the strong sense: animals may of course observe demonstrations, but their actions are not directly forced/"teleoperated" by other animals). Another important note he makes is that even if you just treat pretraining as an initialization of a prior before you finetune with reinforcement learning, Sutton sees the approach as tainted with human bias and fundamentally off course, a bit like when AlphaZero (which has never seen human games of Go) beats AlphaGo (which initializes from them). In Sutton's world view, all there is is an interaction with a world via reinforcement learning, where the reward functions are partially environment specific, but also intrinsically motivated, e.g. "fun", "curiosity", and related to the quality of the prediction in your world model. And the agent is always learning at test time by default, it's not trained once and then deployed thereafter. Overall, Sutton is a lot more interested in what we have common with the animal kingdom instead of what differentiates us. "If we understood a squirrel, we'd be almost done".

As for my take...

First, I should say that I think Sutton was a great guest for the pod and I like that the AI field maintains entropy of thought and that not everyone is exploiting the next local iteration LLMs. AI has gone through too many discrete transitions of the dominant approach to lose that. And I also think that his criticism of LLMs as not bitter lesson pilled is not inadequate. Frontier LLMs are now highly complex artifacts with a lot of humanness involved at all the stages - the foundation (the pretraining data) is all human text, the finetuning data is human and curated, the reinforcement learning environment mixture is tuned by human engineers. We do not in fact have an actual, single, clean, actually bitter lesson pilled, "turn the crank" algorithm that you could unleash upon the world and see it learn automatically from experience alone.

Does such an algorithm even exist? Finding it would of course be a huge AI breakthrough. Two "example proofs" are commonly offered to argue that such a thing is possible. The first example is the success of AlphaZero learning to play Go completely from scratch with no human supervision whatsoever. But the game of Go is clearly such a simple, closed, environment that it's difficult to see the analogous formulation in the messiness of reality. I love Go, but algorithmically and categorically, it is essentially a harder version of tic tac toe. The second example is that of animals, like squirrels. And here, personally, I am also quite hesitant whether it's appropriate because animals arise by a very different computational process and via different constraints than what we have practically available to us in the industry. Animal brains are nowhere near the blank slate they appear to be at birth. First, a lot of what is commonly attributed to "learning" is imo a lot more "maturation". And second, even that which clearly is "learning" and not maturation is a lot more "finetuning" on top of something clearly powerful and preexisting. Example. A baby zebra is born and within a few dozen minutes it can run around the savannah and follow its mother. This is a highly complex sensory-motor task and there is no way in my mind that this is achieved from scratch, tabula rasa. The brains of animals and the billions of parameters within have a powerful initialization encoded in the ATCGs of their DNA, trained via the "outer loop" optimization in the course of evolution. If the baby zebra spasmed its muscles around at random as a reinforcement learning policy would have you do at initialization, it wouldn't get very far at all. Similarly, our AIs now also have neural networks with billions of parameters. These parameters need their own rich, high information density supervision signal. We are not going to re-run evolution. But we do have mountains of internet documents. Yes it is basically supervised learning that is ~absent in the animal kingdom. But it is a way to practically gather enough soft constraints over billions of parameters, to try to get to a point where you're not starting from scratch. TLDR: Pretraining is our crappy evolution. It is one candidate solution to the cold start problem, to be followed later by finetuning on tasks that look more correct, e.g. within the reinforcement learning framework, as state of the art frontier LLM labs now do pervasively.

I still think it is worth to be inspired by animals. I think there are multiple powerful ideas that LLM agents are algorithmically missing that can still be adapted from animal intelligence. And I still think the bitter lesson is correct, but I see it more as something platonic to pursue, not necessarily to reach, in our real world and practically speaking. And I say both of these with double digit percent uncertainty and cheer the work of those who disagree, especially those a lot more ambitious bitter lesson wise.

So that brings us to where we are. Stated plainly, today's frontier LLM research is not about building animals. It is about summoning ghosts. You can think of ghosts as a fundamentally different kind of point in the space of possible intelligences. They are muddled by humanity. Thoroughly engineered by it. They are these imperfect replicas, a kind of statistical distillation of humanity's documents with some sprinkle on top. They are not platonically bitter lesson pilled, but they are perhaps "practically" bitter lesson pilled, at least compared to a lot of what came before. It seems possibly to me that over time, we can further finetune our ghosts more and more in the direction of animals; That it's not so much a fundamental incompatibility but a matter of initialization in the intelligence space. But it's also quite possible that they diverge even further and end up permanently different, un-animal-like, but still incredibly helpful and properly world-altering. It's possible that ghosts:animals :: planes:birds.

Anyway, in summary, overall and actionably, I think this pod is solid "real talk" from Sutton to the frontier LLM researchers, who might be gear shifted a little too much in the exploit mode. Probably we are still not sufficiently bitter lesson pilled and there is a very good chance of more powerful ideas and paradigms, other than exhaustive benchbuilding and benchmaxxing. And animals might be a good source of inspiration. Intrinsic motivation, fun, curiosity, empowerment, multi-agent self-play, culture. Use your imagination.

[Ian Huang @IanHuang3D]

@andrewpprice Would be awesome to see you there! I still remember the convo we had at the last year's Siggraph. So much has changed in the technological landscape, we should chat again!

[Ian Huang @IanHuang3D]

It's time for AI agents in graphics.

On Thursday (tomorrow) #SIGGRAPH2025, I'll be doing a talk on how you can make AI work in YOUR graphical tools, based on 4 of the works I've done during my PhD at Stanford. Come learn about the future of graphics design -- everything from AI controlling #Blender nodes to creating 3D scenes, like below!

Session name: ML in Production
Thursday, 14 August 2025, 2:00pm - 3:30pm PDT
West Building, Rooms 109-110
https://t.co/rfUC0NAFww

[Ian Huang @IanHuang3D]

I wonder if tasks that are harder to verify by conventional methods (sims, tests, groundtruth answers) are being unlocked too, using the reasoning traces of pretrained models.

Visual aesthetics used to be slow and unscalable to verify, but using pretrained models as a proxy has changed this. You could, for instance, iterate the designs of a 3D object/material based on how a model may perceive its visual output (we did this with BlenderAlchemy https://t.co/uDVUVOJSuI and BlenderGym https://t.co/W8DmqgIwrc). And in fact human and AI judgement of visual aesthetics seems quite aligned according to works that have studied this.

This still suffers from not being (1)"objective truth" (since aesthetics is subjective), (4)"low noise" (fixable over time?), or (5) having a "continuous reward" (though you *could* backprop through the VLM).

Maybe in the future a small amount of training data + VLM finetuning = decent verifier for problems that are harder to verify.

[Jason Wei @_jasonwei]

New blog post about asymmetry of verification and "verifier's law": https://t.co/28Ly8NHJWZ

Asymmetry of verification–the idea that some tasks are much easier to verify than to solve–is becoming an important idea as we have RL that finally works generally.

Great examples of asymmetry of verification are things like sudoku puzzles, writing the code for a website like instagram, and BrowseComp problems (takes ~100 websites to find the answer, but easy to verify once you have the answer).

Other tasks have near-symmetry of verification, like summing two 900-digit numbers or some data processing scripts. Yet other tasks are much easier to propose feasible solutions for than to verify them (e.g., fact-checking a long essay or stating a new diet like "only eat bison").

An important thing to understand about asymmetry of verification is that you can improve the asymmetry by doing some work beforehand. For example, if you have the answer key to a math problem or if you have test cases for a Leetcode problem. This greatly increases the set of problems with desirable verification asymmetry.

"Verifier's law" states that the ease of training AI to solve a task is proportional to how verifiable the task is. All tasks that are possible to solve and easy to verify will be solved by AI. The ability to train AI to solve a task is proportional to whether the task has the following properties:
1. Objective truth: everyone agrees what good solutions are
2. Fast to verify: any given solution can be verified in a few seconds
3. Scalable to verify: many solutions can be verified simultaneously
4. Low noise: verification is as tightly correlated to the solution quality as possible
5. Continuous reward: it’s easy to rank the goodness of many solutions for a single problem

One obvious instantiation of verifier's law is the fact that most benchmarks proposed in AI are easy to verify and so far have been solved. Notice that virtually all popular benchmarks in the past ten years fit criteria #1-4; benchmarks that don’t meet criteria #1-4 would struggle to become popular.

Why is verifiability so important? The amount of learning in AI that occurs is maximized when the above criteria are satisfied; you can take a lot of gradient steps where each step has a lot of signal. Speed of iteration is critical—it’s the reason that progress in the digital world has been so much faster than progress in the physical world.

AlphaEvolve from Google is one of the greatest examples of leveraging asymmetry of verification. It focuses on setups that fit all the above criteria, and has led to a number of advancements in mathematics and other fields. Different from what we've been doing in AI for the last two decades, it's a new paradigm in that all problems are optimized in a setting where the train set is equivalent to the test set.

Asymmetry of verification is everywhere and it's exciting to consider a world of jagged intelligence where anything we can measure will be solved.

[Yunqi (Richard) Gu @richard_yunqigu]

We are presenting 17:00-19:00 today at Poster 267 in ExHall D for #CVPR25! Come and check out the first #VLM #3D #Graphics Benchmark! 📣📣📣

[Ian Huang @IanHuang3D]

📣Happening at Poster #269 TODAY between 10:30am - 12:30am at #CVPR2025 ! Come learn about multimodal #VLM reasoning for #3D scene generation!

[Ian Huang @IanHuang3D]

🏡Building realistic 3D scenes just got smarter!

Introducing our #CVPR2025 work, 🔥FirePlace, a framework that enables Multimodal LLMs to automatically generate realistic and geometrically valid placements for objects into complex 3D scenes.

How does it work?🧵👇

[Ian Huang @IanHuang3D]

If you're wondering which multimodal LLMs you should be using to build 3D graphics agents 🧑‍💻 , check out our #CVPR2025 Highlight work, BlenderGym -- not only does BlenderGym benchmark the top open and closed models, it also reveals a trick about *how* you should be allocating your inference compute for graphical editing tasks. With this trick, open source models *can* beat close-source models on 3D graphics editing. Curious? 🧐 👉 https://t.co/q93I6AxOan

[Yunqi (Richard) Gu @richard_yunqigu]

Which multimodal LLM should you be using to edit graphics in Blender?

Today, we’re releasing our #CVPR2025 Highlight🌟 work, #BlenderGym 🏋️‍♀️, the first agentic 3D graphics editing benchmark that will tell you exactly how multimodal LLMs compare in their Blender-editing skills.

What'd we find? 🧵👇

[Hansheng Chen @HanshengCh]

Excited to share our work:
Gaussian Mixture Flow Matching Models (GMFlow)
https://t.co/XWAy2VCJlg
GMFlow generalizes diffusion models by predicting Gaussian mixture denoising distributions, enabling precise few-step sampling and high-quality generation.

[Ian Huang @IanHuang3D]

📣 Happy to share that FirePlace got a #CVPR2025 Highlight ! See you all in Nashville! 🎷

[Ian Huang @IanHuang3D]

🏡Building realistic 3D scenes just got smarter!

Introducing our #CVPR2025 work, 🔥FirePlace, a framework that enables Multimodal LLMs to automatically generate realistic and geometrically valid placements for objects into complex 3D scenes.

How does it work?🧵👇

[Eduardo Alvarado @roboonaut]

Impressive demo and work! Amazing stuff going on with MLLMs.