Tony Lu

Robotics | AI infrastructure | Energy | tracking the systems behind the next industrial shift

[Tony Lu @Tony_Lujian]

There are two types of creativity.

Low-level creativity is when sit on a crouch and think “what if we made X?” AI is already very good at that.

Another type of creativity is where timeless classic gets made. Ones that define an industry/culture.

This creativity usually comes after years of imitation, feedback, and failure.

Taste is a byproduct of insane amounts of data and feedback and we don’t yet understand the exact mechanism.

Similarly, we can’t fully comprehend frontier AI’s capacity. Strong AI models often show emergent abilities: they suddenly get better at reasoning, coding, or abstraction in ways researchers did not perfectly predict from the training data alone.

With new advancements in reasoning models, RL, world models, and systems that learn from real user behavior, I don’t think “taste” stays mystical forever. It may just become the next scaling frontier.

[Tony Lu @Tony_Lujian]

@MarioNawfal Human creativity is also trained on old data.

Van Gogh and Beethoven spent years absorbing and imitating what came before them.

Given more quality data and better analytical models, AI might come up with original ideas in the future.

[Tony Lu @Tony_Lujian]

@SydneyHan176110 The race comes down to who can find a way to make data not just better embodied but also scalable.

The future will prob be a combo of Level 5 data and Nvidia’s simulation lab.

[Tony Lu @Tony_Lujian]

The real bottleneck in physical AI isn’t more data.

It’s better embodied data.

Here’s a breakdown of different levels of data quality and what the future of robotics looks like.

Level 1: Internet videos
This is the lowest quality data: footage of humans doing tasks on YouTube or TikTok.

This is useful for visual priors. Robot can learn what a task looks like and the sequence of it.

However, this type of data is high scale but low embodiment.

The robot can’t feel force, motor intent or action trajectory.

Level 2: Human Teleoperation
This is where a human control the robot arm or humanoid remotely.

This data is better than passive video.

But there’s still a gap: the human operator is using a VR headset or some type of interface.

They can’t necessarily feel what robots “feel.”

Level 3: Simulation
This is where robot learns in a simulated environment through millions of attempts.

Nvidia is heavily invest in this because real world data is scarce, expensive and slow.

The only problem is that there’s still a sim-to-real gap: messy lighting, deformed objects, and edge cases in real life can throw robots off guard.

Level 4: Embodiment-matched human data

This is where humans perform tasks while wearing motion/eye tracking devices, hand sensors, etc.

This reduce the gap between human motion and robot motion.

However, this is hard to scale as it’s limited by the number of human training hours.

Level 5: Full robot-human sensory loop

This is where the industry is headed. This level creates the highest quality data where humans feel what robots feel.

The human operator partially inhabits the robot body, creating the lowest possible embodiment gap.

The future of physical AI belongs to the company that build the infrastructure for human intent, robot motion, sensory feedback, task outcome, and real-world failure — all captured in one loop.

[Tony Lu @Tony_Lujian]

@BostonDynamics The impressive part is whole body contact planning.

This requires coordination of feet, hand, torso, grip, etc all at the same time.

This is meaningful step to robots taking over heavy lifting and other physical labor in real life.

[Tony Lu @Tony_Lujian]

@IlirAliu_ The real impressive part isn’t a dexterous human looking hand.

It’s that a high DOF compliant hand can reduce embodiment gap.

This makes imitation learning much more scalable.

[Tony Lu @Tony_Lujian]

@HarryStebbings This is smart. As models become commoditized, the real value is in learning real enterprise workflow.

By sending engineers into real enterprises, they collect valuable data to facilitate the eventual deployment of autonomous AI agents.

[Tony Lu @Tony_Lujian]

@Seanfrank @ylecun Data center doesn’t explode like nuclear plants but it still drains local acquirer for cooling, which I doubt is completely pollution free.

Furthermore it strains grid capacity and raises power prices. Hard to see it as harmless.

[Tony Lu @Tony_Lujian]

@lukas_m_ziegler This is impressive and exciting.

Specialized robots like this sculptor robot are the predecessors.

We are headed toward a future with general purpose humanoids that can complete almost all human physical labor.

[Tony Lu @Tony_Lujian]

@realBigBrainAI Now AI is just a model that thinks, which is impressive.

But once it can systematically break down tasks and execute them, it will be economically disruptive.

Question is how do we deal with potential massive layoff and tremendous inequality that follow?

[Tony Lu @Tony_Lujian]

@rohanpaul_ai That’s why the frontier of AI is shifting from “can we build it” to “can we predict what it becomes after scale, data, and feedback loops.”

[Tony Lu @Tony_Lujian]

@chamath This is like the enterprise AI version of the railroad switchyard.

As models become commoditized, the firm that capture traffic flow will have the most leverage.

[Tony Lu @Tony_Lujian]

@TFTC21 AI will probably be the next biggest global inequality driver.

The only people preaching of an AI utopia are those with vested interest in the field.

We can’t avoid it, might as well adapt.

[Tony Lu @Tony_Lujian]

@beffjezos @adcock_brett It’s still quite early.

We haven’t yet figured out how to create highly embodies data at scale to train general purpose humanoids.

But it’s just a matter of time that this problem is resolved.

[Tony Lu @Tony_Lujian]

@rohanpaul_ai This is exciting! Softer, safer, and human compatible robots as key to labor shortage in healthcare, particularly in elderly care.

[Tony Lu @Tony_Lujian]

@MarioNawfal The key is who can connect gigawatt scale compute to reliable power the fastest. The AI race is an energy infrastructure race.

[Tony Lu @Tony_Lujian]

@lukas_m_ziegler This is great, but the world is still built around human body.

Maintenance robots are definitely valuable in dangerous work environments.

But humanoids can operate across millions of workflows and unlock trillions in value.

[Tony Lu @Tony_Lujian]

@MarioNawfal This application to AI data centers is interesting.

The bottleneck to AI buildout is shifting from GPU to power.

Whoever can make energy modular and easy to deploy will win big in the AI infrastructure play.

[Tony Lu @Tony_Lujian]

@cryptorover Beijing will never trade control for capital inflow.

The next stage of US China diplomacy looks more like selective deal making rather than globalization 2.0.

[Tony Lu @Tony_Lujian]

@AndrewYNg This is the most honest view on the impact of AI.

At the end, it’s about whether the lowered intelligence barrier leads to more demand or plateau demand.

If AI makes lawyer cheaper, more people will use a lawyer not the opposite. Employment will therefore rise.