bowu33

@bowu33

Agentic and Physical AI Builder

California, USA

Joined June 2021

55 Following

2.8K Followers

188 Posts

bowu33 retweeted

Elon

@Elonspacemedia

5 days ago

Tomorrow, we take another step toward the future. 🚀 We will push the boundaries of what’s possible, proving once again that humanity’s greatest achievements begin with a bold vision and the courage to pursue it. The future launches tomorrow. 🌎✨

307

664

105

82K

bowu33 retweeted

Junfan Zhu 朱俊帆

@junfanzhu98

8 days ago

📚 @saturdayrobotic Robotics & World Models Reading Club 18 Recap: keynote @Guanming717 (@gen_instinct), @ffeng01 (@UCSanDiego, @AetherLab_AI). Scaling video generation alone will never produce reliable embodied intelligence. The next generation of world models will be defined by causality, latent actions, verification and task alignment — not bigger diffusion models. DreamZero, ImageWAM, FastWAM, LeWorldModel, TC-WM, WAV and Unified Latent Action Models all point in the same direction. DreamZero jointly models video and actions instead of treating control as an afterthought. Frames are VAE-encoded into latents that enter Causal DiT blocks together with action noise and proprioception+language. Training uses joint video-action flow matching under teacher forcing. Inference keeps a KV cache, samples action chunks autoregressively from real observations, executes them asynchronously in closed loop, and decodes future frames only on demand. The joint prediction is deliberately factorized into a video term plus an inverse-dynamics term. ImageWAM turns image-editing foundation models into world-action models. A frozen LLM encodes the language instruction; the current observation is VAE-encoded with noise and fed into a tunable image-editing backbone. A lightweight tunable Action Expert then reads the edited future observation and outputs the full action sequence, leaving the powerful visual priors frozen. FastWAM asks a blunt efficiency question: does action prediction really need to attend to future video? Training still mixes three masking regimes — joint video-action denoising, video denoising plus inverse dynamics, and action conditioned only on current observations. At inference the winning strategy predicts actions without ever attending to future video tokens, treating video prediction merely as an auxiliary training signal. Joint training still helps representation learning; decoupling at inference delivers much faster real-time control. LeWorldModel learns planning-oriented latent dynamics instead of pixels. An encoder maps observations to latents; a predictor rolls those latents forward for many steps under actions; a cost module compares the trajectory against a goal latent. An auxiliary regularizer projects the latents onto random univariate directions and forces the distribution toward normality, producing a compact space directly usable for model-based control. “Is latent all you need?” The answer is nuanced. Asymmetric denoising, sparse future imagination and dense action refinement let the model dream only when necessary. Heatmaps show predicted action hotspots tightly aligned with objects and robot end-effectors, proving that latent representations can focus computation on causally relevant regions while ignoring irrelevant background. Classic video generators fail three basic tests: precise action control, object consistency and physical consistency. Choppy knife-cutting sequences and physically implausible 3D navigation are not edge cases — they are symptoms of missing causal structure. Three interlocking questions therefore dominate: Can we recover the hidden state factors behind observations? Can we recover the latent actions that actually drive system dynamics? How do we use those representations to build self-improving systems? Hu & Shum (2013) give a concrete answer to the first: under mild assumptions a short temporal block of observed trajectories is already sufficient to recover the latent context up to an invertible transformation. When clean latent factors are absent, stochastic residuals still matter and the model becomes a pseudo-Bayesian filter. Empirically this approach ranks at or near the top on Kitchen (~70 %), Maze2D (~160 %), Walker (~118 %), LIBERO-object (~93 %) and LIBERO-long (~62 %), beating DD, DF, LDCQ, Diffuser and DP. Latent-factor identification proceeds by feeding raw trajectories through a sequential encoder to obtain latents, then a sequential decoder that reconstructs the trajectories. These latents are subsequently used by Ada-Diffuser-Planning and Ada-Diffuser-Policy modules that inherit the same causal inductive bias (diffusion I/O, masks, inverse dynamics). Task-centric world models go further. Five architectural families are contrasted; the winning TC-WM injects a task signal that aligns and splits latents during training. History is per-patch encoded and aligned with the current embedding; actions condition a latent-dynamics transformer (positional embedding + transformer + per-patch decode) that predicts future embeddings and proprioception. A trainable encoder-decoder sits on top of frozen vision foundation models. Planning uses either cross-entropy method elite selection or latent diffusion guided by inverse dynamics. The same models outperform TD-MPC2, DreamerV3, MuZero and DINO-WM on CEM (Maze 100 %, Wall 100 %, Push-T 92 %, Cheetah 292) and LDP (Lift 60+ %, Can 62 %, Square 40 %, Hopper 46) while producing markedly more physical contact and fewer floating artifacts than Cosmos3-Nano. Identifiable world models can also serve as verifiers. Generative models produce blurry objects, blurry arm motion and interaction hallucinations. Identifiable latents are fed through an Ada-Diffuser; Temporal Difference Verification then applies gradient guidance that penalizes physically inconsistent regions, pushing trajectories onto the valid dynamic manifold for reliable test-time guidance. Unified Latent Action Models abandon robot-specific action spaces. Instead of recovering every hidden factor they recover only the shared latent actions that explain how the world changes across embodiments (partial identifiability, Kong & Xie 2022). A video foundation encoder produces latents; an inverse-dynamics stack of spatio-temporal ViTs and a forward stack of DiTs are trained inside a diffusion process with AdaLN, timestep conditioning, embodiment-ID classification and gradient reversal. At inference a single frame yields a transferable latent action that can be executed zero-shot on a new body. Zero-shot transfer experiments confirm the point: LAD collapses into incoherent frames, LVP produces plausible video that ignores the source actions, while the latent-action approach successfully executes the same intrinsic behavior on the target embodiment. Self-supervised skill learning needs neither demonstrations nor rewards. A skill embedding is contrastively aligned with state-action pairs: positives that move toward a goal are pulled together, negatives are pushed away (InfoNCE). Interaction-weighted resampling from the replay buffer focuses learning on meaningful interactions. Locomotion often follows linear-Gaussian dynamics; manipulation exhibits discontinuous “jumpy modes”. Local causal structure learning is the key that bridges these islands. Hallucination has a clear data-allocation root. Policies only need the narrow distribution of optimal actions; world models need the broad distribution of suboptimal and exploratory actions. Action-free internet video is abundant for learning general dynamics, yet action-labeled robot data remains scarce. On-policy collection (Sailor, VLAW, World-VLA) limits generality; information-maximizing exploration runs into the information paradox where model uncertainty does not correlate with useful learning progress. The practical learning framework therefore combines an adaptive curriculum that generates increasingly difficult tasks, a skill library of latent skill variables, and MIST-style masking: states are randomly masked, the model must maximize mutual information between the masked states and its predictions of observation and reward. This forces compact, task-relevant, causally structured representations. WAV (World Action Verifier, Liu et al., arXiv:2604.01985) reframes the remaining problem as verification. Three core ideas: (1) semi-supervised setting that exploits more data, (2) decomposed verification that replaces one hard check with two easier ones (state plausibility + action reachability), (3) goal-oriented cycle consistency that couples inverse and forward models. Diverse subgoals are sampled from action-free video; sparse inverse dynamics extracts action-relevant features; the agent actively seeks the hardest plausible subgoal. On MiniGrid the method achieves the highest correlation with true error, lowest prediction error and strongest action following (near Oracle). On real robots it adapts to novel appearance, novel objects and shifts in policy optimality with only 200 target samples. The high-level pipeline is now clear: Environment (with exploration that discovers novel representations) → Representation Learning (compress and distill from foundation models) → Structure Learning (incorporate domain knowledge and task feedback) → Decision-Making (adaptivity, compositionality, controllability). Scaling alone makes world models broader but not necessarily physical or controllable. Causal hidden representations identify what the world is; latent actions identify where the control signals come from; simple causal principles enable models that are more physical, controllable and self-improving. The ultimate question left open by all of this work remains: can we build a world model that not only dreams of the world, but also lets agents interact inside it, experiment, discover goals, and continuously refine the model itself?

junfanzhu98's tweet photo. 📚 @saturdayrobotic Robotics & World Models Reading Club 18 Recap: keynote @Guanming717 (@gen_instinct), @ffeng01 (@UCSanDiego, @AetherLab_AI).

Scaling video generation alone will never produce reliable embodied intelligence. The next generation of world models will be defined by causality, latent actions, verification and task alignment — not bigger diffusion models.
DreamZero, ImageWAM, FastWAM, LeWorldModel, TC-WM, WAV and Unified Latent Action Models all point in the same direction.

DreamZero jointly models video and actions instead of treating control as an afterthought. Frames are VAE-encoded into latents that enter Causal DiT blocks together with action noise and proprioception+language. Training uses joint video-action flow matching under teacher forcing. Inference keeps a KV cache, samples action chunks autoregressively from real observations, executes them asynchronously in closed loop, and decodes future frames only on demand. The joint prediction is deliberately factorized into a video term plus an inverse-dynamics term.

ImageWAM turns image-editing foundation models into world-action models. A frozen LLM encodes the language instruction; the current observation is VAE-encoded with noise and fed into a tunable image-editing backbone. A lightweight tunable Action Expert then reads the edited future observation and outputs the full action sequence, leaving the powerful visual priors frozen.

FastWAM asks a blunt efficiency question: does action prediction really need to attend to future video? Training still mixes three masking regimes — joint video-action denoising, video denoising plus inverse dynamics, and action conditioned only on current observations. At inference the winning strategy predicts actions without ever attending to future video tokens, treating video prediction merely as an auxiliary training signal. Joint training still helps representation learning; decoupling at inference delivers much faster real-time control.

LeWorldModel learns planning-oriented latent dynamics instead of pixels. An encoder maps observations to latents; a predictor rolls those latents forward for many steps under actions; a cost module compares the trajectory against a goal latent. An auxiliary regularizer projects the latents onto random univariate directions and forces the distribution toward normality, producing a compact space directly usable for model-based control.

“Is latent all you need?” The answer is nuanced. Asymmetric denoising, sparse future imagination and dense action refinement let the model dream only when necessary. Heatmaps show predicted action hotspots tightly aligned with objects and robot end-effectors, proving that latent representations can focus computation on causally relevant regions while ignoring irrelevant background.

Classic video generators fail three basic tests: precise action control, object consistency and physical consistency. Choppy knife-cutting sequences and physically implausible 3D navigation are not edge cases — they are symptoms of missing causal structure.

Three interlocking questions therefore dominate:
Can we recover the hidden state factors behind observations?
Can we recover the latent actions that actually drive system dynamics?
How do we use those representations to build self-improving systems?

Hu & Shum (2013) give a concrete answer to the first: under mild assumptions a short temporal block of observed trajectories is already sufficient to recover the latent context up to an invertible transformation. When clean latent factors are absent, stochastic residuals still matter and the model becomes a pseudo-Bayesian filter. Empirically this approach ranks at or near the top on Kitchen (~70 %), Maze2D (~160 %), Walker (~118 %), LIBERO-object (~93 %) and LIBERO-long (~62 %), beating DD, DF, LDCQ, Diffuser and DP.

Latent-factor identification proceeds by feeding raw trajectories through a sequential encoder to obtain latents, then a sequential decoder that reconstructs the trajectories. These latents are subsequently used by Ada-Diffuser-Planning and Ada-Diffuser-Policy modules that inherit the same causal inductive bias (diffusion I/O, masks, inverse dynamics).

Task-centric world models go further. Five architectural families are contrasted; the winning TC-WM injects a task signal that aligns and splits latents during training. History is per-patch encoded and aligned with the current embedding; actions condition a latent-dynamics transformer (positional embedding + transformer + per-patch decode) that predicts future embeddings and proprioception. A trainable encoder-decoder sits on top of frozen vision foundation models. Planning uses either cross-entropy method elite selection or latent diffusion guided by inverse dynamics. The same models outperform TD-MPC2, DreamerV3, MuZero and DINO-WM on CEM (Maze 100 %, Wall 100 %, Push-T 92 %, Cheetah 292) and LDP (Lift 60+ %, Can 62 %, Square 40 %, Hopper 46) while producing markedly more physical contact and fewer floating artifacts than Cosmos3-Nano.

Identifiable world models can also serve as verifiers. Generative models produce blurry objects, blurry arm motion and interaction hallucinations. Identifiable latents are fed through an Ada-Diffuser; Temporal Difference Verification then applies gradient guidance that penalizes physically inconsistent regions, pushing trajectories onto the valid dynamic manifold for reliable test-time guidance.

Unified Latent Action Models abandon robot-specific action spaces. Instead of recovering every hidden factor they recover only the shared latent actions that explain how the world changes across embodiments (partial identifiability, Kong & Xie 2022). A video foundation encoder produces latents; an inverse-dynamics stack of spatio-temporal ViTs and a forward stack of DiTs are trained inside a diffusion process with AdaLN, timestep conditioning, embodiment-ID classification and gradient reversal. At inference a single frame yields a transferable latent action that can be executed zero-shot on a new body.

Zero-shot transfer experiments confirm the point: LAD collapses into incoherent frames, LVP produces plausible video that ignores the source actions, while the latent-action approach successfully executes the same intrinsic behavior on the target embodiment.

Self-supervised skill learning needs neither demonstrations nor rewards. A skill embedding is contrastively aligned with state-action pairs: positives that move toward a goal are pulled together, negatives are pushed away (InfoNCE). Interaction-weighted resampling from the replay buffer focuses learning on meaningful interactions. Locomotion often follows linear-Gaussian dynamics; manipulation exhibits discontinuous “jumpy modes”. Local causal structure learning is the key that bridges these islands.

Hallucination has a clear data-allocation root. Policies only need the narrow distribution of optimal actions; world models need the broad distribution of suboptimal and exploratory actions. Action-free internet video is abundant for learning general dynamics, yet action-labeled robot data remains scarce. On-policy collection (Sailor, VLAW, World-VLA) limits generality; information-maximizing exploration runs into the information paradox where model uncertainty does not correlate with useful learning progress.

The practical learning framework therefore combines an adaptive curriculum that generates increasingly difficult tasks, a skill library of latent skill variables, and MIST-style masking: states are randomly masked, the model must maximize mutual information between the masked states and its predictions of observation and reward. This forces compact, task-relevant, causally structured representations.

WAV (World Action Verifier, Liu et al., arXiv:2604.01985) reframes the remaining problem as verification. Three core ideas: (1) semi-supervised setting that exploits more data, (2) decomposed verification that replaces one hard check with two easier ones (state plausibility + action reachability), (3) goal-oriented cycle consistency that couples inverse and forward models. Diverse subgoals are sampled from action-free video; sparse inverse dynamics extracts action-relevant features; the agent actively seeks the hardest plausible subgoal. On MiniGrid the method achieves the highest correlation with true error, lowest prediction error and strongest action following (near Oracle). On real robots it adapts to novel appearance, novel objects and shifts in policy optimality with only 200 target samples.

The high-level pipeline is now clear: Environment (with exploration that discovers novel representations) → Representation Learning (compress and distill from foundation models) → Structure Learning (incorporate domain knowledge and task feedback) → Decision-Making (adaptivity, compositionality, controllability).

Scaling alone makes world models broader but not necessarily physical or controllable. Causal hidden representations identify what the world is; latent actions identify where the control signals come from; simple causal principles enable models that are more physical, controllable and self-improving.

The ultimate question left open by all of this work remains: can we build a world model that not only dreams of the world, but also lets agents interact inside it, experiment, discover goals, and continuously refine the model itself?

bowu33 retweeted

Tianxing Chen

@MarioChan2002

19 days ago

We evaluated 30+ frontier embodied AI models. The result is clear: current generalist robot policies are still far from robust real-world manipulation. This is why we built RoboDojo.

500

394

121K

bowu33 retweeted

Alexandr Wang

@alexandr_wang

20 days ago

we are truly just getting started on video :) look forward to sharing more!

105

136K

Who to follow

Bored Becky 🇨🇦

@iamboredbecky

Canadian. 🍁💪🥱🤘🍁 BAYC 2355

Hosting @wedoalittlepod | 69 IQ Thing Collector

bowu33 retweeted

Zhi (Leo) Wang

@TX_Leo_Wang

19 days ago

🔥Call for contributors — building the world's largest open tactile dataset, Open-X-Tactile! 1/ 🖐️ Robotics can't be solved without touch! 2/ 👁️ Vision scales from the internet. Touch only grows from real contact — a robot can see an object, but not know if it truly grasped, pressed, or inserted it. 🚀 So we're building Open-X-Tactile (OXT): the "Open-X-Embodiment" of touch, toward the world's largest open tactile manipulation dataset — and we're calling for contributors worldwide⬇️ ✦ Any sensor: visuotactile, array, force/torque, dexhand, gripper ✦ Unified representation — mixable, trainable, downloadable ✦ Open-Source & community-built Until Now: ✦ ~400 tasks · ~26K trajectories · ~18M frames — and growing ✦ 37+ institution nodes (~21 labs + 16 industry teams) — and growing ✦ 15+ data sources · 12 sensor type Got tactile data? Come build the foundation with us — any sensor, any scale welcome! 🦾 🌐Website: https://t.co/yJGQG3LBTI 🫧Slack: https://t.co/3BTk9BdpsE 📧 Contact: [email protected] 🛰️or WeChat: tx-leo-wz 🧵 1/n

14K

bowu33

@bowu33

Who to follow

Last Seen Users on Sotwe

Trends for you

Most Popular Users