Dongwon Kim

PhD student

[Dongwon Kim @dngwnkm]

@nilaksh404 Nice work! We did a similar thing for world models (compressing semantic latents from pretrained encoder) and found it leads to better planning/action-following.

https://t.co/faFsb2Lf9w

[Zhengyang Geng ✈️ ICML @ZhengyangGeng]

Very nice blog post. Sander being Sander. : )

My lens: amortization is about where you choose to close the loop.

If inference-time integration is expensive but learnable, push it into training: learn the jump, bake in the trajectory, and turn an iterative loop into an amortized training-time fixed point.

But the dual is just as important: when training can’t compress the computation into weights, push it back to inference — adaptive computation, iterative refinement, verifier/search loops, test-time fixed points.

So the interesting question isn’t 1-step vs iterative.

It’s: which parts of the computation are worth amortizing, and which parts should remain online?

[DailyPapers @HuggingPapers]

CompACT: Planning in 8 Tokens

A compact discrete tokenizer that compresses visual observations into just 8 tokens for latent world models, achieving 40x faster planning while maintaining competitive accuracy for real-time robotic control.

[RLWRLD @RLWRLD_ai]

6 papers from RLWRLD accepted to CVPR 2026! 🎉 Congrats to all the authors and collaborators, led by Professor Joo (@jhugestar) and Professor Cho (@LabyrinthMaker) 👇🏻

📚 DextER: Language-driven Dexterous Grasp Generation with Embodied Reasoning — contact-based embodied reasoning that predicts finger-link contacts on object surfaces for language-driven dexterous grasping
Junha Lee, Eunha Park (@eunha724), Minsu Cho
🔗 https://t.co/EMe7tRYnw2
🔗 https://t.co/ff80Tg9K6r

📚 Dexterous World Models — scene-action-conditioned video diffusion model to simulate embodied dexterous actions in a given static 3D scene
Byungjun Kim*(@byungjun__kim), Taeksoo Kim*(@taeksu98), Junyoung Lee(@junc0ng), Hanbyul Joo
🔗 https://t.co/Hnoz4EdpRU

📚 Affostruction: 3D Affordance Grounding with Generative Reconstruction — generative reconstruction to complete occluded regions and ground affordances on full 3D shapes
Chunghyun Park, Seunghyeon Lee (@llishyun), Minsu Cho
🔗 https://t.co/TqAntMRbdd
🔗 https://t.co/bbgkizuxVq

📚 Planning in 8 Tokens: A Compact Discrete Tokenizer for Latent World Model — CompACT compresses each observation into just 8 discrete tokens, enabling orders-of-magnitude faster planning in latent world models
Dongwon Kim (@dngwnkm), Gawon Seo, Jinsung Lee, Minsu Cho, Suha Kwak
🔗 https://t.co/qB7HohHYq1

📚 MoGaF: Space-Time Forecasting of Dynamic Scenes with Motion-aware Gaussian Grouping — long-term stable scene forecasting via motion-aware Gaussian grouping
Junmyeong Lee* (@ijunmye02373079), Hoseung Choi*, Minsu Cho
🔗 https://t.co/Io0Mv7lJbO

📚 Improving Text-to-Image Generation with Intrinsic Self-Confidence Rewards — post-training T2I generators with the model's own self-confidence as reward, improving compositionality and text-image alignment without external reward models
Seungwook Kim (@1ndependentgrad), Minsu Cho
🔗 https://t.co/EDwsxQCH3V

[Zhou Xian @zhou_xian_]

Everything you love about generative models — now powered by real physics!

Announcing the Genesis project — after a 24-month large-scale research collaboration involving over 20 research labs — a generative physics engine able to generate 4D dynamical worlds powered by a physics simulation platform designed for general-purpose robotics and physical AI applications.

Genesis's physics engine is developed in pure Python, while being 10-80x faster than existing GPU-accelerated stacks like Isaac Gym and MJX. It delivers a simulation speed ~430,000 faster than in real-time, and takes only 26 seconds to train a robotic locomotion policy transferrable to the real world on a single RTX4090 (see tutorial: https://t.co/bEkIlCKqdf).

The Genesis physics engine and simulation platform is fully open source at https://t.co/DhBv7NdyqH. We'll gradually roll out access to our generative framework in the near future.

Genesis implements a unified simulation framework all from scratch, integrating a wide spectrum of state-of-the-art physics solvers, allowing simulation of the whole physical world in a virtual realm with the highest realism.

We aim to build a universal data engine that leverages an upper-level generative framework to autonomously create physical worlds, together with various modes of data, including environments, camera motions, robotic task proposals, reward functions, robot policies, character motions, fully interactive 3D scenes, open-world articulated assets, and more, aiming towards fully automated data generation for robotics, physical AI and other applications.

Open Source Code: https://t.co/DhBv7NdyqH
Project webpage: https://t.co/SBNyhFB0yn

Documentation: https://t.co/3yuBoaealV

1/n

[Dayoung @gongda0e]

Excited to present ActFusion 🎥 at #NeurIPS2024! 🚀
A unified diffusion model that addresses both action segmentation and anticipation, advancing long-term video understanding.
It achieves superior performance 📈 across benchmarks.

Check it out: https://t.co/oGYVMCl51h

[Dongwon Kim @dngwnkm]

#NeurIPS2024 Excited to be in Vancouver presenting "Bootstrapping Top-down Information for Self-modulating Slot Attention"!
Catch our poster (East Exhibit A-C #2105) on Wed 11AM-2PM 👋
https://t.co/zBNTmPlW55

[Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞) @teortaxesTex]

Von Neumann's active preference for noise has always fascinated me. Moreover his memory was audial (as per Ulam)!

My hunch is that some very efficient brains are so good at dampening intrinsic noise that they require external perturbations as "random seeds" for novel thought.

[Sindy Löwe @sindy_loewe]

Excited to share Rotating Features – accepted as oral at #NeurIPS2023

Rotating Features learn to represent object affiliation via their orientation on real-world data without labels.

Let’s dive in!

📜 https://t.co/HbGEn6iR59
🖥️ https://t.co/eGvXqjYRUW

[AK @_akhaliq]

How FaR Are Large Language Models From Agents with Theory-of-Mind?

paper page: https://t.co/wQERlvsGEk

"Thinking is for Doing." Humans can infer other people's mental states from observations--an ability called Theory-of-Mind (ToM)--and subsequently act pragmatically on those inferences. Existing question answering benchmarks such as ToMi ask models questions to make inferences about beliefs of characters in a story, but do not test whether models can then use these inferences to guide their actions. We propose a new evaluation paradigm for large language models (LLMs): Thinking for Doing (T4D), which requires models to connect inferences about others' mental states to actions in social scenarios. Experiments on T4D demonstrate that LLMs such as GPT-4 and PaLM 2 seemingly excel at tracking characters' beliefs in stories, but they struggle to translate this capability into strategic action. Our analysis reveals the core challenge for LLMs lies in identifying the implicit inferences about mental states without being explicitly asked about as in ToMi, that lead to choosing the correct action in T4D. To bridge this gap, we introduce a zero-shot prompting framework, Foresee and Reflect (FaR), which provides a reasoning structure that encourages LLMs to anticipate future challenges and reason about potential actions. FaR boosts GPT-4's performance from 50% to 71% on T4D, outperforming other prompting methods such as Chain-of-Thought and Self-Ask. Moreover, FaR generalizes to diverse out-of-distribution story structures and scenarios that also require ToM inferences to choose an action, consistently outperforming other methods including few-shot in-context learning.

[Michael Tschannen @mtschannen]

New paper! We ask: How important is VQ for neural discrete representation learning? 🤔

I always felt it should be possible to “absorb” the VQ on top of a deep net into the net and use a simple grid-based quantization instead, without sacrificing much expressivity.

Summary 🧵👇

[Jason Wei @_jasonwei]

I reached 10k citations recently, a goal of mine for many years. It’s a nice moment to reflect back, and I mostly feel bittersweet:

(1) When I joined Google Brain back in 2020, I thought I'd stay for 10+ years, doing open-ended research and publishing papers. But the field has quickly changed from "publish or perish" to "publish and perish". Now, Google Brain doesn't even exist anymore (lol), and I haven't written a first-author paper in almost a year. Most state-of-the-art work doesn’t get published because companies want to keep their competitive advantages. We've seen a transition along the "gradient" of publishing: conference papers -> arxiv/tech reports -> blog posts -> (tweet + code), and I expect the trend to continue.

(2) The start of my research journey was in an extremely academic setting, and I miss parts of that culture. I was advised by five professors from 2018-2021, and I always felt like an apprentice. I learned a lot from having a consistent mentor for which I could expect regular feedback and genuine investment in my development. For example, in industry most people give very sparse feedback on papers (mostly just LGTM + some comments on abstract + intro). Conversely, the professors that I had would go through every sentence of the paper and make detailed comments. One time Ryan Cotterell even read my bibliography and pointed out every single punctuation and capitalization error, which was harsh in the moment but I later really appreciated it.

(3) I'm sometimes nostalgic for the old days where there was less hype and things were more rigorous. I remember in 2019 I read the abstract of every paper in the physical EMNLP handbook---it was actually a pretty decent way of figuring out what was hot in the field. Nowadays there is a lot of hype on twitter with iffy methods, questionable evaluations, overly-general claims. Perhaps it is a by-product of too many papers being published

That being said, I'm really excited for the new way we are doing AI. There is less academic research but more ways to impact the world practically. I optimized a lot for the old way of doing things, but part of the fun of being in AI is adapting to new paradigms :)

[Dongwon Kim @dngwnkm]

@Xp96HY Is there an official announcement of postponement?