Furong Huang

Hot take: robots should not dream in pixels.

Pixels are too low-level.
Latents are too opaque.

μ₀ predicts a third thing:
3D motion traces.

On real robots, it beats π₀.₅ — with ~1/100 the data scale and no action labels for world-model pretraining. 🧵

mu0-wm.github.io

(this video features voiceover narration)

I want robots that can watch a person perform a task, remember what matters, and adapt when the situation changes.

That’s the vision connecting our lab’s work at #CoRL2026.

A human demonstration contains much more than motion. It shows how hands and objects interact and how a task unfolds. We’re studying how to make that experience useful for robot learning—from large-scale human video in EgoScale (research.nvidia.com/labs/gear/egos…), to minutes of egocentric video in HumanEgo (humanego-ai.github.io), to a single demonstration in Dex-One2Many (dex-one2many.github.io).

To use that experience, a robot needs to represent and remember the right things. μ₀ (mu0-wm.github.io) builds a world model around 3D interaction traces. What to Attend, What to Keep (what-to-attend-what-to-keep.github.io) studies how skills can guide attention to vision and touch, and which events to retain as a task progresses.

Then comes acting—and responding to what happens. Fly-by-Code (fly-by-code.github.io) brings visual and physical feedback into embodied coding agents. Guava (guava-harness.github.io) explores how to transfer the capabilities of frontier vision-language models into a compact robot agent.

These are complementary steps toward robots that need less task-specific supervision, transfer knowledge across settings, and use feedback to act more reliably.

I won’t be in Austin this year, but I’m proud of the students and collaborators advancing this agenda. Ruijie Zheng is also an invited speaker at the Scaling Human2Robot Workshop!

If you’re attending CoRL, please meet the team at their talks and posters. I’d love to hear your thoughts on this direction.

Presentation details are in the posters below.

#RobotLearning #EmbodiedAI

Such a cool work!
Quite aligned with your idea @JayLEE_0301

How did you make such amazing videos? @jbhuang0604!!

Precompute and reuse. I keep coming back to this idea, and there’s a 4 a.m. slide-making story behind it.

Back at ICLR 2023 in Kigali, Rwanda, a student of mine was helping organize the Reincarnating Reinforcement Learning workshop. A speaker couldn’t make it, and they asked whether I could give a talk.

I said yes of course, it was my student’s workshop! So I found myself making slides at 4 a.m. in my hotel room for the talk the next day. 😂

My talk was “Adaptable Reinforcement Learning in an Ever-Changing World: Reuse to Learn and Adapt Efficiently.” I also joined the panel, “Challenges & Open Problems in Reusing Prior Computation.”

It was absolutely a great workshop. What stayed with me was a simple question: why keep starting over when we’ve already spent so much compute learning something useful?

Today, that sounds almost too obvious. Isn’t that a big part of what foundation models are about? And really, RL post-training follows the same logic: start with a pretrained model, reuse what it has learned, and build on it with RL.

But back then, training a specialist RL agent from scratch for each task was still very common. That workshop helped me put words to something that has become a recurring theme in our research.

Once you take reuse seriously, the interesting question becomes: what should we learn ahead of time, and what should we leave for adaptation when the actual task arrives?

You can see that question across our work, just to name a few:
• SMART: pretrain control representations that transfer across tasks.
• DynaFLIP: learn dynamics-aware perception that different robot policies can build on.
• μ₀: learn a world model from video and reuse it across robot embodiments.
• FlowBank: precompute useful agentic workflows, then choose one for each query.

And now, Meta²: precompute and reuse knowledge about how to improve.

An agent’s harness includes the prompts, tools, memory, hooks etc around the model. Often, you only discover what needs changing once the agent is already working. A missing tool. An instruction that keeps being forgotten. A check that needs to happen repeatedly.

We want the agent to adapt as those needs appear. But reasoning about every possible change from scratch is expensive, and making changes doesn’t automatically mean making progress.

So we borrow a core idea from meta-learning: learn how to adapt.

Offline, Meta² learns a textual Meta-Skill that guides an evolver on when and how to change the harness, including when to leave it alone. We learn this guidance by comparing runs with no evolution, runs with the current evolver, and training runs where the evolver gets privileged verifier feedback.

During a new run, that learned guidance stays fixed. At checkpoints, the evolver combines it with evidence from the current run to decide whether to keep or change the harness. The underlying model stays frozen.

The guidance gets reused. The actual edits depend on what happens.

One result I especially like: carrying one task’s harness edits into future tasks actually hurt performance in our experiments. Knowing how to adapt transferred better than keeping the adaptations themselves.

That distinction matters for RSI. A system that keeps improving needs to figure out what to preserve from experience, what to revise, and what was only useful in one particular situation.

Meta² gives us one concrete piece of that: reusable guidance for improvement, paired with adaptation while the agent is still working.

Longer term, I see RSI as potentially the co-evolution of models, harnesses, and the environments they learn in. Progress in each could help the others improve.

For now, we’re nailing down the harness part! I’m really proud of my students and collaborators for finding an efficient and effective way to make this work.

That 4 a.m. slide-making session stayed with me longer than I expected. 🧬