Reward is always a bottleneck for robotic policy finetuning. Dense rewards introduce bias or require laborious annotations, while still struggling with behavioral nuances. How to achieve dense rewards with minimal annotations at test time? hire-project.github.io #CoRL2026
Haoyi Niu
@t641769919
465 Following 294 Followers 113 Posts
Latest posts 5
I will join the debate online. Honored to be here for the first session of Saturday Robotics @saturdayrobotic held in China. May it be the first of many! @EraAi95973
Check out what we discussed about ODEWorld in this YouTube recording if you missed it on Saturday!
📺 YouTube 22 @saturdayrobotic Robotics & World Models Reading ClubView quoted post on X
ODEWorld: A Continuous Predictive Architecture via Physical-Time Flow
youtu.be/gQ3L67HldyU?si…
Keynote by @t641769919 (@UCBerkeley)
In the physical world we inhabit, space and time are fundamentally continuous. However, existing machine learning paradigms for world modeling are largely confined to discrete-time prediction and inference, thereby exhibiting significant inefficiency in capturing the underlying dynamics of the physical world. To bridge this gap, we introduce Physical-Time Flow (PT-Flow), a novel approach that learns a continuous latent velocity field operating in physical time. Crucially, the underlying dynamics of sequential data are parameterized by an ordinary differential equation (ODE) embedded in a well-structured dynamical representation space. Under this paradigm, the prediction of the future can be recast as temporal integration via an ODE solver in the compressed latent space. Building upon PT-Flow, we construct ODEWorld, a physics-grounded, continuous-time latent world model that is both efficient and versatile. By extracting time-variant features and enforcing ODE properties on both the dynamical representation space and the latent velocity field, ODEWorld effectively addresses the long-standing representation collapse issue in latent world model literature. This also enables high-quality image reconstruction with ODEWorld even after long-horizon prediction. Moreover, its continuous nature allows for arbitrary temporal resolution and even backward prediction, which is impossible for most discrete-time models. Lastly, benefiting from the learned dynamics-centric latent space, ODEWorld can provide rich planning-oriented information to facilitate downstream policy learning. Comprehensive experiments across multiple simulation and real-world datasets demonstrate that ODEWorld successfully reconciles planning-conducive dynamics abstraction with visual realism, excelling in both video generation and robotic control.
Pre-Readings
dstate.github.io/odeworld_websi…
youtu.be/gQ3L67HldyU?si…
Great to be back sharing ODEWorld — and see Saturday Robotics grow from a handful of people into the thriving community it is today. 🎉 @junfanzhu98 @aurorafeng_01 @saturdayrobotic
💥 Reading ODEWorld gave me “@sama watching the atomic bomb explode” moment—the kind where you sit back in your chair.View quoted post on X
@saturdayrobotic Robotics & World Models Reading Club 22: keynote @t641769919 (@UCBerkeley), cohosted with @aurorafeng_01, @RoboticsCtr.
🤖 ODEWorld: What if robot world models stopped predicting frames—and started modeling physical time?
The paper attacks a fundamental mismatch in world modeling—not just model architecture.
🔥 Hot take #1 — Sufficiency:
If the physical world is continuous, a continuous latent velocity field is the natural predictive object. Instead of next-frame prediction, ODEWorld learns how the latent state changes with physical time, then integrates that flow with an ODE solver.
🔥 Hot take #2 — Necessity:
JEPA-style consistency objectives supervise dynamics only indirectly and can entangle reconstruction with dynamics, creating representation-collapse failure modes. ODEWorld instead gives the model a direct first-order velocity target.
🧠 Recipe
1️⃣ Dynamical representation decoupling
The latent representation is conditioned on the initial observation. The decoder also receives the initial frame, forcing the compact latent to encode dynamic change rather than static appearance.
Result: a surprisingly compact single 768-D token, ~256× compression versus typical multi-token representations, while preserving smooth semantic trajectories.
2️⃣ Direct JVP supervision
Instead of noisy finite-difference latent targets, the authors obtain latent velocity through a Jacobian-vector product (JVP): differentiate the dynamics encoder with respect to the observed state and multiply by the state velocity.
The target is stop-gradient, making reconstruction and dynamics learning substantially decoupled.
3️⃣ Better velocity estimation
Raw finite differences are noisy, so ODEWorld uses a Savitzky–Golay derivative filter (window=5), then applies the same smoothing principle in latent space.
Training combines dynamics reconstruction + velocity regression, with equal weights. Pixel reconstruction uses frozen DINO features with a trainable decoder, combining L1, LPIPS and a small GAN term. DINO itself stays frozen.
⚙️ Architecture
Frozen DINOv2 → dynamics encoder → 768-D latent trajectory → lightweight 3-layer MLP velocity field, conditioned via FiLM on normalized physical time and goal/language conditioning.
Inference is simply ODE integration (RK4).
That gives something discrete frame predictors fundamentally don't have: a trajectory you can query at any time.
⏱️ Arbitrary temporal resolution
🔄 Forward + backward planning
🧩 Missing-frame interpolation / temporal super-resolution
🎯 Replanning by restarting integration from a new observation
🧠 Continuous-control-theoretic structure
🔥 What surprised me
PCA trajectories show that direct first-order supervision produces dramatically smoother latent dynamics than next-frame prediction or raw DINO features.
The latent velocity is time-varying and task-specific, not just a constant interpolation between start and goal.
Long-horizon rollouts remain spatially coherent; once a goal configuration is reached, the scene can freeze instead of drifting, even far beyond the training horizon.
Even more interesting: the model exhibits emergent recovery behaviors absent from demonstrations—for example, recovering after a robot drops an object.
Failures remain (e.g. disappearing objects, no-contact interactions), but they look qualitatively different from classic temporal inconsistency or mode collapse.
🚀 Discussions on scaling
GigaTok attacks tokenizer complexity through semantic regularization; SANA-Video attacks transformer cost through hybrid linear/softmax attention and reports ~3.2× faster DiT inference.
ODEWorld takes a different route: compress the dynamics problem itself.
Once the velocity field lives in a 768-D latent space, long-horizon prediction becomes a handful of cheap ODE steps instead of repeatedly evaluating huge visual transformers.
Reported LIBERO long-horizon rollout latency: 0.072 s.
💡 Takeaway: ODEWorld isn't merely putting an ODE wrapper around a video model. It changes the predictive primitive from “what is the next frame?” to “how does the physical state flow through time?”
If physical time is truly fundamental, perhaps continuous dynamics should be an architectural constraint—not a post-hoc interpolation trick.
Thanks for invitation🍓@junfanzhu98 ! Happy to meet you all in SF on 8/8 sharing our continuous world model, ODEWorld. Please visit our public website (dstate.github.io/odeworld_websi…) for more qualitative results and interpretative visualizations🦾🥳
📚 @saturdayrobotic Robotics & World Models Reading Club 22: ODEWorld: A Continuous Predictive Architecture via Physical-Time Flow — San Francisco 08/08View quoted post on X
👉🏻 RSVP: luma.com/7nnl4a5r
🎤 Keynote: @t641769919 (@UCBerkeley)
In the physical world we inhabit, space and time are fundamentally continuous. However, existing machine learning paradigms for world modeling are largely confined to discrete-time prediction and inference, thereby exhibiting significant inefficiency in capturing the underlying dynamics of the physical world. To bridge this gap, we introduce Physical-Time Flow (PT-Flow), a novel approach that learns a continuous latent velocity field operating in physical time. Crucially, the underlying dynamics of sequential data are parameterized by an ordinary differential equation (ODE) embedded in a well-structured dynamical representation space. Under this paradigm, the prediction of the future can be recast as temporal integration via an ODE solver in the compressed latent space. Building upon PT-Flow, we construct ODEWorld, a physics-grounded, continuous-time latent world model that is both efficient and versatile. By extracting time-variant features and enforcing ODE properties on both the dynamical representation space and the latent velocity field, ODEWorld effectively addresses the long-standing representation collapse issue in latent world model literature. This also enables high-quality image reconstruction with ODEWorld even after long-horizon prediction. Moreover, its continuous nature allows for arbitrary temporal resolution and even backward prediction, which is impossible for most discrete-time models. Lastly, benefiting from the learned dynamics-centric latent space, ODEWorld can provide rich planning-oriented information to facilitate downstream policy learning. Comprehensive experiments across multiple simulation and real-world datasets demonstrate that ODEWorld successfully reconciles planning-conducive dynamics abstraction with visual realism, excelling in both video generation and robotic control.
📖 Pre-Readings
Project: anonymous-submission-16982.github.io/ODEWorld/
Hosted by @junfanzhu98 & @aurorafeng_01
📍 San Francisco
📅 Saturday, August 8, 2026 | 2:00 PM – 5:00 PM
Agenda
2:00–2:30 PM — Doors open & social (food 😋, beverages 🧋 and UNLIMITED strawberries 🍓 — official Reading Club fruit!)
2:30–3:30 PM — Keynote by @t641769919 (YouTube recording: TBD — volunteers welcome!)
3:30–5:00 PM — Q&A + open-floor roundtable (spotlight any paper or technical deep dive)
Join the Saturday Robotics Discord: discord.gg/WH7DrTHRXK
Come ready to discuss continuous-time world models, latent dynamics, ODE-based prediction, long-horizon planning, and the next generation of physics-grounded embodied AI.
👉🏻 RSVP: luma.com/7nnl4a5r
#Robotics #WorldModels #EmbodiedAI #ODEWorld #PhysicalAI #LatentDynamics #VideoGeneration #RobotLearning #Sim2Real #BayAreaRobotics