Real-Time EXPO-FT is another iteration in the EXPO series, this time tackling policy reactivity on dynamic tasks.
Sampling actions from a VLA like π0.5 takes time. By the time the actions arrive, the observation they were conditioned on is stale. To make the policy more reactive they apply two fixes: First, the VLA is called asynchronously, ahead of time, conditioned on the actions that will be executed while inference is running (training-time RTC). Second, the small edit policy is cheap enough to run at the control rate, so it corrects each candidate with the latest observation available after the actions from the base policy arrive.
To speed up training they introduce a separate critic for filtering candidates in noise space during the Bellman backup, so only the best one is denoised. (At test time all candidates are denoised.)
Other interesting things from the paper:
- control frequency 30 Hz, up from 10 Hz in EXPO-FT
- N = 32 candidates instead of 8, each with one edit, so 64 in total
- vision tower of the base policy is not frozen anymore
- no human interventions this time, as far as I can tell
- for some tasks the critics and the edit policy get privileged state information like the ball position and velocity for ball balancing, which I think is kind of a bummer
Sampling actions from a VLA like π0.5 takes time. By the time the actions arrive, the observation they were conditioned on is stale. To make the policy more reactive they apply two fixes: First, the VLA is called asynchronously, ahead of time, conditioned on the actions that will be executed while inference is running (training-time RTC). Second, the small edit policy is cheap enough to run at the control rate, so it corrects each candidate with the latest observation available after the actions from the base policy arrive.
To speed up training they introduce a separate critic for filtering candidates in noise space during the Bellman backup, so only the best one is denoised. (At test time all candidates are denoised.)
Other interesting things from the paper:
- control frequency 30 Hz, up from 10 Hz in EXPO-FT
- N = 32 candidates instead of 8, each with one edit, so 64 in total
- vision tower of the base policy is not frozen anymore
- no human interventions this time, as far as I can tell
- for some tasks the critics and the edit policy get privileged state information like the ball position and velocity for ball balancing, which I think is kind of a bummer
EXPO-FT brings EXPO to real-world tasks with action chunks and adds human interventions for faster learning.
The base policy (now π0.5), edit policy and critic are the same setup as in EXPO (see post below), but the critic and edit policy operate on the executed part C ≤ H of each action chunk. During online training a human can intervene with corrections. These corrections overwrite the policy’s actions and go into the replay buffer like any other transition. Even though the policy reaches the same success rate without interventions, they speed up the process.
Additional things I found interesting:
- learner (training and inference) and actor (sensors and motors) run on separate machines
- sparse binary reward is used for the tasks
- for the critic they use a separate leightweight ResNet-50 encoder while the π0.5 encoder stays frozen. The edit policy reuses the critic’s features
- 10 Hz end-effector velocity control (probably in base frame)
- OpenPI-style augmentation on both camera views
- even though the tasks look complex on first sight, the randomisation is very little and the tasks are very shortView quoted post on X
