Evaluated across four benchmarks: LIBERO, RoboTwin, RoboCasa, and RoboDojo.
📖 Blog: tianqi-zh.github.io/BenchMend/
🎥 Rollouts: huggingface.co/spaces/benchme…
tianqi-zh.github.io Agent-Guided Instruction Repair for Robotics Benchmarks Repair the instruction layer. Preserve the demonstrations. A human-proxy policy helps debug LIBERO and RoboTwin. 📖 Blog: tianqi-zh.github.io/BenchMend/
🎥 Rollouts: huggingface.co/spaces/benchme…
A robot can follow a reasonable interpretation of an instruction and still fail the benchmark.
Meet BenchMend: we use GPT-6-Astra to uncover instruction–verifier gaps in robotics benchmarks, then repair the wording while preserving success checkers and recorded demonstrations.
Key takeaways:
📝 Instructions matter. Small edits can substantially improve alignment between task language and benchmark verdicts.
🤖 Even strong agents have limits when performing zero-shot robot control. Tasks requiring precise contact, placement, and coordination remain challenging for agents.
🛠️ An imperfect policy can still help debug benchmarks. Agents that diagnose gaps, propose fixes, and test them could enable self-improving benchmarks and potentially support recursive self-improvement (RSI).
Read the blog: tianqi-zh.github.io/BenchMend/
Watch the rollouts: huggingface.co/spaces/benchme…View quoted post on X
