Strefer auto-generates instruction data for tuning Video LLMs on space-time–focused video tasks.
With just +545 short videos, Strefer-trained models outperform baselines on various tasks, showing stronger space–time–aware perception & reasoning !!
With just +545 short videos, Strefer-trained models outperform baselines on various tasks, showing stronger space–time–aware perception & reasoning !!
(Thread 1/8) 🚨 Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Data 🚨
Introducing Strefer: a novel data engine for auto-generating instruction data that enables Video LLMs to excel at spatiotemporal video understanding 🎬🧩⏳
Key Contributions:
▶️ Automated Pipeline: Eliminates dependence on legacy annotations through fully automatic instruction generation
▶️ Fine-grained Spatiotemporal Information: Produces temporally aligned, object-centric metadata with instruction-response pairs and multimodal prompts
▶️ Data-Efficient: Achieves improvements in space-time referring and reasoning with only 545 extra videos and no proprietary model dependencies
📄 Paper: bit.ly/427rVnw
🌐 Project: bit.ly/4gi4Owr
💻 Code: bit.ly/3I5HbdO
🎥 YouTube (10-min video): bit.ly/4lZL1TS
How does Strefer lay the foundation for perceptually grounded, instruction-tuned Video LLMs? Dive into the researchers' walk-through below! 🧵View quoted post on X
