M- Lab of Adaptive Intelligence @EPFL

Mackenzie Weygandt Mathis, PhD @TrackingActions · May 3, 2023

🦓 Self-supervised multimodal ML is promising the next AI breakthrough - in our new work published in @Nature, we debut @CEBRAai: for self-supervised hypothesis- and discovery-driven science.

📝 doi.org/10.1038/s41586…
💻github.com/AdaptiveMotorC…
🦓 cebra.ai
🧵⬇️

View quoted post on X

Cell @CellCellPress · Jun 11, 2025

Now online! Neuropeptides specify and reprogram division of labor in the leafcutter ant Atta cephalotes dlvr.it/TLJNcY

View quoted post on X

Mackenzie Weygandt Mathis, PhD @TrackingActions · Mar 25, 2025

✨ Introducing a new #SOTA action recognition large multimodal language model: #LLaVAction!

Understanding human behavior requires recognizing actions—a challenging task given the complexity of behavior. Large multimodal language models (#MLLMs) offer a promising path forward, but how well do they perform in action recognition?

In our latest work - by @shaokaiyeah, Haozhe Qi, @TrackingPlumes and me | @EPFL_en - we rigorously evaluate and enhance MLLMs for action recognition in a real-world and challenging settings- egocentric views in the kitchen! 🧑‍🍳🔪🧽🤖

👀 We find that developing a multi-question-answer (#MQA) task serves as a valuable intermediate step in training (and evaluating) MLLMs for action understanding. Namely, we introduce EPIC-KITCHENS-100-MQA, a reformulation of the highly challenging EPIC-KITCHENS-100 dataset into a video multiple-choice question-answering task which allows for rigorous benchmarking of MLLMs in this task.

Next, we propose methods that substantially improve MLLM performance, and even achieving state-of-the-art results 🏆 (#SOTA) on the EPIC-KITCHENS-100 validation set 🔥✨.

Our approach also outperforms GPT-4o by 21 points in accuracy on EPIC-KITCHENS-100-MQA and demonstrates improvements across other action-related video benchmarks, including #VideoMME, #PerceptionTest, and #MVBench.

Our #LLaVAction-7B and -0.5B models can do #MQA and, critically, can do video captioning! 🙏🚀

As MLLMs become central to AI-driven video understanding in such real-world settings, ensuring their robustness in real-world tasks is critical. Excited to push the boundaries of multimodal AI further! 💪

🇨🇭We could not have done this without the amazing support of #SwissAI: the Swiss AI Initiative & the Swiss National Supercomputing Centre (#CSCS). @EPFL_AI_Center

#ProjectPage: mmathislab.github.io/llavaction/
📝 #arXivPaper: arxiv.org/abs/2503.18712
👩‍💻💻 GitHub code & Google #ColabDemo: github.com/AdaptiveMotorC…
🤗 Hugging Face models (use with transformers): huggingface.co/MLAdaptiveInte…

#AI #MultimodalLearning #ActionRecognition #EPICKITCHENS100 #MLLM #LLaVAction #VideoCaptioning #VLMs

View quoted post on X