@mmiemon
The 5th Transformers for Vision and Multimodal AI workshop is happening at #CVPR2026 tomorrow (Wednesday, June 3rd)! We've got a great speaker lineup covering diverse topics across Transformers and Multimodal AI.
When: Wed, June 3rd
Where: Room 607
Website: sites.google.com/view/t4v-cvpr26
Schedule:
1:50 - 2:00 Opening Remarks
2:00 - 2:30 Ranjay Krishna
2:30 - 3:00 Jiatao Gu
3:00 - 3:30 Sherry Yang
3:30 - 4:00 Coffee Break
4:00 - 4:30 Juan Carlos Niebles
4:30 - 5:00 Zhuang Liu
5:00 - 5:30 Peter Tong
See you all tomorrow!
@thoma_gu @RanjayKrishna @sherryyangML @jcniebles @liuzhuang1234 @TongPetersbView quoted post on X
In the second before a play develops, a basketball player can instantly recognize the defensive scheme (perception), anticipate how the defense will rotate (causal reasoning), simulate several possible outcomes (simulation), and choose the best move (decision).
Today's video AI is far from this. These models can describe what they see, but they cannot explain why something happened, predict what comes next, or decide how to respond. We introduce SVI-Bench to measure these capabilities, and to push toward models that can reason over real-world, multi-agent video.View quoted post on X
I recently found this practical guide to building agents from OpenAI while doing some reading on agent evals. Nothing groundbreaking in terms of technical content, but it provides a really nice / rigorous structure around agent concepts and their tradeoffs that is useful.
What is an agent? It is possible to integrate an LLM into an automated workflow in a way that is not agentic; e.g., single-turn LLMs or chatbots. The core characteristic that makes a workflow agentic is whether the LLM is provided control of the workflow execution and allowed to make decisions. For example, an agent can control when the workflow is finished, attempt to recover from issues, and use tools to gather context or take actions.
Agent components. An agent is an LLM-powered system that includes multiple components in addition to the LLM:
- Tools: external functions or APIs for taking actions or gathering context.
- Instructions: written guidelines that describe in detail how the agent is expected to behave. We should draw upon existing documentation for our task
Usually, we are using a reasoning model for the LLM, meaning that the model also has the ability to dynamically reason over the instruction to determine how to decompose a problem and call tools in order to accomplish a desired task.
Beyond single agent. We can handle complex tasks with one agent by simply adding ore tools to provide more capabilities. Multiple agents can be helpful to decompose complex workflows, but the extra complexity can also lead to downsides / lower performance.
We should only use multi-agents systems when necessary. Some signs that using multiple agents could be helpful are:
- Instructions are complex, contain many conditional cases, and are becoming difficult to scale / manage.
- Your single agent is experiencing tool overload, meaning that it struggles to select the correct tools from the large set of tools available due to the presence of many similar tools.
Multi-agent systems. There are two main ways we can create a multi-agent system:
1. Manager setup: we have a central “manager” agent that delegates sub-tasks to multiple specialized sub-agents via tool calls and stitches their results into a final answer.
2. Decentralized setup: we have multiple peer agents that hand tasks off to one another based upon their specific purposes.
Both of these structures are common in practice, and we should aim to make each agent in the multi-agent system flexible / composable to simplify scaling of the system over time. We don’t want a brittle system that breaks every time we need to tweak guidelines or add a new capability.
Good instructions. One of the biggest keys to success is writing the best possible instructions for the agent(s). To write good instructions, we should:
- Draw upon existing documentation for the workflow.
- Clearly define guidelines and desired actions for the task.
- Prompt the agent to break the problem into steps.
- Provide concrete examples of how to handle edge cases.View quoted post on X
