Insightful! I have been thinking about "thinking in video" too
Some further thoughts on the idea of "thinking with images":
1) zero-shot tool use is limited -- you can’t just call an object detector to do visual search. That’s why approaches like VisProg/ViperGPT/Visual-sketchpad will not generalize or scale well.
2) visual search needs to be a native, end-to-end component within multimodal LLMs. This was our focus in V*, but two yrs ago, we didn’t realize how powerful RL would become, so we had to stick with SFT to train detection heads. It worked, but it was slow and kind of a pain.
3) however, when the tools are simple and low-level -- say, basic python image processing functions, rather than a faster R-CNN -- they can be incorporated directly into the end to end system. With RL at scale, these simple tools become visual primitives you can mix and match to build scalable visual skills.
4) we should keep identifying these visual primitives. It’s definitely not just simple image processing functions; think about video and 3D.
5) finally, I think most traditional visual recognition models are dead. As the great @inkynumbers said, they’re parsers (t.co/i3qS4iheU5). But vision itself isn’t dead. i think it’s more alive and exciting than ever.View quoted post on X
