@sangnie
4⃣ This Spotlight in Session 6 on 5/9 from @TianhongLi6, @sangnie, @YonglongT, @Han_Zhang_, @MyNameIsTooLon, #JClinic PI @dina_katabi, @g_lajoie_, @huiwen_chang, & @dilipkay introduces ITIT, allowing vision-language training on unpaired image + text data openreview.net/forum?id=kNjrh…View quoted post on X
Leveraging Unpaired Data for Vision-Language Generative Models via Cycle Consistency
paper page: huggingface.co/papers/2310.03…
Current vision-language generative models rely on expansive corpora of paired image-text data to attain optimal performance and generalization capabilities. However, automatically collecting such data (e.g. via large-scale web scraping) leads to low quality and poor image-text correlation, while human annotation is more accurate but requires significant manual effort and expense. We introduce ITIT (InTegrating Image Text): an innovative training paradigm grounded in the concept of cycle consistency which allows vision-language training on unpaired image and text data. ITIT is comprised of a joint image-text encoder with disjoint image and text decoders that enable bidirectional image-to-text and text-to-image generation in a single framework. During training, ITIT leverages a small set of paired image-text data to ensure its output matches the input reasonably well in both directions. Simultaneously, the model is also trained on much larger datasets containing only images or texts. This is achieved by enforcing cycle consistency between the original unpaired samples and the cycle-generated counterparts. For instance, it generates a caption for a given input image and then uses the caption to create an output image, and enforces similarity between the input and output images. Our experiments show that ITIT with unpaired datasets exhibits similar scaling behavior as using high-quality paired data. We demonstrate image generation and captioning performance on par with state-of-the-art text-to-image and image-to-text models with orders of magnitude fewer (only 3M) paired image-text data.View quoted post on X
I am grateful to have a helping hand in @pierrelux, @riashatislam and @pierthodo when I first started my research career as an MSc student in 2018. The advice they gave me on reading research papers, organizing my thoughts, and thinking like a researcher is still with me. (1/N)View quoted post on X
Join us at the WiML Un-Workshop breakout session on "Role of Mentorship and Networking"! Do not miss the chance to talk with leading researchers Samy Bengio, @suchenzang @hugo_larochelle @SharonYixuanLi @pcastr @JohnCLangford and others!View quoted post on X
#ICML2023 @WiMLworkshop
