How does Multimodal Generation work in LLMs?
With over 50 (!) visuals, I explain how LLMs can be created that not only generate text, but images, audio, and video as well!
We explore how generation of these different modalities works in general (e.g., Diffusion Transformer, Soundstream, Latte). It’s interesting to explore how Transformers have made their way into so many other domains beyond just text generation. We then cover how these multimodal decoding techniques could be connected to LLMs through early or late fusion.
I’m saying “We” because this was originally to be part of the book “An Illustrated Guide to AI Agents” written by @JayAlammar and me. Because the book was starting to become too big, we decided to remove this chapter. However! We love giving back, so we made this into a bonus chapter that is free for everyone to read :)
With over 50 (!) visuals, I explain how LLMs can be created that not only generate text, but images, audio, and video as well!
We explore how generation of these different modalities works in general (e.g., Diffusion Transformer, Soundstream, Latte). It’s interesting to explore how Transformers have made their way into so many other domains beyond just text generation. We then cover how these multimodal decoding techniques could be connected to LLMs through early or late fusion.
I’m saying “We” because this was originally to be part of the book “An Illustrated Guide to AI Agents” written by @JayAlammar and me. Because the book was starting to become too big, we decided to remove this chapter. However! We love giving back, so we made this into a bonus chapter that is free for everyone to read :)
