Pinned post
Can video diffusion models predict a video of boucing balls?
Diffusion is iterative. It looks very much like an RNN.
But. How can you train an RNN without backprop through time?
Then it hits me. Diffusion might not be a true RNN.
RNN can compute any program. Diffusion migth not.
This starts the whole Serial Scaling Hypothesis, which predicts that diffusion models are more like a fixed depth network, than an RNN.
Videos are not supposed to be solved by a fixed depth network.
That’s why we want to test it out — and they can’t.
(@jdiazchao is a wonderful fellow to work with! I'm fortunate to work with him.)
Diffusion is iterative. It looks very much like an RNN.
But. How can you train an RNN without backprop through time?
Then it hits me. Diffusion might not be a true RNN.
RNN can compute any program. Diffusion migth not.
This starts the whole Serial Scaling Hypothesis, which predicts that diffusion models are more like a fixed depth network, than an RNN.
Videos are not supposed to be solved by a fixed depth network.
That’s why we want to test it out — and they can’t.
(@jdiazchao is a wonderful fellow to work with! I'm fortunate to work with him.)
"Video models are world simulators."View quoted post on X
Ok — can they predict a ball bouncing off another ball, off another ball? 🤨
We find standard video diffusion breaks down as the chain of dependent events lengthens — and more compute isn't the fix.
Meet the seriality gap. 🧵
1/n