Twitxy Saved

Konpat Ta Preechakul

Verified account This account is verified. Learn more

@konpatp

What if it's not all about data? PhD student at @berkeley_ai. I'm from Bangkok, Thailand 🐘.
Berkeley, CA konpat.me Joined April 2008
352 Following 982 Followers 1.3K Posts

Latest posts 5

Pinned post

Konpat Ta Preechakul Verified account @konpatp
Can video diffusion models predict a video of boucing balls?

Diffusion is iterative. It looks very much like an RNN.

But. How can you train an RNN without backprop through time?

Then it hits me. Diffusion might not be a true RNN.

RNN can compute any program. Diffusion migth not.

This starts the whole Serial Scaling Hypothesis, which predicts that diffusion models are more like a fixed depth network, than an RNN.

Videos are not supposed to be solved by a fixed depth network.

That’s why we want to test it out — and they can’t.

(@jdiazchao is a wonderful fellow to work with! I'm fortunate to work with him.)
Jorge Diaz Chao Verified account @jdiazchao Jul 16
"Video models are world simulators."

Ok — can they predict a ball bouncing off another ball, off another ball? 🤨

We find standard video diffusion breaks down as the chain of dependent events lengthens — and more compute isn't the fix.

Meet the seriality gap. 🧵

1/n
View quoted post on X
2 replies 19 reposts 130 likes 19K views View on X
Konpat Ta Preechakul Verified account @konpatp
No cfg and actually no conditions even needed for high quality images!
Donghoon Ahn Verified account @donghoon_ahn Oct 13
Can diffusion models generate high-quality images without CFG?

Yes! We introduce ✨Correction Training for Diffusion Models✨, which significantly improves sample quality without relying on classifier-free guidance.

Project page: sunovivid.github.io/correction-tra…

🧵 (1/n)
View quoted post on X
Konpat Ta Preechakul Verified account @konpatp
This gives me the vibes of IMLE. Looks really impressive!
Alexi Gladstone Verified account @AlexiGlad Jul 31
We discovered a third pretraining axis beyond parameters and data: exploration.

Scaling exploration monotonically improves existing models across images/video/language, and unlocks end-to-end generation.

In the simplest case, it's just a for loop.

Introducing Explorative Modeling.

TLDR:
- Gains from exploration grow with scale: 7%→36% as data scales, 13%→23% as parameters scale, and gains double at 3× the compute
- Adding exploration to ~SOTA baselines improves data efficiency by 6.2×, FLOP efficiency by 4.1×, parameter efficiency by 47%, and hits a near-SOTA 1.43 unguided FID on ImageNet
- Exploration lets you trade training compute for generalization, and scales how end-to-end your generative model is
- End-to-end Explorative Models (XMs) match diffusion performance on control tasks with up to 256× less inference compute

🧵Thread:
View quoted post on X
2 replies 3 reposts 32 likes 6.6K views View on X
Konpat Ta Preechakul Verified account @konpatp
This is interesting.

“Any successful policy is thus forced to continuously rebake its effective hidden state into the observations. That might at least be part of the reason why LLMs cannot effectively control their chain of thought in a deceptive manner, because the effective h must always be a shallow function f(x).”

This might explain why Serial Scaling works well in LLMs; looped etc. This also motivates “true state” models.
mike Verified account @mike64_t Jul 18
Glad that more people are finally working on this.
I also might have been at least partially responsible for this haha.

The principle here spans quite wide:
Standard video diffusion doesn't work because diffusion steps don't add depth. How do you expect it to world model bouncing balls when the act of collision is a literal data dependent branch?
You don't. You can't. Make it block-causal and suddenly it starts working. (recall "The Seriality Gap in Video Diffusion Models" from 3 days ago, although the result was proven all the way back in "The Serial Scaling Hypothesis")
Same goes for language. Time steps don't add depth in Sequential-Parallel Duality Sequence Models such as Transformers, linear RNNs and so on.
Assume an un-truncated CoT of length n -> \inf.
Any sequence model must aggregate an ordered set of embeddings such that it represents the sequence after time t.
If the amount of decisions required to make sense of the sequence exceeds the amount of data dependent decisions the model can perform (~layer count), you get generalization failure while true RNNs can proceed.
Now, language models have exploited language to a great amount to externalize any and all computation that actually matters, so CoT has become a very expressive RNN, the unsettled question however remains for very long sequences.
Any successful policy is thus forced to continuously rebake its effective hidden state into the observations. That might at least be part of the reason why LLMs cannot effectively control their chain of thought in a deceptive manner, because the effective h must always be a shallow function f(x).
We know however that this is unsustainable. Memory is finite, context cannot grow forever. You have to learn to compress, you have to learn to forget -- and you have to be expressive while doing it.

Solving this is the key to O(1) space rollouts to free us from the chains of the notion of context and KV cache.
View quoted post on X