Twitxy Saved

Zachary Novack

@zacknovack

zacharynovack.github.io Joined June 2022
675 Following 876 Followers 438 Posts

Latest posts 5

Zachary Novack @zacknovack
my goat has done it again, @hayden_prairie truly a must follow for the state of looped models!
Hayden Prairie Verified account @hayden_prairie Oct 23
Rumor has it frontier labs are using looped LMs to scale quality without scaling parameters. But KV-cache memory still grows with every loop. Is that necessary?

No! Our method BeNice! unlocks near-lossless sharing of the final KV-cache across loops: more depth, same KV memory.

This lossless KV-sharing isn’t guaranteed out of the box with looped models. 👀👇for our method BeNice! to teach your model that sharing is caring!
View quoted post on X
0 replies 0 reposts 14 likes 725 views View on X
Zachary Novack @zacknovack
model seems dope but this is all could think of
Diogo Almeida Verified account @CompleteSkeptic Sep 15
After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI?

I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev

• 20-200x faster
• 40-400x cheaper (w/ output tokens free)
• Frontier composable intelligence optimized for decisions

AFAICT the shortest path to AI-based economic revolution
View quoted post on X
2 replies 0 reposts 10 likes 698 views View on X
Zachary Novack @zacknovack
Video / AR Diffusion folks!

Get rid of your pesky noise schedule + hand crafted samplers, all you need is simple gradient descent given by our Equilibrium Forcing approach.

The GOATs @hansenlillemark and @a5rojas did a fantastic job here, check out the thread below for more!
Hansen Lillemark @hansenlillemark Aug 18
Rigid, hand-designed sampling schedules become inaccurate during autoregressive video generation as errors accumulate.

We introduce Equilibrium Forcing (EqF), a modular framework for video generative modeling that removes noise level conditioning to enable adaptive inference. 🌀
View quoted post on X
0 replies 3 reposts 46 likes 6.4K views View on X
Zachary Novack @zacknovack
@huskydogewoof is truly one of the looped model GOATs, awesome work here!

AND they used Parcae-style stable input injection too🤘
Benhao Huang Verified account @huskydogewoof Jul 31
🔥
𝐍𝐞𝐰 𝐛𝐥𝐨𝐠: 𝐓𝐨𝐰𝐚𝐫𝐝𝐬 𝐋𝐨𝐨𝐩𝐞𝐝 𝐌𝐨𝐝𝐞𝐥𝐬 𝐃𝐨𝐧𝐞 𝐑𝐢𝐠𝐡𝐭 — 𝐏𝐚𝐫𝐭 𝐈

Looped models reuse the same weights across depth, promising a better compute–parameter trade-off, especially for reasoning.

𝐁𝐮𝐭 𝟏) 𝐝𝐨 𝐭𝐡𝐞 𝐠𝐚𝐢𝐧𝐬 𝐬𝐮𝐫𝐯𝐢𝐯𝐞 𝐰𝐡𝐞𝐧 𝐛𝐨𝐭𝐡 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐚𝐧𝐝 𝐢𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞 𝐅𝐋𝐎𝐏𝐬 𝐚𝐫𝐞 𝐦𝐚𝐭𝐜𝐡𝐞𝐝? 𝟐) 𝐀𝐧𝐝 𝐰𝐡𝐢𝐜𝐡 𝐚𝐫𝐜𝐡𝐢𝐭𝐞𝐜𝐭𝐮𝐫𝐚𝐥 𝐜𝐡𝐨𝐢𝐜𝐞𝐬 𝐚𝐜𝐭𝐮𝐚𝐥𝐥𝐲 𝐦𝐚𝐭𝐭𝐞𝐫?

We run 𝐚𝐩𝐩𝐥𝐞𝐬-𝐭𝐨-𝐚𝐩𝐩𝐥𝐞𝐬 ablations spanning Ouro to Huginn. Huginn performs better overall, with the largest gains coming from the loop-in-the-middle (sandwich) design and input injection, though they provide different benefits.

Trained on 𝟓𝟎𝟎𝐁 tokens, an 𝟖𝐁-𝐀𝟎.𝟖𝐁 Huginn MoE approaches or surpasses a 𝟑𝟐𝐁-𝐀𝟑.𝟐𝐁 feedforward MoE on several reasoning benchmarks, including GSM8K (83.6% vs. 80.8%), while using 𝟕𝟓% 𝐟𝐞𝐰𝐞𝐫 resident parameters under 𝐦𝐚𝐭𝐜𝐡𝐞𝐝 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐚𝐧𝐝 𝐢𝐧𝐟𝐞𝐫𝐞𝐧𝐜𝐞 FLOPs.

More details and the blog link in the thread ↓
View quoted post on X