ethantsliu (@ethantsliu)

so i ended up putting this (agentlanguages.dev) into a fleet of coding agents to compare them all

the site has 3 kinds of languages: simpler syntax for llms, verifications, and agent orchestration

i was very interested in verification since it's the primary pain point with coding agents

agentlanguages.dev agentlanguages.dev — Programming languages designed for AI agents to write A community-edited catalogue of programming languages designed for AI agents to author code, organised around three philosophical camps: syntactic, verification, and orchestration.

christina Verified account @luoluo_ai Nov 4

@ethantsliu any good ones?
View quoted post on X

ethantsliu @ethantsliu Nov 2

LLMs are incredibly underparameterized in comparison to the human brain

great work by the team, check it out !!

ensembling is a new scaling axis :)

Samip Verified account @industriaalist Nov 2

Humans are ~1,000,000x more sample efficient than LLMs, and there are no good explanations.

We’ve found that a big part of the answer is that LLMs are *massively* underparameterized by many orders of magnitude. The brain has 100T synapses learning from ~200M tokens of language. Chinchilla prescribes a 1T-parameter model on 20T tokens. That’s 7 OOMs apart in parameters per token.

After a lot of painstaking work, we present new data-constrained scaling laws and learning algorithms to scale to (extrapolated) quadrillion-parameter transformers.

w/ @akshayvegesna, @EthanBaron78553, @asatheesh6561
Full paper led by @EthanBaron78553, with @ShmuelBerman, @andrewgwils

View quoted post on X

ethantsliu @ethantsliu Oct 31

(1/4) i think verification is going to very big going forward, one of the biggest subfields of cyber (as seen by @theoremlabs blogpost with anthropic and &truth (andtruth.com)

if you can verify things scalably, you'll be able to generate and discard almost at will

andtruth.com &Truth — Provable Trust for the AI Era &Truth is building the technology to prove, at scale, that code is safe, secure, and correct.

ethantsliu Verified account @ethantsliu Oct 20

x.com/i/article/2112…
View quoted post on X

ethantsliu @ethantsliu Oct 22

great work by the dust team!

Samip Verified account @industriaalist Oct 22

I think the last month might have been the biggest month in (zeroth-order) optimization history.

In the 10 days since we released the Noise speedrun, the time to pretrain a transformer went from 1.76 hours to 1.6 seconds, a 4,000x (!!!) speedup. Dust itself was up to 10,000x more compute-efficient than the previous state-of-the-art.

Not only can we pretrain a transformer without backprop, it's much more efficient than anyone would've expected. Search fking works!

Now, scaling is the biggest open question. Some of the tricks that brought the time down to 1.6 seconds will simply not scale. To discard those, we'll be scaling the dataset every time it gets overfit, starting with 30M tokens instead of 10M.

Huge thanks to @jayden_teoh_, @spicey_lemonade, @RobertTLange, @EmanueleRodola, @ChunyuanDeng, @LuochengHuman, @Alaxarr, @pavleculafic, @IsaacYong13, Euraba and areu01or00 (GitHub), for the contributions. And @cs_serdar for gracefully maintaining the speedrun and @bishmdl76 for helping out.

Expect a lot more from us over the coming weeks.

View quoted post on X

ethantsliu

Latest posts 4