cider

This account is verified. Learn more

@jeffreycider

my purpose in life is to forget linear algebra 2x a year

San Francisco, CA Joined September 2019

771 Following 2.8K Followers 892 Posts

Latest posts 5

Pinned post

cider @jeffreycider May 31, 2022

linear transformations stretch euclidean space
ReLU folds euclidean space

neural networks are just repeated origami on high-dimensional laffy taffy

cider @jeffreycider 3d

kanto is pokemon australia

It has 33 poison-types. 22% of kanto pokemon are poisonous. 37% of poison-type pokemon are native to kanto

cider @jeffreycider 6d

we discovered the edge of stability for muon
(and also every other non-adaptive optimizer)

we call it bubbleforce.

also we shaved off 100 iterations from the nanoGPT muon record (-3.2%)

0:26

cider @jeffreycider Oct 18

just like me frfr

cider @jeffreycider Oct 5

hiding your y-axis labels when you know that you're benchmarking against the first 3% of a nanogpt train is academic dishonesty

cider Verified account @jeffreycider Oct 5

@industriaalist you're only training until 5.0 nats of fineweb validation loss? that's so high that it's not even on the graph for nanogpt-small training. at this stage, most of the signal is still sending logits of unlikely tokens to neginf, feature learning hasn't really begun yet

View quoted post on X