Twitxy Saved

Jacob Buckman

@jacobmbuckman

SF jacobbuckman.com Joined December 2016
385 Following 5.9K Followers 2.4K Posts

Latest posts 5

Jacob Buckman @jacobmbuckman
I appreciate what @willdepue is aiming for here, but I have always found these compute-constrained competitions to be a bit misguided. Compute is fungible: constraining compute spent on gradient descent just means that the best models will be the ones that spend the most compute elsewhere.

The most obvious way to exploit this is to do a giant hyperparameter sweep and submit the best. But the same principle applies to other parts of the architecture; the more compute you have to spend, the more architectural tweaks you can test (or optimizer variants, loss functions, etc).

In fact, you can even achieve arbitrarily-high-compute performance by just defining your architecture to have no weights and 16MB of hardcoded constants. You can use as much compute as you would like to find your constants! In the actual train loop you do 0 steps of training and have 0MB of weights, so technically, you are within budget and have a valid leaderboard submission.

This is obviously not within the *spirit* of the competition, but it seems certainly allowed by the rules as stated. And sure, you can manually disqualify submissions that do this explicitly, but you can't do much about one-off improvements with a nice just-so story that implicitly rely on the same principle, advantaging researchers with more access to compute.
will depue Verified account @willdepue Mar 18
we're launching a challenge, inspired by @kellerjordan0's NanoGPT Speedrun, to train the best model under extreme parameter limitations

anytime i get asked how to break into ML, i point to challenges like this: just start training models!

also, we're covering $1M of compute
View quoted post on X
3 replies 0 reposts 17 likes 9.5K views View on X
Jacob Buckman @jacobmbuckman
New post: "Gradient Descent Is Not Learning, But Evolution"

It is popular to use the analogy that training a neural network is like teaching a human. But I think there is a better analogy available.

Training ~= human evolution. Inference ~= human learning.

Link below
4 replies 1 reposts 20 likes 1.4K views View on X