Twitxy Saved

Tom Burns

Verified account Aleph Alpha

@tfburns

research @aleph__alpha. previously @Cornell @ICERM @OISTedu @MonashUni. opinions my own. I want this AI stuff to go well (for us all).
tfburns.com Joined April 2009
315 Following 2.1K Followers 5K Posts

Latest posts 5

Tom Burns Verified account @tfburns
There are many subtleties to getting evaluation of models right! As others have reported previously, the location of a single space can sometimes make a large difference. Why and how? Read our blog and paper if you are interested :)
Aleph Alpha @Aleph__Alpha Oct 20
The location of one space in a multiple-choice evaluation changes what it measures. It changes the probability of the correct answer by several orders of magnitude but usually barely changes the accuracy.
View quoted post on X
Tom Burns Verified account @tfburns
What evals do you rely on for model quality?

Many BPB scores are a fixed cost divided by answer length, i.e., cross-task averages can weight tasks by answers lengths. PriorBPB cuts the correlation with length from 0.68 to 0.18.

Paper and Apache 2.0 code below:
Aleph Alpha @Aleph__Alpha Oct 13
Same model, same question, same meaning, but a longer answer can improve the score ~2x. A large share of bits-per-byte (BPB), a common pre-training eval metric, reflects answer length rather than predictive quality. We propose an alternative, PriorBPB. 🧵
View quoted post on X
1 replies 0 reposts 13 likes 513 views View on X
Tom Burns Verified account @tfburns
Benchmaxxing is cheating in so many ways and just plain bad.

If you engage in it, not only do you cheat the community out of trustworthy scores on your model, you also cheat yourself and your team. And if you don't have trustworthy metrics, how can you make evidence-based decisions?

This blogpost from our team has a some great nuggets of wisdom in it. We're still learning, along with y'all :)
Aleph Alpha @Aleph__Alpha Oct 6
Benchmark text is all over training data. A model that has read the test set will nail it. That score measures memory, not skill.

We scanned our mid-training pool and removed 1.5M documents. Our finding: trustworthy scores need decontaminated data.
View quoted post on X
1 replies 1 reposts 35 likes 1.7K views View on X
Tom Burns Verified account @tfburns
Kolibri Base is the best-performing model among its active param class that we tested. Evals are how we measured this, and we used an open-source (Apache 2.0) eval-framework to generate the reported scores. Here's some more context 🧵