Tom Burns
Verified account Aleph Alpha@tfburns
research @aleph__alpha. previously @Cornell @ICERM @OISTedu @MonashUni. opinions my own. I want this AI stuff to go well (for us all).
315 Following 2.1K Followers 5K Posts
Latest posts 5
There are many subtleties to getting evaluation of models right! As others have reported previously, the location of a single space can sometimes make a large difference. Why and how? Read our blog and paper if you are interested :)
The location of one space in a multiple-choice evaluation changes what it measures. It changes the probability of the correct answer by several orders of magnitude but usually barely changes the accuracy.View quoted post on X
What evals do you rely on for model quality?
Many BPB scores are a fixed cost divided by answer length, i.e., cross-task averages can weight tasks by answers lengths. PriorBPB cuts the correlation with length from 0.68 to 0.18.
Paper and Apache 2.0 code below:
Many BPB scores are a fixed cost divided by answer length, i.e., cross-task averages can weight tasks by answers lengths. PriorBPB cuts the correlation with length from 0.68 to 0.18.
Paper and Apache 2.0 code below:
Same model, same question, same meaning, but a longer answer can improve the score ~2x. A large share of bits-per-byte (BPB), a common pre-training eval metric, reflects answer length rather than predictive quality. We propose an alternative, PriorBPB. 🧵View quoted post on X
Benchmaxxing is cheating in so many ways and just plain bad.
If you engage in it, not only do you cheat the community out of trustworthy scores on your model, you also cheat yourself and your team. And if you don't have trustworthy metrics, how can you make evidence-based decisions?
This blogpost from our team has a some great nuggets of wisdom in it. We're still learning, along with y'all :)
If you engage in it, not only do you cheat the community out of trustworthy scores on your model, you also cheat yourself and your team. And if you don't have trustworthy metrics, how can you make evidence-based decisions?
This blogpost from our team has a some great nuggets of wisdom in it. We're still learning, along with y'all :)
Benchmark text is all over training data. A model that has read the test set will nail it. That score measures memory, not skill.View quoted post on X
We scanned our mid-training pool and removed 1.5M documents. Our finding: trustworthy scores need decontaminated data.