Really excited to share this work! We look at a fundamental question in SAE training: How much data do you actually need to train a Sparse Autoencoder (SAE)?
Give it a read and let us know what you think!
Give it a read and let us know what you think!
Exciting work led by @sambitghsh and @MadelynMathai on the diminishing returns of data in Sparse Autoencoders. Standard practice often uses millions of activation tokens, but it’s unclear whether that scale is necessary for different SAE goals. We investigate this systematically by training different SAE architectures on different LLM families with training budgets ranging from 5M to 500M activation tokens.View quoted post on X