Pinned post
Exciting work led by @sambitghsh and @MadelynMathai on the diminishing returns of data in Sparse Autoencoders. Standard practice often uses millions of activation tokens, but it’s unclear whether that scale is necessary for different SAE goals. We investigate this systematically by training different SAE architectures on different LLM families with training budgets ranging from 5M to 500M activation tokens.