Ryan Yixiang Wang

MoEs are everywhere in frontier models, and they are deployed as a monolith system.

But many applications only need a narrow slice of capabilities, e.g., math, code, biomedical, etc.

So what if "modularity" is actually the missing opportunity for MoEs?

Today, we're releasing EMO: an end-to-end pretrained MoE where modularity emerges naturally, enabling selective use of experts!

Almost all model behavior can be traced back to its training data. Unexpected LLM behaviors are often just hard to notice patterns in the data!

Human inspection just can’t catch this sort of stuff 🥸

When I use coding agents these days, I often get the feeling that some questions are inherently harder than others. I'd be comfortable asking an agent, “Is this README still up to date?”, but much less comfortable asking something broad like, “Are there any remaining bugs in this codebase?” This notion of “hardness” has always been based mostly on vibes, and it's made me wonder if there's a more concrete way to think about it.

I came across this paper by @prasann_singhal that gives a really interesting lens for thinking about "hardness" of questions over large corpora. It defines Corpus Task Complexity (CTC) based on how the number of operations needed to answer a question scales as the corpus gets larger. Some tasks, like retrieval, can be solved by examining each piece of the corpus independently, so the required work grows roughly linearly. Others require reasoning across pieces of the corpus—for example, finding all contradictions may require considering a quadratically growing number of pairs, while some tasks scale even faster. What I find really impressive from the paper is that the difference in model performance between low-complexity and high-complexity tasks exists and gets bigger as the corpus size gets larger (even after direct finetuning on task-specific training data!!), which means there's a lot more to do in this space to improve language models!

I think this is a really promising lens to view the complexity of tasks and questions on large corpuses of text. I'd highly recommend ppl check the paper out!

Presenting our work at ACL on using data watermarks for detecting if an LM used your data during pretraining (with statistical guarantees)!

Come find me and @johntzwei Monday 5:45 pm!

Also happy to chat about how hot Bangkok is, or anything LM pre-training/memorization, etc

We could detect SHA hashes that only occurred in BLOOM-176B pre-training data 90 times! For reference BLOOM has a training corpus of 341B tokens 🫣🥸