Sewon Min

This account is verified. Learn more

@sewon__min

Assistant professor @Berkeley_EECS @berkeley_ai || Research scientist at @allen_ai || PhD from @uwcse @uwnlp

Seattle, WA sewonmin.com Joined November 2017

912 Following 18K Followers 1.2K Posts

Latest posts 5

Pinned post

Sewon Min @sewon__min May 8

As MoEs grow larger and sparser, they become memory-bottlenecked.

What if experts were actually composable - so you only keep the subset relevant to your task?

We show that this doesn't emerge in standard MoEs (their training makes this hard), but you can pre-train MoEs to support this kind of modularity!

I hope everyone sees the right figure from @RyanYixiang 's original post - I was so excited when I saw this result!!

Ryan Yixiang Wang Verified account @RyanYixiang May 8

MoEs are everywhere in frontier models, and they are deployed as a monolith system.

But many applications only need a narrow slice of capabilities, e.g., math, code, biomedical, etc.

So what if "modularity" is actually the missing opportunity for MoEs?

Today, we're releasing EMO: an end-to-end pretrained MoE where modularity emerges naturally, enabling selective use of experts!

View quoted post on X

Sewon Min @sewon__min 2h

Model competition is heating up, and distillation is 🔥🔥

Distillation detection is hard, so watermarking can help make it detectable.

But are existing watermarks robust?

We find two ways to break them: distill a model, then distill that model again; or distill only selected tokens that matter.

Then what do we do? We introduce SubWM, which outperforms prior methods and is more robust to laundering.

Harshit Varma Verified account @hrshtv 1d

Excited to present our new work on distillation detection: Subliminal Watermarks (SubWM). SubWM is much more robust compared to existing distillation detectors, is able to distinguish different watermarking keys, and also works reliably for out-of-domain prompts!
Blog: subliminal-watermarks.github.io
🧵

View quoted post on X

Sewon Min @sewon__min Nov 3

"No OCR, read text natively from images" continues!

We showed this with deep research agents: it not only improves performance but also reduces tool calls!

Post-training to work with page images gives a big boost - likely because these open-weight models haven't been trained to work with page images yet?

Exciting work led by @Hughie_Hu43 !

Tiansheng Hu @Hughie_Hu43 Nov 3

(1/7) 🚀 Introducing DR Pixel, the first open-source deep research agent that searches and reasons directly over page images, no OCR. On 1.06M pages, our trained 27B model matches GPT-5.4.

📄 arxiv.org/abs/2611.03097
Joint work from @BerkeleyNLP, @rtnldynamics!

View quoted post on X

Sewon Min @sewon__min Oct 29

Good slides are extremely intentional. When someone presents a paper, the paper already has all the information. But the speaker would make deliberate choices about things like, what to say and what not to say, what to put on a slide and what to say only verbally.

What to frame as a "side note" and when to say "this is the part I found particularly exciting."

Even individual word choices, like the subject, verb, and adjective, and when to pause, are intentional.

In fact, a good speaker would never really give a talk about a paper, even when that's seemingly what they're doing.

They would talk about what they want to say, like the story they want to tell, the kind of research they like, the vision they have, etc. Then they use the paper to support / back up their message.

Some tools are perfectly fine to use. But some skills take so much time and efforts, like years, and some tools proactively prevents learning them.

Tal Linzen Verified account @tallinzen Oct 29

my hypothesis for why AI-generated slides for technical presentations are so awful:

people who don't have a lot of experience presenting think that to make a talk you just put all of the stuff you want to talk about on the slides, then you go through each slide and say the stuff the slide says.

how you're actually supposed to prepare a presentation: you decide what to say, write it out as a rough script, then put maybe 3% of the text on the slide, plus a bunch of visual aides (e.g. plots that only include data you're planning to discuss).

AI-generated presentations are mostly the former. they have a ton of stuff the presenter never mentions, and the stuff that is most useful for the presenter is absent or not highlighted. they include enough content for a presentation 10 times longer than the actual presentation is supposed to be.

maybe try doing it the other way: first write the script for what you actually want to say, and make sure you can say all of it in the time you have. then prompt the model to create the slides based on the script. ask it to use as little text a possible, and a minimalist visual style rather than the claude house style. you can thank me later (I accept both venmo and zelle).
View quoted post on X

Sewon Min @sewon__min Oct 6

Simply prepending "\n\nOkay" to the response makes the Olmo base model match its RL counterpart. And the same for Qwen3-14B - but with " Alright," (!!) 🤯

Sophie Wang @SophieLWang Oct 6

Can “chicken” make a base model reason? 🐔

Yes!

Why? With the right first tokens, base models can match RL in reasoning. We trace this to learned training data associations and use a simple data edit to make “chicken” elicit reasoning too.

blog: sophielwang.com/cues

View quoted post on X