Sean Yun-Shiuan Chuang

LLM role-playing agents can look human-like in short interactions. But once they debate each other in groups over multiple rounds, do their collective opinions evolve like real human groups?

We introduce DEBATE: the first large-scale benchmark built specifically to evaluate the human-likeness of simulated opinion dynamics in role-playing LLM agents.

💬 To build DEBATE, we collected real human data from a demographically diverse pool of 2,832 participants in 708 groups across 107 controversial topics through a structured online discussion platform, yielding 30,707 messagesand privately reported beliefs.

📊 Main result: LLM groups over-converge toward consensus relative to human groups. Post-training with SFT/DPO narrows this gap and improves alignment, even on unseen topics.

🌐 Website: debate-benchmark.vercel.app
📄 Paper: arxiv.org/abs/2510.25110
🤗 Dataset: huggingface.co/datasets/seant…

Excited to present our new LLM agent benchmark at the SEA @ #NeurIPS2025! 🎉

Paper: “DEBATE: A Large-Scale Benchmark for Role-Playing LLM Agents in Multi-Agent, Long-Form Debates”

TL;DR: We release DEBATE, a large-scale empirical benchmark of 29k+ messages from 2,700+ U.S. participants debating 107 controversial topics. We turn humans into “digital twins” for LLM agents and evaluate opinion dynamics alignment at utterance, individual, and group levels.

We find that role-playing LLM agent groups over-converge, show positive drift in stance, and are more influenced by partners than humans, and that vanilla SFT on human debate mostly improves surface style (ROUGE, length) but not deeper opinion alignment.

Shout out to my amazing collaborators: Tim Rogers, @JunjieHu12, @dvshah, @sijiayang_camer, Ruixuan Tu, Chengtao Dai, Smit Vasani, @Binnie8545 , Michael Henry Tessler, Robert Hawkins, @JunjieHu12 🙌

Link to the paper: arxiv.org/abs/2510.25110
See you at SEA @ NeurIPS 2025 Dec 7th! 🤖🧠📊

Paper accepted at EMNLP 2025 (main track)!
Can LLMs "Guesstimate" - making rough but educated guesses about real-world quantities?

For example,
💡 How many marbles fit in a cup?
📈 What will US GDP be in Q2 2025?
🇺🇸 What % of votes will Kamala Harris get in Ohio in 2024 US?

🕒These questions refer to events after the LLMs' training cutoff — so models can't just memorize. They must reason using their internal world model.

Despite being common in real life, guesstimation has been overlooked in LLM research.

🔍Inspired by “Wisdom of Crowds” in cognitive science, we propose WOC decoding — taking the median across multiple reasoning paths.

✅WOC decoding consistently improves LLM accuracy.
📄 Paper Link: arxiv.org/pdf/2501.17310

Honored to have our paper "No Preference Left Behind: Group Distributional Preference Optimization (GDPO)" accepted to #ICLR2025! 🎉 Huge thanks to @Binnie8545 for leading this work.

TL;DR: Preference-based alignment methods like DPO eliminate minority opinions—how can we ensure LLMs reflect the full spectrum of human preferences? 🤔

We proposed GDPO to tackle this, enabling LLMs to generate responses with diverse viewpoints.

Paper: arxiv.org/pdf/2412.20299

Excited to present my paper on role-playing LLM agents at #EMNLP2024! 🎉

Paper Title: “Beyond Demographics: Aligning Role-playing LLM-based Agents Using Human Belief Networks”

Come say hi and let's chat about some exciting research! 🤖🧠✨

TL;DR: How can we make LLM agents serve as “digital twins” and role-play people with diverse backgrounds? We showed that feeding demographic information alone is not enough to align LLM agents’ belief with corresponding human individuals. Instead, we showed that incorporating “belief networks” latent information estimated from human empirical data boosts the LLM-human alignment by 23% on average across 64 topics.

Shout out to my amazing collaborators: Tim Rogers, @JunjieHu12, @dvshah, @sijiayang_camer, Krirk Nirunwiroj, @studdiford41709, @agam_goyal,

Link to the paper: arxiv.org/abs/2406.17232

Poster at EMNLP: Nov 14 (Thu) 2:00-3:30pm