Pinned post
Ken Gu
@kenqgu
160 Following 223 Followers 38 Posts
Latest posts 5
Even as agents gain greater autonomy, it’s critical to consider human collaboration in agent design and evaluation. This work is an important step in this direction. Check it out!
Today's AI agents are optimized to complete tasks in one shot. But real-world tasks are iterative, with evolving goals that need collaboration with users.
We introduce collaborative effort scaling to evaluate how well agents work with people—not just complete tasks 🧵View quoted post on X
True intelligence = reasoning about new information, not memorized facts.
How can we scalably create benchmarks that are completely novel yet have known answers?
Meet SynthWorlds, an eval & data-gen framework to disentangle reasoning and knowledge⬇️🧵
📄arxiv.org/pdf/2510.24427
How can we scalably create benchmarks that are completely novel yet have known answers?
Meet SynthWorlds, an eval & data-gen framework to disentangle reasoning and knowledge⬇️🧵
📄arxiv.org/pdf/2510.24427
Excited to be at #EMNLP in Miami this week! ☀️
I’ll be sharing our work on evaluating LLM agents for scientific analyses. Join me Thursday, 10:30am-noon to learn more about how we automate evaluations for complex open-ended tasks.
I’d love to connect and chat about AI agents!
I’ll be sharing our work on evaluating LLM agents for scientific analyses. Join me Thursday, 10:30am-noon to learn more about how we automate evaluations for complex open-ended tasks.
I’d love to connect and chat about AI agents!
🚀 New Benchmark Alert: ⚙️ BLADE — A benchmark with expert-annotated analyses, evaluating how #LLM agents tackle open-ended, data-driven scientific research questions. Results show top models reach only ~44% F1.
📄 arxiv.org/abs/2408.09667
👨🏻💻 github.com/behavioral-dat…
🧵1/9View quoted post on X
🚀 New Benchmark Alert: ⚙️ BLADE — A benchmark with expert-annotated analyses, evaluating how #LLM agents tackle open-ended, data-driven scientific research questions. Results show top models reach only ~44% F1.
📄 arxiv.org/abs/2408.09667
👨🏻💻 github.com/behavioral-dat…
🧵1/9
📄 arxiv.org/abs/2408.09667
👨🏻💻 github.com/behavioral-dat…
🧵1/9