Twitxy Saved

Joachim Baumann

Verified account This account is verified. Learn more

@joabaum

Postdoc @StanfordNLP @StanfordAILab / Prev: @MilaNLProc @UZH_en @MPI_IS @CarnegieMellon. CompSocSci, LLMs, algorithmic fairness.
Stanford joe-baumann.com Joined February 2021
1.8K Following 1K Followers 267 Posts

Latest posts 5

Pinned post

Joachim Baumann Verified account @joabaum
We present SWE-chat: the first large-scale dataset of coding agent interactions from real users in the wild.

In 40% of real coding sessions, the agent writes ~all the code. Users push back 39% of the time – agents almost never stop to check.

Data, paper, & findings in the 🧵👇
14 replies 77 reposts 482 likes 73K views View on X
Joachim Baumann Verified account @joabaum
it has never been easier to simulate how users interact with coding agents. @harborframework now supports simulated user interactions with claude code, codex, opencode, and gemini-cli!

the setup is super simple: just choose two agents. one plays the user and holds the task, the other runs like any standard coding agent. after each turn, the user agent reviews what was produced and sends the next prompt.

docs.harborframework.com/core-concepts/…

thanks @alexgshaw @kobe0938 for @harborframework
docs.harborframework.com Simulate a user - Harbor Evaluate an agent through a multi-turn conversation with a simulated user.
Kobe Verified account @kobe0938 Sep 22
huge shoutout to @joabaum for adding Codex and OpenCode as simulated-user targets in harbor. i instant-merged his PR.

here's why:
View quoted post on X
6 replies 6 reposts 35 likes 3.4K views View on X
Joachim Baumann Verified account @joabaum
Auditing a model via the API doesn't tell you what users get in the chatbot. Our new EMNLP paper led by @jennjwang is timely evidence that the third-party access debate should be about systems, not just models!
Jennifer Wang @jennjwang Sep 15
Third-party auditors often audit models via the API, but do those findings transfer to chatbot interfaces?

We audited seven systems across ChatGPT, Claude, and Gemini and found that they don’t. (EMNLP 2026)
View quoted post on X