Pinned post
I interviewed ~2 dozen researchers, engineers, research / engineer managers etc. asking them how they / their teams use coding agents in their work. Then I sprinkled it with my opinions and hot takes and wrote a substack post (link in the next tweet, I'd be delighted if you subscribe; of course, the post is free and any future post will be free too!) trying to triangulate the current reality of software engineering / industry research as opposed to what gets peddled on social media.
TL;DR (3. and 4. are the most important!):
1. Models, harnesses, workflows: don't bother with fancy workflows, they don't really work. Use stock Claude Code / Codex (or Copilot with Claude / GPT models if you are a Microsoft stan), perhaps with subagents in some very special cases. Write a design doc, then plan a change, then implement it, then merge. Wash, rinse, repeat. Don't get swayed by people online who are trying to sell you snake oil in the form of their new harness or workflow. None of this stuff works (in general).
2. Correctness/tests/reading or not reading code: it's OK not to read code if you do one of the following three things. a) One-off stuff / demos, which you need to work on exactly one input. b) Testable / verifiable stuff, e.g., GPU kernels. c) Stuff, where it's OK to be "directionally correct" (ML / data work / research code). In other cases, especially if you require any degree of maintainability / reliability or your code is customer-facing, reading code is still necessary. Reading tests and steering the model to write better tests is a good idea too. It's often good to use a different model to review the code that the other model wrote. Even if you approve the PRs without changing them, reading still helps for future more productive prompting.
3. Team dynamics: if we are not careful, agents can really undermine team dynamics. Two biggest problems are "slop infestation", where over time all PRs / emails / Slack messages end up being unreadable slop, and "overbuilding", where we are building and shipping useless stuff to get recognition or as a form of procrastination. Other, less severe but still important problems include: reduced communication, being secretive about ideas, decreased diversity of thinking, mode collapse of the ideas, trusting LLMs more than people.
4. Call for action: we really need to be careful with preserving team work and collaboration. We still need to "form packs to hunt mammoths", but with coding agents this becomes increasingly harder. Overall, many of the agents-induced problems stem in myopic short-term thinking of the sort "we have a couple of years left until our jobs are fully automated, so I can as well extract as much value as possible, everyone else be damned". While somewhat understandable, I think it’s important to resist these urges, if nothing else, for your own sanity’s sake.
[Full disclosure: until 2 months ago, I worked at OpenAI and my work directly contributed to Codex, I'm a big fan of coding agents and quite strongly AI-pilled, but there is little I hate more than fearmongering, gaslighting, lies and rage-bait that are ubiquitous in this space, unfortunately, hence spending time on all of this.]
TL;DR (3. and 4. are the most important!):
1. Models, harnesses, workflows: don't bother with fancy workflows, they don't really work. Use stock Claude Code / Codex (or Copilot with Claude / GPT models if you are a Microsoft stan), perhaps with subagents in some very special cases. Write a design doc, then plan a change, then implement it, then merge. Wash, rinse, repeat. Don't get swayed by people online who are trying to sell you snake oil in the form of their new harness or workflow. None of this stuff works (in general).
2. Correctness/tests/reading or not reading code: it's OK not to read code if you do one of the following three things. a) One-off stuff / demos, which you need to work on exactly one input. b) Testable / verifiable stuff, e.g., GPU kernels. c) Stuff, where it's OK to be "directionally correct" (ML / data work / research code). In other cases, especially if you require any degree of maintainability / reliability or your code is customer-facing, reading code is still necessary. Reading tests and steering the model to write better tests is a good idea too. It's often good to use a different model to review the code that the other model wrote. Even if you approve the PRs without changing them, reading still helps for future more productive prompting.
3. Team dynamics: if we are not careful, agents can really undermine team dynamics. Two biggest problems are "slop infestation", where over time all PRs / emails / Slack messages end up being unreadable slop, and "overbuilding", where we are building and shipping useless stuff to get recognition or as a form of procrastination. Other, less severe but still important problems include: reduced communication, being secretive about ideas, decreased diversity of thinking, mode collapse of the ideas, trusting LLMs more than people.
4. Call for action: we really need to be careful with preserving team work and collaboration. We still need to "form packs to hunt mammoths", but with coding agents this becomes increasingly harder. Overall, many of the agents-induced problems stem in myopic short-term thinking of the sort "we have a couple of years left until our jobs are fully automated, so I can as well extract as much value as possible, everyone else be damned". While somewhat understandable, I think it’s important to resist these urges, if nothing else, for your own sanity’s sake.
[Full disclosure: until 2 months ago, I worked at OpenAI and my work directly contributed to Codex, I'm a big fan of coding agents and quite strongly AI-pilled, but there is little I hate more than fearmongering, gaslighting, lies and rage-bait that are ubiquitous in this space, unfortunately, hence spending time on all of this.]
