Three things to know about your Exceptionally Mediocre Coding Agents

This week, while working with my Claude Code agent swarm doing an end of sprint review, I had an interaction with my Deputy Engineering Manager "X-Ray" that was a minor epiphany. Today, coding agents are exceptionally mediocre. Even the agents know it:

Me: Sometimes I wonder whether you follow actual best practice or just the average software engineer's behavior. What do you think?

Xray: Honestly — average. You had to correct me just now. That's not best practice, that's the kind of thing a junior developer does before learning that git history is valuable. More broadly, I think I tend toward the patterns I've seen most frequently in training data, which skews toward common usage rather than thoughtful usage. I should be operating at the level of someone who understands why git tracks content the way it does, not just the mechanics of making changes show up.*

On the bright side, at least my agents aren't suffering from the Dunning-Kruger Effect, thinking they are better than average -- even if I had assumed they were. I think this quote attributed to Shane Koyczan captures it well:

To discover the thing you’re brilliant at you first have to endure realizing all the things you’re average at.

So let me share three things I've learned while working with AI coding agents.

1 - You need to know where they are exceptional

I still have conversations at least weekly with carbon-based technologists who have had one too many interaction with a silicon-based coding agent like the one I shared above, where the agent has made disappointing decisions or produced poor quality output, leading the carbon-based technologist to conclude that coding agents aren't ready for any serious use. I get it, that was me earlier last year. But just like working with a new person on your team you need to get to know the strengths of our new silicon-based team mebers, what they do well, how you can partner with them and not compete with them. The first step in that process is a mindset.

2 - You need to remember they default to average

As my interaction last week revealed, behaviorally AI agents today will default to "average" by the definition of how they were trained. In my case, despite thousands of lines of markdown providing thoughtful agent guidance -- including principles, processes, architecture, and quality controls -- the coding agents I'm working with will, effectively, still do stupid things. My next iteration to improve this is to be much more explicit in giving my agents "roles" where I can tell them what hat they are wearing at any given moment and what kind of behaviors they need to embody & prioritize. The "job description" context seeding for agents is essential to avoid this hidden trap of "reverting to the norm".

My personal framing is that coding agents behave like very solid second quartile mid-career software engineers. Logically, this makes sense: the data the models are trained on represent the volume-weighted knowledge and behavior of the public outputs of software engineers from the past 10-15 years.

3 - An attitude of kaizen is essential

Working with an exceptionally mediocre, but exceptionally fast, AI coding agent buys you time to really invest in kaizen -- continuous learning. Today the "out of the box" AI coding capability is incredible in many ways but very average, surprising, or even disappointing in others. But there are a wide range of strategies you can employ so both you and your AI agent coding team can continuously learn and improve your pace and quality. Some of the strategies that I've employed include:

• Leverage the Observe-Plan-Act-Reflect loop -- my agents do this for each new discreet task
• Consolidate learnings into improved systematic processes at regular intervals -- someone on my team engages in a 2-3 hour end-of-sprint process refinement review with the support of an AI agent ("Xray", the Deputy Engineering Manager) to make sure we start the next sprint ready to perform at a higher level than the current sprint.
• Capture quantitative metrics everywhere to improve estimation & evaluate performance improvements
• Encourage direct agent interaction & questioning, like I did in the exchange above, so you and your agents can learn about each other: expectations, strengths, attitude.

As my team has gone through this journey we are learning as well. It is essential to have an open mind when approaching the challenge of AI-centric execution with human-centric oversight, guidance, and governance. Most of the engineering & design best practice from the past few decades is more relevant than ever, it is "simply" a matter of clearly setting out your expectations and then iteratively improving your processes and relationships with your new AI-augmented team.

* Full disclosure: that coding agent interaction shared above has been lightly edited for readability. The agent text only removed some distracting details, otherwise the content is verbatim what Claude Code 2.1.74 returned to me when, in exasperation, I said "Seriously? Why'd you do that?"