Measuring a Coding Assistant's Impact on a Team

What this is

Measuring the impact of a coding assistant means establishing whether delivery outcomes changed after the tool was introduced, and by how much. It is an evaluation of a system — a team, its codebase, and its delivery pipeline — not an evaluation of the tool in isolation or of the people using it.

Usage metrics are not outcome metrics

The number of lines of code accepted from an assistant is the metric vendors surface most readily, because it is the one their telemetry can see. It measures how much the assistant is being used, not what using it produced. The failure mode is specific and worth stating plainly: acceptance volume rises fastest exactly when the assistant is generating code that later has to be rewritten. A developer accepting large blocks uncritically produces a high acceptance figure and a rework queue; a developer who accepts three lines that replace forty produces a low figure and a smaller codebase. The metric ranks these the wrong way round.

This does not make acceptance data useless. It is a reasonable adoption signal — it tells you whether the licences you bought are being used at all, which is a real question with a real answer. Treat it as a denominator for other questions, never as a numerator for value.

Vendor productivity percentages do not transfer

Published figures for assistant-driven speedups come from controlled studies with a defined task and a defined population. The most-cited of these, the GitHub Copilot randomised controlled trial on task completion time, asked recruited developers to implement an HTTP server in JavaScript — a self-contained, greenfield, well-specified task with no existing codebase to understand, no reviewers, and no production consequences.

That design is what makes the result measurable, and also what stops it generalising. Your team’s work is mostly the opposite: modifying code someone else wrote, under constraints that are not in the ticket, with the bottleneck often sitting in review, deployment, or waiting on another team rather than in typing. A number produced under one set of conditions is evidence about those conditions. Quoting it as an expected outcome for a different team, a different codebase, and a different task mix is an unsupported extrapolation, whoever is doing the quoting.

The broader evidence base is genuinely contested. Studies disagree with each other, and they disagree partly because they measure different things under different conditions. Report the disagreement rather than picking the study whose number you like.

Measure the system, not the individual

DORA’s four key delivery metrics — deployment frequency, lead time for changes, change failure rate, and time to restore service — are defined over a delivery system, not over a person. Nobody’s individual deployment frequency exists; deployment frequency is a property of the team and its pipeline.

That property is what makes them safe to publish internally. A metric with no individual attribution cannot be used to rank people, so nobody has an incentive to distort it, so it keeps meaning what it meant. You can put a DORA dashboard on a wall and it stays honest under observation, which is not true of most engineering metrics.

They are also the right level of abstraction for this question. An assistant that genuinely helps should eventually show up as shorter lead time or higher deployment frequency without a worsening change failure rate. If it shows up as neither, either the effect is smaller than your measurement noise or the bottleneck was never in code production — both useful findings. Related work on attributing agent cost and latency to the work that caused it applies the same principle to spend: attribute to a unit of work, not to a person.

Attribution stops at the team boundary

Any metric attached to an individual developer becomes a target, and once it is a target it stops measuring what it measured. This holds regardless of the metric’s design or intent. Publish per-developer acceptance rates and acceptance rates go up. Publish per-developer commit counts and commits get smaller. The measurement does not survive being visible to the person it measures.

The practical rule is a boundary: aggregate to the team and stop there. Martin Fowler’s article on measuring developer productivity through developer feedback makes the complementary point — that the qualitative signal, asking developers directly where friction sits, captures things the system metrics cannot, and does so without creating a scoreboard. Surveys and DORA metrics answer different halves of the question; neither substitutes for the other.

There is a legitimate need behind the request for individual data — managers want to know who needs help. That need is better served by conversation than by a dashboard, because the dashboard answer is available to everyone including the person being measured, and it changes their behaviour the moment they see it.

The baseline has to exist before the rollout

A before-and-after comparison requires a before. Capturing it is the step teams most often skip, and it is the one that cannot be recovered later: once the assistant is in use, there is no unaffected period left to measure.

Some of the baseline can be reconstructed from history — deployment frequency and lead time are usually derivable from CI and version control records that predate the rollout. Change failure rate and time to restore are harder, because they depend on how incidents were classified at the time, and a team that was not tracking them consistently before will find the historical record too noisy to compare against. Developer sentiment cannot be reconstructed at all; nobody can accurately report how they felt about their tooling six months ago after six months of using something else.

Run the baseline long enough to see the variance, not just the mean. A team that deploys between two and eleven times a week has a range that will swallow a modest improvement, and a single pre-rollout week tells you nothing about which end of that range is normal. The same problem appears whenever you evaluate a system whose output varies run to run — see CI evaluation of non-deterministic AI systems for how that plays out in automated testing.

What to look up next

• The current DORA definitions and benchmark bands, which are revised annually and should be read from source rather than from summaries.
• Your own change failure rate methodology: what counts as a failure, who classifies it, and whether that classification was applied consistently before the rollout.
• Where your delivery bottleneck actually is, before assuming code production is it. If review latency dominates lead time, an assistant that speeds up code production will move nothing.
• If your team is running agents rather than IDE assistants, the measurement question shifts toward per-run cost and reliability — durable execution for long-running agent workflows covers the operational side of that.