In 2025, a nonprofit research group ran the kind of study the AI productivity debate had been missing: a randomized controlled trial. Sixteen experienced open-source developers worked on 246 real issues from repositories they knew intimately, with each task randomly assigned to allow or forbid AI tools. The developers predicted AI would make them 24 percent faster. Afterward, they estimated it had made them 20 percent faster.
The measured result: with AI allowed, tasks took 19 percent longer.
The headline number matters less than the gap it exposed. These weren’t skeptics or novices; they were experts who experienced the slowdown firsthand and still felt sped up. Felt productivity and measured productivity can point in opposite directions, and nothing about that is specific to programmers.
When Does AI Actually Cost More Time Than It Saves?
AI tends to cost net time when you’re already expert at the task, when verifying the output takes as long as doing the work, when your context is too large or private for the tool to hold, and when the task punishes small errors severely. It tends to save time on first drafts, unfamiliar domains, formatting, and volume work where errors are cheap and obvious.
The METR study’s own caveats define the boundary honestly. The setting was maximally unfavorable: deep experts, mature codebases with strict standards, tasks the developers could nearly do from memory. METR itself has since noted the result is historical, tied to early-2025 tools, and that both tools and workflows have shifted since. The durable finding isn’t “AI slows everyone down.” It’s that self-assessment of AI’s help is unreliable enough that it shouldn’t be the basis of the decision.
The Four Situations Where AI Underperforms
You’re the expert and the task is in your muscle memory. The study’s developers averaged five years in the codebases they were modifying. When you can nearly do the work as fast as you can describe it, describing it to a model, reviewing its attempt, and fixing the near-misses is pure overhead. Delegation only pays when the delegate is faster than you, including the management time.
Verification costs as much as production. For a fabricated-fact-intolerant task, legal citations, financial figures, medical details, the checking work is the work. If every output line needs verification against a source, the model produced a draft of your real job, not a completion of it. The tasks where this bites hardest are exactly the high-fabrication-risk categories covered in our guide to fact-checking AI output.
The context won’t fit or can’t be shared. Tasks that depend on knowing your whole system, your team’s history, or confidential material the tool shouldn’t hold push the model toward plausible-but-wrong output, since it fills the gaps it can’t see. Partial context in, confident nonsense out.
Small errors are expensive and hard to spot. AI errors don’t look like human errors; they’re fluent, well-formatted, and hiding in the one changed number or inverted condition. Tasks where a subtle error survives review and detonates later, contracts, configs, dosage tables, invert the economics of a fast draft.
Where AI Reliably Earns Its Time
The mirror image, for balance, since this is a boundary question rather than a verdict:
- First drafts of anything, where blank-page cost is high and your editing pass was going to happen anyway
- Domains where you’re the novice, where the model’s baseline beats yours and you’re learning, not certifying
- Format transformations: summarizing, restructuring, converting between formats, where correctness is checkable at a glance
- Volume work with cheap errors: tagging, sorting, boilerplate, where a 5 percent miss rate costs seconds
- Search-shaped questions you’d have spent twenty minutes googling
A Two-Column Test Before You Delegate
| Ask yourself | Lean AI if… | Lean manual if… |
|---|---|---|
| How fast am I at this already? | You’d fumble or research | You’d finish before finishing the prompt |
| What does checking cost? | Errors are obvious at a glance | Every line needs verification against a source |
| Does the model have the context? | The task is self-contained in what you can paste | It depends on unwritten or unsharable knowledge |
| What does an escaped error cost? | Embarrassment at most | Money, safety, or legal exposure |
| Am I measuring or vibing? | You’ve timed both paths | It just “feels faster” |
That last row is the study’s real lesson. The developers weren’t lying about feeling faster; offloading effort genuinely feels like speed, because the hard, boring parts get quieter even when the clock says otherwise. The only fix is occasionally timing both paths on a recurring task, the same evidence-first habit behind our one-week trial method: a few timed runs beat a month of impressions.
Two adjacent mistakes are worth naming. Abandoning AI entirely after one bad fit wastes the categories where it genuinely pays. And treating “AI was slow here” as final sometimes just means the wrong tool shape was used, a chat window where the task needed nothing at all, or a frontier model where a task needed none, a sizing question we take up in do you need a frontier model.
Frequently Asked Questions
Didn’t that study only cover programmers? Directly, yes: 16 experienced open-source developers on their own repositories. What generalizes isn’t the 19 percent; it’s the demonstrated gap between felt and measured productivity, and the mechanism, expertise plus verification cost, applies to any skilled work.
Does the study mean AI coding tools don’t work? No, and METR itself flags the result as specific to early-2025 tools and an unusually expert setting. It means the benefit isn’t universal, varies with your expertise and the task, and can’t be judged by how productive the session felt.
Why does AI feel faster even when it’s slower? Because it removes the effortful parts, typing, recalling, starting, and effort is what we perceive as time. Waiting, reviewing, and correcting feel lighter than they cost. Only measurement catches the difference.
So when should I definitely still use AI? Drafts, unfamiliar territory, reformatting, summarization, and high-volume low-stakes work: places where you’d accept a decent-but-imperfect result and can check it cheaply. That covers a lot of real work, which is why the answer is boundaries, not abstinence.
How do I actually measure this for myself? Pick one recurring task. Do it your old way twice and with AI twice, timing all four, including your review and fix time. Twenty minutes of measurement settles what months of impressions won’t.
Where This Leaves You
The question was never “is AI useful,” which is settled, but “useful for which of my tasks,” which almost nobody measures. Expertise, verification cost, context fit, and error price mark the boundary; your clock, not your impression, finds where you sit relative to it. Delegate what the test supports, keep what it doesn’t, and re-test as the tools change, because they do. More of our boundary-drawing coverage lives under productivity.