QWE AI Academy The question I keep seeing on Hacker News threads and in Slack DMs this week: “If AI only touches 14% of my day, why does everyone keep saying it doubled their output?” That number comes from the paper everyone’s arguing about – Eight Myths on Software Engineering and GenAI, published in ACM Queue and co-authored by five Microsoft researchers plus Margaret-Anne Storey of the University of Victoria. It landed a few weeks ago and it’s still trending because it does something rare: it puts real data behind claims that most of us have been arguing about with vibes.
This isn’t a summary. Plenty of those exist. This is the version where you actually change something on Monday morning.
What the paper is (and why the eight myths on software engineering and GenAI hit a nerve)
The paper groups its findings into three buckets: how developers actually spend their time, how to measure AI’s impact, and how AI gets adopted in real organizations. The through-line is that the gap between AI’s promise and its measured impact has less to do with the models and more to do with the surrounding system of work.
Translation for a solo dev or small team: your Cursor subscription isn’t the bottleneck. Your workflow around it is.
The stat that broke containment on Twitter was the 14% one. A 2025 Microsoft study found developers spend closer to 14% of their time writing code – earlier research put it at 18% on a good day, 11% on a bad one. A lot of people read this as “AI is useless.” That’s the wrong read. The right read is: if AI only helps with 14% of the day, doubling coding speed still only saves you 7% of the day. The interesting question becomes what the other 86% looks like – and whether AI can touch that.
The three myths that should actually change your workflow this week
Rather than march through all eight, I’m picking the three that translate directly into things you can do. The full list is in the ACM Queue article, which is open access.
1. “A better prompt is always a small tweak away”
Here’s the finding almost nobody is talking about, and it’s the most useful one in the whole paper. As of the paper’s publication in 2025, one cited study found that rewriting a prompt while keeping it semantically equivalent led to different code in 46% of cases and a change in correctness in 28% of cases.
Read that twice. If you rephrase your prompt in a way you think means the same thing, there’s roughly a 1-in-3 chance the correctness of the output flips. This means “cleaning up” your prompt is not free – you’re rolling dice.
2. “Experienced devs will just figure prompting out”
Nope. According to the ACM Queue paper, years of professional software development experience is inversely correlated with confidence in writing effective prompts. Senior engineers are the ones most likely to under-prompt and least likely to notice.
3. “AI ships PRs faster, so we ship faster”
Faster generation just moves the queue. For one internal AI coding agent studied in the paper, only about half of the PRs were ultimately accepted, with 15% abandoned and 15% stuck waiting on a human reviewer – a finding documented in the DX blog’s breakdown of the paper. If your team celebrates AI-generated PR volume without measuring merge rate, you’re measuring the wrong number.
A practical setup: what to actually change on Monday
Three concrete adjustments, in order of effort:
- Version your prompts like code. If your prompt is doing real work, put it in a file, git-track it, and never “just clean it up” without re-testing the output. Given the 46% variance figure, treat prompt edits as behavior changes, not typos.
- Add a personal merge-rate counter. For two weeks, track: PRs opened with AI help / PRs merged unchanged / PRs I had to rewrite. That ratio is your actual ROI. Volume is not.
- Save the diff you rejected. When you throw away AI output, paste it into a scratch file with a one-line note about why. After 20 rejections a pattern emerges – usually one specific type of mistake you can pre-empt in the system prompt.
Here’s a template for step 1 – a prompt file with a locked “do not touch” block and an editable section:
# prompts/refactor.md
## LOCKED (do not edit without re-running eval)
You are refactoring TypeScript. Preserve public API exactly.
Return only a unified diff. No prose.
## EDITABLE
Context: {{paste_current_file}}
Goal: {{one_sentence_goal}}
Keep the LOCKED block frozen. When you tweak the EDITABLE block, note it in git. If output quality drops, you know which line to blame.
Pro tip: If you’re a senior dev, assume your prompts are worse than you think. Ask a junior on your team to review your system prompts the way you’d review their PRs. The paper’s inverse-correlation finding is genuinely uncomfortable, and pretending it doesn’t apply to you is the surest way to get bitten by it.
The trust gap nobody at your standup is naming
As of the paper’s publication (2025), 80% of developers use AI tools but only 29% trust the accuracy of the output – and many report spending more time debugging AI output than writing code themselves. That’s not a niche complaint. That’s the majority of your teammates, quietly.
The practical implication: if your team’s AI “strategy” is “everyone uses Copilot,” you have adoption without trust, which is the worst combination. The METR randomized controlled trial from July 2025 made this concrete in the other direction: when developers were allowed to use AI tools, they took 19% longer to complete issues. They had predicted a 24% speedup going in – and even after experiencing the slowdown, they still believed AI had sped them up by 20%.
You can be measurably slower and feel measurably faster, at the same time. That’s the actual problem.
Advanced: run a two-week micro-experiment on yourself
The paper is aimed at engineering leaders, but the methodology scales down to one person. Here’s a stripped-down version you can run alone:
- Week 1: Pick 10 tasks of similar shape (say, bug fixes in your main repo). For 5 of them, prohibit yourself from using AI. For the other 5, use it freely. Log start/end time honestly.
- Week 2: Do the same, but flip which tasks get AI (as best you can with fresh tickets).
- Compare medians, not averages. One 4-hour outlier destroys the signal in a sample this small.
This isn’t a real RCT – n=10 is nothing statistically. But you’re not trying to publish. You’re trying to catch yourself in the perception-reality gap the METR study documented.
Honest limitations – where this paper won’t help you
The paper is a synthesis of existing research, not new empirical work on every myth. Some claims lean on a single study. The 46% prompt-variance figure, for example, comes from one cited paper – not a meta-analysis. Treat it as directional, not gospel.
It’s also written primarily from a large-enterprise (Microsoft) perspective. If you’re a solo indie hacker shipping a side project, the myths about org-wide adoption and “innovating at startup speed” are less relevant to you. The prompt-brittleness and merge-rate findings, however, apply at any scale.
One number worth watching carefully: the paper cites an 18% increase in implementation time from a 2025 study, while METR’s own published RCT reports 19%. The one-point gap has no public explanation – the paper is citing the same METR study, just rounding differently. Treat these as one data point, not two independent confirmations of a slowdown effect.
As of 2025, the paper also documents what it calls a “competence penalty” – developers, especially women and older engineers, receive harsher evaluations for AI-assisted work even when the output is identical. The paper names it; it doesn’t solve it. Neither can this tutorial. But if you’re doing performance reviews, this is the finding to sit with.
FAQ
Do I need to read the full paper before applying this?
No. It’s about 5,000 words and worth it if you lead a team, but the three findings above cover most of what changes your actual behavior.
My team’s leadership wants a “lines of code shipped with AI” dashboard. What do I tell them?
Show them the 50/15/15 split on that internal AI coding agent – half the PRs accepted, 15% abandoned, 15% stuck in review. Lines of code shipped and lines of code kept are different numbers, and only the second one matters. A decent replacement metric: PR merge rate for AI-assisted vs. human-written PRs. It has its own biases (reviewers may hold AI PRs to a different bar – the paper’s competence-penalty finding actually predicts this), so track both and compare quarterly.
Does this mean AI coding tools aren’t worth it?
That’s not the paper’s argument, and it’s not mine either. The argument is that the gains are real but smaller and more context-dependent than the marketing implies – and that measuring them badly is worse than not measuring them at all. Use the tools. Just don’t build your entire workflow on the assumption they’re making you 10x faster when the data suggests something closer to break-even for experienced devs on complex code.
Next action: Open your last five AI-assisted commits. For each one, note whether the diff you shipped was closer to the AI’s first suggestion or your third revision of it. If it’s mostly the third revision, that’s your prompt-tuning signal – start a prompts file today.