QWE AI Academy The #1 mistake people make with the whole “AI writing on arXiv” story: they read a headline like 17.5% of CS papers are AI-generated, then try to apply that number to a single paper. The method behind those numbers can’t tell you anything about one document – and the researchers who invented it say so plainly. More on that in a moment.
This tutorial walks you through the actual technique used to measure AI writing across arXiv, so you can run a mini version yourself, understand what the headlines really mean, and see the three places the measurement quietly breaks.
What the arXiv studies actually measured
Liang et al., published in Nature Human Behaviour in 2025, analyzed 1,121,912 preprints from January 2020 to September 2024 across arXiv, bioRxiv, and Nature portfolio journals. Their finding: up to 22% of computer science papers showed LLM modification by September 2024, versus up to 9% for Nature portfolio journals.
The method – distributional GPT quantification – works like this. Two reference word-frequency distributions: one from known human-written abstracts (pre-ChatGPT), one from LLM-rewritten versions of those same abstracts. Maximum likelihood estimation then finds what mixture of the two best explains a real 2024 corpus. Code and datasets are public on the official GitHub repo. It’s clever. It’s also strictly population-level – it estimates the fraction α of a large corpus that was LLM-touched. Single abstracts? Out of scope by design.
The hands-on version: measure it yourself in 30 minutes
You need a CSV of arXiv abstracts, a text editor, and about 30 minutes. No research budget required.
Step 1 – Grab the data
Download the Cornell arXiv metadata dump on Kaggle – every arXiv abstract, updated regularly. Filter to cs.* categories and pull two slices: one from 2021 (pre-ChatGPT baseline) and one from 2024.
Step 2 – Count the marker words
Skip the MLE for now. Just count frequencies of known LLM-favored words. Kobak et al. published an annotated list of roughly 900 “excess words” – words whose frequency spiked after late 2022. The strongest signals, per Kobak et al. (arXiv:2406.07016):
word freq_ratio_2024_vs_baseline
delves ~28.0x
underscores ~13.8x
showcasing ~10.7x
intricate high
pivotal highTurns out “look into” alone jumped from 0.056% of tokens in 2022 to 0.793% in 2024 across OpenAlex – a 14x shift in two years, per the Kobak et al. follow-up (arXiv:2511.15872). That’s not a rounding error. That’s a vocabulary earthquake.
Step 3 – Compute a corpus-level excess
For each word w in your marker list: freq_2024(w) - freq_2021(w). Sum the positive deltas. That total is your rough excess vocabulary – a crude but real proxy for LLM influence. Not the full MLE, but it moves in the same direction.
Watch out: Do this by arXiv sub-category, not across all of cs.*. The signal is much stronger in fast-moving areas (cs.CL, cs.AI) than in slower ones (cs.LO, cs.CG). Averaging across the whole field blurs the picture substantially.
And here’s something worth sitting with before we get to the breakdowns: what you’re actually building is a linguistic thermometer. It doesn’t tell you who used AI. It tells you the temperature of the corpus. That distinction matters enormously when someone tries to cite your mini-analysis in a tweet.
Where the measurement breaks
Break #1 – Estimates disagree, wildly
Same field. Overlapping timeframes. A 2025 review (arXiv:2512.01560) put the competing estimates in one table:
| Source | Corpus | Estimate |
|---|---|---|
| Bao 2025 | arXiv preprints, to 2023.08 | 12.6% |
| Liang 2024b | arXiv CS, to 2024.01 | up to 17.5% |
| Geng 2024 | arXiv CS, to 2024.01 | up to 35% |
| Liang 2025 | arXiv CS, to 2024.09 | up to 22% |
Numbers differing by 20+ percentage points. Whichever one gets tweeted becomes “the” statistic for a week. No consensus number exists – a fact that news coverage reliably ignores.
Break #2 – The signal is moving
The Coevolution paper caught something odd: authors now consciously strip out marker words like “look into” and “pivotal.” But they miss smaller shifts. Per the Human-LLM Coevolution paper (arXiv:2502.09606), the words “is” and “are” are quietly decreasing in arXiv abstracts – a subtler signal of LLM influence that’s much harder to scrub without rewriting entire sentences.
Every headline about “AI marker words” is a tutorial for authors on what to remove next. The trackable signal is already shifting from vocabulary to grammar. Next year’s method will need different features entirely.
Break #3 – Off-the-shelf detectors don’t work at scale
Docs say word-frequency works; trained detectors don’t – Geng et al. tested both. Geng et al. (arXiv:2409.13686) found that Fast-DetectGPT, GLTR, and RADAR showed no significant difference between pre- and post-ChatGPT abstracts at the corpus level. The word-frequency method worked; the tools built specifically for this task didn’t. Make of that what you will.
Four pitfalls when interpreting these numbers
- “22% of papers are AI-generated” is not what the paper says. It’s the fraction of text showing LLM modification, not fully AI-authored papers. Copyediting counts.
- Non-native English authors show up more. Prior work by the same Stanford group found detectors carry a bias against non-native writers (as reported in earlier Liang et al. work); marker-word methods likely inherit that bias – non-native authors leaned on ChatGPT for polish earlier and harder.
- You need a pre-ChatGPT baseline. Studying a subfield that didn’t exist before late 2022? No reference distribution. The method silently returns garbage.
- Shorter papers may score higher. According to Liang et al. 2025, higher LLM-modification estimates appear in shorter papers and more crowded research areas – probably because abstract polish matters more when your paper competes for attention.
What the numbers actually tell us
Somewhere between 10% and 25% of recent CS abstracts on arXiv show meaningful LLM influence, depending on method and cutoff. Real signal. Just not the sharp “1 in 5 papers is fake” claim it gets rewritten into online.
The more interesting finding is the direction. Every study, using different methods, finds the same monotonic rise starting in early 2023. That agreement across methods is stronger evidence than any single percentage.
When NOT to use this method
Don’t use word-frequency analysis to accuse an individual author. It’s like using the national average income to estimate one specific person’s salary – the math runs fine, the output is meaningless. The method estimates a fraction across a large corpus. On a single abstract, the number has no defensible interpretation.
Don’t use it on a corpus without a clean pre-2023 baseline. And don’t use it in a field where vocabulary is changing for reasons unrelated to AI – a new subfield might introduce words like “pivotal” naturally because a genuinely important paper coined the term.
For individual-document decisions, the honest answer as of 2026 is that no reliable method exists. Detectors have false-positive rates that make them unsuitable for high-stakes calls. That’s not a gap waiting to be filled – it’s the current state of the art.
Which raises the question the papers don’t answer: at what point does “LLM-assisted polish” become something worth measuring at all? Is a copyedited abstract meaningfully different from a human one? The field is still working that out.
FAQ
Can I use this method to prove a specific paper was written by AI?
No. Single paper, single abstract – the method produces a number with no defensible interpretation. It’s a corpus tool, not a document classifier.
Which arXiv fields show the strongest AI signal right now?
Computer science leads by a wide margin – Liang’s 2025 Nature paper puts it at up to 22% by September 2024, versus roughly 9% for Nature portfolio journals. Inside CS, the sharpest jumps are in cs.CL and cs.AI (where LLMs are also the research subject – go figure). Slow-moving theoretical areas like mathematical logic barely register a shift at all. Crowded, fast-moving subfields are where the signal concentrates, because abstract polish matters most when 50 papers drop on the same topic the same week.
If authors are now avoiding words like “look into,” doesn’t that make all the estimates outdated?
Here’s the misconception: marker-word avoidance hurts the word-frequency approach, but it doesn’t erase the signal. Newer methods now track subtler patterns – the drop in common words like “is” and “are” (documented in the Human-LLM Coevolution paper) is much harder to strip out consciously. You’d have to rewrite entire sentence structures, not just swap vocabulary. The measurement game has become a co-evolution, not a one-shot detection problem. The methods will keep changing. So will the writing.
Next step: pull 1,000 abstracts from cs.CL for January 2022 and January 2024 off the Kaggle arXiv dump, count “look into,” “underscore,” and “showcase” in each, and plot the ratio. You’ll reproduce the core finding in an afternoon – and feel exactly how fragile it is when you try to apply it to a single paper.