Detect LLM Text with Classical ML: A Hands-On Guide

A blog post making the rounds on Hacker News right now argues something a lot of people don’t want to hear: you can detect text from modern LLMs – GPT-class, Gemini, DeepSeek, Kimi, all of them – using techniques from a 2005 NLP textbook. No transformers. No GPU. Just scikit-learn and TF-IDF. The author reports around 85% accuracy per sentence, which compounds to near-certainty across a full document. The comments section is loudly split between “impossible, LLMs write like humans now” and “actually, base models do, but instruction-tuned ones absolutely don’t.”

If you’ve been assuming this problem requires a fine-tuned RoBERTa or a paid API, this guide is for you. We’re going to build a working detector for LLM-generated text using classical ML – the same recipe from that viral post – and explain why it works, when it breaks, and what to do about it.

Why classical ML still works for detecting LLM-generated text

Instruction-tuned LLMs aren’t trained to mimic any single human – they’re trained to produce the average of many humans, filtered through RLHF preferences. That averaging leaves statistical fingerprints. Certain function words, certain punctuation rhythms, certain transition phrases appear at frequencies no individual writer would ever hit. TF-IDF is basically a word-frequency histogram with weighting. Feed it enough labeled examples and a linear classifier finds those fingerprints in the bag of words.

How well? A 2026 arXiv paper by Alikhanov measured it: TF-IDF + logistic regression hit 82.87% accuracy on a combined HC3 + DAIGT v2 benchmark. DistilBERT reached 88.11%. That ~5-point gap comes with a steep cost – DistilBERT trained in 159 minutes and ran inference in 64.67 seconds across 13,311 samples. Logistic regression: 3.4 minutes to train, 0.01 seconds to infer. Roughly 50x slower at inference for a 5-point gain.

Then there’s PAN 2024 – an academic bake-off for AI text detection. Third place (Lorenz et al., LOG-AID paper): TF-IDF term counts, linear SVM. Mean score: 0.886. Above most BERT and DeBERTa entries. Classical ML isn’t obsolete for this task.

The dataset: HC3

You need labeled human vs. LLM text. The default choice is HC3. The Human ChatGPT Comparison Corpus is free on Hugging Face – around 37k human answers and 37k ChatGPT answers to the same questions, drawn from Reddit ELI5, StackExchange, finance, medicine, and Wikipedia CSAI domains. Same questions, different authors. Clean controlled comparison.

One thing the dataset card doesn’t flag upfront: HC3’s ChatGPT half was generated by GPT-3.5. A detector trained purely on HC3 learns 2023-era ChatGPT style. Whether it generalizes to newer Claude, Gemini, or DeepSeek V3.2 outputs is a real open question – and exactly why the viral blog post’s author augmented with 7 different modern models (gemini-3-pro, qwen-coder-plus, glm-5, glm-4.7, kimi-k2.5, doubao-seed-code, deepseek-v3.2, as of early 2026).

Building the detector step by step

Install scikit-learn, pandas, and the Hugging Face datasets library first. Then:

from datasets import load_dataset
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

# 1. Load HC3
ds = load_dataset("Hello-SimpleAI/HC3", "all")["train"]

texts, labels = [], []
for row in ds:
 for h in row["human_answers"]:
 texts.append(h); labels.append(0) # 0 = human
 for c in row["chatgpt_answers"]:
 texts.append(c); labels.append(1) # 1 = LLM

X_train, X_test, y_train, y_test = train_test_split(
 texts, labels, test_size=0.2, random_state=42, stratify=labels
)

# 2. TF-IDF with unigrams + bigrams
vec = TfidfVectorizer(ngram_range=(1, 2), max_features=25000, stop_words="english")
X_train_v = vec.fit_transform(X_train)
X_test_v = vec.transform(X_test)

# 3. Logistic regression
clf = LogisticRegression(C=1.0, max_iter=1000)
clf.fit(X_train_v, y_train)

print(classification_report(y_test, clf.predict(X_test_v)))

Those hyperparameter choices – ngram_range=(1,2), max_features=25000, C=1.0 – come from the Alikhanov paper’s grid search over vocab sizes {15000, 25000, 35000} and C ∈ {0.1, 1, 10}. Winning defaults. Expect accuracy in the low-to-mid 80s on HC3’s test split.

Want to see why the model classifies the way it does? Print the top positive and negative coefficients from clf.coef_ against vec.get_feature_names_out(). Phrases like “it is important to”, “To sum up”, and “as an AI” spike toward the LLM class – hard. That interpretability is the one thing classical ML has over BERT: you can literally read what the model learned.

The sentence-aggregation trick nobody talks about

Most tutorials treat this as a per-document problem. Per-sentence is more powerful. At 85% per-sentence accuracy, a 20-sentence essay gives you 20 near-independent samples. Average the LLM-probability across sentences and noise cancels. This is exactly what the viral post exploited to move from “okay classifier” to “confident detector” – no model change, just aggregation.

Split each document into sentences (NLTK’s sent_tokenize or a regex on . ! ?), score each one, then take the mean probability. Documents where the mean exceeds a threshold – say 0.7 – get flagged. The threshold is a policy call: lower for recall-first use cases (spam moderation), higher for precision-first ones. Academic integrity accusations, where false positives ruin careers, fall firmly in the second group.

Watch out: before trusting your accuracy number, hold out an entirely different topic as your test set. The viral post’s author saw a naive classifier score 99.45% – because it had memorized topic vocabulary, not writing style. A topic-based split (train on finance, test on medicine) reveals this instantly.

Common pitfalls

Pitfall #1 – topic leakage. That 99.45% story is the clearest example. HC3’s human and ChatGPT answers cover the same questions, so a random split lets you train and test on the same topics. The model memorizes domain vocabulary, not authorship style. Fix: group your split by question ID. The accuracy drop when you do this is the most instructive experiment in this entire space.

Pitfall #2 – domain shift. The catch here is severe. Turns out the HC3 Plus paper (arXiv:2309.02731) tested existing detectors on translation output and news summaries. On WMT translation data, precision and recall dropped for both classes. On CNN/DailyMail summaries, the model just… predicted “human” for almost everything – high human recall, near-zero ChatGPT recall. Translation wrecks lexical fingerprints. A detector trained on Q&A text won’t survive it.

Pitfall #3 – paraphrasing attacks. Feed LLM output through a second LLM with “rewrite this casually” and the fingerprint smears. Classical ML degrades harder than transformers here because you’ve directly attacked the surface features TF-IDF depends on. No clean fix – only ensembling with a statistics-based detector (perplexity, entropy) partly compensates.

Pitfall #4 – short text. Below ~50 tokens the TF-IDF vector is too sparse to be reliable. Tweets, chat messages, single-sentence comments – don’t use classical ML here. Either fine-tune a transformer or don’t detect at all.

There’s a harder question underneath all of this. If cheap detection keeps working, is it because instruction-tuned LLMs are still genuinely statistically distinguishable – or because we just haven’t trained the next generation yet? The HN thread has strong opinions and no consensus. Worth keeping in mind every time you see a “95% accuracy” headline.

How this compares to the alternatives

Three families of detection exist (per recent survey literature, as of 2026): watermarking, ML-based classifiers, and statistics-based methods like perplexity and log-likelihood.

Honest read: if you own the labels and the deployment budget is small, classical ML gives you most of the utility for a fraction of the cost. Fine-tune a transformer if you need production-grade robustness and have a GPU to spare. And if you’re a policy team hoping to catch cheaters at scale – accept that neither approach is trustworthy enough to accuse individuals. Both have false-positive rates that cause real harm when deployed as verdicts rather than signals.

FAQ

Can I use this in production to flag student essays?

Not as a verdict. The ~85% sentence accuracy sounds high until you remember that non-native English writers, heavily edited human text, and technical writing all trigger false positives at higher rates than average – and those populations overlap badly with the students you’d most likely flag. Use it as a signal that goes to a human reviewer. Never as the final decision.

Do I need a GPU to train this?

No. That’s the whole point. The Alikhanov benchmark clocked logistic regression at 3.4 minutes training, 0.01 seconds inference on 13,311 samples.

Why does an obviously “dumb” model beat some fancy transformers at PAN 2024?

A common misconception: that more model complexity always wins. LLM-generated text has strong lexical fingerprints – specific bigrams, specific formality patterns – that a linear model over TF-IDF captures directly. Transformers capture those signals plus deeper structure, but they also overfit to their training distribution more aggressively. Under distribution shift, the simpler model sometimes generalizes better. The PAN 2024 result (mean 0.886 for TF-IDF + linear SVM, placing 3rd above BERT/DeBERTa entries) isn’t a fluke – it’s a feature of how narrow and consistent LLM writing style actually is.

Next step: pull HC3 from Hugging Face right now, paste the 30-line pipeline above into a notebook, and run it. Then re-run with a group-based split on question ID and compare the two accuracy numbers. The gap between them is exactly how much of your score was topic memorization versus real signal – and it’s the single most instructive experiment you can do in this space.