How do AI text detectors work? Perplexity, burstiness and classifiers explained

The two main approaches

Statistical scoring. The older method asks a language model how surprised it is by each word. GPTZero's own explainer describes perplexity as a measure of how likely an AI model would have chosen the exact same set of words found in the document. Low perplexity looks machine-like. Burstiness is the variation in that score across a document: people tend to mix plain sentences with odd ones, while models are described as more even. The same page carries a note that GPTZero moved to a deep-learning architecture in autumn 2023, so treat these two terms as a description of the early approach, not of every tool today.

Trained classifiers. The second method fine-tunes a language model on pairs of human and AI text. OpenAI described its own classifier this way in its January 2023 announcement: a model fine-tuned on human and AI answers to the same prompts, with the confidence threshold set to keep false positives low. Vendors rarely publish the full recipe of their current products, so what a given tool combines is often not documented.

Approach	What it looks at	Where it tends to fail
Perplexity and burstiness	How predictable each word is, and how much that varies	Plain, formulaic or constrained writing looks "predictable" even when a person wrote it
Trained classifier	Patterns learned from labelled human and AI samples	Text unlike its training data, other languages, short passages, edited or mixed text
Process records (version history, writing replay)	How a document was built over time	Only exists if tracking was on while you wrote; see below

Why they misfire

OpenAI listed the limits of its own classifier plainly. It was very unreliable on short texts (below 1,000 characters), performed worse outside English, and could not judge text that is very predictable: its example was a list of the first 1,000 prime numbers, where the right answer is always the same. It also said human text was sometimes labelled AI confidently. The company withdrew the tool in July 2023 because of its low rate of accuracy.

The best known study of bias is Liang and colleagues at Stanford. They tested several widely used detectors and found they consistently misclassified writing by non-native English speakers as AI-generated, while native samples were identified accurately. Their reading: detectors may penalise writers with constrained linguistic expression. The authors cautioned against using detectors in evaluative or educational settings. Our thread on why detectors flag non-native writers more often goes through what members have seen.

Mixed text is its own problem. Turnitin's product team has said that its sentence-level false positive rate is around 4 percent, that these sentences cluster near real AI writing (54 percent of the time they sit right next to it), and that its document-level false positive rate was under 1 percent for documents with 20 percent or more AI writing. Those are vendor figures from June 2023 on one product, so read them as a statement of method, not as a promise about any particular essay.

How to read a score

1. Treat the number as a lead, not a verdict. Turnitin's own guidance says there is no "right" or "target" score, and to use the information to start a conversation, not to draw a conclusion.
2. Check length. A few short paragraphs give a classifier very little to work with.
3. Check language and genre. Lab reports, legal boilerplate, abstracts and second-language prose are all low in variation by nature.
4. Ask what else exists. Drafts, notes, version history and the author's ability to explain the work are better evidence than any single score.
5. Run the same text through a second tool only to learn how much they disagree. Disagreement is common and tells you about the tools, not the author.

For the broader question of whether any of this is reliable enough for grading, see the thread on whether AI detectors are accurate enough for teachers to rely on, and for what vendors actually publish about their error rates, this comparison of detector accuracy claims.

What a detector cannot tell you

A detector cannot say who wrote a text, which tool was used, or whether a policy was broken. It only reports that the text resembles patterns in its training data or looks statistically smooth. Proof of how a document was written has to come from the writing process itself, which is why process records matter more every year.