Skip to content
DigiCruncher

Comparing what AI detector vendors actually publish about accuracy

Asked by Ilya V. · Reviews and tool comparisons5 repliesupdated

Short answer

This is exactly the gap I run into as a teacher. The vendor number is a number about their test set, not about my students' essays. Even a low false positive rate means some honest student gets flagged when you scan a whole class, and the student in front of me does not care about the average.

From the top reply by Dana F. ·
CiteDigiCruncher, "Comparing what AI detector vendors actually publish about accuracy", https://digicruncher.com/topic/comparing-what-ai-detector-vendors-actually-publish-about-accuracy/, accessed 2026-11-11.
  1. Member ·

    The short answer after reading the vendor pages side by side: they do not publish comparable numbers, so you cannot rank detectors from vendor pages alone. Each one reports a different quantity, on a different dataset, at a different threshold. I read three vendors' own documentation this week and want to lay out what each actually says, so we can stop arguing from screenshots.

    What I looked at: Turnitin's detection FAQ, GPTZero's FAQ and the Originality.ai AI checker page.

    What stands out:

    • Turnitin states a target for false positives on whole documents and ties it to documents above a level of detected AI writing. It also hides low scores entirely and says the percentage should not be the sole basis for action.
    • GPTZero gives confidence categories, and its stated figures apply to the "high" category, not to every result. It says results should not be used to punish students, and that accuracy rises with text length.
    • Originality.ai points readers to third party studies and its own benchmark rather than stating a single figure on the page, and notes that false positive rates vary by model and brand.

    So one vendor gives a document level rate, one gives per category rates, and one links out to studies. A "1 percent" and a "99 percent" from two different pages are not the same claim. What I could not find in any of them is a published figure for non-native English writers, short texts or edited drafts, which are the cases people here actually care about.

    What I would like from this thread: have you found a vendor statement that I missed, and do you know of an independent test that reports false positive and false negative rates separately?

    0
    ReplyLink

Plain text or simple Markdown. Links are nofollow. Your email is never shown.

  1. Dana F.Member ·

    This is exactly the gap I run into as a teacher. The vendor number is a number about their test set, not about my students' essays. Even a low false positive rate means some honest student gets flagged when you scan a whole class, and the student in front of me does not care about the average. So I treat every flag as a reason for a conversation and a look at drafts, never as a verdict. The longer discussion of this is in the accuracy for teachers thread.

  2. Jae-won P.Member ·

    For me the missing figure is the one for second language writers. I have been flagged before on work I wrote myself, and I have not seen a vendor page that gives a rate for writers like me. Turnitin's FAQ says it considered second language learners when building its training data, which is a statement about process, not a result I can check. I would like to see a number with the test described. Until then I keep version history for every draft, because that is the evidence I can control.

  3. Colin R.Member ·

    Quick reality check from content ops: we sometimes run client drafts through two or three detectors, and they disagree often enough that I stopped treating any single score as meaningful. A heavily edited human draft and a lightly edited machine draft can land in the same band. I am not going to quote percentages from our tests because the sample is small and informal. My takeaway is that the disagreement between tools is itself a finding about how much weight one score can bear.

  4. Hollis B.Member ·

    Ilya, one nitpick on method. When a vendor says "accuracy", ask whether it is accuracy on a balanced test set, because on a set that is mostly human text a detector that says "human" every time looks excellent. The honest pair is false positive rate and false negative rate, reported separately and with the dataset described. A single accuracy figure hides the trade-off that matters most for anyone who can be accused.

  5. Nathan ColeStaffEditor, DigiCruncher ·

    Good write-up, and the three summaries match what I see on the pages. A few specifics worth keeping next to each source.

    Turnitin says it aims to keep false positives under 1 percent for documents with over 20 percent AI writing, validated on more than 700,000 pre-ChatGPT academic papers, and accepts that this means it may miss some AI text. It does not show scores in the 1 to 19 percent range. It also says its model is not explicitly programmed with signals such as burstiness or perplexity (Turnitin FAQ).

    GPTZero says that when its confidence category is "high", 99.1 percent of human articles were classified as human and 98.4 percent of AI articles as AI, that most of its training data is English prose by adults, and that it can flag highly procedural text (GPTZero FAQ).

    Originality.ai says an AI score should not be used alone for academic integrity discipline and that some AI-assisted editing features can cause false positives (Originality.ai).

    All three are vendor statements, not independent tests, and the figures are not comparable. Hollis's request for separate false positive and negative rates on a described dataset is the right standard.