Knowledge Engine

How reliable are AI text detectors?

·no read data

The idea

Since chatbots became widely available, a market has grown for tools that estimate whether a piece of writing came from a person or a language model. Many work by measuring perplexityperplexity A measure of how surprised a language model is by a piece of text; predictable wording gives low perplexity.: roughly, how predictable each word is to a language model. Machine text tends to be predictable, so low perplexity is read as a sign of AI.

In 2023 Weixin Liang, James Zou and colleagues at Stanford tested seven widely used detectors, in work published in Patterns. On 88 essays by US eighth-grade students, the detectors did reasonably well. On 91 essays written by non-native English speakers for the TOEFL exam, they flagged an average of 61.22 percent as AI-generated. Eighteen essays were flagged by all seven detectors, and 89 of the 91 by at least one. The authors attribute this to perplexity: writers with a smaller vocabulary and simpler grammar produce predictable text, which is what the detectors look for.

Why it might change a decision

The key arithmetic is about base ratesbase rate How common something is in the group being tested, before any test result is known.. Take a detector, illustrative rather than drawn from any study, that catches 90 percent of AI essays and wrongly flags 2 percent of human ones. Run it on 1,000 essays.

If 100 of them are AI-written, it flags 90 of those plus 2 percent of the 900 human essays, which is 18. So 18 of 108 flags, about one in six, accuse a human.

If only 20 are AI-written, it flags 18 of those plus 2 percent of 980 human essays, about 20. Now roughly half of all flags land on human writers. The detector has not changed; only the mix of writing has.

The testable prediction is that the share of flagged work that is actually human should rise as genuine AI use becomes rarer, and rise further for groups, such as non-native writers, on whom the false positive ratefalse positive rate The share of cases that are truly negative which a test nonetheless wrongly labels as positive. is higher. A 2 percent false positive rate is not a 2 percent chance that a flagged essay is innocent.

How solid is this?

Liang’s study is a single study with small samples: 91 TOEFL essays from one online forum and 88 American essays. The pattern was large and the mechanism plausible, and in a further test, rewording the TOEFL essays with richer vocabulary cut the average false positive rate to 11.77 percent, while simplifying the American essays raised theirs from 5.19 to 56.65 percent. These figures come from the preprintpreprint A research paper shared publicly before, or without, formal peer review by a journal. version of the paper; detectors are updated often, so the exact rates may not hold for current versions.

A second line of work asks whether detection can be made reliable at all. Vinu Sankar Sadasivan, Soheil Feizi and colleagues, in a 2023 preprint, attacked several kinds of detector, including watermarkwatermark A hidden statistical pattern deliberately built into generated text so that it can later be recognised as machine-made.-based, trained-classifierclassifier A model that sorts inputs into categories, such as flagged or not flagged. and zero-shot tools, by passing AI text of about 300 tokenstoken A small chunk of text, often a word or part of a word, that a language model reads or writes as one unit. repeatedly through a paraphrasing model. Detection rates fell substantially, often with only a slight loss of quality. They also showed spoofing: making human text appear watermarked. Their theoretical result ties the best achievable accuracy to how different human and machine text are; as models write more like people, the ceiling falls. This is an argument about limits, not a measurement of any one product.

The strongest counterposition is watermarking. John Kirchenbauer, Tom Goldstein and colleagues, at ICML in 2023, proposed that a model gently favour a pseudo-randomly chosen “green” subset of words as it writes. Anyone with the key can count green words and compute a p-valuep-value The probability of seeing a result at least this extreme if there were really nothing going on.. At their threshold the chance of flagging unwatermarked text is about 3 in 100,000, and in their tests 98.4 percent of watermarked generations were detected. The authors note limits: it only works on text from a model that applies it, and rewording attacks weaken it, though they report that removing it from a long passage requires changing a large share of the words.

The popular claim that a detector can tell you whether a given student cheated outruns the data. Whether watermarking can hold up against determined rewording is still being argued.

Sources

  • Liang, W., Yuksekgonul, M., Mao, Y., Wu, E., & Zou, J. (2023). GPT detectors are biased against non-native English writers. Patterns, 4(7), 100779. doi:10.1016/j.patter.2023.100779. Figures quoted from the preprint: arXiv:2304.02819
  • Sadasivan, V. S., Kumar, A., Balasubramanian, S., Wang, W., & Feizi, S. (2023). Can AI-generated text be reliably detected? arXiv preprint 2303.11156. arxiv.org/abs/2303.11156
  • Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., & Goldstein, T. (2023). A watermark for large language modelslarge language model A computer program trained on very large amounts of text to predict and produce written language.. Proceedings of the 40th International Conference on Machine Learning, PMLR 202, 17061-17084. proceedings.mlr.press/v202/kirchenbauer23a

Check yourself

Answers are checked in your browser. Nothing is recorded or sent anywhere.

  1. In Liang and colleagues' study, what happened when seven detectors were run on 91 human-written TOEFL essays?

  2. A detector catches 90 percent of AI-written essays and wrongly flags 2 percent of human ones. In a class where only 2 percent of essays are AI-written, roughly what share of flagged essays are actually human?

  3. How does the Kirchenbauer watermark work?

  4. Which statement about the evidence is fairest?

Questions without answers

  1. A university is considering using a detector to open misconduct cases. What information about its error rates, and about the students it will be used on, would you want before judging whether a flag means much?
  2. If watermarking only works when the model provider chooses to apply it, what does that imply about detecting text from openly available models?

Back to the archive