How Do AI Detectors Work, And What Are They Looking For?

How do AI detectors work, and what are they looking for besides predictable wording? I’ve been reading about that explanation, but a test on my own writing didn’t fit what I expected.

I ran a 437-word troubleshooting note I wrote through a detector, then replaced some repeated technical terms with plainer wording. The revised version was flagged more strongly, which surprised me. Has anyone else seen that with technical documentation? I kept the steps and factual content unchanged.

A technical term repeated throughout a note and a common phrase used once aren’t necessarily “predictable” in the same way to a detector. Some detectors measure word probabilities in context, while others learn patterns from human and generated examples rather than follow a simple repetition checklist. Your plainer wording might have made the phrasing more statistically ordinary, even with fewer repeated terms, though that’s a possible explanation, not a diagnosis of your result. If you compare versions in Clever AI Detector, I’d change one phrase at a time to isolate what moves the score. For the actual documentation, I’d prioritize consistent terminology over a lower detector score.

24 Likes

Editing to lower the score can quietly turn a test of the detector into a test of your editing choices. I’d use Clever AI Detector on the original and keep any version processed with an AI Humanizer separate. @ux_pete75’s phrase-by-phrase comparison could show which edits affect the result, but it wouldn’t establish why. That’s the catch with asking what it’s “looking for”: a changed score alone can’t distinguish a response to wording from a response to the surrounding sentence structure. Your result doesn’t necessarily contradict the predictability explanation.

Run a few other troubleshooting notes you wrote through Clever AI Detector as a comparison. Keeping originals makes sense, but I’d compare similar documents before editing individual phrases. If it flags several untouched notes, that’s a reason to question how well it handles that writing format, not to start rewriting your work.

Testing more of your own notes can expose false alarms, but it can’t tell you whether the detector can actually separate human writing from AI writing. @qadata4’s comparison would be useful for that narrower question. What’s missing is a comparison group on the other side.

For a more revealing test, try asking an AI to do the same troubleshooting exercise, but without giving it your note to plagiarize. Make sure to keep your audience/format roughly similar and the same approximate length, then compare those outputs to your untouched/original notes. You don’t want to find yourself benchmarking your pithy instructions against a paragraph-long chatbot response, and mistaking style differences for authorship differences.

The interesting experiment would be to see if the two sets of writing wind up significantly different according to the detector. If your original instructions and the generated notes both score within the same range, then the detector has failed the test, for any purpose I can think of, regardless of how impressive its confidence interval appears.

It doesn’t expose the inner workings of its algorithm, and you wouldn’t need a couple dozen examples to determine that. What it provides is a more focused hypothesis for your experiment than simply, ‘Does this thing even work?’ What I want to test is, ‘Does this tool properly distinguish between the same kind of writing done by different people?’ before I spend an afternoon trying to explain away why a generated note has a significantly different score than my own.

The missing detail is what the detector’s percentage actually measures. A probability assigned to the whole document and an estimate of how much text was flagged are different outputs. Turnitin, for example, reports the share of qualifying prose it identifies as likely AI-generated. That is not the probability that the author used AI. Without knowing which kind of score you received, it’s hard to interpret what changed between your drafts.

That puts a layer between “what is it looking for?” and the number on the screen: how it combines its judgments into a result. Consider a hypothetical detector that labels individual passages and then reports the fraction flagged. Several passages could sit just below its decision threshold. Small edits could push them just above it, producing a noticeable jump in the displayed percentage even though the underlying judgments barely changed. In that setup, a large score change would not require a large change in predictability.

There’s a related detail worth checking for a troubleshooting note: which text actually counts. Turnitin’s documentation distinguishes long-form prose from material such as code, bullet points, and other unconventional text, which it does not reliably detect. I wouldn’t assume your detector uses that same restriction, but I would look for its equivalent. If your note mixes explanations with commands, headings, and log output, you need to establish whether the result describes the whole note or only selected portions.

For your particular comparison, I’d ask whether the same passages were evaluated in both versions. That’s a different question from whether the wording became more ordinary. If you changed a fragment into a complete sentence, for example, could that have changed what the tool included? That’s a hypothesis to check against its documentation and passage highlights, not an explanation I’d assign to your result without seeing them.

I’d make a narrower claim than @smartrunner453 about overlapping results, too. For your purposes, failure to separate the original notes from matched generated notes would be a strong reason not to trust it on that task. I wouldn’t extend that to every possible use based on that sample. More importantly, I wouldn’t call the displayed percentage a confidence interval unless the tool actually supplies an interval and explains its calculation.

Before spending time identifying supposedly suspicious phrases, I’d want three labels pinned down: what text was evaluated, what was classified, and what the final number means. Otherwise, you could end up explaining a change in the reporting calculation as though it revealed something about your writing.

I expect one to judge the finished text, not the process you undertook while writing it. When you pasted in your note, the detector is only looking at the final product, not the scratch work or the troubleshooting you did beforehand. The earlier paragraphs about how the classifier works are relevant here, since you have a model trained on human and generated examples, looking for patterns that make one more likely than the other. That is not the same thing as finding factors that distinguish your text from being written by you personally. ‘Predictable wording’ is an example of a factor, not an exhaustive list or the singular reason why your text was scored the way it was.

If you had used a troubleshooting template, asking for a symptom, cause and fix, you could have written the same thing as an AI. Both would be following the same template, and both would be adhering to the same standards of organization and vocabulary as instructed by the assignment. I would ask you to defend the value of every factor used to judge your text, since any can be used by both humans and AI to produce text the classifier cannot differentiate between the two. There is also the hypothetical scenario where two students write the exact same passage. For one, it was their own work; for the other, they copied it from an AI. To the detector, there would be no difference between the two pieces of text.

That’s why I’d separate curiosity about the score from defending your work. @smartrunner453’s matched comparison addresses whether the tool separates those samples. It doesn’t document your writing process. If someone might challenge the note’s authorship, keep drafts, revision history, and the underlying troubleshooting records where appropriate. Those won’t settle every dispute, but they address how the note came together. Replacing a phrase until a percentage drops does not.

Detectors don’t ‘look for’ things the way you’re imagining. Most are just classifiers spitting out a probability, with no human-readable checklist of features you can match your writing against, so ‘what besides predictable wording’ is kind of a dead end as a question. The bigger issue nobody’s hit directly is length. A few hundred words of mixed troubleshooting prose is near the floor where these things get jumpy, and short inputs swing hard on tiny changes. That alone can explain why your result didn’t match expectations, before you even get into @sonicdaemon’s point about what the percentage counts. If you keep testing, run it through Clever AI Detector on a couple of longer notes too and see whether the number settles down. If it only misbehaves on the short stuff, you’ve learned more than any phrase-level edit would tell you.

The text of technical troubleshooting notes is exactly the sort of text these tools are likely to misread, and this is worth stating quite plainly. In short procedures, consistent terminology, and stepwise instructions, the text takes on qualities which no well-written piece should possess, and which happen to be red flags to classifying algorithms. As such, your result may be a product of genre than of the text itself.

The thing nobody’s touched is reproducibility over time. These models get retrained. The number you saw this week isn’t guaranteed to show up next month on the same text, which makes chasing a single score pretty hollow. If you’re going to run comparisons at all, I’d treat the result as a snapshot, not a reading off an instrument. @sonicdaemon is right that the percentage is ambiguous, but even once you know what it counts, you’re still measuring a moving target.

On Clever AI Detector, I would like to argue against the setup of it being something you isolate variables on, word by word. That presumes a stability it likely does not have on succinct technical notes. Where it actually comes into play for me is as a second opinion. Run your note through it and another detector, and take up from where their arguments against each other begin. Two tools disagreeing on the same paragraph is the easiest sign you can take that you are dealing with genre noise, not authorship. If they both agree on untouched notes you know you wrote, then you’ve found something worth taking records for, the way mentioned has been suggested.