Substack shipped an AI detector this week. Every post, note, and comment over 100 words can now be scanned through Pangram to see how much of it re...
For further actions, you may consider blocking this person and/or reporting abuse
I ran into the same issue: blog posts from 2017 that detectors claimed were AI-written with 97% certainty—based on sentence structure, phrasing, and likely other criteria—even though I had written them myself!
That said, I don't claim never to use AI; I use it primarily for formatting. French is my native language, so I do ask for rephrasing to make my English flow more smoothly—but this applies to articles I have already written, often spending anywhere from 6 to 20 hours on them before submission.
Pascal, the 2017 detail is the part that actually breaks the argument. A stylistic case, sentence rhythm, phrasing, can always be argued around. A publish date from before the technology existed can't. That's not a false positive with room for doubt, it's one with an alibi.
The 97% is doing more damage than a lower number would, too. Confident and wrong reads as authoritative in a way "uncertain and wrong" never does.
Did whichever tool flagged it change its read once you pointed out the year, or did the number just sit there unmoved?
Oh! The score was so absurd that I didn't even bother arguing with it. It mostly made me question how reliable these tools really are. I tried the same test with several other pieces I'd written, and every single one came back as "AI-generated." Then again... maybe I'm an AI after all. 😄
if every piece you've written comes back flagged, that's not really a false positive rate anymore. That's the tool deciding your writing style is what AI writing looks like. It's not making isolated mistakes on your work, it's using you as the reference class.🤣
That's both the funniest and the most disturbing explanation I've heard so far. 😄
If AI detectors have decided that my writing is what AI writing looks like, then we've reached a rather ironic point in history: I spent 35+ years learning to write clearly, and now clarity is apparently suspicious.
35 years is the detail that sticks. Clarity isn't incidental to what got flagged, it's the actual mechanism. A classifier trained on regularities reads consistency as your fingerprint and consistency as AI's fingerprint the same way, because to a statistical model those look identical.
This is the same failure mode from the devto AI-flag debate a few weeks back: the detector measures "does this look statistically consistent with model output," and consistency is exactly what careful writers optimize for. Marco's case is the cleanest proof, a non-native speaker using AI to clean up phrasing gets flagged the same as someone who wrote nothing themselves, because the tool has no way to tell polish apart from authorship.
The fix probably isn't a better classifier. It's moving the question from "does this look like AI" to "can the author defend what they wrote, line by line." Harder to automate, but it's the only thing that actually separates the two cases.
Mike, "can the author defend what they wrote" is the better question and it's also a heavier one to hand someone.
Right now the accused defends nothing until Pangram or Sloan, makes a claim. Move the standard to defend-it-line-by-line and defense becomes the default state every writer owes, whether or not anyone's accused them yet.
That's less an alternative to the detector than a description of what happens once cases like Marco's start piling up: authorship turns into something you have to be ready to prove, not something assumed until challenged.
Is there a version of this that doesn't just relocate the burden from the tool to the writer??
"Write worse, look more human. Write well, get flagged." — This paradox perfectly captures why probabilistic AI detectors are fundamentally broken.
As a Russian systems engineer with twenty years of experience, writing complex technical documentation in English, this hits incredibly close to home. When you are documenting hard engineering, your language must be highly structured, logical, and precise.
I deal with this exact demand for precision in my domain. I build strictly deterministic C++ state sync cores for Medium-Frequency Trading (MFT). I recently finalized the monolithic architecture for my own engine (TolmachЁv SDK v36.0.0). When your baseline requirement is hitting 41.5M TPS with ~24ns physical RTT and absolute zero CPU validation waste, your technical specs cannot afford a single ambiguous word.
I actively use AI to ensure my English explanations of this C++ architecture are as flawless and deterministic as the code itself. If a detector like Pangram or Sloan flags my documentation because it is "too perfectly structured" or lacks human "flaws," it is actively penalizing engineering rigor. They are treating precision as a crime.
You absolutely nailed the conclusion. The only metric that actually matters in technical writing is accountability: "Did you architect this system, and do you own the logic behind these words?" Excellent piece.
Puts me on abstract mode 🤔 forensic. I think a 2017 article claimed AI generated was based on the fact that a human knows a human voice to a good extent similarly an assistant was used to translate to French would be a case of grammar formating not idea. But how do you really know if an egg was actually fried or boiled by looking at the eggshell 🤔
Ekong, the eggshell metaphor is dope and it points at the real limit of it. You can't tell frying from boiling by the shell because the shell was never part of the cooking, it's packaging, not process.
Writing doesn't work that way. The words are the process. Every choice, which clause comes first, what gets cut, how a sentence closes, is baked directly into the same surface a detector is scanning.
That's what makes the grammar-versus-idea line so hard to hold in practice.
Pascal's case wasn't formatting laid over finished thinking, it was rephrasing woven into the sentence itself. A classifier reading the sentence has no way to separate the 2 because there's no shell to look at separately from the egg.
I don't think this gets solved by looking harder at the surface. You'd need to watch it get made.
The bit about disclosure creating a dishonesty incentive is the piece I keep coming back to, because that's a labeled dataset problem, not just a policy one. If the classifier was trained on posts marked AI-assisted, it's calibrated to the population that admits it, and the invisible population (equally assisted, undisclosed) becomes the false-negative tail nobody measures. Curious if Pangram publishes precision at the writers-who-know-their-craft slice, because that's where structural failure lives.
Kartik, that's a different failure mode than the one the piece is actually about, and it's the sharper one. I was pointing at false positives: real human writing that happens to read AI-shaped.
You're pointing at the mirror case: AI-assisted writing skilled enough to read human, undisclosed, sitting quietly in whatever "human" bucket the training set uses. That second tail is close to unlabelable in principle, since ground truth requires someone admitting to something they have no incentive to admit.
The mirror-prompt method Pangram describes doesn't obviously close that gap either. It generates synthetic AI counterparts matched to real human documents, that's a different distribution than a real writer's genuine, undisclosed, AI-assisted draft. One is manufactured contrast. The other is a population that opted itself out of the dataset by staying quiet...
I haven't seen Pangram publish precision sliced by disclosure status or writer skill, just an aggregate false-positive cap. That's exactly the number that would tell you whether the classifier is calibrated against real skilled writers or just against its own synthetic negatives, and it's the one number missing from everything they've published.
As someone who's starting to build with AI every day, I don't think AI use itself is the problem. The question should be whether the author understands what they're publishing and can defend it. AI should help people communicate better, not replace their thinking.
If it flags everything, then the tool isn't really doing its job.
Was it vibe coded 🤣