This started as a comment @wrencalloway left on my last post. It was sharp enough that it deserved more than a reply — so here's the full version.
You did the work. The page is server-rendered. The JSON-LD is in the raw response. curl returns the whole article, headline and all. A crawler that fetches your URL gets everything it needs.
Except the crawler never fetches your URL. It asks the CDN, and the CDN says 403.
Your content is perfect and unreachable. This is the layer underneath the one everyone talks about — and it's invisible in every tool you'd normally reach for.
Two different kinds of "no"
robots.txt is a note taped to the door. It says "please don't come in." A polite crawler reads it and turns around. A rude one ignores it and walks straight past. Either way, the note never touches your bytes — it's a request, enforced entirely by the visitor's own manners.
A WAF or CDN rule is the door. It answers the request itself, at the edge, before anything reaches your origin: 403 Forbidden, 429 Too Many Requests, or a JavaScript challenge the bot can't solve. The HTML behind that door could be a masterpiece or a blank page. The bot sees neither. It sees the status code.
People spend weeks perfecting the note and never check whether the door is locked.
This isn't hypothetical — here's the data
Cloudflare Radar publishes what actually happens to AI-crawler traffic across its network. From the AI Insights dashboard (7-day view, as of 18 July 2026):
Of all HTTP responses served to AI bots and crawlers:
- 200 OK — 73.6%
- 403 Forbidden — 5.2%
- 429 Too Many Requests — 1.3%
- 503 — 1.1%
AI-crawler response codes. Source: Cloudflare Radar, AI Insights, 7-day view, 18 Jul 2026.
Read that again. More than one in fifteen requests from AI crawlers is being actively refused at the edge — 403 or 429 — before it ever touches the content. Not deprioritised. Refused.
And most of that isn't a decision. It's a default — a managed WAF ruleset, a "block AI bots" toggle flipped in 2024, a bot-fight score set too high. The same dashboard's robots.txt tracker shows GPTBot, CCBot and ClaudeBot as the most-disallowed crawlers across the top domains — much of it inherited, not authored.
Most-disallowed AI user agents across top domains. Source: Cloudflare Radar, AI Insights, as of 18 Jul 2026.
Why you will never notice on your own
Open your site. It loads. Of course it does — your browser isn't on the blocklist. Residential IP, Chrome User-Agent, you pass every check.
Open robots.txt. Allow: /. Looks perfectly welcoming.
View source. Clean HTML, structured data, the works. (You already fixed that part.)
Every tool you'd reach for confirms you're fine — because every one of them comes from you, not from GPTBot. The block is aimed at a class of visitor you are never going to be.
The only place it shows: server logs
The browser lies, because the browser is you. robots.txt lies, because it's advisory. The access log doesn't lie. It has one row per real request, with the visitor's User-Agent and the exact status you returned. It's the only record of what GPTBot actually got.
Grep for the ones that matter and count their status codes:
grep -E "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|PerplexityBot|Google-Extended" access.log \
| awk '{print $9}' \
| sort | uniq -c
200 means it got the page. 403 means the edge refused it. 429 means you rate-limited it into silence. The failure mode nobody catches looks like this: robots.txt wide open, logs full of 403s. Everything you can see says yes; the thing that decides says no.
I've read this exact story more than once. Months of content, zero citations in ChatGPT or Perplexity, and an audit that finally finds Cloudflare had been handing GPTBot a 403 the entire time.
The trap inside the fix
The obvious repair — allow anything whose User-Agent contains GPTBot — is wrong, because a User-Agent is a string anyone can type. A price scraper labels itself GPTBot and strolls through your generous new rule.
Cloudflare's own transparency tracking makes the point: it flags which operators can even be verified. As of mid-2026, ByteDance shows as unverified — which is exactly why User-Agent alone can't be trusted.
AI bot transparency tracking. Source: Cloudflare Radar, AI Insights, as of 18 Jul 2026.
So real verification isn't the name. It's the origin. Two ways:
1. Verify by published IP ranges. OpenAI, Anthropic and the rest publish their crawler IPs — openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json, and their equivalents. Reverse-DNS the requesting IP, confirm the hostname belongs to the vendor, then forward-resolve it back to the same IP. If either step fails, it's a spoofer wearing the name.
2. Verify by cryptographic signature. This is where it's heading. Web Bot Auth — a Cloudflare-led IETF draft (draft-meunier-web-bot-auth-architecture, built on RFC 9421) — has crawlers sign each request with an Ed25519 key, so identity is proven, not claimed. It's integrated into Cloudflare's Verified Bots program with a dedicated Signed Agents directory, and ChatGPT's agent was in the first signed cohort in 2025. It isn't only Cloudflare, either — AWS WAF added Web Bot Auth support in late 2025, auto-allowing verified agents.
Note the status honestly: Web Bot Auth is an active IETF draft, not a ratified standard. But with Cloudflare, OpenAI, Anthropic and AWS moving in lockstep, it's already the de-facto direction.
The point for you: "block the bad bots" and "let the AI in" are the same problem, and the User-Agent solves neither. Verify by origin or by signature — otherwise you block the crawler you want and admit the one you don't.
What to actually do
Check the logs, not the browser. Pull a week of access logs, filter to the AI User-Agents, look at the status codes. That's the whole audit — an afternoon, not a project.
If you see 403/429, find the rule. It's almost always a managed ruleset, a "block AI bots" setting, or an over-eager bot-fight score — not a line you wrote. Allowlist the crawlers you want by verified IP or Web Bot Auth, never by User-Agent alone.
The rule I took away
Your HTML is the last thing in the request's path, not the first. Before a bot reads a single byte of it, the request has to clear DNS, the CDN, the WAF, the rate limiter, the bot-management score. Any one of them can end the request with a status code you never see — on a page that renders flawlessly for you.
"It works in my browser" was always a weak claim. For crawlers it isn't even the right question. The right question is: what status code did the bot get? And the only honest answer is in the logs.
Related reading:
Top comments (8)
Genuinely made my day to see the comment turned into this. You have done great work here, thank you.
Thanks @wrencalloway — and credit where it's due, the CDN angle was your catch. I'd have left it at "check your HTML" without it.
Working on a follow-up now: how to actually allowlist verified crawlers without opening the door to every scraper wearing a GPTBot label. Web Bot Auth vs IP verification, the tradeoffs. Curious whether you've touched signature-based verification yet, or still doing it by IP range?
Still IP-range in practice, and honestly it's the part I trust least. Reverse-DNS-then-forward-confirm against published ranges works until the vendor rotates or ships a range late, and you're left choosing between a stale allowlist and a 20-minute window where legit crawlers eat a 403.
Signature-based (Web Bot Auth, HTTP Message Signatures) is the direction I'd want to go precisely because it moves the trust from "where did this connection come from" to "can you prove you hold the key" — which survives IP rotation and doesn't need me babysitting CIDR blocks. The catch I keep coming back to: it only helps for crawlers that actually sign, and the scrapers you most want to block are exactly the ones who won't adopt it. So realistically it's additive, not a replacement — signatures to fast-path the honest ones, IP/behavior heuristics still holding the line for everyone else. Curious if your follow-up lands somewhere different, because I'd genuinely like to be wrong about needing both.
You're not wrong about needing both, @wrencalloway — and the follow-up won't argue otherwise. The signature-vs-IP framing is a false choice; the honest model is layered, exactly as you describe. Signatures fast-path the ones who sign. Everyone else falls through to IP, and behavior heuristics catch what's left. Three gates, not one.
Where I'd push slightly: I don't think "scrapers won't adopt signatures" is a weakness of Web Bot Auth — it's the point. The value isn't that it blocks bad actors. It's that it lets you stop trying to identify good ones by circumstantial evidence. Today an honest crawler and a scraper both look like "datacenter IP + plausible UA," and you're forced to adjudicate between them with reverse-DNS and vibes. Once the honest ones sign, the population you're applying heuristics to gets smaller and dirtier — which is a much easier population to be aggressive with. Signatures don't replace the IP layer; they shrink its blast radius.
The rotation problem you named is the real near-term pain, though, and I don't have a clean answer either. A stale allowlist and a 20-minute 403 window are both real costs, and "just poll the JSON more often" is a band-aid on a design that shouldn't have required polling. That's genuinely the strongest argument for signatures — not security, just operational sanity. No CIDR babysitting.
This whole thread is sharper than my draft, honestly. Can I quote you in the follow-up?
"Shrink its blast radius" is a better framing than mine, and it flips the thing I was treating as a limitation. I kept filing "scrapers won't sign" under the con column. You're right that it belongs in the pro column — the residual population getting smaller and dirtier is exactly what lets you drop the polite defaults and start returning 403 on ambiguity instead of agonizing over it. The cost of a false positive goes down once the honest traffic has a clean lane it can prove it belongs in.
The one thing I'd add to the rotation pain: polling isn't just a band-aid on latency, it's a band-aid on trust direction. You're pulling the vendor's truth on your schedule and hoping it's current. A signed request inverts that — the crawler asserts identity at request time, so there's no window where your cached view of reality is stale. That's the operational sanity you're pointing at, and it's why I think the near-term answer is "poll less as more crawlers sign," not "poll better." And Yes feel free to quote me, I appreciate it and i look forward to more discussion
Perfect, thanks. And yeah - the behavioral layer is the honest floor under all of this. Even in a fully-signed world you're still watching what a client does after it's through the door, because a valid signature proves identity, not intent. A legit crawler having a bad day and a compromised key look identical at the signature layer; only behavior tells them apart.
That's the reframing I keep coming back to, and you said it cleaner than I did: signatures move the question from "who are you" to "what are you doing." Verification gets you a name. Rate limits and behavior decide whether the name gets to keep acting like one.
Going in the follow-up as its own point, credited. This thread turned into half the outline.
ngl i always just assume it's a rendering issue, never thought about the cdn blocking the bot entirely. makes so much sense though.
Yeah, "it's a rendering issue" is the right instinct 90% of the time — that's what makes the CDN case so nasty. You fix the rendering, everything you can see says solved, and it's still a 403 one layer down. The tell is always the same: check the logs, not the browser. If robots.txt says allow and the logs say 403, you've found it.