Your robots.txt is now a distribution strategy

Only 8.11% of sites fully block GPTBot in their robots.txt. The number that moved over the past year was not blocking — it was explicit allowing, which tripled from 2.51% to 7.69% of files. Meanwhile Cloudflare has started grading bots on whether they actually obey that file. Put those together and robots.txt stops being a politeness note and becomes a distribution policy with enforcement behind it — which is a problem, because almost nobody wrote theirs on purpose.

What changed this week

On October 16, 2026, Cloudflare announced continuous accountability for Verified bots: passing identity verification once is no longer enough to stay verified. The system now watches behaviour over time and scores it.

Two signals do most of the work:

• robots.txt consistency — whether a bot's observed traffic matches the preferences a site published, judged on repeated patterns rather than one-off violations.
• Response to HTTP 429 — when a site answers "Too Many Requests", does the crawler slow down, pause, or keep the same pace?

The output is an explainable trust band rather than a permanent verdict, and Cloudflare says it will publish those scores on Radar and let customers set their own thresholds for Verified bot traffic. Independent coverage frames it the same way: trust becomes continuous and revocable.

There is precedent for the teeth. Cloudflare delisted Perplexity's bot in 2025 after it accessed resources on sites that had opted out and then tried to work around the controls.

The data most people get backwards

The common story is that publishers are walling themselves off from AI. The measured picture says otherwise. From an analysis of 4,242 robots.txt files sampled across Cloudflare's network on 28 September 2026:

Over twelve months, GPTBot's full-disallow share moved barely at all — 7.15% to 8.11%. What actually tripled was explicit Allow directives, from 2.51% to 7.69% of parsed files.

Two conclusions follow, and they point the same way:

1. Over 90% of sites are not blocking anything. Not as a decision — as a default. Their robots.txt is inherited boilerplate that predates any of this.
2. The teams being deliberate are mostly opting in, not out. The movement is toward explicit permission, not refusal.

Crawl purpose is shifting underneath too: in Q3 2026, training-purpose traffic was 47.62% and search plus user-action traffic was 11.92% — up from 6.51% the previous quarter. The share of crawling done to answer a question right now, rather than to train, is growing fast.

Training bots and answering bots are not the same trade

This is the distinction that makes robots.txt a strategy question instead of a compliance checkbox.

A training crawler takes your content into a model. You get no attribution, no traffic, and no event you can measure. The trade is pure: content out, nothing back.

An answering crawler fetches your page because a user asked something right now, and your content may be cited in the response. That is closer to a referral than to extraction — and it is the mechanism behind every "how do we get cited by LLMs" conversation.

Collapsing those two into one Disallow: * is how a team blocks its own distribution while believing it protected an asset. The reverse is just as common: leaving everything open and calling it a strategy.

The honest framing is that these are different trades and deserve different answers. If being cited is how you get found — which, for most B2B companies publishing technical content, it increasingly is — then blocking answering bots is self-harm, and the real work is being worth citing.

What to actually do

1. Read your robots.txt today. Not metaphorically. Most teams have never opened it, and the odds are it was generated by a framework or a plugin years ago.
2. Decide the two cases separately. Training crawlers and answering crawlers are different bargains. Write the file so it says what you actually mean about each.
3. Make Allow explicit where you want to be found. Silence used to be permission. With scoring on both sides, explicit beats implicit — and explicit Allow is the directive that tripled for a reason.
4. Stop rate-limiting the crawlers you want. If you return 429 to an answering bot you would like citations from, you are training it to visit less. Rate limits are now read as preferences, and preferences are now scored.
5. Watch the trust bands when they land on Radar. A crawler with a bad band is one you can block with evidence rather than suspicion.

This is the same shift we keep running into when taking AI-built products to production: the defaults that were harmless when nobody enforced them become the thing that decides your outcome once someone starts measuring. And it is a sibling of an argument we made recently — settled ideas have expiry dates. A robots.txt nobody has read since 2019 is exactly that kind of fossil.

Key takeaways

• Cloudflare now scores Verified bots continuously on robots.txt consistency and HTTP 429 response, producing a trust band it plans to publish on Radar.
• Only 8.11% of sites fully disallow GPTBot; ClaudeBot 6.91%; Google-Extended 5.94%. Over 90% block nothing, mostly by default rather than by decision.
• The real year-over-year change is explicit Allow tripling from 2.51% to 7.69% — publishers opting in deliberately, not walling off.
• Training crawlers and answering crawlers are different trades: one takes content with nothing back, the other can send citations. One Disallow for both blocks your own distribution.
• Returning 429 to a crawler you want citations from teaches it to come back less.

FAQ

Is robots.txt legally binding?

No, and that has not changed. What changed is enforcement by intermediaries: Cloudflare now measures whether Verified bots honour it over time and can revoke verification — as it did with Perplexity's bot in 2025 — which makes the file consequential without making it a contract.

Should we block AI crawlers?

Separate the cases. Blocking training crawlers is a defensible trade if your content is the product. Blocking answering crawlers removes you from the surface where people increasingly ask questions. Most teams should be deliberate about the first and permissive about the second.

How many sites actually block AI crawlers?

Very few. As of 28 September 2026, across 4,242 parsed robots.txt files: 8.11% fully disallow GPTBot, 6.91% ClaudeBot, 5.94% Google-Extended.

What is an "explainable trust band"?

Cloudflare's output for a Verified bot: a range showing how consistently it respects site preferences over time, rather than a one-off pass/fail. The stated plan is to publish these on Radar and let customers set thresholds.

What is the smallest useful action?

Open your robots.txt and confirm it says what you intend about AI crawlers. For most teams that is a ten-minute task that has never been done.

Sources: Cloudflare — Continuous accountability for Verified bots · Runtimewire · robots.txt analysis across Cloudflare's network