Cloudflare Replaces Its AI Bot Blocker With Three Separate Switches
Agentic AIThe internet’s bot-to-human traffic ratio flipped sometime in 2025 — more automated requests than human ones — and Cloudflare’s response is to give site owners surgical controls rather than a single off switch.
What Happened
Cloudflare is replacing its July 2024 blanket AI-block toggle with three independent controls. The updated system lets site owners configure access separately for:
- Search crawlers — bots that index content for AI-powered search results
- Training scrapers — bots collecting data for model training datasets
- Agent crawlers — bots acting on behalf of end users, such as ChatGPT browsing or autonomous research agents
Starting September 15, 2026, training and agent bots will be blocked by default on pages displaying ads, while search crawlers remain permitted. All controls extend to free-tier users.
Cloudflare also launched BotBase — a searchable database that classifies each known bot by type and behavior, showing whether it links back to content or reproduces it in full.
Why It Matters
The distinction between these three categories matters because they have fundamentally different value propositions for site owners. A search crawler that links users back to the original content is a referral channel. A training scraper that ingests content into a model’s weights with no attribution or revenue is extraction. An agent crawler is somewhere in between — it may drive engagement or it may replace the visit entirely.
Cloudflare CEO Matthew Prince has publicly criticized Google for bundling search and AI crawlers under the same identifier, making it impossible to allow one while blocking the other. The new controls address this directly by forcing each bot type to declare its intent through the classification system.
What This Means for Builders
For teams deploying agentic systems that need to crawl external content — research agents, monitoring systems, competitive intelligence tools — this policy shift has operational consequences. Sites will increasingly block agent-class bots by default, particularly after September 15. Builders should expect: stricter robots.txt enforcement, higher rates of 403 responses from Cloudflare-protected sites, and a growing requirement to negotiate explicit access with content owners rather than assume permissive crawl.
The BotBase taxonomy also gives compliance teams a reference point. If your agent crawler is registered and correctly classified, site owners can allow it specifically. If it isn’t, it defaults to the most restrictive policy.
The So What
The agentic web is real, and access to it is no longer free by default. The Cloudflare change codifies a distinction that matters: indexing, training, and acting are three different things, and content owners are entitled to allow different answers to each. For teams building AI systems that depend on broad web access, registering your crawler’s identity and purpose is shifting from a courtesy to a prerequisite.
Source: The Decoder
Content created with AI assistance and reviewed for accuracy.
Join the conversation
Stack Insiders is our free community for readers who want to go deeper — share resources, ask questions, and connect with others across every vertical we cover.
Join Stack Insiders →