Overview
- Cloudflare replaced its single "Block AI Bots" toggle with three separate traffic categories — Search, Agent, and Training — available to every customer, including Free plan.
- Beginning September 15, 2026, Cloudflare is changing how these controls work, including new defaults for newly onboarded domains and new enforcement rules for crawlers that serve multiple purposes.
- Googlebot, Bingbot, and Applebot are classified as both Search and Training crawlers. If Training is set to Block, Cloudflare will also block these crawlers from their Search activity, even if Search itself is set to Allow. This can prevent Google and Bing from crawling the site and put organic search visibility at risk.
- Transistor recommends that clients keep Training set to Allow unless there is a specific business requirement to block AI training and the implications for search crawling have been carefully addressed.
- Enterprise customers with Bot Management get BotBase, a new searchable database showing how any crawler is classified.
- None of this requires a Cloudflare plan upgrade — the core protections are available on Free.
Reference article: https://blog.cloudflare.com/content-independence-day-ai-options/
Change 1: AI Traffic Split Into Three Categories
What it is: Cloudflare now lets every site owner set a separate policy for three kinds of AI-related bot traffic, each configurable as Block (all pages), Block (pages with ads only), or Allow:
- Search — bots that index content to serve it in search results (Googlebot, Bingbot).
- Agent — bots acting in real time on someone's behalf, like a chat assistant fetching a page or a browser-use agent completing a task.
- Training — crawlers pulling content into a model's training data (GPTBot, ClaudeBot, Amazonbot, CCBot).
Why it matters: Previously, "block AI bots" was a single switch. This gives much finer control — a site can now allow being indexed and cited while still keeping training crawlers out.
What clients need to do: Go to Security → Settings → AI bot policies and set each category deliberately rather than leaving it on whatever the old legacy toggle was doing.
For the most common client goal — stay crawlable/indexable by search engines and stay accessible to AI assistants like ChatGPT and Claude — recommend:
- Search: Allow — preserves Googlebot, Bingbot, and other search-indexing crawlers.
- Agent: Allow — preserves real-time access for AI assistants, since a browsing agent (ChatGPT fetching a page, Claude pulling a URL a user shared) falls under the Agent category, not Search.
- Training: Allow — the only setting that's guaranteed not to also block Googlebot/Bingbot, since those crawlers are multi-purpose and get blocked across all their behaviors if Training is blocked (see Change 3).
Change 2: New Default Settings Effective September 15, 2026
What it is: Beginning September 15, Cloudflare is changing how its AI bot controls apply to crawlers that serve multiple purposes.
For new domains added to Cloudflare, the default settings will be:
- Search: Allow
- Agent: Block on pages with ads
- Training: Block on pages with ads
For existing domains, the more important change is how Cloudflare will enforce Training blocks. Starting September 15, crawlers classified for both Search and Training — including Googlebot, Bingbot, and Applebot — will be subject to the most restrictive applicable setting. If Training is blocked, these crawlers will also be blocked from Search crawling, even if Search is set to Allow. This also applies to sites using Cloudflare's legacy "Block AI Bots" setting.
Why it matters: A site owner may reasonably assume that setting Search to Allow will ensure Google and Bing can continue crawling the site. Beginning September 15, that is not necessarily true. If Training is blocked, Cloudflare will also block multi-purpose crawlers such as Googlebot and Bingbot. A setting intended to prevent AI training could therefore interfere with search engine crawling and put organic search visibility at risk.
What clients need to do:
- Review Cloudflare's AI bot policies before September 15.
- Make sure Training is set to Allow. Do not assume that Search: Allow is sufficient to protect Googlebot and Bingbot.
- For our recommended configuration, set Search: Allow, Agent: Allow, and Training: Allow.
- Pay particular attention to sites that previously enabled the legacy "Block AI Bots" setting, because Cloudflare says the September 15 change will also cause that configuration to block mixed-purpose Search/Training crawlers.
Change 3: Multi-Purpose Crawlers Are Blocked Across All Their Behaviors
What it is: Googlebot, Applebot, and Bingbot serve more than one purpose — they can contribute to both Search indexing and AI Training. Cloudflare has confirmed that when a bot has multiple behaviors, it is blocked according to all of them, enforced by whichever rule is most restrictive:
"multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training — either through the new options to manage AI traffic, or through the legacy Block AI bots service."
Why it matters: This is the biggest risk for indexability. A client who blocks the Training category — or who left the old "Block AI Bots" toggle on — can end up blocking Google and Bing outright, not just AI model training. That's the opposite of what most clients want.
What clients need to do:
- If a client wants AI training blocked but search indexing preserved, do not rely on the Training category alone. Use the Verified Bots list (Pro plan and up) to allowlist Googlebot, Bingbot, and Applebot by name, separate from the Training block (see recommendations in Change 1).
- Re-check any client still using the legacy "Block AI Bots" one-click toggle — it carries this same risk.
- Review the site's managed robots.txt to make sure it reflects the same intent, since some AI crawlers respect robots.txt voluntarily even when they'd otherwise be blocked at the edge.
Change 4: BotBase — New Bot Visibility Database (Enterprise)
What it is: A searchable database, new as of this announcement, that lets Enterprise Bot Management customers look up exactly how any given crawler is classified (Search, Agent, Training, or a combination).
Why it matters: Removes guesswork before setting a policy — useful for clients who want to verify a specific bot's classification rather than assume it.
What clients need to do: No action required for most clients. For Enterprise clients making granular bot-policy decisions, point them to BotBase to confirm classification before changing a rule.
