Briefing · AI crawlers
Since 1 July 2026, Cloudflare sorts AI visitors into three kinds. Which does your website let in?
On 1 July 2026 Cloudflare, which sits in front of websites to deliver and protect them, replaced its single “Block AI bots” switch with separate settings for Search, Agent and Training. On 15 September 2026 it announced a setting to refuse AI training while staying in search. Whether your own site is set that way is a separate check. And no crawl setting makes what AI says about you correct.
Published Updated
Cloudflare, 1 July 2026
One switch became three decisions
In July, every Cloudflare customer, including the Free plan, could set each kind of AI traffic on its own: allow it, block it on all pages, or block it only on pages that show ads. From 15 September, Training has a fourth option next to those: Disallow AI Training.
Search
Crawlers that help you be found
- Most businesses
- Keep it allowed if you want to stay reachable.
They read your pages so search engines, including AI search, can find and index them.
Agent
Assistants acting for a person, right now
- Worth thinking about
- Blocking it can stop a real customer’s assistant from finishing a task.
An AI agent visiting on a customer’s behalf, for example to check something or complete a task.
Training
Crawls that collect content to train AI models
- Since 15 September
- Cloudflare says its new Disallow AI Training setting refuses training while you stay in search.
They take your content to build models, and may send nothing back.
What new websites get from 15 September
Cloudflare’s July changelog said that from 15 September 2026 new domains would by default block Agent and Training on pages with ads while Search stayed allowed.
Its 15 September post describes it differently: a new domain is offered one of two recommended presets when it’s set up, and the owner can change them at any time. For a site that earns money from ads, the preset sets Training to Disallow AI Training rather than blocking it.
Cloudflare, 15 September 2026
Block is stricter than it sounds
Cloudflare says the new setting “lets you easily stay indexed for search while refusing to let that same crawler train on your content”. It applies to Training only.
Disallow AI Training
- Cloudflare says it refuses AI training while you stay in search
- Crawlers that do both and that Cloudflare calls Accountable remain allowed for search; that includes operators that only commit to meeting its requirements
- Doesn’t yet pass the no-training preference to Bing through robots.txt: Cloudflare says Microsoft’s support is targeted for early 2027
Block
- Cloudflare says Block, and Block on pages with ads, now also stop crawlers that serve both search and training, such as Googlebot, Bingbot and Applebot
- So blocking Training can cost you search visibility too
If you want to be found but not used for training, Cloudflare points to Disallow AI Training, not Block.
What “Accountable” and Bot Preference Sync mean
Cloudflare gives the Accountable label to crawler operators that meet, or commit to meeting, four requirements it lists. One is “Assurance that opting out of AI training will not affect traditional search results”. In Cloudflare’s words the label recognises “capabilities available today and concrete commitments to deliver them”.
Cloudflare also says “Managed Robots.txt will be deprecated in favor of Bot Preference Sync. Customers who enabled Managed Robots.txt will migrate to the new system.” Its Bot Preference Sync post, published 21 August 2026 and updated 15 September 2026, states the aim as “the preference you set is the preference you publish”, and says it’s available on every plan from Free to Enterprise.
New reporting
Is a bot sending anything back?
Alongside the settings, Cloudflare added two ways to see what AI bots do on a site.
Attribution Business Insights
Crawl-to-referral ratio
- Who gets it
- Bot Management Enterprise customers only, not every Cloudflare customer.
It compares how much a bot crawls with how many visitors it sends back, for the whole site and per bot operator, over 24 hours, 7 days or 30 days. Cloudflare describes it as visibility, not a new control.
BotBase
Which bot is which
It shows how Cloudflare classifies each bot as Search, Agent or Training, gives each a detection ID, and lists every tracked bot in Cloudflare Radar’s public directory of bots and agents.
Our read: use the ratio to question bots that take a lot and send little back. It counts visits, not recommendations, sales or revenue.
What these numbers can’t tell you
The ratio measures crawling against referred visitors. It isn’t conversion lift, recommendation quality, citation quality, revenue or return on investment.
Cloudflare’s classifications don’t perfectly capture every AI request or every operator’s intent.
Access is not accuracy
A crawl setting decides who reads. Not what they say.
Getting access right is necessary. But the facts behind your page are where most AI answers about a local business go wrong.
What a crawl setting decides
- Which AI crawlers may read your site
- Whether they may use it for search, agents or training
- Which bots Cloudflare recognises and how it labels them
What only you can make true
- Your name, address and category, the same everywhere
- Current opening hours, special hours and availability
- Booking, deposit, cancellation and service terms
- What you actually offer, in the words customers use
Nothing here claims these controls improve rankings or recommendations in ChatGPT, Gemini, Perplexity, Google AI Mode or any other AI system.
Where it goes wrong with the settings right
A site allows Search and blocks Training, and an AI still repeats opening hours that changed three weeks ago.
A business blocks Agent traffic to protect its content, then finds that a real customer’s assistant can’t complete a task it could have handled.
A team reads that Managed Robots.txt customers will move to Bot Preference Sync, assumes the training refusal is now in force, and never opens its own settings to check.
What to do
Five things to check on your own site
An announcement tells you what a product offers. Only your own settings tell you what your site does.
Open your own settings
If your website runs on Cloudflare, read what Search, Agent and Training are actually set to for your domain. Not sure it does? Ask whoever set up your website or domain.
Keep Search open if you want to be found
To refuse AI training without leaving search, use Disallow AI Training rather than Block, which can also stop Googlebot, Bingbot and Applebot.
Read your robots.txt
It’s the short public file that tells automated visitors what they may read, for example yourrestaurant.com/robots.txt. Check what it now says, and whether an older Managed Robots.txt setting has moved.
Treat the ratio as a question
If you have crawl-to-referral reporting, use it to challenge bots that take much and send little back, not as proof of recommendations or sales.
Fix the facts, then report them apart
Check that hours, address, services and policies are right, and report “can AI read us” and “is what AI says correct” as two separate results.
Keep reading
Reading your site is the first step
Whether AI can reach you and whether it describes you correctly are two different questions.
- Your website can now be scored for AI agentsA readiness score says whether agents can read your site, not whether what it says is true.Read the Signal
- You can now measure how AI sees you. The number won’t fix the answerWhy a dashboard that shows what AI says about you is not the same as correcting the facts behind it.Read the Signal
- AI visibility, explainedWhy being found, read and recommended by AI rests on the facts a business publishes about itself.Read more
Questions about this research?
Write to Obenan Research about the sources, or about what they could mean for your business.
Sources
Where this comes from
Built from seven official Cloudflare pages on the 1 July and 15 September 2026 releases, each with the date we last checked it; the 15 September post and the Bot Preference Sync post were read again on 5 October 2026. We report what Cloudflare announced. No Cloudflare account, customer domain, robots.txt file or migration was checked, so nothing here describes how any real site is set. Obenan is not affiliated with, partnered with, integrated with or endorsed by Cloudflare, OpenAI, Microsoft, Google or any source named here, and doesn’t run edge infrastructure, crawl control or bot classification.
01Cloudflare Published Checked
Cloudflare: Your site, your rules, new AI traffic options for all customersThe Search, Agent and Training settings for every customer including Free, and the new-domain default Cloudflare then planned for 15 September 2026.
02Cloudflare developer docs Published Checked
Cloudflare: New options to manage AI traffic (changelog)The allow, block and block-on-ad-pages options, and the new-domain default the changelog then planned.
03Cloudflare Published Checked
Cloudflare: Unmasking the crawls with Attribution Business InsightsWhat the dashboard is for, and the crawl-to-referral ratios it shows.
04Cloudflare developer docs Published Checked
Cloudflare: Attribution Business Insights (docs)Bot Management Enterprise availability, the ratio’s definition, and that it gives visibility rather than control.
05Cloudflare developer docs Published Checked
Cloudflare: More visibility into bot traffic with BotBase and Attribution Business Insights (changelog)BotBase classifications, per-bot detection IDs and the public bots directory in Cloudflare Radar.
06Cloudflare Published Checked
Cloudflare: Have it both ways: stay discoverable in search while disallowing AI trainingDisallow AI Training, what Block now stops, the Accountable requirements, the presets for new domains, Microsoft’s early 2027 target and the move to Bot Preference Sync. Read again on 5 October 2026.
07Cloudflare Published Checked
Cloudflare: Say it once: introducing Bot Preference SyncHow a preference is published into robots.txt, the stated aim, and availability on every plan. Updated by Cloudflare on 15 September 2026; read again on 5 October 2026.