Obenan

Briefing · AI crawlers

Since 1 July 2026, Cloudflare sorts AI visitors into three kinds. Which does your website let in?

On 1 July 2026 Cloudflare, which sits in front of websites to deliver and protect them, replaced its single “Block AI bots” switch with separate settings for Search, Agent and Training. On 15 September 2026 it announced a setting to refuse AI training while staying in search. Whether your own site is set that way is a separate check. And no crawl setting makes what AI says about you correct.

Published Updated

Illustrative example, not a real website or Cloudflare account.

Cloudflare, 1 July 2026

One switch became three decisions

In July, every Cloudflare customer, including the Free plan, could set each kind of AI traffic on its own: allow it, block it on all pages, or block it only on pages that show ads. From 15 September, Training has a fourth option next to those: Disallow AI Training.

  • Search

    Crawlers that help you be found

    Most businesses
    Keep it allowed if you want to stay reachable.

    They read your pages so search engines, including AI search, can find and index them.

  • Agent

    Assistants acting for a person, right now

    Worth thinking about
    Blocking it can stop a real customer’s assistant from finishing a task.

    An AI agent visiting on a customer’s behalf, for example to check something or complete a task.

  • Training

    Crawls that collect content to train AI models

    Since 15 September
    Cloudflare says its new Disallow AI Training setting refuses training while you stay in search.

    They take your content to build models, and may send nothing back.

What new websites get from 15 September

Cloudflare’s July changelog said that from 15 September 2026 new domains would by default block Agent and Training on pages with ads while Search stayed allowed.

Its 15 September post describes it differently: a new domain is offered one of two recommended presets when it’s set up, and the owner can change them at any time. For a site that earns money from ads, the preset sets Training to Disallow AI Training rather than blocking it.

Cloudflare, 15 September 2026

Block is stricter than it sounds

Cloudflare says the new setting “lets you easily stay indexed for search while refusing to let that same crawler train on your content”. It applies to Training only.

Disallow AI Training

  • Cloudflare says it refuses AI training while you stay in search
  • Crawlers that do both and that Cloudflare calls Accountable remain allowed for search; that includes operators that only commit to meeting its requirements
  • Doesn’t yet pass the no-training preference to Bing through robots.txt: Cloudflare says Microsoft’s support is targeted for early 2027

Block

  • Cloudflare says Block, and Block on pages with ads, now also stop crawlers that serve both search and training, such as Googlebot, Bingbot and Applebot
  • So blocking Training can cost you search visibility too

If you want to be found but not used for training, Cloudflare points to Disallow AI Training, not Block.

What “Accountable” and Bot Preference Sync mean

Cloudflare gives the Accountable label to crawler operators that meet, or commit to meeting, four requirements it lists. One is “Assurance that opting out of AI training will not affect traditional search results”. In Cloudflare’s words the label recognises “capabilities available today and concrete commitments to deliver them”.

Cloudflare also says “Managed Robots.txt will be deprecated in favor of Bot Preference Sync. Customers who enabled Managed Robots.txt will migrate to the new system.” Its Bot Preference Sync post, published 21 August 2026 and updated 15 September 2026, states the aim as “the preference you set is the preference you publish”, and says it’s available on every plan from Free to Enterprise.

New reporting

Is a bot sending anything back?

Alongside the settings, Cloudflare added two ways to see what AI bots do on a site.

  • Attribution Business Insights

    Crawl-to-referral ratio

    Who gets it
    Bot Management Enterprise customers only, not every Cloudflare customer.

    It compares how much a bot crawls with how many visitors it sends back, for the whole site and per bot operator, over 24 hours, 7 days or 30 days. Cloudflare describes it as visibility, not a new control.

  • BotBase

    Which bot is which

    It shows how Cloudflare classifies each bot as Search, Agent or Training, gives each a detection ID, and lists every tracked bot in Cloudflare Radar’s public directory of bots and agents.

Our read: use the ratio to question bots that take a lot and send little back. It counts visits, not recommendations, sales or revenue.

What these numbers can’t tell you

The ratio measures crawling against referred visitors. It isn’t conversion lift, recommendation quality, citation quality, revenue or return on investment.

Cloudflare’s classifications don’t perfectly capture every AI request or every operator’s intent.

Access is not accuracy

A crawl setting decides who reads. Not what they say.

Getting access right is necessary. But the facts behind your page are where most AI answers about a local business go wrong.

What a crawl setting decides

  • Which AI crawlers may read your site
  • Whether they may use it for search, agents or training
  • Which bots Cloudflare recognises and how it labels them

What only you can make true

  • Your name, address and category, the same everywhere
  • Current opening hours, special hours and availability
  • Booking, deposit, cancellation and service terms
  • What you actually offer, in the words customers use

Nothing here claims these controls improve rankings or recommendations in ChatGPT, Gemini, Perplexity, Google AI Mode or any other AI system.

Where it goes wrong with the settings right

A site allows Search and blocks Training, and an AI still repeats opening hours that changed three weeks ago.

A business blocks Agent traffic to protect its content, then finds that a real customer’s assistant can’t complete a task it could have handled.

A team reads that Managed Robots.txt customers will move to Bot Preference Sync, assumes the training refusal is now in force, and never opens its own settings to check.

What to do

Five things to check on your own site

An announcement tells you what a product offers. Only your own settings tell you what your site does.

  1. Open your own settings

    If your website runs on Cloudflare, read what Search, Agent and Training are actually set to for your domain. Not sure it does? Ask whoever set up your website or domain.

  2. Keep Search open if you want to be found

    To refuse AI training without leaving search, use Disallow AI Training rather than Block, which can also stop Googlebot, Bingbot and Applebot.

  3. Read your robots.txt

    It’s the short public file that tells automated visitors what they may read, for example yourrestaurant.com/robots.txt. Check what it now says, and whether an older Managed Robots.txt setting has moved.

  4. Treat the ratio as a question

    If you have crawl-to-referral reporting, use it to challenge bots that take much and send little back, not as proof of recommendations or sales.

  5. Fix the facts, then report them apart

    Check that hours, address, services and policies are right, and report “can AI read us” and “is what AI says correct” as two separate results.

Questions about this research?

Write to Obenan Research about the sources, or about what they could mean for your business.

Sources

Where this comes from

Built from seven official Cloudflare pages on the 1 July and 15 September 2026 releases, each with the date we last checked it; the 15 September post and the Bot Preference Sync post were read again on 5 October 2026. We report what Cloudflare announced. No Cloudflare account, customer domain, robots.txt file or migration was checked, so nothing here describes how any real site is set. Obenan is not affiliated with, partnered with, integrated with or endorsed by Cloudflare, OpenAI, Microsoft, Google or any source named here, and doesn’t run edge infrastructure, crawl control or bot classification.

  1. 01Cloudflare Published Checked

    Cloudflare: Your site, your rules, new AI traffic options for all customers

    The Search, Agent and Training settings for every customer including Free, and the new-domain default Cloudflare then planned for 15 September 2026.

  2. 02Cloudflare developer docs Published Checked

    Cloudflare: New options to manage AI traffic (changelog)

    The allow, block and block-on-ad-pages options, and the new-domain default the changelog then planned.

  3. 03Cloudflare Published Checked

    Cloudflare: Unmasking the crawls with Attribution Business Insights

    What the dashboard is for, and the crawl-to-referral ratios it shows.

  4. 04Cloudflare developer docs Published Checked

    Cloudflare: Attribution Business Insights (docs)

    Bot Management Enterprise availability, the ratio’s definition, and that it gives visibility rather than control.

  5. 05Cloudflare developer docs Published Checked

    Cloudflare: More visibility into bot traffic with BotBase and Attribution Business Insights (changelog)

    BotBase classifications, per-bot detection IDs and the public bots directory in Cloudflare Radar.

  6. 06Cloudflare Published Checked

    Cloudflare: Have it both ways: stay discoverable in search while disallowing AI training

    Disallow AI Training, what Block now stops, the Accountable requirements, the presets for new domains, Microsoft’s early 2027 target and the move to Bot Preference Sync. Read again on 5 October 2026.

  7. 07Cloudflare Published Checked

    Cloudflare: Say it once: introducing Bot Preference Sync

    How a preference is published into robots.txt, the stated aim, and availability on every plan. Updated by Cloudflare on 15 September 2026; read again on 5 October 2026.