How to Block AI Training in Cloudflare Without Blocking Google Search

Cloudflare changed its AI bot controls on September 15, 2026, and if you want to block AI training in Cloudflare without losing Google Search, that change decides which setting you pick. The setting is Disallow AI Training, and the older Block option now stops Googlebot as well. So, if you are looking to set this up on your own, this guide can surely help.

I run Cloudflare in front of my own sites and have set it up for a good number of client sites over the years.

Every site has its own mix of hosting, plugins and old robots.txt rules, so one guide cannot cover every setup. But the Cloudflare part is the same for everyone, and in this one I will try my best to cover the settings, the checks that prove they worked and the traps around Google, Bing and WordPress properly.

If you would rather have someone look at your setup, I am always available for a chat. Let’s get started.

Which Cloudflare Settings to Use

Wanting people to find your website is not the same as agreeing to let your content train a model. These are two separate decisions and your crawler settings should reflect that.

There are broadly two ways people handle this today. One group picks Block on everything that looks like an AI bot and finds out weeks later that Googlebot went with it. The other group leaves everything on Allow, because search traffic pays the bills and nobody wants to risk it. Neither extreme is a good idea. Cloudflare’s new setting exists precisely so that you do not have to choose between these.

You can say both things at once, and Disallow AI Training is how you say them.

Cloudflare now gives you 3 separate controls for AI bots, one each for Search, Training and Agent, and this is the combination I recommend as a starting point to block AI training in Cloudflare while search stays open:

Control	Set it to	What it does
Search	Allow	Googlebot, Bingbot and Applebot keep crawling for search
Training	Disallow AI Training	Publishes a no-training preference for mixed-use crawlers and blocks training-only crawlers at the edge
Agent	Leave as it is	Governs assistants fetching a page for a user, which is a separate decision
Bot Preference Sync	On	Writes your Search, Training and Agent choices into robots.txt

The Agent setting is about assistants such as ChatGPT-User visiting a page because a person asked for it. That is a different question from training, so I would not touch it just because you are restricting training. The training change should go first, then the search check, and the agent policy can wait for another day. Changing 3 policies in one sitting makes troubleshooting harder than it needs to be.

Bot Preference Sync is the piece that connects the dashboard choice to the file crawlers actually read. It is available on every plan, including Free, and it prepends its block to whatever robots.txt you already serve rather than replacing the file.

There is one limit I want you to hold in mind from the start. A robots.txt preference is a request, not a wall. Google’s own documentation says a crawler can ignore it, and the whole setup below is a mix of real access restrictions for the crawlers Cloudflare can block and published preferences for the ones it can only ask. If you treat it that way, it will not disappoint you.

Disallow AI Training vs Block

The distinction matters because of how the big crawlers are built. Googlebot, Bingbot and Applebot are mixed-use crawlers, which means the same crawler fetches your pages for search and for AI training, and Cloudflare cannot block the training half without blocking the search half. Until September 2026 Cloudflare’s Block setting for Training skipped these crawlers. It no longer does.

Cloudflare’s announcement defines the 4 Training options like this:

• Allow: every crawler is allowed unless another setting or a WAF rule blocks it.
• Disallow AI Training: Bot Preference Sync publishes the no-training preference in robots.txt, the mixed-use crawlers stay allowed for search, and Cloudflare blocks every training-only crawler at the edge, including the training crawlers run by Amazon, Anthropic, Meta and OpenAI. This option exists only for Training, not for Search or Agent.
• Block on pages with ads: every crawler, mixed-use ones included, is blocked on pages Cloudflare detects as carrying an ad.
• Block: every crawler is blocked, Googlebot included.

Put against each other, the two settings people confuse look like this:

Crawler	Disallow AI Training	Block
Googlebot	Allowed for search; no-training preference published through the Google-Extended token	Blocked, search included
Bingbot	Allowed for search; no robots.txt preference until Microsoft ships support	Blocked, search included
Applebot	Allowed for search; no-training preference published through the Applebot-Extended token	Blocked, search included
GPTBot, ClaudeBot and the training crawlers from Meta and Amazon	Blocked at the edge	Blocked at the edge
OAI-SearchBot and other search-only crawlers	Governed by the Search setting	Governed by the Search setting

The two settings sit one option apart in the same dropdown, and the difference between these is whether your site stays in Google.

Keeping Search set to Allow does not make a Training Block harmless. Cloudflare says so directly: pick Block for Training and Googlebot is gone, search included. If you want the mixed-use crawlers gone entirely, Block is now the honest way to say it. For everyone else, Disallow AI Training is the setting, and I would go as far as saying it is the only Training option a site that lives on search traffic should touch.

Existing settings migrate on their own. A domain that had the old Block AI Bots switch turned on lands on Search Allow, Training Disallow AI Training and Agent Allow, and a domain that had already set Training to Block or Block on pages with ads is moved by Cloudflare to Disallow AI Training. Cloudflare says you need to do nothing in almost every case. I would still open the page and read what it shows, because “nothing to do” and “nothing to check” are not the same thing.

Googlebot and Google-Extended

Googlebot and Google-Extended do different jobs, and most of the confusion I read about this setting comes from mixing them up. Googlebot is the crawler, and Google-Extended is not a crawler at all. It is a robots.txt token that tells Google what it may do with the pages Googlebot has already fetched. Google’s crawler documentation says Google-Extended has no user agent string of its own, so a firewall rule that looks for that string matches nothing, and the robots.txt rule is the only place the preference can live.

The rule itself is 2 lines:

User-agent: Google-Extended
Disallow: /

Google says this does not affect inclusion or ranking in Search. What it does cover is training for Gemini models and, in Google’s own words, grounding in Gemini Apps and in Grounding with Google Search on Vertex AI. Grounding is the step where a model reads a live page to build its answer and cites it. So the token is not a pure training switch. Disallow it and you refuse both the training and the citations.

That tradeoff deserves a proper thought before you save anything. On my own site the file keeps Google-Extended in the allowed group, with a comment right above it that says why: search, citation and grounding stay open. The cost of that choice is that my pages remain available for Gemini training, because the same token covers both, and Cloudflare’s Disallow AI Training setting would flip it the other way for the whole domain. Neither choice is wrong, but you should know which one you are making.

Apple draws the same line. Applebot crawls, and Applebot-Extended only decides whether the crawled data may train Apple’s foundation models. Apple’s documentation says a page that disallows Applebot-Extended can still show up in its search results.

The principle that falls out of all this is simple: restrict the use you object to, rather than blocking every crawler a company runs.

The Cloudflare Setup

Before you change anything, save the current state. Copy your public robots.txt into a file, take screenshots of the AI bot settings and of any custom firewall rules that mention bots and note the time. That is your rollback point and, later, your before-and-after evidence.

Confirm Traffic Passes Through Cloudflare

Cloudflare can only enforce the edge-level part of this setup if it sits in the request path. Using Cloudflare as your registrar or as your DNS host is not enough on its own.

Open the DNS records for your domain and look at the hostname people actually visit, including www if that is what you use. A proxied record shows the orange cloud and sends web traffic through Cloudflare. A DNS only record, grey cloud, sends visitors straight to your host and skips every rule you are about to set. I have written up the full DNS, SSL and cache side in my Cloudflare for WordPress setup guide if you are starting from zero.

Proxying should stay off for mail, verification and unrelated service records while you are in there. And if your host manages Cloudflare for you, ask the host where these policies are controlled before you touch DNS, because in that arrangement the settings you see may not be the ones in the request path.

Open the AI Bot Policy Controls

In the Cloudflare dashboard, pick the account and the domain, go to Security, then Settings, and find the AI bot policy controls. Cloudflare’s documentation calls the page Configure AI bot policies. You will see the 3 separate settings for Search, Training and Agent.

Apply the table from the top of this guide: Search Allow, Training Disallow AI Training, Agent unchanged. Save, reload the page and confirm the saved values are still what you picked. A settings page that silently reverts is rare, but a reload costs 5 seconds and a wrong assumption can cost you days of lost crawling.

If the option you expect is missing, check Cloudflare’s current documentation or ask support. A control with a similar name should not be assumed to do the same thing. A few of Cloudflare’s own docs pages still carry the pre-September wording, which is one more reason to read the label carefully. On my own Free zones the new dropdowns had not appeared in the dashboard as of September 17, 2026, even though the API already exposes the fields, so a missing control can simply mean the rollout has not reached your account yet.

Turn On Bot Preference Sync

Make sure Bot Preference Sync is on. New domains get it on by default, older ones may need the switch. Cloudflare exposes it as its own switch, on the Signals page of AI Crawl Control and again under Security and then Bots, and its API describes it plainly: when enabled, Cloudflare serves robots.txt content derived from the zone’s Search, Agent and Training preferences.

Two different things happen here and I want you to keep them apart. For the cooperating mixed-use crawlers, the sync publishes a usage restriction while search access stays open, and the crawler has to honor that on its own. For the dedicated training crawlers, Cloudflare refuses the request at the edge, and that does not depend on the crawler’s cooperation at all.

Cloudflare adds its generated block before your existing robots.txt rather than replacing the file, so your sitemap line and your existing Disallow rules survive. What the sync does not do is read your custom firewall rules. It works category-wide, so if you have a special arrangement with one company, a paid deal or a research exception, the sync will not know about it. Cloudflare’s own advice for that case is to turn the sync off and maintain the file by hand, and I agree with it. A published prohibition and a private access exception should not contradict each other by accident.

Review Older Rules

Cloudflare migrates the old settings, but it does not audit the rules you wrote yourself. A crawler marked Allow in the AI bot controls can still be blocked by a custom WAF rule that runs before it, and a skip rule that bypasses WAF custom rules can let a blocked crawler straight through. Cloudflare documents both directions of that conflict.

Look for these in particular:

• rules that target every bot, or every request with a crawler-like user agent string
• rules that block an entire company’s network range
• country blocks that happen to cover a crawler’s data center
• skip rules added during an incident and never removed

When you find a conflict, change the matching rule and nothing else. Disabling the whole firewall to make a crawler test pass is how a site ends up with no bot protection and no memory of why.

Verify the Setup

A saved dashboard setting is the beginning, not the proof. I would check three things separately: the file the public sees, Google’s own view of your pages and what real crawler requests actually got.

Inspect the Public robots.txt

Open the file on your live hostname:

https://example.com/robots.txt

Replace example.com with your domain, and read what the public URL returns, not what a plugin’s editor shows you. Confirm these:

• the file is readable plain text, not an HTML error page
• your sitemap lines and your own Disallow rules are still there below the Cloudflare block
• a group that names Google-Extended disallows the content you meant to protect
• no group aimed at Googlebot, or at every crawler through User-agent: *, disallows the whole site

Cloudflare may stack several user agent lines above a single Disallow, so the layout need not match the 2-line example above. What matters is the group. Google picks the most specific group that matches the crawler and follows only that group, so a broad User-agent: * block with Disallow: / is a problem even when a friendlier group sits above it, and a separate Allow line elsewhere is not a universal override.

Subdomains need the same check, because a robots.txt file applies only to the host, protocol and port it is served on, so blog.example.com has its own file and its own policy. The Signals page of AI Crawl Control lists every hostname’s robots.txt with the status it returned, and on one of my own domains that list showed one subdomain answering 404 and another redirecting, which no check of the main file would have caught.

Run a Live URL Test in Search Console

In Google Search Console, inspect your homepage and choose Test live URL. Then repeat it on an important article, a category page and a product or service page, because a rule that spares the homepage can still catch a path deeper in.

Read the crawl permission, the page fetch result and the indexing permission, and open the tested page to confirm Google received your real content rather than an error or a challenge screen. If you have never set the tool up, my Google Search Console setup guide covers verification and the reports worth reading.

A green result here is useful evidence and nothing more. It proves the inspection tool can reach that URL right now. It does not prove that every ordinary Googlebot request clears every security rule, which is exactly why the next check exists.

Check Real Crawler Requests

Use Cloudflare’s security events, its traffic analytics or your server logs to look at Googlebot’s real requests after the change, and confirm these reach the pages they should.

A request you send yourself with a Googlebot user agent string proves nothing, because anyone can type that string. Google documents 2 ways to verify a request: a reverse DNS lookup on the IP that must resolve to googlebot.com, google.com or googleusercontent.com, followed by a forward lookup back to the same IP, or a match against Google’s published IP range files.

For the AI crawlers, open AI Crawl Control. Its Security page lists each crawler with its operator, category, bytes transferred, allowed and unsuccessful request counts and a Block switch of its own, and its Signals page holds the robots.txt availability and violation reports. Read the unsuccessful numbers with care, though. Cloudflare says an unsuccessful request may come from any rule or response error, not only from the block you just set. On the Free plan the detection is based on user agent strings, so it sees the well-known self-identifying crawlers and not the ones that pretend to be a browser.

Note the exact time you changed the policy and compare the traffic after that point with the same period before it. A random dip in requests is not proof of anything.

Check the File Google Actually Fetched

Search Console’s robots.txt report shows the version Google last fetched, whether the fetch succeeded and previous versions from the last 30 days. It also lets you request a recrawl of the file, which is the right move after a real correction.

Google generally caches robots.txt for up to 24 hours and can hold on to it longer when a refresh fails. A correct file in your browser does not mean Google has read that version yet, so give it a day before you conclude anything from Google’s side.

There is a second reporting trap on the Cloudflare side. The robots.txt violation report on the Signals page of AI Crawl Control compares your current directives with past requests, and Cloudflare says a newly added rule will make older, previously permitted requests appear as violations. A spike right after you save is old requests being judged against a new rule, not a new wave of crawlers ignoring you.

Access logs can tell you who received a page. They cannot tell you what happened to that content inside a training pipeline. These are different kinds of evidence, and the setup you are building gives you the first kind only.

The WordPress Layer

WordPress generates robots.txt on the fly when no physical file exists, and plugins filter that output. Its default includes the crawl rules for the admin area, so replacing the whole file with a copied list of AI bots throws away useful lines you did not know were there.

The bigger problem is that several layers can each think they own the file. A physical robots.txt in the web root answers before WordPress loads. If no file exists, the SEO plugin’s setting usually wins over a theme or plugin filter, unless that filter runs late enough to replace it. Then Cloudflare prepends its block on top of whichever of those won. On one of my own sites a carefully maintained plugin-level AI policy never reached a single crawler for months, because a stale physical file in the docroot answered first. Nothing there was broken. The wrong layer was just winning.

So keep one clear owner for the ordinary crawl directives, let Cloudflare handle the category-level policy and inspect the combined public response after every change. Installing a second robots.txt editor to fix a problem the first one is not causing is not a good idea. If the public file still shows old content after a change, find the cache layer serving that URL, purge only that and fetch again while logged out. I have published my own WordPress robots.txt, with the reason behind each line, if you want a known-good base to compare against.

One more thing, because I see it suggested in forums as a shortcut. The noindex directive should not be used as an AI training workaround. It tells search engines to drop the page from results, which is precisely the thing this whole guide is trying to keep.

A Manual robots.txt Alternative

Cloudflare’s sync is convenient, but nothing about blocking AI training through robots.txt requires Cloudflare. If you maintain the file by hand, these are the starter groups for Google, Apple and OpenAI:

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

The Google and Apple tokens carry the documented meaning above. OpenAI separates GPTBot, which collects content that may be used for training, from OAI-SearchBot, which powers search inside ChatGPT, and its documentation says each can be set independently. So you can disallow GPTBot and still appear in ChatGPT search. Copying a blanket “block everything OpenAI” rule from a template throws that visibility away for no gain.

This is not a complete robots.txt, not an exhaustive list of bots and not what Cloudflare’s generated block looks like. Merge these into your existing file, and if a group already names one of these tokens, edit that group instead of adding a second one, because a crawler obeys exactly one group and the newer one may not be the one it picks.

My own file also carries a Content-Signal line, a newer convention that Cloudflare promotes for stating training and search preferences in one place. I keep it there because it costs nothing, but Google’s robots.txt documentation does not mention it, so it does not replace the Google-Extended group.

Manual directives, like Cloudflare’s generated ones, are preferences. They tell a cooperating operator what you want. They do not stop a scraper that never reads the file.

Bing and the noarchive Tag

As of September 2026, Cloudflare’s robots.txt preference does not reach Bing. Microsoft is building support for a domain-level no-training preference in robots.txt, and Cloudflare says the target is early 2027. Until then, choosing Disallow AI Training tells Bing nothing new.

Microsoft’s documented route today is the noarchive robots meta tag. It goes in the HTML head, not in robots.txt, and the basic form is:

<meta name="robots" content="noarchive">

Bing says content marked NOARCHIVE is left out of its generative foundation model training and out of the Bing Chat answers and links described in its policy, while staying eligible for ordinary search results. That is a wider tradeoff than training alone, because it also removes you from Bing’s AI answers, so decide it deliberately instead of stamping it on every page.

Check for an existing nocache directive while you are in there. When both are present, Microsoft treats the page as NOCACHE, which permits more use than NOARCHIVE on its own, and the stricter tag you just added loses.

Bing’s URL removal tools are the wrong instrument for a training-only policy, because removing pages from Bing search is not the objective here. Bing Webmaster Tools has its own AI performance report, which is the better place to see how Bing’s AI features are using you.

The AI Overviews Control

Blocking training is not the same as opting out of AI Overviews. Google now offers a control for that inside Search Console, under Settings and then Search generative AI, and its help page says the control reached all websites worldwide on August 31, 2026.

The control governs whether your site’s links and content appear in AI Overviews, AI Mode and the generative AI features in Discover. Google says it is not used as a ranking or inclusion signal for the rest of Search, and it does not affect AI training at all. It also comes with a plain cost: exclude your site and you receive no traffic or impressions from those features, while content from other sites still fills them.

For a training-only policy, leave this control alone. Wanting search and AI-search referrals while refusing the documented training uses is a reasonable position, and it is roughly the one I hold, which is why I spend more effort on earning citations in AI search than on hiding from it. The opposite is also a legitimate editorial choice. It is a separate decision with its own consequence, not an extra box you have to tick to finish the Cloudflare setup.

Troubleshooting

When something looks wrong, narrow the diagnosis instead of adding another blocking rule on top.

Problem	Check first
Google cannot fetch an important page	The saved Training value, the matching security event and the robots.txt group that applies to Googlebot
The dashboard looks right but a crawler is blocked	A separate custom rule, a skip rule or a host-level restriction that runs first
A blocked crawler still receives pages	Whether the request comes from that crawler at all, the hostname it hit and rules that skip enforcement
Your plugin’s robots.txt differs from the public file	Which layer generates the final response: physical file, WordPress, plugin or Cloudflare
AI Crawl Control shows no relevant requests	Proxied DNS on that hostname, the hostname selected in the report and the time range

For rollback, restore the saved values for the one change that caused the problem, repeat the same tests and keep the evidence. Turning off unrelated protections to make a single test pass is the habit that costs the most later.

The Limits

This setup cannot stop a scraper that ignores robots.txt. Google’s documentation is open about that, and Cloudflare’s Free plan identifies AI crawlers by their user agent strings, so a crawler that presents itself as a browser walks past the AI controls. Advanced detection sits on the paid Bot Management plans. What you get on Free is a clear published policy plus a real block on the crawlers that identify themselves, and for most blogs and business sites that is the part worth having.

It cannot undo training that already happened. Changing an access rule or a preference is not a deletion request, and nothing in this setup gives you evidence that earlier copies or earlier training runs were reversed.

It cannot tell Bing anything through robots.txt until Microsoft ships its support, so the noarchive tag remains a separate job on the page.

It cannot separate Google’s training from Google’s grounding. The Google-Extended token covers both, and Cloudflare’s setting flips the token for the whole domain, so a site that wants Gemini citations but no Gemini training has no clean way to say that today.

And it cannot show you what happens after the fetch. Your logs end at the moment a page is delivered.

What Quietly Ruins It

Choosing Block because it sounds firmer than Disallow. It reads like the stronger setting, and it is, in the sense that it also stops Googlebot. The stronger word costs you search.

Changing Search, Training and Agent in one sitting. Three changes at once means three suspects when something breaks, and the suspect people blame first is usually the wrong one.

Testing with a curl command that carries a Googlebot user agent string. It feels like a crawler test, but Cloudflare and your server may treat a verified Googlebot request differently from a request that only claims to be one, so the result tells you little.

Fixing a robots.txt mismatch by installing another editor. Each plugin adds one more layer that can win, and the crawler reading the file only ever sees one of them.

Reading the first violation spike as proof that the world is ignoring you. The report compares today’s rules with yesterday’s requests, so the spike is old traffic being graded by a new rule, not a new wave of bad crawlers.

Switching the firewall off to make a test go green. The test passes, the site is left without protection, and three months later nobody remembers why the firewall is off.

FAQs on Blocking AI Training

Final Remarks

If you have followed the steps, your site now says one clear thing to every crawler that listens: index me, cite me if you like, but do not train on me. The idea I want you to keep is that these were always three different questions, and the mistake most sites make is answering all of them with one Block.

My advice at this point comes down to three things. Verify from the outside, not from the dashboard, because the file the public sees and the settings page you saved are different objects and only one of them reaches a crawler. Change one policy at a time and write down the time you changed it, because a before-and-after with a timestamp is the only evidence that survives a “what did you touch?” question later. And review the Bing tag and the AI Overviews control as separate decisions on separate days, because each one trades something away and neither trade is forced on you by the Cloudflare setting.

If you want a second pair of eyes on your Cloudflare and robots.txt layers together, I am available for a chat.

That is all from my side. I hope it helps.