TL;DR: Starting today, September 15, 2026, Cloudflare enforces new default rules for AI crawlers. If your site has ever had the legacy “Block AI Bots” toggle switched on, Cloudflare may treat Googlebot as blocked too, because Google’s crawler does double duty for search indexing and AI training. The fix is a five-minute check in your Cloudflare dashboard, but only if you know where to look.
A recent update to Cloudflare’s bot management may be unintentionally shutting out search crawlers. This episode explains how settings designed to stop AI scrapers can trigger 403 errors for Googlebot and cause sudden ranking drops. Learn which accounts are most at risk and how to fix the problem.
What actually changed today
Cloudflare classifies crawlers into three buckets: Search, Agent, and Training. As of September 15, 2026, Training and Agent bots are blocked by default on ad-supported pages for new domains, new sites added to existing accounts, and any Free tier account that hasn’t touched its settings. Search crawlers are supposed to stay allowed.
Here’s the part that trips people up. Googlebot doesn’t fit neatly into one bucket. It crawls for search indexing, and it feeds Google’s AI training pipeline, so Cloudflare classifies it as a mixed-purpose crawler. Cloudflare’s own guidance is blunt about what that means: block Training, even through the old one-click toggle, and the most restrictive rule wins across every job that bot performs. Googlebot gets caught in a rule aimed at something else entirely.
Why this isn’t a hypothetical
This isn’t a theory. A site owner posted in r/SEO after testing Cloudflare’s AI Crawlers and Scrapers feature directly. They set AI Training to Block and watched Googlebot and Bingbot start pulling HTTP 403 errors on their own sitemap. Flip the training block off, and the 403s disappear. Google’s John Mueller responded publicly, asking for a closer look, which tells you this wasn’t dismissed as user error on Google’s side either.
The uncomfortable reality is that a lot of site owners flipped “Block AI Bots” back when it first shipped, purely to keep their content out of model training, with zero intention of touching how Google indexes their pages. That decision is now doing something nobody signed up for.
Who this actually affects, and who it doesn’t
Not every Cloudflare customer needs to panic today. The new default only auto-applies to new domains, new sites on existing accounts, and Free tier accounts that never changed their settings. If you’re on a paid plan with an established zone and you never touched the AI bot controls, your existing configuration likely carries forward unchanged.
The sites at real risk are the ones where somebody, possibly a past developer, a hosting provider, or an agency that no longer manages the account, flipped Block AI Bots at any point in the last two years. That setting has been sitting there quietly, and today is the day it starts behaving differently.
How to check your own site in five minutes
- Log in to the Cloudflare dashboard and go to Security, then Bots.
- Look for AI Crawl Control, or the older Block AI Bots section if your dashboard hasn’t updated its labels yet.
- Check whether Training is set to Block. If it is, that’s your flag.
- Cloudflare gives you an explicit opt-out for multi-purpose crawlers. Confirm you want Search allowed regardless of your Training setting, and save it.
- Pull up Google Search Console’s URL Inspection tool on two or three important pages and confirm live crawling still works.
If step three comes back clean, you’re done. If it doesn’t, you just found a problem before it cost you three weeks of rankings.
What this means for your business
This is exactly the kind of issue that a routine SEO checklist misses. Nobody’s monthly audit template has a line item for “re-check a bot toggle nobody remembers setting.” It’s not a ranking factor, not a content gap, not a backlink problem. It’s an infrastructure setting quietly overriding your indexing without touching a single page on your site.
That’s the difference between a checklist audit and an actual technical audit. Most audits stop at a checklist. We just spent a full sprint on Austin Code Monkey’s own WordPress theme finding the kind of bug that only shows up when someone actually reads the code: a noindex flag silently misfiring from a substring-matching error. This Cloudflare situation is the same category of problem: invisible until someone with real infrastructure experience goes looking for it.
FAQ
Does Cloudflare block Googlebot automatically for everyone today? No. It only applies where Training is set to Block, either through the new controls or the legacy Block AI Bots toggle, and only fully by default for new domains, new sites, and unmodified Free tier accounts.
How do I know if my site is affected right now? Check Security, then Bots, then AI Crawl Control in your Cloudflare dashboard. If Training shows as Blocked, confirm your multi-purpose crawler opt-out is set to allow Search regardless.
Will this show up in Google Search Console? Yes, eventually. Watch for a spike in “Blocked by robots.txt” style errors or a drop in crawl requests under Settings, then Crawl Stats. Catching it in Cloudflare directly is faster than waiting for Search Console to reflect it.
Does this affect paid Cloudflare plans too? The automatic new default mainly targets new domains and unmodified Free tier accounts. But any plan, paid or free, can still be affected if the Training block was manually enabled at any point.
Is this Cloudflare’s fault or Google’s? Neither, really. It’s a side effect of Googlebot doing two jobs at once and Cloudflare’s rules treating multi-purpose crawlers by their most restrictive applicable setting. Understanding that mechanic is the whole fix.
Need help with your SEO?
The rules of local search are being rewritten in real time, and most businesses don’t find out until a competitor shows up in Gemini’s answer instead of them. Austin Code Monkey specializes in feeding the AI what it needs: business data that’s structured, verified, and descriptive enough for Gemini, ChatGPT, and Google’s AI Overviews to choose you over the competition.
Contact us today to run an AI Visibility Audit and see exactly how your business appears (or doesn’t) across ChatGPT and the other AI search tools your customers are already using.
Austin Code Monkey
Phone: (737) 932-7532
Hours: Monday – Friday, 10:00 AM – 10:00 PM
Website: https://austincodemonkey.com/
Get Started Today:
- Schedule your complimentary SEO audit
- Discover how SEO and AI Search Optimization can transform your business
Don’t let your competitors dominate your business in search queries. Partner with Austin’s leading SEO company and ensure your business appears when and where your customers are searching. Contact Austin Code Monkey today and take the first step toward complete digital dominance in the Austin market.
