If you are asking, “how do you check which AI bots Cloudflare is blocking?” you are already ahead of many business owners. Plenty of sites turn on a bot-blocking setting, assume it is protecting them, and never check whether it is also keeping their business out of useful AI discovery experiences. The answer is in your Cloudflare zone dashboard, but it takes more than one screen to get the full picture.
Start with the site, or zone, you want to review. Look at your AI crawler controls, then verify what actually happened in Security Events. Finally, inspect any custom firewall rules, bot settings, rate limits, and robots.txt directives that may be producing the block. A setting can look harmless while a separate rule is doing the real damage.
How Do You Check Which AI Bots Cloudflare Is Blocking?
Cloudflare labels and feature availability can vary by plan and dashboard version, but the practical workflow stays the same. First, open the Cloudflare dashboard and select your domain. Search the navigation for AI Crawl Control, Bots, or the AI crawler controls within your bot-management settings.
This area is the fastest way to see whether Cloudflare is configured to block known AI crawlers at the account or zone level. Depending on your setup, you may see controls for categories of AI bots, individual verified crawlers, or a policy that blocks AI scrapers by default. Review the current status carefully. “Block” means Cloudflare is actively denying requests that match the policy, while “allow” or “managed” can mean another setting still has the final say.
Next, go to Security Events. This is where configuration becomes evidence. Set the date range to at least the past 7 to 30 days, then filter for blocked actions. Search for the bot name or user agent when the interface allows it. You may see events tied to familiar crawlers such as GPTBot, ClaudeBot, PerplexityBot, Google-Extended, or other AI-related user agents.
The event record should tell you the action Cloudflare took, the rule or product that triggered it, the request path, the country or IP, and the user agent. That rule source matters more than the bot name alone. If the event says an AI crawler was blocked by a custom WAF rule, turning off an AI-crawler toggle will not solve the problem.
Check the rule that made the decision
Open an event and look for language such as “rule ID,” “service,” “source,” or “matched rule.” Then review the corresponding area of Cloudflare:
- WAF custom rules can block requests based on user agent, URI path, country, ASN, IP reputation, or a custom expression.
- Bot Fight Mode or Super Bot Fight Mode can challenge or block automated traffic based on Cloudflare’s bot detection.
- Rate limiting rules can stop a crawler that requests pages too quickly, even if you intended to permit it.
- Managed rules may flag suspicious behavior that resembles scraping.
- robots.txt management or site directives can tell compliant crawlers not to access content, even when Cloudflare itself is not blocking them.
For a lean business team, the key distinction is simple: Cloudflare may be blocking a bot, challenging a bot, or allowing the request while your robots.txt denies crawling. All three affect discoverability differently.
Why the AI Bot Name Is Not the Whole Story
A user agent is a label a crawler sends with its request. Bad actors can impersonate GPTBot or ClaudeBot with a copied user agent. That is why you should not whitelist every request that claims to be an AI crawler.
Cloudflare’s verified bot signals and its event data are more useful than a user-agent string by itself. When a crawler is recognized as verified, you have stronger evidence that it is associated with the company it claims to represent. If the traffic is unverified, arrives in huge bursts, or hits unusual URLs, treat it as a security question first, not an AI visibility opportunity.
There is also a business decision hiding inside this technical check. Some AI crawlers may collect information for model training. Others may support search, answer engines, or product discovery features that could send people toward your brand. The right policy depends on the crawler and your goals.
For example, Google-Extended controls whether Google can use content for certain generative AI purposes. It does not control your standard Google Search ranking. Blocking it is not the same thing as blocking Googlebot. A local service business that wants calls from Google Search should be especially careful not to confuse the two.
Use Security Events to Separate a Real Block From a Crawl Gap
A missing AI referral does not prove Cloudflare blocked a crawler. AI platforms may not crawl every page, may use cached information, may respect robots.txt, or may simply choose not to cite your site. Security Events gives you a more defensible answer.
When reviewing logs, look for a pattern rather than one isolated request. Are requests from a recognized AI crawler repeatedly receiving a 403 or challenge? Is the same rule appearing each time? Are the blocked URLs important pages such as services, locations, product categories, or articles?
A 403 generally indicates access was denied. A challenge may mean the crawler could not complete the verification step. A 429 often points to rate limiting. A 5xx error may be an origin or server issue rather than an intentional Cloudflare block. Each outcome calls for a different fix.
Keep in mind that event retention, log depth, and available filtering depend on your Cloudflare plan. If you do not see enough history in the dashboard, document the rule settings now and check events regularly going forward. Small businesses do not need a giant security operations center. They do need a repeatable check before changing a rule that affects lead-generating content.
A Smart Review Process for Small Business Websites
Before changing anything, make a short list of the AI crawlers that matter to your visibility strategy. Then confirm whether you want to allow, limit, or block each one. There is no universal “allow all AI bots” answer, and anyone who says otherwise is selling simplicity where strategy is needed.
If you decide to allow a verified crawler, use the narrowest exception possible. Allow the verified bot or a specific verified signal rather than opening a broad exemption for every visitor with a matching user agent. This reduces the risk of fake bots slipping through under a familiar name.
After making a change, monitor Security Events again over the next several days. Confirm that legitimate crawler requests are no longer being blocked and that suspicious automated traffic has not increased. Also check whether your important public pages return a normal 200 response and are not accidentally restricted by password protection, geoblocking, or an overly aggressive rate limit.
For many local businesses, a sensible baseline is to keep malicious and anonymous scrapers out while allowing the verified platforms that support the discovery channels you actively want to grow. Your website should not be wide open. It also should not be a locked storefront with great services inside and no way for customers, search engines, or trusted AI assistants to see them.
Common Mistakes That Hide the Real Problem
The most common mistake is checking only the AI Crawl Control screen. That tells you what one feature is configured to do, not what every security layer did in practice. A custom WAF rule written months ago, a bot-management setting, or a rate limit can override your expectations.
Another mistake is treating all AI crawler blocks as bad. If your site has proprietary content, large media files, sensitive customer areas, or a history of abusive scraping, selective blocking may be the right call. Public marketing pages and private account areas should not necessarily have the same policy.
Finally, do not confuse crawler access with AI visibility. Allowing a crawler is only permission. To earn mentions in AI answers, your pages still need clear services, accurate location details, original expertise, trustworthy proof, logical structure, and content that answers the questions customers actually ask.
Quick FAQ
Can Cloudflare show every AI bot that visited my site?
Not always. What you can see depends on your Cloudflare plan, event retention, and whether the requests triggered a logged security action. Security Events is most useful for identifying blocks, challenges, and the rules responsible for them.
Should I allow every verified AI bot?
No. Decide based on what the bot does, your content policy, your security needs, and whether the platform fits your customer acquisition strategy. Verify the crawler first, then create narrow allowances when needed.
Will blocking AI bots hurt SEO?
It depends on the bot. Blocking a training-related crawler is different from blocking a search crawler. Accidentally blocking Googlebot, Bingbot, or other search crawlers can harm organic visibility. Blocking certain AI crawlers may limit exposure in AI-generated answers without changing traditional rankings.
The best move is not to choose between security and visibility. Treat them as two parts of the same growth system: protect what needs protection, measure what is being denied, and make intentional access decisions around the channels that can bring your business its next customer.
