Blocking AI bots on your website: what Cloudflare offers for free and what to watch out for

Website owners have long complained that their texts are read by more programs than people. Some of these bots collect content to train language models, some use it for AI search engine responses, and some act as agents on behalf of the user. Cloudflare, one of the largest networks for website protection and delivery, offers tools—including a free version—to monitor this kind of traffic. However, headlines about a “new free weapon” misrepresent what is new and what isn’t.

What’s New and What Isn’t

Free, one-click blocking of AI bots isn’t a recent development. Cloudflare made this feature available to all customers, including those on the free plan, back in July 2024. All it took was flipping a toggle for AI scrapers and crawlers in one place on the dashboard.

The change the company announced on July 1, 2026, lies elsewhere: the single toggle has been replaced by three separate categories. These are available to all customers, including those on the free plan. The claim that advanced protection was previously only available on expensive plans is therefore inaccurate.

Three Types of AI Visitors

Cloudflare categorizes AI traffic based on what the bot is doing on the page.

• Search: AI search crawlers that can send readers back to your website.
• Agent: bots that act on behalf of a user.
• Training: crawlers that collect content for training or fine-tuning models.

For each category, the site owner can choose whether to block it on all pages, only on pages with ads, or not at all. The rules in the robots.txt file are voluntary: according to the company’s own explanation, nothing forces a bot to follow them. That is why there is also enforced blocking at the network level.

Change effective September 15, 2026

Starting September 15, 2026, updated default settings will apply to new domains connected to Cloudflare: bots classified as “Training” or “Agent” will be blocked on pages with ads, while “Search” remains allowed. Existing customers had the option to opt out of the new default settings before this date.

Beware of Search Engines

It is often claimed that the new feature does not block legitimate search engines. According to available analyses, the situation is not so clear-cut. Some crawlers serve multiple purposes simultaneously—that is, both searching and training. Following the change on September 15, if the “Training” category is enabled, the block may also affect such multipurpose bots. The analysis cites examples of Googlebot, Bingbot, and Applebot, even with the old toggle. This is a secondary source that cites a statement from Cloudflare. Therefore, please verify the exact behavior in your own settings.

How to Check This

• In your security settings, check whether you have the “Training” category block enabled or the old “Block AI bots” toggle.
• Find out which bots are actually visiting your site. Cloudflare offers an activity overview for individual crawlers and the option to set exceptions.
• Monitor search engine crawl statistics (for example, in Search Console) before and after the change.

How to Enable It

The exact names of these options have changed over time. In the original version, it was a toggle in the “Security” and “Bots” sections; in newer descriptions, it refers to security settings filtered by bot traffic. According to the descriptions, the setting takes effect immediately and does not require any changes to the website’s code. This assumes that your website is already running through Cloudflare. Deploying the network in front of your website is a separate step that also involves DNS configuration.

What This May and May Not Solve

Content Protection

Blocking training bots reduces the amount of content that crawlers use to train models. However, this is not an absolute guarantee, especially for bots that masquerade as regular browsers.

Server Load

We have not substantiated the claim that these bots significantly slow down your website with data. If you notice a slowdown, first check your analytics and server logs to determine where the traffic is coming from.

Visibility in AI Search

Blanking out AI bots across the board can also reduce your visibility in AI search results. That’s why it’s worth distinguishing between categories and not blocking access for bots that bring you readers.

Conclusion

Today, website owners have free tools at their disposal that can restrict AI bots’ access to content. However, they aren’t a magic bullet: the key lies in properly configuring categories, monitoring side effects, and understanding what you want to protect and what you want to allow. The most practical approach is to combine a robots.txt file for legitimate bots, a network enforcement tool, and regular monitoring of statistics.