Enable Cloudflare's new AI Scrapers and Crawlers block

[Steve Krause]

Although I can't find a mention on the official Cloudflare blog, a new setting just showed up for all Cloudflare plans which allows you to block known AI Scrapers and Crawlers. These are bots which crawl your site to train LLM's.

Cloudflare > Security > Bots > Configure Super Bot Fight Mode > AI Scrapers and Crawlers > On/Off Toggle

Looking a bit deeper, it appears what it's actually doing is just adding a new WAF rule configured to block known AI Scrapers.

Looking at the WAF rule, it's blocking all Verified Bots that have a category of AI Crawler. Pretty straight forward if you want to customize the rule or add it to an existing rule.

Before enabling it, my next question was, "What exactly is Cloudflare blocking?"

Big thanks to Cloudflare for documenting how each Verified Bot is categorized on its Verified Bots page.

 The page also has a search to easily narrow down who the AI Crawlers are (as of 6/30/2024).

Unfortunately, the list only includes the top 3 platforms. Although this will help prevent the continued theft of copyrighted content from Amazon, Google, and OpenAI, it doesn't appear to be blocking other AI crawlers yet.

Anyway, it's a curious finding this morning while drinking my coffee....

[shockersh]

That's all great but also a bit risky... Never know how Google many punish you from an Organic standpoint if you don't feed it. Thoughts?

[Jeff Butts]

19 hours ago, shockersh said:

Never know how Google many punish you from an Organic standpoint

Google claims it doesn't....but I'm not sure how much I trust that claim. That being said, it can't really punish a site much more than it would by scraping all of its content and then feeding it to users without ever sending the traffic to our pages,