Cloudflare Launches Tool to Block Unauthorized AI Crawlers
Cloudflare has announced the launch of a new tool designed to automatically block unauthorized AI crawlers. The tool, now enabled by default for all new websites using Cloudflare’s services, also introduces a groundbreaking “Pay Per Crawl” model that allows publishers to charge AI companies for accessing their data.
The move comes amid growing concerns from publishers, news outlets, and creators that AI models are exploiting online content without permission or compensation. AI crawlers used by companies like OpenAI and Google often ingest vast amounts of publicly available data to train models, while sending little to no traffic or credit back to the original sites.
“This is about putting control back into the hands of content creators,” said Matthew Prince, CEO of Cloudflare. “Publishers should be able to decide whether, how, and under what terms AI companies use their content.”
Blocking by Default, Monetization by Choice
Effective immediately, all new domains added to Cloudflare will block known AI crawlers by default. Website owners can opt to allow specific bots or negotiate terms for access. The company is also offering site owners the option to monetize their data via a “Pay Per Crawl” feature — a first-of-its-kind commercial framework for AI data use.
In addition to blocking, Cloudflare is rolling out a verification system to help distinguish between legitimate AI crawlers and those attempting to bypass restrictions. This ensures that only verified, policy-compliant bots are able to access content.
Widespread Industry Support
The initiative has already received backing from major players in publishing and online content. Supporters include The Atlantic, Gannett, Associated Press, Condé Nast, BuzzFeed, Stack Overflow, Reddit, and Pinterest. These organizations have increasingly pushed back against the use of their content to train AI models without proper attribution or payment.
Industry groups such as the IAB Tech Lab and The Arena Group have also voiced their support for Cloudflare’s new model, calling it a positive step toward restoring balance between AI innovation and content ownership.
Addressing the AI Traffic Disparity
One of the core concerns driving this change is the imbalance in value exchange. According to Cloudflare and several publishers, AI bots often consume data at massive scale while delivering very little traffic in return. For example, Google’s AI crawl-to-click ratio recently dropped from 6:1 to 18:1. OpenAI’s ratio is reportedly as high as 1,500:1 — meaning for every 1,500 pages crawled, only one visitor is referred back to the site.
This disparity not only deprives publishers of ad revenue but also reduces the visibility and recognition of their original work.
A New Standard for AI Access
With Cloudflare powering roughly 20% of global internet traffic, the policy is expected to set a new standard for how AI models interact with the web. It aligns with broader efforts by publishers and lawmakers worldwide to enforce ethical data use and protect intellectual property in the AI era.
Cloudflare’s tool builds on earlier initiatives, including the AI Labyrinth honeypot system designed to trap and detect rogue AI bots.
As AI systems become increasingly dependent on real-world data, the launch of this tool marks a turning point in the digital economy — one where content creators finally have a say in how their work is used, and a pathway to be compensated when it is.