Cloudflare just flipped the switch on a new model for the internet. Starting July 1, 2025, every new domain using Cloudflare blocks AI crawlers by default. No scraping. No silent indexing. If an AI company wants access, it has to pay. That’s the core of Pay-Per-Crawl, a system now in private beta that lets websites charge bots for every crawl. The company already powers 20% of the web and works with 30% of the top 1 million sites.
The mechanism is built into HTTP itself. If a crawler hits a site without permission, it gets a 402 Payment Required response. That’s a rarely used status code, now repurposed to signal pricing. Site owners can set a flat rate or whitelist specific bots. Cloudflare acts as the merchant of record, handling billing and payouts. If the crawler doesn’t have a billing relationship, it’s blocked. No exceptions.
AI companies must disclose their intent. Training, inference, search—each use case has to be declared. That’s part of the transparency layer. Crawlers are verified through cryptographic headers. Spoofing is blocked. The system is designed to prevent unauthorized scraping and enforce economic accountability. Over 1 million Cloudflare customers have already opted to block AI bots. That number is rising.
The incentive model is broken. AI crawlers scrape content, generate answers, and send zero traffic back to the source. Cloudflare says OpenAI’s crawler hit sites 1,700 times for every referral. Anthropic’s ratio was 73,000 to one. That’s not sustainable. Publishers lose revenue. Creators lose visibility. The web loses incentive to produce quality content. Pay-Per-Crawl is designed to reverse that.
Major publishers are backing the move. TIME, The Atlantic, Reddit, Stack Overflow, and the Associated Press have signed on. The system lets them charge for access, block bad actors, and negotiate licensing deals. It’s not just about defense. It’s about building a market. Cloudflare says the real potential lies in agentic systems, AI agents with budgets that negotiate access dynamically. That’s the next layer.
If this scales, Cloudflare won’t just be a CDN. It’ll be the toll booth for the AI economy. Every model training on public content will have to pass through their gate.
Sources