Cloudflare’s new policy pushes AI companies to pay for publishers’ content

Cloudflare has introduced a new policy requiring AI companies to distinguish between web crawlers used for search and those used for AI training by September 15. If companies fail to comply, their crawlers risk being blocked on many publisher sites. This move aims to protect publishers' content and ensure fair compensation for AI training data usage.

NowLaunch

Signal trust

Single sourceEarly signal

Market reactionNET ↓ -1.73% by next close

Before $251.03After $246.69

PublishedWednesday, July 1, 2026 at 7:48 PMJul 1, 07:48 PM

Freshness3h live

Story ID#4758

Back to feed Original report

Original article excerpt

Server-side extracted preview paragraphs from the original source.

Cloudflare is giving AI companies until September 15 to separate web crawlers used for search from those used for AI training and agents, or risk being blocked by default on many publisher sites.

Cloudflare has just issued the AI industry a new deadline to separate the web crawlers used for traditional search purposes, like Google Search, from those used for AI agents and training. Starting on September 15, 2026, Cloudflare’s default settings will block “mixed-use” crawlers from any pages that host ads, the company announced on Wednesday.

That means that the crawlers that blend search, agent use, and training will be blocked from crawling these sites by default, unless the site owner adjusts the settings otherwise. These changes to the defaults will apply to new Cloudflare customers, new sites set up by existing customers, and all existing free customers, the company says.

The move could impact how AI model providers are able to access web content for training purposes and to help power their agentic services.

Cloudflare points out that most website owners want their content to be discoverable via search and often through AI services as well, but they want protections against having their intellectual property given away for free.

Cloudflare specifically calls out the “world’s largest search engine” (clearly a Google reference!) as having access to about “2x more information” than other AI companies because the search giant makes it difficult for customers to remain discoverable without being used for AI.

Google has pushed back against this generalization in the past, noting that it provides a bot called Google Extended that lets site owners opt out of having their content used for training and AI products and services like Gemini Apps and Vertex API. Its use doesn’t impact a site’s inclusion in Google Search. However, the tech giant’s flagship Googlebot crawls for Search, including AI features like AI Overviews and AI Mode.

Opening the briefing

Cloudflare’s new policy pushes AI companies to pay for publishers’ content

Original article excerpt