Cloudflare’s new policy pushes AI companies to pay for publishers’ content
Which summary reads better? Pick one — models revealed after.Both summaries are AI-generated.
Starting September 15, 2026, Cloudflare will block mixed-use AI/search crawlers by default on ad-supported pages for new sites, new customers, and existing free customers unless owners opt out. If you rely on web crawling for training, retrieval, or agents, you need separate, transparent crawler identities for search versus AI use and should expect more publisher content to move behind permissioned or paid access rather than being freely fetchable.
Starting September 15, 2026, Cloudflare will block by default any mixed-use crawler—one that blends search, training, and agent fetching—from ad-hosting pages, hitting all free customers and new sites unless owners opt out. If your crawlers don't cleanly separate search from training/agent use, you risk losing default access to a large slice of the web, and Cloudflare's Pay Per Use model means you may soon be billed when content drives value, not just when fetched—so plan for segmented, transparent bot identities and a metered content-cost line item.