Episode
Two Thousand Crawls for One Click: Cloudflare Flips the Web's AI Default, September 15, 2026
- Published
- Sep 15, 2026
- Duration seconds
- 799
- Processing state
not_requested
Actions
POST https://stenobird.com/v1/public/podcasts/dx-today-no-hype-podcast-news-about-ai-dx-6434212/episodes/two-thousand-crawls-for-one-click-cloudflare-flips-the-web-s-ai-default-september-15-2026/transcription-requests
Idempotently request low-priority transcript generation for this episode.GET https://stenobird.com/podcast/dx-today-no-hype-podcast-news-about-ai-dx-6434212/two-thousand-crawls-for-one-click-cloudflare-flips-the-web-s-ai-default-september-15-2026.md
Read the agent-friendly Markdown representation of this episode resource.
Summary
Two Thousand Crawls for One Click: Cloudflare Flips the Web's AI Default Today Cloudflare's new crawler defaults take effect, sorting automated visitors into three declared purposes, Search, Agent, and Training, and blocking the last two by default on ad supported pages. Chris and Laura dig into the mixed use crawler problem that puts Googlebot, Applebot, and Bingbot on notice, the crawl to referral ratios showing some AI companies taking thousands of pages for every visitor they send back, ...