# Two Thousand Crawls for One Click: Cloudflare Flips the Web's AI Default, September 15, 2026

Page: https://stenobird.com/podcast/dx-today-no-hype-podcast-news-about-ai-dx-6434212/two-thousand-crawls-for-one-click-cloudflare-flips-the-web-s-ai-default-september-15-2026
Text version: https://stenobird.com/podcast/dx-today-no-hype-podcast-news-about-ai-dx-6434212/two-thousand-crawls-for-one-click-cloudflare-flips-the-web-s-ai-default-september-15-2026.md
Podcast: [DX Today | No-Hype Podcast & News About AI & DX](https://stenobird.com/podcast/dx-today-no-hype-podcast-news-about-ai-dx-6434212)
Published: 2026-09-15T10:44:37+00:00
Episode link: https://www.buzzsprout.com/2207817/episodes/19808459-two-thousand-crawls-for-one-click-cloudflare-flips-the-web-s-ai-default-september-15-2026.mp3
Audio file: https://www.buzzsprout.com/2207817/episodes/19808459-two-thousand-crawls-for-one-click-cloudflare-flips-the-web-s-ai-default-september-15-2026.mp3
Processing state: not_requested
JSON: https://stenobird.com/v1/public/podcasts/dx-today-no-hype-podcast-news-about-ai-dx-6434212/episodes/two-thousand-crawls-for-one-click-cloudflare-flips-the-web-s-ai-default-september-15-2026
Duration seconds: 799

## Resource

Two Thousand Crawls for One Click: Cloudflare Flips the Web's AI Default Today Cloudflare's new crawler defaults take effect, sorting automated visitors into three declared purposes, Search, Agent, and Training, and blocking the last two by default on ad supported pages. Chris and Laura dig into the mixed use crawler problem that puts Googlebot, Applebot, and Bingbot on notice, the crawl to referral ratios showing some AI companies taking thousands of pages for every visitor they send back, ...

## Actions

- request_transcript: `POST https://stenobird.com/v1/public/podcasts/dx-today-no-hype-podcast-news-about-ai-dx-6434212/episodes/two-thousand-crawls-for-one-click-cloudflare-flips-the-web-s-ai-default-september-15-2026/transcription-requests` — Idempotently request low-priority transcript generation for this episode.
- read_markdown: `GET https://stenobird.com/podcast/dx-today-no-hype-podcast-news-about-ai-dx-6434212/two-thousand-crawls-for-one-click-cloudflare-flips-the-web-s-ai-default-september-15-2026.md` — Read the agent-friendly Markdown representation of this episode resource.

A page view does not enqueue transcription. Agents should invoke `request_transcript` explicitly when they need this episode processed.

## Transcript

Full transcripts are not published on public pages unless there is a clear rights basis.