> Markdown version of [/videos/1366-fireside-chat-with-cloudflare-s-chief-strategy-officer-stephanie-cohen-with-mike-butcher-mbe?t=1080](https://www.wearedevelopers.com/videos/1366-fireside-chat-with-cloudflare-s-chief-strategy-officer-stephanie-cohen-with-mike-butcher-mbe?t=1080). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md). --- # Fireside Chat with Cloudflare's Chief Strategy Officer, Stephanie Cohen (with Mike Butcher MBE) Stephanie Cohen declares that uncompensated AI scraping is unsustainable. Learn how Cloudflare is driving a permission-based web by blocking AI training crawlers by default. - **Speakers:** [Stephanie Cohen](https://www.wearedevelopers.com/@stephanie-cohen), [Mike Butcher](https://www.wearedevelopers.com/@mike-butcher) - **Event:** World Congress 2025 - **Published:** July 17, 2025 - **Duration:** 24:15 - **URL:** https://www.wearedevelopers.com/videos/1366-fireside-chat-with-cloudflare-s-chief-strategy-officer-stephanie-cohen-with-mike-butcher-mbe ## Summary In a recent fireside chat, Cloudflare's Chief Strategy Officer, Stephanie Cohen, discusses the evolving relationship between AI and internet infrastructure, highlighting a pressing challenge: AI models scraping the web for training data without attributing or compensating content creators. Viewing this dynamic as unsustainable, Cloudflare announced that it now defaults to blocking AI training crawlers for all new customers. This strategic shift moves the web toward a "permission-based ecosystem" where content creators can choose to block, allow, or charge for LLM crawls, ensuring a continuous incentive to produce the high-quality human content that AI models ultimately depend on. To address the rise of AI agents and increasingly sophisticated web scraping, Cloudflare introduced Web Bot Auth, offering cryptographic verification to prevent malicious actors from spoofing robot protocols. As the internet transitions from being historically human-optimized to accommodating autonomous machines, Cloudflare is expanding its developer platform to support the "agentic future." This includes rolling out an AI Agent SDK and Remote MCP hosting, which handle backend scalability and logging so developers can focus strictly on agent innovation. Ultimately, the dialogue underscores an industry-wide push for trust, noting Cloudflare's participation in C2PA for content authenticity to combat web manipulation. By bridging robust network defense against escalating DDoS attacks with transparent monetization mechanisms, Cloudflare aims to cultivate a secure ecosystem where human creativity and machine intelligence can concurrently thrive. **Keywords:** cloudflare developer platform, AI training crawlers, pay per crawl monetization, web bot auth, cryptographic bot verification, LLM data scraping, permission-based internet, AI agent SDK, remote MCP hosting, C2PA content authenticity, DDoS attack mitigation, automated bot traffic, content creator compensation, machine-optimized internet, agentic applications ## Chapters 1. **Blocking AI training crawlers by default on networks** (00:58) — Enabling website owners to block AI crawlers or charge for content scraping. 1. **Balancing content creator incentives and AI training needs** (02:29) — High-quality AI development demands sustainable models that compensate original content authors. 1. **Accessing crawler management tools through a free proxy** (05:24) — Routing traffic through specific gateways enables detailed visibility and immediate control over web scrapers. 1. **Integrating AI management tools directly into developer platforms** (06:27) — Infrastructure optimized for AI environments enables scalable agent deployment through distributed physical networks. 1. **Verifying bot identities through cryptographic authentication protocols** (08:08) — Cryptographically signing bot traffic prevents malicious actors from bypassing indexing rules or spoofing legitimate agents. 1. **Optimizing web infrastructure for machine traffic and agents** (09:34) — Adapting the internet for programmatic access requires robust host interfaces and specialized software development kits. 1. **Defending networks against massive distributed denial of service attacks** (11:57) — Increasing automated bot volume amplifies the urgency for exact threat identification techniques at edge locations. 1. **Adapting to changes in traditional web search and traffic** (14:49) — The shift from standard search indexing to generative summaries fundamentally alters established internet navigation pathways. 1. **Verifying digital content authenticity through open validation standards** (18:00) — Implementing tracking standards like C2PA ensures transparent tracing of media origination and generative alterations. 1. **Engaging globally and expanding infrastructure operations engineering teams** (19:45) — Opening direct feedback loops and recruiting across new international tech hubs accelerates the deployment of novel capabilities. ## Related Moments - [Monetizing automated web scraping with Cloudflare pay per crawl](https://www.wearedevelopers.com/videos/1381-402-payment-required) (from "402 - Payment Required") - [Analyzing the surge in internet bot and crawler traffic](https://www.wearedevelopers.com/videos/1901-wearedevelopers-live-smoover-who-s-best-at-devrel) (from "WeAreDevelopers LIVE - Smoover - Who's Best at DevRel?") - [Identifying technical hurdles with aggressive AI web crawlers](https://www.wearedevelopers.com/videos/100133-ai-search-insights-from-otterlyai-what-we-tested-what-failed-and-what-actually-works) (from "AI Search Insights from OtterlyAI: What We Tested, What Failed, and What Actually Works") - [Managing security risks introduced by autonomous AI agents](https://www.wearedevelopers.com/videos/1403-five-things-in-tech-that-matter-and-we-have-to-make-work) (from "Five things in tech that matter and we have to make work") - [Q&A on platform discovery, security, and prompt injection](https://www.wearedevelopers.com/videos/100305-api-mcp-or-mcp-app-choosing-the-right-surface-for-ai-agents) (from "API, MCP or MCP App? Choosing the right surface for AI agents") - [Balancing free public access with commercial AI licensing requirements](https://www.wearedevelopers.com/videos/1116-the-data-phoenix-the-future-of-the-internet-and-the-open-web) (from "The Data Phoenix: The future of the Internet and the Open Web") ## Related Articles - [Who Owns Your Content in the Age of LLMs?](https://www.wearedevelopers.com/magazine/610-who-owns-your-content-in-the-age-of-llms) - [Stephan Gillich - Bringing AI Everywhere](https://www.wearedevelopers.com/magazine/489-stephan-gillich-bringing-ai-everywhere) - [SEO in an AI world - Google vs. ChatGPT and survival tips for content creators](https://www.wearedevelopers.com/magazine/534-seo-in-an-ai-world-google-vs-chatgpt-and-survival-tips-for-content-creators) - [WebMCP: Empowering Agents as First-Class Citizens of the Web](https://www.wearedevelopers.com/magazine/696-webmcp-empowering-agents-as-first-class-citizens-of-the-web) ## Related Jobs - [Principal Field Architect - AI Agents](https://www.wearedevelopers.com/jobs/ext/1442858-principal-field-architect-ai-agents) at **Twilio** - [Security Architect - AI](https://www.wearedevelopers.com/jobs/ext/1581899-security-architect-ai) at **ZEISS Group** - [Senior Engineer, Infrastructure Platform](https://www.wearedevelopers.com/jobs/ext/328836-senior-engineer-infrastructure-platform) at **Intercom, Inc.** - [Twilio's next Senior Principal Field Architect - AI Agents](https://www.wearedevelopers.com/jobs/ext/1487390-twilio-s-next-senior-principal-field-architect-ai-agents) at **Twilio** - [Principal Software Engineer, Enterprise AI Platform](https://www.wearedevelopers.com/jobs/ext/1467292-principal-software-engineer-enterprise-ai-platform) at **GitHub** - [AI Software Engineer (Germany)](https://www.wearedevelopers.com/jobs/48317-ai-software-engineer-germany) at **Sunhat**