Coffee With Developers • Jun 24, 2026

Web Scraping, AI Agents, and the Future of Open Source - Kevin Lewis (Apify)

Kevin Lewis argues the AI free lunch is over. Discover how equipping agents with deterministic web scrapers and crypto wallets ensures secure workflows that fund open-source maintainers.

Pause

Mute

Enter Fullscreen

#1 about 4 min

Understanding the Apify marketplace and web scraping actors

The Apify marketplace provides developers with containerized web scraping scripts called actors, simplifying persistent storage and proxy management.

#2 about 2 min

Defining actor schemas and programmatic integration interfaces

Actors function as isolated programs executing defined schemas directly via web consoles, REST APIs, or MCP servers.

#3 about 3 min

Distinguishing large language models from autonomous AI agents

While base language models handle generative tasks, true agents combine custom instructions, multi-turn chat loops, and external tools to accomplish goals autonomously.

#4 about 4 min

Establishing security guardrails and mitigating local agent risks

Using local open source models, monitoring deterministic tool behavior, and managing dynamic permissions mitigate the risks of runaway agent costs and malicious actions.

#5 about 11 min

Monetizing autonomous agent interactions with HTTP 402 payments

Integrating the HTTP 402 payment protocol equips agents to securely execute microtransactions for paywalled resources without exposing primary user funds.

#6 about 6 min

Preserving agent context windows through direct CLI integrations

Defaulting to well-documented command-line interfaces instead of complex MCP servers preserves an agent's context window while reliably accessing real-time platforms.

#7 about 7 min

Securing competitive intelligence using automated web extraction crawlers

Developers deploy automated data extraction actors for large-scale competitive intelligence, running targeted crawls across e-commerce product listings and influencer profiles.

#8 about 7 min

Fostering sustainable open source systems via web monetization

Monetizing automated platform access creates an equitable ecosystem that gives content creators and open source maintainers kickbacks for parsed data and tools.

Matching moments

3:00 min

Building automated actors and software agents with Apify

Chris Heilmann +4 · LIVE

10:12 min

Challenges with open source monetization and AI scraping

Chris Heilmann +2 · LIVE

53 sec

Autonomous AI agents and the need for digital payments

Brian Whippo

Brian Whippo · World Congress 2026 Europe

2:18 min

Managing security risks introduced by autonomous AI agents

Christian Heilmann

Christian Heilmann · World Congress 2025

3:35 min

Navigating token economics and open source artificial intelligence

Mike Swift · Coffee With Developers

5:09 min

Securing AI agents against malicious MCP servers

Gbadebo Bello

Gbadebo Bello · Europe 2026 Virtual

More from Coffee With Developers

Using Agents to Curate the Content You See Online - Dan Gish thumbnail

Using Agents to Curate the Content You See Online - Dan Gish

Dan Gish

AI Vector Search at Scale - Ewa Szyszka - Ewa Szyszka thumbnail

AI Vector Search at Scale - Ewa Szyszka - Ewa Szyszka

Ewa Szyszka

What is Cross-Origin Storage - Thomas Steiner thumbnail

What is Cross-Origin Storage - Thomas Steiner

Thomas Steiner

Women in AI - Nalini Garg thumbnail

Women in AI - Nalini Garg

Nalini Garg

Agents and People Dynamics in Shared Chats - Sam Liu thumbnail

Agents and People Dynamics in Shared Chats - Sam Liu

Sam Liu

Why Token Reduction Isn’t Cost Reduction - Dave Anderson & Sarel Weinberger, PhD. thumbnail

Why Token Reduction Isn’t Cost Reduction - Dave Anderson & Sarel Weinberger, PhD.

Dave Anderson, Sarel Weinberger, Phd

Upcoming sessions on this topic

World Congress 2026 North America

September 25, 2026 · 11:40–12:10

Stage 7

x402: Building Pay-Per-Request APIs for AI Agents

Brian Whippo

Senior Director Integrations and Developer Tooling, Algorand Foundation

Brian Whippo

World Congress 2026 North America

September 24, 2026 · 17:30–18:00

Stage 5

Securing AI Agent Infrastructure: Identity, Attestation, and Trust at Scale

Abdel Fane

Founder of OpenA2A

Abdel Fane

World Congress 2026 North America

September 25, 2026 · 10:20–10:50

Outdoor Stage

MicroAgents: The Microservices of the Agentic Era

Ashish Shubham

Runs the show bussiness at ThoughtSpot

Ashish Shubham

World Congress 2026 North America

September 25, 2026 · 12:20–12:50

Stage 4

Give the Agent a Budget, Not a Token

Sachin Malhotra

MTS @Anthropic

Sachin Malhotra

World Congress 2026 North America

September 24, 2026 · 11:40–12:10

Mainstage

I Don't Trust AI Agents (And Neither Should You): Building Production-Ready Architectures

Darko Mesaros

Distinguished Developer Advocate at AWS

Darko Mesaros

World Congress 2026 North America

September 25, 2026 · 12:20–12:50

Stage 9

Designing APIs That Survive AI Agents at Scale

Phani Pendurthi

Mastercard, Principal Software Engineer

Phani Pendurthi

Related videos

How to scrape modern websites to feed AI agents thumbnail

World Congress 2025 • about 1 year ago

How to scrape modern websites to feed AI agents

Jan Curn

HTTP 402: Teaching the Web to Let Agents Pay thumbnail

World Congress 2026 Europe • about 2 months ago

HTTP 402: Teaching the Web to Let Agents Pay

Ed Marquez

200 OK: Payment settled, resource delivered thumbnail

World Congress 2026 Europe • 2 months ago

200 OK: Payment settled, resource delivered

Brian Whippo

WeAreDevelopers LIVE – Web Scraping, Agents, Actors and more thumbnail

LIVE • 9 months ago

WeAreDevelopers LIVE – Web Scraping, Agents, Actors and more

Chris Heilmann, Daniel Cranney, Ondra Urban, Coo, Gtm At Apify

Teaching agents to pay: what devs need to know thumbnail

World Congress 2026 Europe • about 2 months ago

Teaching agents to pay: what devs need to know

Benjamin Smith

MCP Mashups: How AI Agents are Reviving the Programmable Web thumbnail

World Congress 2025 • about 1 year ago

MCP Mashups: How AI Agents are Reviving the Programmable Web

Angie Jones

Related articles

BS

Ben Smith

E-Commerce in the Agentic Age: When "Buy" Becomes a Function Call

For more than thirty years, we’ve built online commerce around one simple assumption: the customer is human. Humans browse websites. They compare products, read reviews, add items to a shopping car...

about 1 month ago

A mockup of a UI, with a 'Buy' button in the centre and handwritten labels around it, one calling a function and the other saying "Will agents click this?"

CH

Chris Heilmann

The Web We Broke (And Why AI Agents Are Paying the Price) - AgentCon Berlin

This is the accompanying post to the talk Chris Heilmann gave at AgentCon in Berlin on 19/05/2026, you can also see the slides and listen to it in this screencast: Thirty years of developer shortc...

4 months ago

Yahoo Pipes allowing you to remix HTML content

A

Apify

What Developers Are Building to Win $1 Million with Apify

Apify started as a web scraping product, but quickly evolved into a full-blown platform and marketplace for developers to write code, and monetise it by creating Actors, tools that simplify the sc...

8 months ago

What developers are building to win $1 million

CH

Chris Heilmann

WebMCP: Empowering Agents as First-Class Citizens of the Web

WebMCP is an exciting W3C proposal that just landed in Chrome Canary to try out . The idea is that you can use some HTML attributes on a form or register JavaScript tool methods to give agents di...

7 months ago

WebMCP: Empowering Agents as First-Class Citizens of the Web

From learning to earning

Jobs that call for the skills explored in this talk.

18 days ago • Verified

Software Engineer, Fullstack

San Francisco, United States

Expert €165k–210k Python

18 days ago • Verified

Senior AI Serving Engineer, Backend

San Francisco, United States

Expert €190k–250k Rust Python

18 days ago • Verified

ML Engineer

San Francisco, United States

Expert €165k–210k Python

18 days ago • Verified

LLM Training Engineer

San Francisco, United States

Expert €155k–220k Python

13 days ago • Verified

Partner Sales Director - AI Alliances - Model Providers

San Francisco, United States

Expert Remote DevOps AI Frameworks Machine Learning

20 days ago • Verified

Senior AI/ML Engineer

Lisbon, Portugal

Expert Remote AI Frameworks AI-assisted coding tools