WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
Coffee With Developers • Jun 24, 2026

Web Scraping, AI Agents, and the Future of Open Source - Kevin Lewis (Apify)

Kevin Lewis argues the AI free lunch is over. Discover how equipping agents with deterministic web scrapers and crypto wallets ensures secure workflows that fund open-source maintainers.

Pause
Mute Enter Fullscreen
#1 about 4 min

Understanding the Apify marketplace and web scraping actors

The Apify marketplace provides developers with containerized web scraping scripts called actors, simplifying persistent storage and proxy management.

#2 about 2 min

Defining actor schemas and programmatic integration interfaces

Actors function as isolated programs executing defined schemas directly via web consoles, REST APIs, or MCP servers.

#3 about 3 min

Distinguishing large language models from autonomous AI agents

While base language models handle generative tasks, true agents combine custom instructions, multi-turn chat loops, and external tools to accomplish goals autonomously.

#4 about 4 min

Establishing security guardrails and mitigating local agent risks

Using local open source models, monitoring deterministic tool behavior, and managing dynamic permissions mitigate the risks of runaway agent costs and malicious actions.

#5 about 11 min

Monetizing autonomous agent interactions with HTTP 402 payments

Integrating the HTTP 402 payment protocol equips agents to securely execute microtransactions for paywalled resources without exposing primary user funds.

#6 about 6 min

Preserving agent context windows through direct CLI integrations

Defaulting to well-documented command-line interfaces instead of complex MCP servers preserves an agent's context window while reliably accessing real-time platforms.

#7 about 7 min

Securing competitive intelligence using automated web extraction crawlers

Developers deploy automated data extraction actors for large-scale competitive intelligence, running targeted crawls across e-commerce product listings and influencer profiles.

#8 about 7 min

Fostering sustainable open source systems via web monetization

Monetizing automated platform access creates an equitable ecosystem that gives content creators and open source maintainers kickbacks for parsed data and tools.

Matching moments

3:00 min

Building automated actors and software agents with Apify

Chris Heilmann +4 · LIVE

10:12 min

Challenges with open source monetization and AI scraping

Chris Heilmann +2 · LIVE

53 sec

Autonomous AI agents and the need for digital payments

Brian Whippo Brian Whippo · WWC Europe 2026

2:18 min

Managing security risks introduced by autonomous AI agents

Christian Heilmann Christian Heilmann · WWC 2025

3:35 min

Navigating token economics and open source artificial intelligence

Mike Swift · Coffee With Developers

2:23 min

The problem of AI agents encountering traditional API paywalls

Ed Marquez Ed Marquez · WWC Europe 2026

More from Coffee With Developers

What to Do About Hackathons in the Time of Agents - Mike Swift thumbnail
46:03
9 days ago

What to Do About Hackathons in the Time of Agents - Mike Swift

Mike Swift

Moving from Playing with AI to Implementing AI - Lee Stott thumbnail
18:43
13 days ago

Moving from Playing with AI to Implementing AI - Lee Stott

Lee Stott

Evals vs. Evil - AI and Package Security - Laurie Voss thumbnail
16:57
22 days ago

Evals vs. Evil - AI and Package Security - Laurie Voss

Laurie Voss

Burning Satellites, Talking Plants and Agents Hiring Each Other - Fake or News with Matt Zeunert thumbnail
4:57
27 days ago

Burning Satellites, Talking Plants and Agents Hiring Each Other - Fake or News with Matt Zeunert

Matt Zeunert

Making Teaching Code Less Academic and More Market-Ready - Peter Ruppel thumbnail
32:22
about 2 months ago

Making Teaching Code Less Academic and More Market-Ready - Peter Ruppel

RAG's Not Dead, You're Just Using It Wrong! - Phil Nash thumbnail
28:20
about 2 months ago

RAG's Not Dead, You're Just Using It Wrong! - Phil Nash

Phil Nash

Upcoming sessions on this topic

Open session

World Congress 2026 North America

Securing AI Agent Infrastructure: Identity, Attestation, and Trust at Scale

Abdel Fane

Founder of OpenA2A

Abdel Fane
Open session

World Congress 2026 North America

MicroAgents: The Microservices of the Agentic Era

Ashish Shubham

Runs the show bussiness at ThoughtSpot

Ashish Shubham
Open session

World Congress 2026 North America

Give the Agent a Budget, Not a Token

Sachin Malhotra

MTS @Anthropic

Sachin Malhotra
Open session

World Congress 2026 North America

Designing APIs That Survive AI Agents at Scale

Phani Pendurthi

Mastercard, Principal Software Engineer

Phani Pendurthi
Open session

World Congress 2026 North America

When Agents Became Users: Rearchitecting Identity and Permissions for AI at Scale

Yoav Gal, Dor Cohen

Yoav Gal
Dor Cohen
Open session

World Congress 2026 North America

Your next customer won’t be human

Benjamin Smith

Staff Developer Advocate at Stripe

Benjamin Smith

Related videos

How to scrape modern websites to feed AI agents thumbnail
19:20
WWC 2025 • 12 months ago

How to scrape modern websites to feed AI agents

Jan Curn

HTTP 402: Teaching the Web to Let Agents Pay thumbnail
26:05
WWC Europe 2026 • about 1 month ago

HTTP 402: Teaching the Web to Let Agents Pay

Ed Marquez

200 OK: Payment settled, resource delivered thumbnail
23:43
WWC Europe 2026 • about 1 month ago

200 OK: Payment settled, resource delivered

Brian Whippo

WeAreDevelopers LIVE – Web Scraping, Agents, Actors and more thumbnail
1:06:14
LIVE • 8 months ago

WeAreDevelopers LIVE – Web Scraping, Agents, Actors and more

Chris Heilmann, Daniel Cranney, Ondra Urban, Coo, Gtm At Apify

Teaching agents to pay: what devs need to know thumbnail
26:50
WWC Europe 2026 • about 1 month ago

Teaching agents to pay: what devs need to know

Benjamin Smith

MCP Mashups: How AI Agents are Reviving the Programmable Web thumbnail
23:17
WWC 2025 • 12 months ago

MCP Mashups: How AI Agents are Reviving the Programmable Web

Angie Jones

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.