WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
Coffee With Developers • Jun 24, 2026

Web Scraping, AI Agents, and the Future of Open Source - Kevin Lewis (Apify)

Kevin Lewis argues the AI free lunch is over. Discover how equipping agents with deterministic web scrapers and crypto wallets ensures secure workflows that fund open-source maintainers.

Pause
Mute Enter Fullscreen
#1 about 4 min

Understanding the Apify marketplace and web scraping actors

The Apify marketplace provides developers with containerized web scraping scripts called actors, simplifying persistent storage and proxy management.

#2 about 2 min

Defining actor schemas and programmatic integration interfaces

Actors function as isolated programs executing defined schemas directly via web consoles, REST APIs, or MCP servers.

#3 about 3 min

Distinguishing large language models from autonomous AI agents

While base language models handle generative tasks, true agents combine custom instructions, multi-turn chat loops, and external tools to accomplish goals autonomously.

#4 about 4 min

Establishing security guardrails and mitigating local agent risks

Using local open source models, monitoring deterministic tool behavior, and managing dynamic permissions mitigate the risks of runaway agent costs and malicious actions.

#5 about 11 min

Monetizing autonomous agent interactions with HTTP 402 payments

Integrating the HTTP 402 payment protocol equips agents to securely execute microtransactions for paywalled resources without exposing primary user funds.

#6 about 6 min

Preserving agent context windows through direct CLI integrations

Defaulting to well-documented command-line interfaces instead of complex MCP servers preserves an agent's context window while reliably accessing real-time platforms.

#7 about 7 min

Securing competitive intelligence using automated web extraction crawlers

Developers deploy automated data extraction actors for large-scale competitive intelligence, running targeted crawls across e-commerce product listings and influencer profiles.

#8 about 7 min

Fostering sustainable open source systems via web monetization

Monetizing automated platform access creates an equitable ecosystem that gives content creators and open source maintainers kickbacks for parsed data and tools.

Matching moments

3:00 min

Building automated actors and software agents with Apify

Chris Heilmann Chris Heilmann +4 · LIVE

10:12 min

Challenges with open source monetization and AI scraping

Chris Heilmann Chris Heilmann +2 · LIVE

53 sec

Autonomous AI agents and the need for digital payments

Brian Whippo Brian Whippo · World Congress 2026 Europe

2:18 min

Managing security risks introduced by autonomous AI agents

Christian Heilmann Christian Heilmann · World Congress 2025

3:35 min

Navigating token economics and open source artificial intelligence

Mike Swift · Coffee With Developers

5:09 min

Securing AI agents against malicious MCP servers

Gbadebo Bello Gbadebo Bello · Europe 2026 Virtual

More from Coffee With Developers

Using Agents to Curate the Content You See Online - Dan Gish thumbnail
12:47
27 days ago

Using Agents to Curate the Content You See Online - Dan Gish

Dan Gish

AI Vector Search at Scale - Ewa Szyszka - Ewa Szyszka thumbnail
11:14
27 days ago

AI Vector Search at Scale - Ewa Szyszka - Ewa Szyszka

Ewa Szyszka

What is Cross-Origin Storage - Thomas Steiner thumbnail
34:51
27 days ago

What is Cross-Origin Storage - Thomas Steiner

Thomas Steiner

Women in AI - Nalini Garg thumbnail
42:19
28 days ago

Women in AI - Nalini Garg

Nalini Garg

Agents and People Dynamics in Shared Chats - Sam Liu thumbnail
37:41
about 1 month ago

Agents and People Dynamics in Shared Chats - Sam Liu

Sam Liu

Why Token Reduction Isn’t Cost Reduction - Dave Anderson & Sarel Weinberger, PhD. thumbnail
41:09
about 1 month ago

Why Token Reduction Isn’t Cost Reduction - Dave Anderson & Sarel Weinberger, PhD.

Dave Anderson, Sarel Weinberger, Phd

Related videos

How to scrape modern websites to feed AI agents thumbnail
19:20
World Congress 2025 • about 1 year ago

How to scrape modern websites to feed AI agents

Jan Curn

HTTP 402: Teaching the Web to Let Agents Pay thumbnail
26:05
World Congress 2026 Europe • 3 months ago

HTTP 402: Teaching the Web to Let Agents Pay

Ed Marquez

200 OK: Payment settled, resource delivered thumbnail
23:43
World Congress 2026 Europe • 3 months ago

200 OK: Payment settled, resource delivered

Brian Whippo

WeAreDevelopers LIVE – Web Scraping, Agents, Actors and more thumbnail
1:06:14
LIVE • 10 months ago

WeAreDevelopers LIVE – Web Scraping, Agents, Actors and more

Chris Heilmann, Daniel Cranney, Ondra Urban, Coo, Gtm At Apify

Teaching agents to pay: what devs need to know thumbnail
26:50
World Congress 2026 Europe • 3 months ago

Teaching agents to pay: what devs need to know

Benjamin Smith

MCP Mashups: How AI Agents are Reviving the Programmable Web thumbnail
23:17
World Congress 2025 • about 1 year ago

MCP Mashups: How AI Agents are Reviving the Programmable Web

Angie Jones

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.