> Markdown version of [/videos/1764-wearedevelopers-live-web-scraping-agents-actors-and-more?t=2514](https://www.wearedevelopers.com/videos/1764-wearedevelopers-live-web-scraping-agents-actors-and-more?t=2514). Every page supports `.md` or `Accept: text/markdown`. Links point to the HTML versions so they work for humans too. Agent guide: [/agents.md](https://www.wearedevelopers.com/agents.md).

---

# WeAreDevelopers LIVE – Web Scraping, Agents, Actors and more

Apify COO Ondřej Urban reveals how containerized AI agents and headless browsers bypass anti-bot mechanics. Turn your automation skills into profit with their $1 million developer challenge.

- **Speakers:** [Chris Heilmann](https://www.wearedevelopers.com/@chris-heilmann), [Daniel Cranney](https://www.wearedevelopers.com/@daniel-cranney), Ondra Urban, Coo, Gtm At Apify
- **Event:** WeAreDevelopers LIVE
- **Published:** December 3, 2025
- **Duration:** 1:06:14
- **URL:** https://www.wearedevelopers.com/videos/1764-wearedevelopers-live-web-scraping-agents-actors-and-more

## Summary

This episode of WeAreDevelopers LIVE opens with a roundup of pressing industry news, from the surge of AI-generated fake shopping sites during Cyber Monday to the proliferation of AI "slop" degrading search engine quality. The hosts examine the creative bankruptcy in AI-generated music via platforms like Suno, persistent web performance gaps, and the nostalgic appeal of discovering browser-based retro coding environments and terminal emulators.

Guest Ondřej Urban, COO of Apify, joins the stream to discuss how the company transitioned from a specialized web scraping service to a vibrant developer marketplace. He introduces the concept of Apify "actors"—containerized automation tools and AI agents that developers can build, host, and sell to users. The discussion explores the underlying technologies powering these actors, including Python, JavaScript, headless browser frameworks like Playwright, and Apify's own open-source web scraping library, Crawlee.

Ondřej highlights Apify's ongoing $1 million developer challenge, designed to stimulate the creator economy and empower developers worldwide to profit from building niche automation workflows. The conversation unpacks the nuances of modern web scraping, navigating anti-bot mechanics, and the growing demand for ethical data extraction to train highly specific AI models. Emphasizing the value of hands-on algorithmic problem-solving, the episode concludes that despite the rise of AI coding assistants, foundational engineering curiosity remains indispensable.

**Keywords:** web scraping techniques, AI data extraction, apify developer marketplace, python crawler development, javascript browser automation, serverless automation actors, anti-bot evasion strategies, playwright browser framework, crawlee open-source library, software monetization models, developer creator economy, AI generated search slop, web performance optimization, headless browser architectures, retro browser terminal emulators

## Chapters

1. **Transitioning from engineer and lawyer to startup operations** (00:07) — How shifting responsibilities from engineering to executive operations define modern startup leadership.
1. **Combating fake ecommerce sites and deceptive holiday discounts** (04:05) — How automated web scraping exposes deceptive retail pricing and fake stores during major seasonal sales events.
1. **Examining the persistent threat of weak user passwords** (11:37) — Why widely compromised default credentials continue to misrepresent overall internet security metrics.
1. **Implications of massive artificial intelligence content generation** (14:36) — The creative and legal impacts of using machine learning models to mass-produce music and web articles.
1. **The persistent performance inequality gap in mobile web** (18:20) — How delivering bloated JavaScript payloads limits worldwide internet accessibility on heavily constrained mobile networks.
1. **Visualizing one million websites via infinite canvas rendering** (19:27) — Building an intuitive and heavily optimized scrolling interface to explore a massive repository of website screenshots.
1. **Why large language models still fail at basic geometry** (21:44) — Understanding the limitations of multimodal models in solving lateral logic and visual intersection puzzles.
1. **Emulating vintage computer environments using WebAssembly architectures** (24:37) — How modern browser capabilities enable complete software development environments and terminal effects for retro systems.
1. **Exploring the deterioration of organic search content discovery** (28:06) — How algorithm-driven search and artificial intelligence summaries are disrupting organic research and content monetization.
1. **Building automated actors and software agents with Apify** (38:54) — Enabling developers to monetize encapsulated scripts and automation workflows on a centralized creation marketplace.
1. **The technical evolution of modern web scraping infrastructure** (41:54) — Navigating anti-bot technologies and shifting public APIs to maintain reliable data extraction pipelines at scale.
1. **Fostering the creator economy for software developers** (47:48) — Incentivizing global developer participation through significant financial challenges and scalable platform monetization structures.
1. **Leveraging modern languages and browser libraries for automation** (53:53) — Understanding the underlying execution environments and libraries required to build robust automated software actors.
1. **Expanding automation boundaries with global developer communities** (58:06) — Discovering unconventional software solutions and diverse technical approaches sourced from international developer ecosystems.
1. **Preserving the joy and curiosity of human coding** (62:41) — A reflection on how the expanding influence of artificial intelligence impacts the creative satisfaction of software engineering.

## Related Moments

- [Challenges with open source monetization and AI scraping](https://www.wearedevelopers.com/videos/1786-wearedevelopers-live-ai-freelancing-keeping-up-with-tech-and-more) (from "WeAreDevelopers LIVE – AI, Freelancing, Keeping Up with Tech and More")
- [Exploring AI agent usage within the software engineering industry](https://www.wearedevelopers.com/videos/1814-wearedevelopers-live-markdown-liquid-and-checkouts) (from "WeAreDevelopers LIVE - Markdown, Liquid and Checkouts")
- [Shifting developer workloads and realistic AI productivity gains](https://www.wearedevelopers.com/videos/1830-wearedevelopers-live-speculaitions) (from "WeAreDevelopers LIVE - SpeculAItions")
- [Differentiating developer skills in the era of artificial intelligence](https://www.wearedevelopers.com/videos/1753-wearedevelopers-live-spicy-vanilla-web-css-magic-more) (from "WeAreDevelopers LIVE – Spicy Vanilla Web, CSS Magic & More")
- [Using artificial intelligence to reimagine developer experience](https://www.wearedevelopers.com/videos/1546-ai-pair-programming-with-github-copilot-at-sap-looking-back-looking-forward) (from "AI Pair Programming with GitHub Copilot at SAP: Looking Back, Looking Forward!")
- [The transforming role of developers in the AI era](https://www.wearedevelopers.com/videos/100337-user-1st-technology-2nd-stop-building-ai-nobody-uses-start-delivering-real-business-outcomes) (from "User 1st! Technology 2nd! Stop building AI nobody uses - start delivering real business outcomes")

## Related Articles

- [How to Monetise Your Code and Win a Share of $1 Million with Apify](https://www.wearedevelopers.com/magazine/676-how-to-monetise-your-code-and-win-a-share-of-1-million-with-apify)
- [What Developers Are Building to Win $1 Million with Apify](https://www.wearedevelopers.com/magazine/677-what-developers-are-building-to-win-1-million-with-apify)
- [Dev Digest 132 - Binging WADFlix?](https://www.wearedevelopers.com/magazine/473-dev-digest-132-binging-wadflix)
- [Dev Digest 108 - Git off my cloud!](https://www.wearedevelopers.com/magazine/407-dev-digest-108-git-off-my-cloud)

## Related Jobs

- [AI Full Stack Engineer](https://www.wearedevelopers.com/jobs/ext/1354435-ai-full-stack-engineer) at **Almedia**
- [AI Software Engineer (Germany)](https://www.wearedevelopers.com/jobs/48317-ai-software-engineer-germany) at **Sunhat**
- [Senior AI Agent Software Engineer (Go, Python) (m/f/x)](https://www.wearedevelopers.com/jobs/48277-senior-ai-agent-software-engineer-go-python-m-f-x) at **Dynatrace**
- [Senior Engineer, Infrastructure Platform](https://www.wearedevelopers.com/jobs/ext/328836-senior-engineer-infrastructure-platform) at **Intercom, Inc.**
- [Principal Software Engineer](https://www.wearedevelopers.com/jobs/ext/267587-principal-software-engineer) at **Twilio**
- [Mid/Senior Full-Stack Engineer (Web-first)](https://www.wearedevelopers.com/jobs/ext/1210833-mid-senior-full-stack-engineer-web-first) at **SMG Swiss Marketplace Group**