WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2025 • Aug 20, 2025 • Session details

Scrape, Train, Predict: The Lifecycle of Data for AI Applications

Vidas Bacevičius

Public datasets are too stale to fuel modern LLMs. Learn how engineers are using AI-driven adaptive parsing to bypass anti-bot walls and build resilient data extraction pipelines.

Pause
Mute Enter Fullscreen
#1 about 3 min

Understanding the basic mechanics of automated web scraping

Proxy servers mask scraper locations to safely automate HTTP requests for large-scale data extraction.

#2 about 3 min

Analyzing historical data and predicting future market trends

Continuous web scraping enables tracking search engine rankings, pricing strategies, and demand forecasting.

#3 about 3 min

Addressing the limitations of common public training datasets

Static public datasets lack the recency, scale, and multimodality needed for competitive AI training operations.

#4 about 2 min

Tailoring custom web scrapers for artificial intelligence training

Building customized in-house scrapers enables access to real-time, targeted, and multimodal web data streams.

#5 about 3 min

Enhancing generative models with real-time external data retrieval

Integrating fast web scraping with retrieval-augmented generation feeds language models with live external knowledge.

#6 about 4 min

Overcoming modern anti-bot mechanisms and network access restrictions

Sophisticated blocking methods, CAPTCHA challenges, and geolocation restrictions complicate modern automated web scraping workflows.

#7 about 4 min

Identifying deceptive server content and parsing messy HTML

Brittle XPath selectors and deceptive server responses make traditional manual HTML parsing difficult to maintain.

#8 about 2 min

Training classification models for resilient automated HTML parsing

Machine learning algorithms process labeled historical documents to automatically identify structural element patterns across websites.

#9 about 3 min

Validating website responses and adapting to layout transformations

Artificial intelligence feedback loops validate true server responses and dynamically extract targeted elements from evolving layouts.

#10 about 4 min

Automating data extraction pipelines using natural language assistants

AI assistants parse natural language prompts to automatically generate reliable extraction instructions for complex e-commerce platforms.

Matching moments

5:54 min

The technical evolution of modern web scraping infrastructure

Chris Heilmann +4 · LIVE

2:54 min

Technical workflow for generating self-healing scrapers

Ariel Shulman Ariel Shulman +1 · WWC Europe 2026

2:14 min

Assessing the future of AI in web performance optimization

Perf + AI

49 sec

Role of web scraping in training language models

Jan Curn Jan Curn · WWC 2025

1:35 min

Hypocrisy in artificial intelligence data scraping practices

Chris Heilmann +2 · LIVE

4:28 min

Exploring artificial intelligence as a solution for web accessibility

Chris Heilmann +2 · LIVE

More from World Congress 2025

GenAI Security: Navigating the Unseen Iceberg thumbnail
24:33
10 months ago

GenAI Security: Navigating the Unseen Iceberg

Maish Saidel-Keesing

Behind the Code: How Women Are Powering the Future of AI thumbnail
30:28
12 months ago

Behind the Code: How Women Are Powering the Future of AI

Alexandra Wudel, Laura Moritz, Madalina Florean

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise? thumbnail
28:12
12 months ago

The AI Hype Filter: What’s Real, What’s Investable, What’s Noise?

Justus Lumpe, Konstantin Vinogradov, Nils Eiteneyer, Stephan Wirris

AI in Leadership: How Technology is Reshaping Executive Roles thumbnail
28:19
12 months ago

AI in Leadership: How Technology is Reshaping Executive Roles

Bernd Datler, Jasmin Kaiser, Jeff Hausmann, Sonja Alvarez

The AI-Ready Stack: Rethinking the Engineering Org of the Future thumbnail
28:09
12 months ago

The AI-Ready Stack: Rethinking the Engineering Org of the Future

Alex Laubscher, Jan Oberhauser, Keno Dreßel, Mirko Novakovic

AI Sovereignty: What Does It Take? thumbnail
29:13
12 months ago

AI Sovereignty: What Does It Take?

Boris Hecker, Clemens Wasner, Robin Hermann, Sonja Alvarez

Upcoming sessions on this topic

Open session

World Congress 2026 North America

Small LLM in your Browser: Huge Opportunities for Web Applications

Daniel Ostrovsky

UI/UX Architect at Payoneer | AI Architect | Full Cycle Development Expert | Public Speaker | Open Source Contributor |

Daniel Ostrovsky
Open session

World Congress 2026 North America

Building Pragmatic AI: 10 AI Features Your Users Actually Want

Jonathan "J." Tower

.NET Foundation Board | 12x Microsoft MVP | Founder & Consultant

Jonathan "J." Tower
Open session

World Congress 2026 North America

AI-Driven API Design

Mike Amundsen

Author, Adviser, Trainer

Mike Amundsen
Open session

World Congress 2026 North America

Building Stuff with GenAI - The Open Minded Workshop beyond OpenAI

Andreas Erben

CTO for Applied AI and Metaverse at daenet

Andreas Erben
Open session

World Congress 2026 North America

You Can’t Re-Run Sunlight: Designing ML Data Architectures for Physical AI

An Phan

Senior Data Infrastructure Engineer @ Hippo Harvest

An Phan
Open session

World Congress 2026 North America

The Broken Rung: How AI is Rebuilding Software Development from the Ground Up

Tomislav Tipurić

Chief Technology Officer, Nephos

Tomislav Tipurić

Related videos

How to scrape modern websites to feed AI agents thumbnail
19:20
WWC 2025 • 12 months ago

How to scrape modern websites to feed AI agents

Jan Curn

From clicks to cribs - How to find your dream home with web scraping thumbnail
26:08
WWC 2023 • almost 3 years ago

From clicks to cribs - How to find your dream home with web scraping

Alexander Lichter

AI in the Open and in Browsers - Tarek Ziadé thumbnail
53:54
Coffee With Developers • 7 months ago

AI in the Open and in Browsers - Tarek Ziadé

Tarek Ziadé

Unlocking Value from Data: The Key to Smarter Business Decisions- thumbnail
28:29
WWC 2025 • about 1 year ago

Unlocking Value from Data: The Key to Smarter Business Decisions-

Farooq Sheikh, Kapil Gupta, Taqi Jaffri, Tomislav Tipurić

WeAreDevelopers Live: Browser Extensions, Honey Scam, Jailbreaking LLMs and more thumbnail
1:01:25
LIVE • over 1 year ago

WeAreDevelopers Live: Browser Extensions, Honey Scam, Jailbreaking LLMs and more

Chris Heilmann, Daniel Cranney

WeAreDevelopers LIVE - Web Performance in an AI World thumbnail
1:07:59
LIVE • 26 days ago

WeAreDevelopers LIVE - Web Performance in an AI World

Chris Heilmann, Daniel Cranney

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.