Lars Kölker
Data is Key: Scraping Metadata from Websites
#1about 2 minutes
How social media sites generate link previews
Social media platforms scrape hidden metadata like titles and descriptions from URLs to transform a simple link into a rich preview.
#2about 1 minute
Defining web scraping and its primary use cases
Web scraping is the practice of gathering data from websites without an API, often used when APIs are missing, rate-limited, or too expensive.
#3about 2 minutes
Why CSS selector-based scraping is brittle
Relying on specific CSS selectors for scraping creates a fragile solution that is tied to a single site and breaks whenever the source code changes.
#4about 4 minutes
Generic scraping with schema.org and JSON-LD
Schema.org provides a standardized vocabulary for structured data, enabling the creation of generic scrapers using formats like JSON-LD.
#5about 5 minutes
Using meta tags for structured data extraction
Protocols like Open Graph (OGP) and Twitter Cards extend standard HTML meta tags to provide rich, structured metadata for social sharing and scraping.
#6about 4 minutes
The oEmbed protocol for embedded content
The oEmbed protocol offers a standardized endpoint for retrieving embeddable representations of a URL, which is essential for sites like Instagram.
#7about 1 minute
Showcasing a powerful multi-protocol scraper
A demonstration shows how combining different scraping techniques can extract rich information, including product prices and author images, from various websites.
#8about 3 minutes
Q&A on legality, rate limits, and frameworks
The speaker addresses audience questions regarding the legality of scraping, managing rate limits, and recommended frameworks like Beautiful Soup.
Related jobs
Jobs that call for the skills explored in this talk.
Matching moments
40:33 MIN
Technical SEO strategies for LLMs and new crawlers
WeAreDevelopers LIVE – SEO, GEO, AI Slop & More
17:41 MIN
Presenting live web scraping demos at a developer conference
Tech with Tim at WeAreDevelopers World Congress 2024
29:02 MIN
Designing a scalable architecture for data collection
Cracking the Code: Decoding Anti-Bot Systems!
04:01 MIN
Navigating the complexities of modern web scraping
How to scrape modern websites to feed AI agents
19:38 MIN
Solving scaling challenges in web data collection
Tech with Tim at WeAreDevelopers World Congress 2024
07:29 MIN
Cleaning messy HTML and scaling data extraction
How to scrape modern websites to feed AI agents
1:09:47 MIN
Extracting structured data from web pages
How to achieve web automation with UiPath
22:16 MIN
A step-by-step methodology for building robust scrapers
Cracking the Code: Decoding Anti-Bot Systems!
Featured Partners
Related Videos
26:08From clicks to cribs - How to find your dream home with web scraping
Alexander Lichter
26:15Scrape, Train, Predict: The Lifecycle of Data for AI Applications
Vidas Bacevičius
19:20How to scrape modern websites to feed AI agents
Jan Curn
48:10Data Science on Software Data
Markus Harrer
28:16The Great API Debate: REST, GraphQL, or gRPC?
Alexis Yushin
32:12Bringing Clarity to Event Streams: Enabling Analytics and AI Through Rich Metadata
Clemens Vasters
28:55Awful APIs: A History Lesson in Industry Mistakes and Mishaps
James Seconde
43:53Web APIs you might not know about
Sasha Shynkevich
From learning to earning
Jobs that call for the skills explored in this talk.
Domain Architect Ricardo Platform (f/m/d) | 80-100% | Hybrid working model | Valbonne France
SMG Swiss Marketplace Group
Canton de Valbonne, France
Senior
Domain Architect Ricardo Platform (f/m/d) | 80-100% | Hybrid working model | Zürich Switzerland
SMG Swiss Marketplace Group
Sachseln, Switzerland
Senior
Senior PHP Developer (NL based only)
Online Payment Platform
Delft, Netherlands
€75-95K
Senior
PHP
MySQL
Laravel
