Lars Kölker
Data is Key: Scraping Metadata from Websites
#1about 2 minutes
How social media sites generate link previews
Social media platforms scrape hidden metadata like titles and descriptions from URLs to transform a simple link into a rich preview.
#2about 1 minute
Defining web scraping and its primary use cases
Web scraping is the practice of gathering data from websites without an API, often used when APIs are missing, rate-limited, or too expensive.
#3about 2 minutes
Why CSS selector-based scraping is brittle
Relying on specific CSS selectors for scraping creates a fragile solution that is tied to a single site and breaks whenever the source code changes.
#4about 4 minutes
Generic scraping with schema.org and JSON-LD
Schema.org provides a standardized vocabulary for structured data, enabling the creation of generic scrapers using formats like JSON-LD.
#5about 5 minutes
Using meta tags for structured data extraction
Protocols like Open Graph (OGP) and Twitter Cards extend standard HTML meta tags to provide rich, structured metadata for social sharing and scraping.
#6about 4 minutes
The oEmbed protocol for embedded content
The oEmbed protocol offers a standardized endpoint for retrieving embeddable representations of a URL, which is essential for sites like Instagram.
#7about 1 minute
Showcasing a powerful multi-protocol scraper
A demonstration shows how combining different scraping techniques can extract rich information, including product prices and author images, from various websites.
#8about 3 minutes
Q&A on legality, rate limits, and frameworks
The speaker addresses audience questions regarding the legality of scraping, managing rate limits, and recommended frameworks like Beautiful Soup.
Related jobs
Jobs that call for the skills explored in this talk.
Featured Partners
Related Videos
43:52Web-based Information Visualization
Johanna Schmidt
50:38Python Data Visualization @ Deepnote (w/ PyViz overview)
Radovan Kavický
26:08From clicks to cribs - How to find your dream home with web scraping
Alexander Lichter
26:15Scrape, Train, Predict: The Lifecycle of Data for AI Applications
Vidas Bacevičius
19:20How to scrape modern websites to feed AI agents
Jan Curn
48:10Data Science on Software Data
Markus Harrer
28:16The Great API Debate: REST, GraphQL, or gRPC?
Alexis Yushin
32:12Bringing Clarity to Event Streams: Enabling Analytics and AI Through Rich Metadata
Clemens Vasters
From learning to earning
Jobs that call for the skills explored in this talk.
