WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2023 • Sep 27, 2023

Data is Key: Scraping Metadata from Websites

Lars Kölker

Tired of scrapers breaking after minor UI updates? Stop relying on brittle CSS selectors. Build resilient metadata extractors using JSON-LD and Open Graph protocols.

Pause
Mute Enter Fullscreen
#1 about 2 min

How social media platforms generate link previews

Social platforms execute transformations that read hidden markup to display titles, descriptions, and dynamic content on links.

#2 about 2 min

Defining web scraping and recognizing proper use cases

Scraping becomes essential when target systems lack transparent APIs or when existing endpoints become prohibitively expensive.

#3 about 3 min

Limitations of specific CSS selectors for web scraping

Hardcoding CSS selectors restricts scraper versatility and creates breaking points whenever an underlying website updates its structure.

#4 about 2 min

Using schema.org for generic structured data extraction

Schema.org libraries define standard rules for machine-readable types and generalized properties across various content models.

#5 about 2 min

Extracting schema attributes mapped to JSON-LD structures

Adopting Google's JSON-LD format reveals precise relational metadata stored inside predefined script elements.

#6 about 2 min

Extracting basic key-value data with common meta elements

Standard HTML attributes like application name and description offer a foundational layer for key-value extraction routines.

#7 about 3 min

Using Open Graph Protocol to construct rich objects

Evaluating Open Graph specification tags accesses dedicated attribute values originally standardized by Facebook for media sharing.

#8 about 2 min

Extracting Twitter Cards markup as valuable scraping fallbacks

Falling back on Twitter-specific metadata summary tags preserves robust link scraping capabilities whenever Open Graph formats miss details.

#9 about 4 min

Utilizing the oEmbed format to query rich endpoints

Fetching an endpoint defined via oEmbed's alternate link attributes returns complex API representations of deep resources.

#10 about 2 min

Demonstrating successful metadata scraping techniques for generic previews

Chaining standard schema fallbacks populates functional link previews containing critical information like live pricing and stock changes.

#11 about 2 min

Navigating legality boundaries and request rate limits safely

Reviewing platform terms of service and restricting endpoint concurrency significantly reduces the compliance risk involved in external scraping protocols.

#12 about 2 min

Selecting suitable extraction frameworks for structured schema parsers

Dedicated Python libraries streamline generalized document parsing, though JavaScript environments often require custom logic to evaluate JSON-LD schemas effectively.

Matching moments

5:54 min

The technical evolution of modern web scraping infrastructure

Chris Heilmann +4 · LIVE

1:09 min

Exploring data collection challenges from user-generated platforms

Linda Mohamed · LIVE

2:54 min

Technical workflow for generating self-healing scrapers

Ariel Shulman Ariel Shulman +1 · WWC Europe 2026

1:45 min

Concluding thoughts on scraper tools and audience questions

Alexander Lichter · WWC 2023

1:20 min

Optimizing social media link previews with header attributes

Chris Heilmann +2 · LIVE

6:57 min

Scraping web data by intercepting JSON HTTP responses

Miki Lombardi · JS Congress

More from World Congress 2023

The year 3000, a brief history of Web Development thumbnail
25:30
over 2 years ago

The year 3000, a brief history of Web Development

Lorenzo Pieri

Demografie nutzen um jedem Bewerber das passende Angebot zu machen! thumbnail
8:52
over 2 years ago

Demografie nutzen um jedem Bewerber das passende Angebot zu machen!

Nathalie Gierth

Guided Memory Management: Rust's Ownership Model thumbnail
0:18
over 2 years ago

Guided Memory Management: Rust's Ownership Model

Stefan Baumgartner

Passwordless future: WebAuthn and Passkeys in practice thumbnail
32:32
over 2 years ago

Passwordless future: WebAuthn and Passkeys in practice

Clemens Hübner

External Secrets Operator: the secrets management toolbox for self-sufficient teams thumbnail
30:07
over 2 years ago

External Secrets Operator: the secrets management toolbox for self-sufficient teams

Moritz Johner

Building Reliable Serverless Applications with AWS CDK and Testing thumbnail
25:18
over 2 years ago

Building Reliable Serverless Applications with AWS CDK and Testing

Raphael Manke

Upcoming sessions on this topic

Open session

World Congress 2026 North America

Bluesky's Open Source Moderation Tools: LLM-based Event Detection in Python

Alex Garnett

Developer Relations, Bluesky

Alex Garnett
Open session

World Congress 2026 North America

From Guesswork to Governance: Data Contracts Bring API Discipline to Apache Kafka

Sandon Jacobs

Senior Developer Advocate at IBM

Sandon Jacobs
Open session

World Congress 2026 North America

Beyond SQL Generation: How to Teach Agents What Your Database Actually Means

Kris Jenkins

Lead Developer Advocate at Snowflake

Kris Jenkins
Open session

World Congress 2026 North America

Small LLM in your Browser: Huge Opportunities for Web Applications

Daniel Ostrovsky

UI/UX Architect at Payoneer | AI Architect | Full Cycle Development Expert | Public Speaker | Open Source Contributor |

Daniel Ostrovsky
Open session

World Congress 2026 North America

Headroom: A Context Optimization Layer for LLM Applications

Tejas Chopra

Senior Software Engineer at Netflix

Tejas Chopra
Open session

World Congress 2026 North America

You Can’t Re-Run Sunlight: Designing ML Data Architectures for Physical AI

An Phan

Senior Data Infrastructure Engineer @ Hippo Harvest

An Phan

Related videos

Scrape, Train, Predict: The Lifecycle of Data for AI Applications thumbnail
26:15
WWC 2025 • 12 months ago

Scrape, Train, Predict: The Lifecycle of Data for AI Applications

Vidas Bacevičius

From clicks to cribs - How to find your dream home with web scraping thumbnail
26:08
WWC 2023 • almost 3 years ago

From clicks to cribs - How to find your dream home with web scraping

Alexander Lichter

How to scrape modern websites to feed AI agents thumbnail
19:20
WWC 2025 • 12 months ago

How to scrape modern websites to feed AI agents

Jan Curn

Bringing Clarity to Event Streams: Enabling Analytics and AI Through Rich Metadata thumbnail
32:12
WWC 2025 • 12 months ago

Bringing Clarity to Event Streams: Enabling Analytics and AI Through Rich Metadata

Clemens Vasters

WeAreDevelopers Live: Browser Extensions, Honey Scam, Jailbreaking LLMs and more thumbnail
1:01:25
LIVE • over 1 year ago

WeAreDevelopers Live: Browser Extensions, Honey Scam, Jailbreaking LLMs and more

Chris Heilmann, Daniel Cranney

WeAreDevelopers LIVE – SEO, GEO, AI Slop & More thumbnail
1:02:24
LIVE • 9 months ago

WeAreDevelopers LIVE – SEO, GEO, AI Slop & More

Chris Heilmann, Daniel Cranney, Simon Cox

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.