WeAreDevelopers Logo WeAreDevelopers
Sign in
For You LIVE
World Congress 2023 • Oct 6, 2023

How building an industry DBMS differs from building a research one

Markus Dreseler

Building a planetary-scale DBMS requires radically different engineering than academic research. Learn why defensive programming, petabyte-scale telemetry, and obsessive reliability beat rapid prototyping in the real world.

Pause
Mute Enter Fullscreen
#1 about 3 min

Building a research database prototype from scratch

Developing an end-to-end open-source in-memory database enables unobstructed academic experimentation.

#2 about 4 min

Understanding decoupled compute and central storage architecture

Separating central storage from an independent compute layer scales resources efficiently without hardware constraints.

#3 about 3 min

Comparing textbook query planning to industry reality

Examining how commercial databases parse, optimize, and execute logical query plans mirrors academic prototypes.

#4 about 4 min

Implementing complex customer requirements and obscure features

Handling collations, evolving time zones, and niche operations like match recognize introduces significant engineering overhead.

#5 about 6 min

Leveraging telemetry to target query performance improvements

Running widespread execution profilers extracts actionable production trends instead of relying on artificial benchmarks.

#6 about 4 min

Implementing extensive automated testing for query correctness

Guaranteeing result consistency requires continuous static analysis, query permutation, and historical query re-execution checks.

#7 about 3 min

Safeguarding code deployments via granular parameter protection

Isolating new code paths with internal feature parameters mitigates release rollbacks and enables progressive rollouts.

#8 about 5 min

Managing query edge cases and hardware failures

Rotating engineers onto support reveals nondeterministic queries, distributed race conditions, and hidden hardware degradation.

#9 about 3 min

Reconciling development speed with huge operational impact

While rigorous safety mechanisms prevent quick iteration, operating at massive scale compounds the value of optimizations.

Matching moments

2:36 min

Modern application stacks and real-time data requirements

Tim Faulkes · LIVE

7:52 min

Audience questions on database performance, deployments, and data migrations

Gregor Bauer Gregor Bauer · LIVE

2:14 min

Solving complex platform architecture challenges at an enterprise scale

Maria Apazoglou · Coffee With Developers

3:16 min

Q&A on database vendor lock-in and alternative architectural choices

George Asafev · World Congress 2023

3:29 min

Database infrastructure and tech industry history

10:58 min

Exploring query boundaries, data storage, and architecture limits

Denis Washington +1 · World Congress 2021

More from World Congress 2023

The year 3000, a brief history of Web Development thumbnail
25:30
over 2 years ago

The year 3000, a brief history of Web Development

Lorenzo Pieri

Demografie nutzen um jedem Bewerber das passende Angebot zu machen! thumbnail
8:52
almost 3 years ago

Demografie nutzen um jedem Bewerber das passende Angebot zu machen!

Nathalie Gierth

Guided Memory Management: Rust's Ownership Model thumbnail
0:18
almost 3 years ago

Guided Memory Management: Rust's Ownership Model

Stefan Baumgartner

Passwordless future: WebAuthn and Passkeys in practice thumbnail
32:32
almost 3 years ago

Passwordless future: WebAuthn and Passkeys in practice

Clemens Hübner

External Secrets Operator: the secrets management toolbox for self-sufficient teams thumbnail
30:07
almost 3 years ago

External Secrets Operator: the secrets management toolbox for self-sufficient teams

Moritz Johner

Building Reliable Serverless Applications with AWS CDK and Testing thumbnail
25:18
almost 3 years ago

Building Reliable Serverless Applications with AWS CDK and Testing

Raphael Manke

Related videos

Leveraging Real time data in FSIs thumbnail
41:51
LIVE • almost 3 years ago

Leveraging Real time data in FSIs

Tim Faulkes

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases thumbnail
32:46
World Congress 2024 • about 2 years ago

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases

Wei Hu

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases thumbnail
28:41
World Congress 2025 • about 1 year ago

Fault Tolerance and Consistency at Scale: Harnessing the Power of Distributed SQL Databases

Wei Hu

Swapping a Data Warehouse at Runtime: Zero-Downtime Migration Without Changing a Single Client thumbnail
30:28
World Congress 2026 Europe • 3 months ago

Swapping a Data Warehouse at Runtime: Zero-Downtime Migration Without Changing a Single Client

Max Fischer, Michael O'Toole

TiDB, One Layer at a Time: How Distributed SQL Became an Agentic AI Backbone thumbnail
27:27
World Congress 2026 Europe • 3 months ago

TiDB, One Layer at a Time: How Distributed SQL Became an Agentic AI Backbone

Daniël van Eeden, Mattias Jonsson

From Messy Queries to Scalable Systems - How Data Engineering actually works thumbnail
31:15
World Congress 2026 Europe • 3 months ago

From Messy Queries to Scalable Systems - How Data Engineering actually works

Sandhya Menon

Related articles

From learning to earning

Jobs that call for the skills explored in this talk.