Adnan Rahic

Making Data Warehouses fast. A developer's story.

A developer went from 1.5-second BigQuery responses to under 500ms with 100 concurrent users. Here's the open-source tool they used to do it.

Making Data Warehouses fast. A developer's story.
#1about 3 minutes

The developer's struggle with data warehouse latency

High latency in applications built on data warehouses creates a poor user experience and presents a significant challenge for developers.

#2about 5 minutes

Differentiating between OLAP and OLTP database workloads

Data warehouses use OLAP for complex, low-volume queries on large datasets, contrasting with OLTP's high-volume, simple transactions.

#3about 3 minutes

Understanding the key factors of query latency

User-perceived performance is impacted by network delays and data scan times, making sub-second responses a critical goal.

#4about 7 minutes

Exploring BigQuery's caching and concurrency limitations

BigQuery's cache only works for identical queries and its concurrency is capped per project, impacting real-world application performance.

#5about 4 minutes

Benchmarking BigQuery's performance under concurrent load

Load testing reveals that BigQuery maintains a consistent query latency of around two seconds regardless of user concurrency up to its hard limit.

#6about 2 minutes

Introducing Cube as a semantic analytics API layer

Cube provides a semantic layer over data warehouses, enabling caching, pre-aggregations, and access control to build fast data apps.

#7about 3 minutes

Setting up a local Cube development environment

A local Cube instance can be configured using Docker Compose to connect to BigQuery and automatically generate data schemas.

#8about 5 minutes

How pre-aggregations dramatically improve query speed

Pre-aggregations act as materialized views that store condensed query results, reducing a query's response time from seconds to milliseconds.

#9about 3 minutes

Comparing benchmark results of Cube vs direct BigQuery

Benchmarks show that using Cube's pre-aggregation layer results in a nearly five-fold performance increase over querying BigQuery directly.

#10about 8 minutes

Answering questions on Cube's architecture and use cases

The discussion covers when to implement a caching layer, how Cube improves performance, and its utility for medium-sized databases.

Related jobs
Jobs that call for the skills explored in this talk.

Featured Partners

Related Videos

Database Magic behind 40 Million operations/s24:22

Database Magic behind 40 Million operations/s

Jürgen Pilz

Things I learned while writing high-performance JavaScript applications27:27

Things I learned while writing high-performance JavaScript applications

Michele Riva

Scaling: from 0 to 20 million users28:12

Scaling: from 0 to 20 million users

Josip Stuhli

1, 2, 3... Fastify!42:08

1, 2, 3... Fastify!

Matteo Collina

Single Server, Global Reach: Running a Worldwide Marketplace on Bare Metal in a Cloud-Dominated World50:06

Single Server, Global Reach: Running a Worldwide Marketplace on Bare Metal in a Cloud-Dominated World

Jens Happe

Enjoying SQL data pipelines with dbt26:53

Enjoying SQL data pipelines with dbt

Matthias Niehoff

Uncharted Territories of Web Performance - Andrew Burnett-Thompson and David Burleigh49:57

Uncharted Territories of Web Performance - Andrew Burnett-Thompson and David Burleigh

Andrew Burnett-Thompson, David Burleigh

How building an industry DBMS differs from building a research one30:34

How building an industry DBMS differs from building a research one

Markus Dreseler

From learning to earning

Jobs that call for the skills explored in this talk.

Databricks SQL Data Engineer

Ubique Systems
Municipality of Madrid, Spain

Intermediate
JIRA
YAML
Python
Agile Methodologies
Amazon Web Services (AWS)