High latency in applications built on data warehouses creates a poor user experience and presents a significant challenge for developers.
Data warehouses use OLAP for complex, low-volume queries on large datasets, contrasting with OLTP's high-volume, simple transactions.
User-perceived performance is impacted by network delays and data scan times, making sub-second responses a critical goal.
BigQuery's cache only works for identical queries and its concurrency is capped per project, impacting real-world application performance.
Load testing reveals that BigQuery maintains a consistent query latency of around two seconds regardless of user concurrency up to its hard limit.
Cube provides a semantic layer over data warehouses, enabling caching, pre-aggregations, and access control to build fast data apps.
A local Cube instance can be configured using Docker Compose to connect to BigQuery and automatically generate data schemas.
Pre-aggregations act as materialized views that store condensed query results, reducing a query's response time from seconds to milliseconds.
Benchmarks show that using Cube's pre-aggregation layer results in a nearly five-fold performance increase over querying BigQuery directly.
The discussion covers when to implement a caching layer, how Cube improves performance, and its utility for medium-sized databases.
49:57Andrew Burnett-Thompson & David Burleigh
24:22Jürgen Pilz
42:08Matteo Collina
27:27Michele Riva
28:12Josip Stuhli
37:21Mario Meir-Huber
22:32George Asafev
30:02Abdelrahman Awad
Jobs that call for the skills explored in this talk.
AllianceIT Inc
American CyberSystems
Google LLC
Atlanta, United States of America
Google LLC
New York, United States of America
