There Is No Such Thing as a Fair DBaaS Benchmark

If you’ve ever tried to benchmark a managed database, you know the frustration: same vCPU, same RAM, same PostgreSQL version — and completely different results. Designing a “fair” DBaaS benchmark today is not just about query workloads and metrics. It’s about understanding hidden IO limits, burst credits, storage backends, noisy neighbors, throttling behavior, and pricing tiers that influence performance in ways your dashboard doesn’t show. In this talk, we break down what “fair” actually means in practice. Should you compare systems at equal cost, equal resources, or equal performance targets? Each approach answers a different engineering question — and can lead to opposite conclusions. Using real-world examples from benchANT benchmarking projects, we’ll show where common benchmarking assumptions fail, how cloud abstractions distort performance expectations, and why reproducibility alone doesn’t guarantee comparability. You’ll leave with practical guidance on how to design benchmarks that reflect production reality — so you can evaluate DBaaS platforms based on engineering truth, not marketing claims.