Your Thread Pool Is Lying to You — Sizing Concurrency from Rate Limits and Latency, Not Guesswork

Here’s a scenario most API teams have lived through: you set a rate limit of 40 TPS, your downstream averages 2.3 seconds per call, and your executor tops out at 25 threads. The limiter is happily approving work. The pool can’t actually run it. Throughput collapses and nobody can explain why, because every component looks correctly configured in isolation. The root cause is that teams size thread pools, HTTP connection pools, and database pools independently — usually from defaults or past incidents — instead of from the relationship between them. The math isn’t complicated. It’s just rarely done. This talk covers one formula and its consequences: in-flight work equals throughput times latency. We’ll apply it to executor pools, HTTP client pools, database connection pools, and distributed limiter client pools. I’ll show the configuration mistakes that create each mismatch, what the symptoms look like in production, and the sizing cascade that keeps the gates consistent with each other. If you’ve ever stared at a dashboard where the rate limiter says everything is fine but p99 latency is through the roof, this talk is for you.