01engineering.zalando.com
Thundering herd of circuit breakers
A client-side circuit breaker trapped traffic in a loop, blocking about 95% of requests for hours despite a healthy service. A custom OpenTelemetry trace visualizer showed that half-open probes succeeded on a three-pod, under-provisioned service, but closing the breaker released a thundering herd; queues exceeded 100 ms, requests timed out, and the breaker reopened. Pre-scaling or force-closing allowed HPA capacity to catch up. CPU-based HPA missed three-second bursts, exposing limits of aggregate metrics and representative probes.