Kevin Tung·Nov 1Our Server Could Handle 100 Requests a Second. In a 3x Spike, At Most 8 Beat the Deadline.Unbounded FIFO goodput fell to 2.5–8.0/s in the spike and 0 after. Adaptive LIFO held ~100/s with p99 ~52–58ms.
thataidev·Oct 24What Is Backpressure in AI Systems? When Requests Arrive Faster Than They Can Be ProcessedImagine your AI application is working perfectly.
Android Expert·Aug 3Taming the 9 AM Thundering Herd: A Layered Defense for Distributed Load ShapingHow client jitter, edge caching, rate limiting, and single-flight coalescing tame a 12,500 req/s spike
Jasmine Park·Jul 15Our LLM service had no backpressure. The provider got 7x slower and our p99 got 25x worse.TL;DR. Our summarization endpoint holds a p99 under 3 seconds. On 21/05 our provider degraded: median call time went from 620ms to about…A response icon1
