Tag: slo

5 articles

• LLM Serving Overload Protection in Production: Guarding Tail Latency with Queue SLO, Backpressure, and Elastic Autoscaling

  A production-grade guide for LLM inference services: using queue wait time, TTFT SLO, backpressure, and elastic autoscaling to handle traffic spikes, avoid scaling lag, tail latency blowups, and unbounded queuing. Includes key metrics, configuration, launch checklist, and common pitfalls.

  08/26/2026 llm-serving / autoscaling / backpressure / slo / vllm

• LLM Serving Autoscaling in Production: Safe Scaling with Queue Depth, P95 Latency, and GPU Utilization

  A deep dive into production-grade autoscaling for LLM serving. Build a multi-layer control loop using queue depth, P95 latency, token throughput, and GPU utilization, with warm pools, admission control, cost boundaries, and cooldown strategies to meet inference SLOs under cost constraints.

  07/09/2026 LLM Serving / Autoscaling / GPU Inference / P95 Latency / SLO / Scaling

• LLM Inference Capacity Planning in Production: Estimating GPU Headroom with Token Throughput Stress Tests and Queue Depth

  A practical guide to transforming LLM inference capacity from reactive scaling to measurable, estimable, and alertable capacity engineering. Covers token throughput, queue depth, P99 TTFT, GPU headroom, and a pre-launch checklist to help teams identify bottlenecks and make quantitative scaling decisions.

  07/05/2026 LLM Serving / Capacity Planning / GPU Inference / SLO / vLLM / Token Throughput

• LLM Serving Overload Protection: SLO-Driven Autoscaling and Admission Control

  A practical guide to building overload protection for LLM inference services using SLO-driven autoscaling and admission control. Covers queue length, token pressure, TTFT/TPOT, and tenant quotas to prevent system collapse under traffic spikes.

  07/01/2026 LLM Serving / Autoscaling / SLO / Admission Control / Overload Protection / Inference Infrastructure

• Prefill-Decode Disaggregation in Practice: Governing TTFT and TPOT for Long-Context LLM Serving

  A practical guide to Prefill-Decode disaggregation for LLM serving: principles, production architecture, monitoring, and a go-live checklist to govern TTFT and TPOT under long-context, high-concurrency workloads.

  06/30/2026 LLM Serving / Prefill Decode / Inference Optimization / KV Cache / SLO