Kubernetes scaling best practices for production workloads

Scaling in Kubernetes is often treated as an HPA configuration task. In production, the result depends on a much larger chain: container resource requests, metrics freshness, application startup time, node provisioning, scheduling constraints, traffic readiness, and downstream dependencies.

The objective is not simply to add replicas. It is to meet a service objective during normal traffic, bursts, deployments, and infrastructure disruption—without leaving permanently idle capacity behind.

1. Set resource requests from measured demand

CPU and memory requests influence scheduling, node capacity, and utilisation-based autoscaling. Requests that are too high waste node capacity. Requests that are too low increase throttling, evictions, and the chance that the scheduler packs workloads onto nodes that cannot support real demand.

Begin with workload-level evidence: CPU usage and throttling, memory working set, OOM kills, request latency, queue depth, and behaviour during deployments or batch work. Use percentiles and known peaks rather than one average value.

Separate requests from limits intentionally

• Set CPU requests so scheduling and HPA calculations have a useful baseline.
• Set memory requests with enough headroom for normal variation and startup behaviour.
• Treat CPU limits carefully when throttling would damage latency.
• Treat memory limits as a hard failure boundary because exceeding them can terminate the container.
• Review sidecars separately; their usage can distort pod-level utilisation.

Resource recommendations are a starting point. Validate changes against application service-level objectives before rolling them across every workload.

2. Configure Horizontal Pod Autoscaler around demand

The Kubernetes Horizontal Pod Autoscaler periodically adjusts replica count to match observed metrics. When scaling on CPU utilisation, the calculation depends on CPU requests. If the relevant requests are missing, HPA cannot calculate that utilisation correctly.

Choose a signal that represents load

CPU works well for CPU-bound services whose usage increases with traffic. It is less useful for queue workers, network-bound services, or applications where latency rises before CPU does. In those cases, consider custom or external metrics such as queue depth, concurrent work, requests per second, or a carefully selected business workload signal.

• Keep a minimum replica count that can serve normal traffic safely.
• Set a maximum that respects downstream and regional constraints.
• Use multiple metrics when a single signal can miss real saturation.
• Configure scale-up behaviour for bursts and scale-down stabilisation to avoid oscillation.
• Account for application warm-up before new replicas receive traffic or contribute misleading metrics.

HPA is not capacity planning. If the cluster cannot provision or place the replicas HPA requests, the workload still fails to scale.

3. Coordinate workload scaling with node autoscaling

Pod and node autoscaling solve different problems. HPA changes workload replicas. Node autoscaling adds or removes compute capacity when pods cannot be scheduled or when nodes remain underutilised.

Kubernetes node autoscaling guidance highlights that node provisioning decisions rely on pod resource requests. Inflated requests can force unnecessary nodes; unrealistic low requests can create unstable bin-packing and performance.

Design node pools around scheduling reality

• Keep node families and sizes understandable; too many pools make placement and consolidation harder to reason about.
• Check topology constraints, affinities, taints, persistent volumes, GPUs, architecture, and zone availability.
• Measure time from an unschedulable pod to a ready application replica.
• Use disruption-tolerant workloads for Spot capacity and maintain fallback capacity where required.

4. Protect availability during scaling and change

A replica should not receive production traffic merely because its process started. Kubernetes startup, readiness, and liveness probes serve different purposes:

• Startup probes protect slow-starting applications from premature liveness failures.
• Readiness probes decide whether a pod should receive traffic.
• Liveness probes recover containers that are running but cannot make progress.

Add graceful termination, appropriate rolling-update settings, and a PodDisruptionBudget where voluntary disruption must preserve a minimum level of availability. Ensure that the budget does not make legitimate cluster maintenance impossible.

5. Measure whether scaling actually works

A healthy autoscaler is one that keeps the application inside its service objective with acceptable cost—not one that changes replica counts frequently.

• Track current and desired replicas alongside the driving metric.
• Alert on pending pods, failed scheduling, and node-provisioning delay.
• Correlate scale events with latency, errors, throttling, and saturation.
• Watch readiness time and the delay before new capacity serves traffic.
• Measure node fragmentation and unused requested capacity.
• Review scale-down events for connection draining and workload churn.

Keep the view workload-centric. Cluster-average CPU can look healthy while a single customer-facing deployment is throttled or waiting for nodes.

6. Validate scaling with production-like load

Test the complete scaling loop before trusting it in production: demand rises, metrics update, HPA requests replicas, the scheduler places pods, nodes are provisioned if necessary, the application starts, probes pass, and traffic reaches healthy replicas.

1. Reproduce normal traffic, a rapid burst, and a sustained peak.
2. Test when the cluster has spare capacity and when new nodes are needed.
3. Observe downstream dependencies such as databases and external APIs.
4. Verify behaviour during deployment, node drain, and zone disruption.
5. Confirm that scale-down does not terminate active work or connections.
6. Record recovery time, service-level impact, and cost after the test.

Production scaling is successful only when the whole path meets the required response time. A fast HPA decision cannot compensate for ten-minute node provisioning or a five-minute application warm-up.

7. Production Kubernetes scaling checklist

Common Kubernetes scaling mistakes

• Enabling HPA before fixing missing or unrealistic requests.
• Scaling only on CPU when another signal represents demand better.
• Ignoring application startup and readiness time.
• Adding replicas faster than a database or API can support them.
• Leaving topology and scheduling constraints out of capacity tests.
• Optimizing node utilisation without protecting workload reliability.