Kubernetes scaling best practices for production workloads

Scaling in Kubernetes is often treated as an HPA configuration task. In production, the result depends on a much larger chain: container resource requests, metrics freshness, application startup time, node provisioning, scheduling constraints, traffic readiness, and downstream dependencies.

The objective is not simply to add replicas. It is to meet a service objective during normal traffic, bursts, deployments, and infrastructure disruption—without leaving permanently idle capacity behind.

1. Set resource requests from measured demand

Requests tell the scheduler how much capacity to reserve; CPU requests also provide the denominator for utilisation-based HPA. Oversized requests leave capacity unused. Undersized requests can pack too much real demand onto a node. A low request does not itself impose a CPU throttle.

CPU limits constrain CPU time and can cause throttling. Memory limits can lead to an out-of-memory termination when usage exceeds the boundary. Node-pressure eviction is a separate mechanism. See the official Kubernetes resource requests and limits documentation before changing either setting.

Begin with workload-level evidence: CPU usage and throttling, memory working set, OOM kills, request latency, queue depth, and behaviour during deployments or batch work. Use percentiles and known peaks rather than one average value.

Separate requests from limits intentionally

• Set CPU requests so scheduling and HPA calculations have a useful baseline.
• Set memory requests with enough headroom for normal variation and startup behaviour.
• Treat CPU limits carefully when throttling would damage latency.
• Treat memory limits as a hard failure boundary because exceeding them can terminate the container.
• Review sidecars separately; their usage can distort pod-level utilisation.

Resource recommendations are a starting point. Validate changes against application service-level objectives before rolling them across every workload.

2. Configure Horizontal Pod Autoscaler around demand

The Kubernetes Horizontal Pod Autoscaler periodically adjusts replica count to match observed metrics. When scaling on CPU utilisation, the calculation depends on CPU requests. If the relevant requests are missing, HPA cannot calculate that utilisation correctly.

Choose a signal that represents load

CPU works well for CPU-bound services whose usage increases with traffic. It is less useful for queue workers, network-bound services, or applications where latency rises before CPU does. In those cases, consider custom or external metrics such as queue depth, concurrent work, requests per second, or a carefully selected business workload signal.

• Keep a minimum replica count that can serve normal traffic safely.
• Set a maximum that respects downstream and regional constraints.
• Use multiple metrics when a single signal can miss real saturation.
• Configure scale-up behaviour for bursts and scale-down stabilisation to avoid oscillation.
• Account for application warm-up before new replicas receive traffic or contribute misleading metrics.

HPA is not capacity planning. If the cluster cannot provision or place the replicas HPA requests, the workload still fails to scale.

Worked example: three replicas become four

Consider a CPU-bound API with three ready pods, each containing one container requesting 500m CPU. If each uses 400m, observed utilisation is 80%. At a 60% target, the simplified calculation is:

observed utilisation = 400m / 500m × 100 = 80%
desired replicas = ceil(3 × 80 / 60) = 4

Four pods reserve 2 vCPU instead of 1.5 vCPU. That is requested capacity, not measured usage or a cloud bill. This example assumes fresh metrics, ready pods and evenly distributed load; readiness, missing metrics, tolerance and scaling policies can alter the controller's actual decision.

An autoscaling/v2 HPA manifest

This HPA configuration targets an existing example-api Deployment in a test namespace named scaling-demo. It does not create an application or install a metrics provider. Resource metrics must be available, commonly through Metrics Server, and the containers need CPU requests. Choose replica bounds from your own capacity tests.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: example-api
  namespace: scaling-demo
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: example-api
  minReplicas: 3
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Pods
          value: 1
          periodSeconds: 60

The 60% target is relative to requested CPU. The lower bound keeps three replicas; the upper bound caps this workload at ten. The five-minute scale-down window considers recent recommendations, while the policy permits at most one pod removal per minute. Adjust these values to your workload and capacity needs. See the official HPA walkthrough for a complete sample application and metrics setup.

Before adopting it, confirm that ten replicas fit your node, quota and downstream limits. Keep GitOps or manual replica updates from fighting the autoscaler. Before rollout, test the manifest with your application's startup, readiness and traffic patterns.

3. Coordinate workload scaling with node autoscaling

Pod and node autoscaling solve different problems. HPA changes workload replicas. Node autoscaling adds or removes compute capacity when pods cannot be scheduled or when nodes remain underutilised.

Kubernetes node autoscaling guidance highlights that node provisioning decisions rely on pod resource requests. Inflated requests can force unnecessary nodes; unrealistic low requests can create unstable bin-packing and performance.

Design node pools around scheduling reality

• Keep node families and sizes understandable; too many pools make placement and consolidation harder to reason about.
• Check topology constraints, affinities, taints, persistent volumes, GPUs, architecture, and zone availability.
• Measure time from an unschedulable pod to a ready application replica.
• Use disruption-tolerant workloads for Spot capacity and maintain fallback capacity where required.

4. Protect availability during scaling and change

A replica should not receive production traffic merely because its process started. Kubernetes startup, readiness, and liveness probes serve different purposes:

• Startup probes protect slow-starting applications from premature liveness failures.
• Readiness probes decide whether a pod should receive traffic.
• Liveness probes recover containers that are running but cannot make progress.

Add graceful termination, appropriate rolling-update settings, and a PodDisruptionBudget where voluntary disruption must preserve a minimum level of availability. Ensure that the budget does not make legitimate cluster maintenance impossible.

5. Measure whether scaling actually works

A healthy autoscaler is one that keeps the application inside its service objective with acceptable cost—not one that changes replica counts frequently.

• Track current and desired replicas alongside the driving metric.
• Alert on pending pods, failed scheduling, and node-provisioning delay.
• Correlate scale events with latency, errors, throttling, and saturation.
• Watch readiness time and the delay before new capacity serves traffic.
• Measure node fragmentation and unused requested capacity.
• Review scale-down events for connection draining and workload churn.

Keep the view workload-centric. Cluster-average CPU can look healthy while a single customer-facing deployment is throttled or waiting for nodes.

Read the evidence before changing replica limits

These commands inspect the example workload. Use the intended cluster context and replace the namespace and resource names with your own. For a pending pod, use its actual name in the last command. The official pod debugging guide explains how pod state and scheduler events narrow down a failure.

kubectl config current-context
kubectl -n scaling-demo get hpa example-api
kubectl -n scaling-demo describe hpa example-api
kubectl -n scaling-demo top pods --containers
kubectl -n scaling-demo get pods -o wide
kubectl -n scaling-demo describe pod <pending-pod-name>

6. Validate scaling with production-like load

Test the complete scaling loop before trusting it in production: demand rises, metrics update, HPA requests replicas, the scheduler places pods, nodes are provisioned if necessary, the application starts, probes pass, and traffic reaches healthy replicas.

1. Record the baseline. In an agreed test environment, capture the manifest revision, resource settings, node pool configuration and a representative traffic profile. Set acceptable p95 latency, error rate, readiness delay and dependency load before testing.
2. Run normal demand. Record current and desired replicas, CPU usage and requests, ready pods, latency and errors. Confirm the scaling metric is available and follows demand.
3. Apply a repeatable burst and sustained peak. Record the load rate, duration and timestamps for metric change, HPA recommendation and new ready capacity. Repeat with spare nodes and with node provisioning required.
4. Observe dependencies and recovery. Track database connections, queue depth and API limits. Stop the test at the agreed service thresholds; use the documented recovery procedure if a change causes regression.
5. Return to normal load. Watch scale-down through the configured stabilization period and until replicas settle. Check connection draining, active jobs and service health. Test rollout or drain scenarios separately under an approved disruption plan.
6. Compare and document. Repeat the same load profile after one scoped change. Compare service health, time to ready capacity, pod/node time and estimated cost over equal periods. Save results, remaining risks and the rollout decision.

Production scaling is successful only when the whole path meets the required response time. A fast HPA decision cannot compensate for ten-minute node provisioning or a five-minute application warm-up.

7. Production Kubernetes scaling checklist

Common Kubernetes scaling mistakes

• Enabling HPA before fixing missing or unrealistic requests.
• Scaling only on CPU when another signal represents demand better.
• Ignoring application startup and readiness time.
• Adding replicas faster than a database or API can support them.
• Leaving topology and scheduling constraints out of capacity tests.
• Optimizing node utilisation without protecting workload reliability.