Load Balancing Architecture: HAProxy vs Nginx in Production

A load balancing architecture is the control point between clients and application capacity. It decides where a connection or request goes, when an unhealthy backend should stop receiving traffic, whether a failed request may be retried, how TLS is terminated, and what happens when a deploy, node, zone, or dependency starts failing. The wrong design can turn a partial backend problem into retry amplification, duplicate writes, connection storms, or an outage hidden behind a healthy proxy process.

This guide is for engineers choosing between L4 and L7 balancing, HAProxy and Nginx, or a combination of edge and internal traffic layers. It focuses on production decisions rather than benchmark claims: health semantics, timeout budgets, retry safety, connection reuse, draining, failover, observability, and validation. The examples use private placeholder addresses and must be tested in staging before production. Reviewed August 2026; validate every directive against the version you actually run.

A useful starting scenario is a stateless HTTP API behind two application nodes. Normal traffic is healthy, but during a rolling deploy one node becomes slow before it becomes unavailable. The load balancer must stop sending new work to the unhealthy node, avoid replaying unsafe writes, preserve enough capacity on the remaining backend, expose the change in metrics, and give the operator a clear stop condition before customer impact expands.

How to choose L4 vs L7 load balancing architecture

Layer 4 and Layer 7 solve different problems. L4 balancing routes TCP or UDP connections using network-level information. L7 balancing understands application protocols such as HTTP and can route using hostnames, paths, headers, cookies, or other request attributes. Neither layer is automatically more reliable; the right choice depends on what decision the balancer must make.

Choose L4 when the proxy should stay protocol-agnostic and the application does not need HTTP-aware routing. Typical cases include raw TCP services, database frontends, TLS pass-through, or a simple connection-distribution layer where application semantics belong elsewhere.

Choose L7 when the routing decision depends on HTTP behavior or when the edge must own HTTP-specific controls. Typical cases include multiple services behind one hostname, path-based routing, canary traffic, header manipulation, HTTP-aware retries, redirects, or detailed upstream response telemetry.

Before choosing the layer, answer these questions:

• Does the proxy need to inspect HTTP host, path, method, headers, or cookies?
• Must TLS terminate at the load balancer, or should it pass through to the backend?
• Is connection affinity required, and what failure behavior is acceptable when the selected backend disappears?
• Can the application tolerate a connection being moved to another backend?
• Which requests are idempotent and therefore candidates for carefully bounded retries?
• Does readiness require an application-level check rather than an open TCP port?
• Where will rate limiting, authentication, caching, compression, or request normalization live?
• Which layer owns client IP preservation and trusted proxy boundaries?
• What is the maximum acceptable queueing and connection establishment time?
• How will the operator remove a backend without terminating in-flight work?

A common production pattern is layered rather than exclusive: an external L4 service distributes connections across multiple L7 proxies, while the L7 proxies route requests to application backends. That pattern can improve failure isolation, but it also adds another place where timeouts, health checks, source addresses, and observability must agree.

HAProxy vs Nginx: define the decision boundary first

HAProxy and Nginx overlap significantly for HTTP reverse proxying and load balancing, but they are often chosen for different operational reasons.

HAProxy is a strong fit when explicit backend state, connection behavior, health checks, traffic steering, and failover logic are central to the design. Its configuration model makes frontend/backend separation, server state, active checks, connection limits, and routing policy visible in one place. HAProxy also supports both TCP and HTTP modes, which is useful when the same operating team manages L4 and L7 traffic paths.

Nginx is a strong fit when the same edge also owns HTTP serving concerns such as TLS termination, static assets, caching, compression, redirects, or application routing. Open-source Nginx provides passive upstream failure handling; periodic active upstream health checks are an NGINX Plus capability, so the health model must be understood before treating Nginx and HAProxy configurations as equivalent.

Do not choose based only on requests-per-second comparisons. A more useful decision is operational:

• Choose the product whose failure semantics the team can explain during an incident.
• Prefer a configuration that keeps timeout and retry behavior explicit.
• Avoid duplicating the same routing policy in several layers unless there is a clear ownership boundary.
• Keep the health signal aligned with actual application readiness.
• Make backend draining part of the deploy and maintenance procedure.
• Preserve enough observability to distinguish proxy saturation from backend saturation.

The load balancer is not a substitute for application resilience. It cannot make a non-idempotent request safe to replay, repair a saturated database, or create capacity that does not exist.

Health checks, timeouts, retries, and draining checklist

Most production failures in a load-balancing layer come from interactions between several individually reasonable settings. A short connect timeout can be good, but a large retry count can multiply backend load. A readiness endpoint can be useful, but if it checks only the process and ignores a critical dependency it may keep an unusable backend in rotation. A long drain window can protect in-flight requests, but only if deployment tooling waits for it.

Use this checklist before enabling or changing production traffic:

• Readiness: the health endpoint should represent the minimum conditions required to accept new work.
• Liveness: do not use a restart-oriented liveness signal as the only traffic-readiness signal.
• Failure threshold: require enough consecutive failures to avoid removing a backend because of one transient probe error.
• Recovery threshold: require enough successful checks before returning a backend to normal traffic after a fault.
• Connect timeout: bound how long the proxy waits to establish a backend connection.
• Response timeout: align proxy read/server timeout with the application SLO and known long-running operations.
• Client timeout: do not allow the edge to wait indefinitely after the upstream has already abandoned the request.
• Retry count: keep retries bounded and treat them as additional load during partial failure.
• Retry methods: do not enable replay of non-idempotent writes unless the application has an explicit idempotency mechanism.
• Queue limit: bound queue time or queued connections so overload fails predictably instead of accumulating hidden latency.
• Connection reuse: verify keepalive behavior against backend connection limits and deployment draining.
• Draining: remove a backend from new traffic before stopping the process; allow existing work to finish within a defined window.
• Capacity headroom: confirm that the remaining healthy backends can carry the expected traffic when one backend is removed.
• TLS ownership: define exactly where TLS terminates and where certificate validation is required again upstream.
• Source identity: preserve client IP or proxy protocol information only through trusted boundaries.

A configuration is incomplete until each timeout has an owner and a reason. The client, CDN, L4 balancer, L7 proxy, application server, database pool, and downstream APIs can all have different timeout values; misalignment creates ambiguous 499/502/503/504 errors and difficult incident diagnosis.

HAProxy example with application readiness and bounded connections

The following is a deliberately small HTTP example. The addresses, certificate path, timeout values, and capacity limits are examples, not universal recommendations. Validate syntax with the exact HAProxy binary that will run the configuration, and test the readiness endpoint before sending production traffic.

# Preconditions:
# - /readyz returns 200 only when the application can accept new requests.
# - both backend addresses are reachable from the proxy host.
# - the TLS certificate path exists and permissions are correct.

global
  log stdout format raw local0

defaults
  mode http
  timeout connect 3s
  timeout client 30s
  timeout server 30s

frontend https_in
  bind :443 ssl crt /etc/haproxy/certs/service.pem
  default_backend app_nodes

backend app_nodes
  balance leastconn
  option httpchk GET /readyz
  http-check expect status 200
  retries 2

  server app1 10.0.0.11:8080 check inter 2s fall 3 rise 2 maxconn 200
  server app2 10.0.0.12:8080 check inter 2s fall 3 rise 2 maxconn 200

Validate before reload:

sudo haproxy -c -f /etc/haproxy/haproxy.cfg
curl -fsS http://10.0.0.11:8080/readyz
curl -fsS http://10.0.0.12:8080/readyz

Expected result: HAProxy reports a valid configuration and each candidate backend returns the readiness response the load balancer is configured to expect. Do not continue if the readiness endpoint is not representative of real serving ability, if only one backend is healthy without enough capacity, or if the proposed retry behavior has not been reviewed for write requests.

For planned maintenance or deploys, use the established runtime/admin mechanism to drain or mark a server out of new traffic rather than killing it first. The exact control path depends on how HAProxy is operated in your environment and should be documented in the deployment runbook.

Nginx example with passive failure handling and bounded next-upstream tries

Open-source Nginx can distribute traffic across an upstream group and use passive failure information such as connection or response errors. Periodic active upstream health checks are part of NGINX Plus, so an open-source deployment often combines passive upstream handling with external or orchestration-level readiness controls.

This example explicitly bounds connection and response timeouts and limits how many upstream attempts may occur. It does not enable non_idempotent retries.

# Preconditions:
# - backends expose HTTP on the listed private addresses.
# - TLS certificate/key paths are valid.
# - application deploy tooling removes unready instances from service discovery
#   or otherwise prevents traffic before process shutdown.

upstream app_backend {
    least_conn;
    server 10.0.0.11:8080 max_fails=3 fail_timeout=10s;
    server 10.0.0.12:8080 max_fails=3 fail_timeout=10s;
    keepalive 32;
}

server {
    listen 443 ssl;
    server_name service.example.com;

    ssl_certificate     /etc/nginx/tls/fullchain.pem;
    ssl_certificate_key /etc/nginx/tls/privkey.pem;

    location / {
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Connection "";

        proxy_connect_timeout 3s;
        proxy_read_timeout 30s;
        proxy_send_timeout 30s;

        proxy_next_upstream error timeout http_502 http_503 http_504;
        proxy_next_upstream_tries 2;
        proxy_pass http://app_backend;
    }
}

Validate before reload:

sudo nginx -t
curl -fsS https://service.example.com/readyz

Expected result: nginx -t succeeds and the public readiness path returns the expected response through the same route clients will use. Pause the rollout if a retry can cross a non-idempotent operation boundary, if a backend can remain discoverable after it stops accepting work, or if connection reuse can exceed the backend connection budget.

Failure modes and stop conditions

A load balancer can stay healthy while the service behind it is failing. That is why proxy process health is not a sufficient release signal. Treat the following as explicit failure modes and define a stop or rollback action for each one.

• Health-check flapping: backends repeatedly enter and leave rotation. Stop the change and inspect readiness semantics, probe thresholds, dependency health, and network stability.
• Retry amplification: upstream request volume increases faster than client request volume during failure. Reduce or disable retries and verify application idempotency before continuing.
• Duplicate writes: customers or workers observe repeated state-changing operations. Stop traffic migration immediately and investigate retry/replay boundaries and idempotency keys.
• Queue growth: proxy queues or connection wait time increase while backend utilization is saturated. Stop adding traffic and restore capacity or reduce concurrency.
• Connection storm: backend connection count spikes after proxy restart, failover, or scaling. Check keepalive, pooling, worker count, connection limits, and restart sequencing.
• Slow backend masking: a backend still passes a shallow check but has high latency or dependency errors. Strengthen readiness or use application-level signals for removal decisions.
• TLS failures: handshake errors increase after certificate, cipher, SNI, or termination changes. Roll back the TLS change before altering unrelated routing settings.
• Uneven traffic: one backend receives a disproportionate share because of affinity, long-lived connections, weights, or topology. Verify the chosen algorithm against the real workload.
• Drain failure: a node is terminated while it still owns in-flight requests or long-lived connections. Fix the deployment/maintenance sequence before continuing.
• Observability gap: the team cannot identify which backend served a failed request. Add upstream address/status/latency context before using the design for critical traffic.

Do not treat “the proxy returns 200” as proof of success. A safe change must also preserve the customer transaction and the downstream systems that transaction depends on.

Validate a production load balancer before rollout

Validation should happen at four levels: configuration, backend state, controlled failure, and business behavior. Run the checks first in a staging environment that resembles the production traffic path.

Configuration validation

• Parse the exact configuration with haproxy -c or nginx -t before reload.
• Confirm certificate files, DNS names, upstream addresses, and included configuration files resolve as expected.
• Verify no old backend or deprecated route remains reachable accidentally.

Backend validation

• Confirm every backend intended to receive traffic is ready.
• Confirm an unready backend is removed from new traffic by the mechanism actually used in production.
• Confirm one backend can be drained without creating an immediate capacity violation.
• Check connection counts, queue depth, backend latency, and error rate before and during the test.

Controlled failure validation

• In staging, make one test backend unavailable using the normal maintenance/deploy mechanism.
• Confirm new traffic moves away from that backend within the expected health or discovery window.
• Confirm existing requests either complete or fail according to the documented drain policy.
• Confirm retries do not create duplicate business operations.
• Restore the backend and confirm it does not return to full traffic before readiness is true.

Business validation

• Perform the critical user transaction through the public endpoint, not directly against a backend.
• Verify authentication, one representative read, and one safe write flow when the service supports them.
• Confirm logs or traces identify the selected upstream and the result.
• Compare customer-facing latency and errors with the agreed baseline or SLO threshold.

A rollout should stop if any customer-facing validation fails, if the remaining capacity is insufficient for a backend loss, if retries are unsafe, or if the team cannot explain an observed error path.

Observability and capacity evidence

The load balancer is a useful observability boundary because it sees client traffic and backend selection at the same point. The exact metrics differ by product, but the evidence package should answer the same operational questions.

Capture and retain:

• request or connection rate by frontend/listener;
• response status or connection outcome;
• backend selection and backend availability state;
• connect time, queue time, response time, and total request time where available;
• active and queued connections;
• retry/redispatch/next-upstream counts;
• TLS handshake errors and certificate expiration monitoring;
• backend saturation signals from the application and database layers;
• deploy or configuration-change timestamps;
• the result of the controlled drain/failure test;
• the customer-facing validation result;
• the configuration revision or commit that produced the tested behavior.

Capacity planning must include the failure case. If the architecture assumes two equal backends but neither can safely carry the workload when the other is drained, the design has no maintenance headroom. The correct response may be additional capacity, lower concurrency, a different deployment strategy, or a different traffic-distribution topology; the load-balancing algorithm alone cannot solve the constraint.

Load balancing decision matrix

Official references

• HAProxy 3.0 configuration manual — Primary reference for HAProxy health checks, timeouts, retries, server state, connection limits, and TCP/HTTP routing behavior.
• Nginx: Using nginx as HTTP load balancer — Official overview of upstream algorithms and passive health behavior in open-source Nginx.
• Nginx HTTP proxy module — Official reference for proxy timeouts, proxy_next_upstream, retry limits, request forwarding, and upstream communication behavior.
• Nginx HTTP upstream module — Official reference for upstream server groups, weights, failure accounting, connection limits, keepalive, and balancing methods.
• Kubernetes Service — Official reference for Service traffic policies, ready backends, and traffic distribution when the load-balancing path includes Kubernetes.

Related SteadyOps reading

• Kubernetes Production Readiness Checklist — Verify readiness, disruption, resources, rollout safety, rollback, and evidence before putting Kubernetes workloads behind production traffic.
• Zero-Downtime Blue/Green Deployments — Connect traffic switching to release validation, compatibility, and rollback decisions.
• PostgreSQL at Scale — Understand how PgBouncer, HAProxy, Patroni, connection budgets, and failover interact on database-heavy systems.
• Scalable Infrastructure Architecture — Place load balancing inside the broader capacity, queueing, latency, and failure-aware architecture decision.

Key takeaways

• Choose L4 or L7 from the routing decision the proxy must make, not from a generic performance claim.
• HAProxy and Nginx overlap, but their health-check and operational models are not identical.
• Health checks, timeouts, retries, connection reuse, draining, and capacity must be designed as one system.
• Never enable broader retries until non-idempotent operations and duplicate-write risk are understood.
• Validate a backend drain and a controlled failure before relying on the architecture during a real incident.
• Production evidence must include the customer transaction, upstream state, latency, errors, and the exact configuration revision.

defaults
    mode http
    timeout connect 3s
    timeout client 30s
    timeout server 30s

backend api
    balance leastconn
    option httpchk GET /ready
    http-check expect status 200
    server api1 10.0.0.11:8080 check inter 2s fall 3 rise 2
    server api2 10.0.0.12:8080 check inter 2s fall 3 rise 2