Before this:Container orchestration
Scaling & load balancing
Key takeaways Two ways to handle more load: scale up (a bigger machine) or scale out (more copies). Scaling up is simple but capped and single-point-of-failure; scaling out goes far further and survives failures — but only if the service is stateless, so any instance can serve any request. A load balancer spreads traffic across the instances by an algorithm like round-robin or least-connections.
Orchestration can run many replicas — but running them only helps if traffic is spread across them and the service is built to be spread. This lesson is the mechanics of both: the two directions you can scale, why statelessness is the enabler, and how a load balancer divides the work.
Scale up or scale out
- Vertical (up) — give one machine more CPU and memory. Dead simple, no code changes, but capped by the largest machine available, and that one box is a single point of failure. A great first move.
- Horizontal (out) — run more copies behind a load balancer. Scales far past any single machine, and losing one instance just means the rest carry on. But it demands a stateless service.
Start by scaling up — it’s free of architectural change. Scale out when you hit the ceiling of one machine or need to survive an instance dying.
Statelessness is the enabler
Horizontal scaling works only if any instance can serve any request. That’s true when the instance keeps no important state of its own — the state lives in a shared database or cache that all instances read. Then the load balancer can send request 1 to instance A and request 2 to instance B and nothing breaks.
Put session data in one instance’s local memory and you’ve broken that: a user’s next request might land on a different instance that’s never heard of them. The fix is to push state out of the instances into a shared store, making the instances interchangeable — the same stateless-vs-stateful split that decided what to back up.
The load balancer and its algorithms
A load balancer sits in front of the instances (often the same reverse proxy you already run) and picks which one gets each request. Common algorithms:
| Algorithm | Picks the instance… | Good when |
|---|---|---|
| Round-robin | Next one in rotation | Requests are roughly equal cost |
| Least-connections | With the fewest active requests | Request durations vary a lot |
| IP hash | Determined by client IP | You need a client pinned to one instance |
| Weighted | Proportional to capacity | Instances have different sizes |
An nginx upstream spreading traffic across three instances:
upstream app {
least_conn; # send each request to the least-busy instance
server 127.0.0.1:8081;
server 127.0.0.1:8082;
server 127.0.0.1:8083;
}
server {
location / { proxy_pass http://app; }
}
The balancer also does health checking: an instance that fails its health check is pulled from rotation until it recovers, so traffic only goes to instances that can serve it.
Sticky sessions: the escape hatch
Sometimes you can’t fully remove per-instance state and need a given client to keep hitting the same instance. Sticky sessions (session affinity) pin a client to one instance, usually via a cookie or IP hash. It’s a pragmatic patch, but treat it as a smell: it re-introduces the coupling that statelessness removed, weakening both scaling and failover. Prefer shared state; use stickiness only when you must.
Quick check: what makes horizontal scaling possible?
Recap
- Scale up (bigger machine) is simple but capped and a single point of failure; scale out (more copies) goes further and survives failures.
- Scaling out needs stateless services — state pushed to a shared store so instances are interchangeable.
- A load balancer spreads traffic by round-robin, least-connections, and other algorithms, and drops unhealthy instances via health checks.
- Sticky sessions pin a client to one instance — a pragmatic patch, but prefer shared state.
Next up: replacing a running version without dropping a request — zero-downtime deploys.
Frequently asked questions
What's the difference between vertical and horizontal scaling?
Vertical scaling (scaling up) means giving one machine more resources — more CPU, more memory. It’s simple but capped by the biggest machine you can buy, and that machine is a single point of failure. Horizontal scaling (scaling out) means running more copies of the service behind a load balancer. It scales far further and survives one instance dying, but only works if the service is stateless.
Why does horizontal scaling need stateless services?
If each instance holds its own important state — a user’s session in local memory, say — then which instance a request lands on matters, and you can’t freely spread traffic. A stateless instance keeps no such state locally (it lives in a shared database or cache), so any instance can serve any request. That’s what lets a load balancer treat all instances as interchangeable and scale them freely.