System Design #2: Scalability — Horizontal vs Vertical Scaling

In the previous article, you learned what system design is and how to approach any design problem. Now let us talk about the most fundamental concept: scalability.

Scalability is the ability of a system to handle more work by adding resources. Every system starts small. The question is: what happens when it needs to grow?

What is Scalability?

Imagine you own a coffee shop. On Monday, you serve 50 customers. Your single barista handles it fine. But on Saturday, 500 people show up. What do you do?

You have two options:

1. Replace your barista with a super-fast robot that makes coffee 10x faster (vertical scaling)
2. Hire 9 more baristas and add more counters (horizontal scaling)

This is the core of scalability. You either make your existing resources more powerful, or you add more resources.

Vertical Scaling (Scale Up)

Vertical scaling means making a single machine more powerful. You add more CPU, more RAM, more storage, or a faster network to your existing server.

Before:
  [Server: 4 CPU, 16 GB RAM, 500 GB SSD]
  Handles: 1,000 requests/second

After vertical scaling:
  [Server: 64 CPU, 512 GB RAM, 4 TB NVMe]
  Handles: 20,000 requests/second

Advantages of Vertical Scaling

• Simple. No code changes needed. Your application runs the same way, just faster.
• No distributed systems. One server means no network latency between components, no data synchronization issues.
• Easy maintenance. One machine to monitor, one machine to back up, one machine to secure.

Disadvantages of Vertical Scaling

• There is a hardware limit. The most powerful server in the world still has a maximum. High-end cloud instances and bare-metal servers in 2026 offer hundreds of CPU cores and several terabytes of RAM — but there is always a ceiling, and the cost grows faster than the performance gain.
• Expensive. Doubling the power of a server often costs more than double the price. A 64-core server costs much more than four 16-core servers.
• Single point of failure. If your one powerful server goes down, everything goes down. There is no backup.
• Downtime for upgrades. You usually need to stop the server to add hardware (unless you use cloud instances that can resize).

When to Use Vertical Scaling

Vertical scaling works well for:

• Databases — many databases (PostgreSQL, MySQL) perform better on a single powerful machine than on many small machines
• Small to medium applications — if your app serves under 10,000 users, one good server is often enough
• Early-stage startups — do not over-engineer. Start with one server and scale vertically until you hit limits
• Stateful workloads — applications that keep a lot of data in memory (like in-memory databases or game servers)

Horizontal Scaling (Scale Out)

Horizontal scaling means adding more machines. Instead of one powerful server, you use many smaller servers working together.

Before:
  [Server 1: 4 CPU, 16 GB RAM]
  Handles: 1,000 requests/second

After horizontal scaling:
  [Server 1: 4 CPU, 16 GB RAM]  \
  [Server 2: 4 CPU, 16 GB RAM]   |
  [Server 3: 4 CPU, 16 GB RAM]   |--> Together: 10,000 requests/second
  ...                             |
  [Server 10: 4 CPU, 16 GB RAM] /

A load balancer sits in front of these servers and distributes incoming requests across them (we will cover load balancers in detail in the next article).

Advantages of Horizontal Scaling

• No hardware limit. Need more capacity? Add more servers. There is no ceiling.
• Fault tolerant. If one server dies, others keep running. Users might not even notice.
• Cost efficient. Many small servers (or cloud instances) are often cheaper than one giant server.
• Geographic distribution. You can put servers in different regions to serve users closer to them.

Disadvantages of Horizontal Scaling

• Complex. Your application must handle multiple servers. Data synchronization, session management, and deployment become harder.
• Network latency. Servers communicate over the network, which is slower than in-memory communication on a single machine.
• Data consistency. Keeping data the same across multiple servers is a hard problem (we will cover this in the CAP Theorem article).
• More operational overhead. More servers means more monitoring, more logging, more things that can fail.

When to Use Horizontal Scaling

Horizontal scaling works well for:

• Web applications — stateless API servers scale horizontally very well
• Large-scale systems — anything serving millions of users needs horizontal scaling
• Microservices — each service can scale independently based on its own traffic
• Read-heavy workloads — add more read replicas for databases

Vertical vs Horizontal — Side by Side

Stateless vs Stateful Services

The key to horizontal scaling is making your services stateless. This is one of the most important concepts in system design.

Stateful Services

A stateful service stores information about each user session on the server:

Stateful server:

[User Alex] --> [Server 1] (stores Alex's session in memory)
[User Sam]  --> [Server 2] (stores Sam's session in memory)

Problem: If Server 1 crashes, Alex loses their session.
Problem: If Alex's next request goes to Server 2,
         Server 2 does not know about Alex's session.

Stateful services are hard to scale horizontally because each server has different data. You need sticky sessions (always routing a user to the same server), which limits your flexibility.

Stateless Services

A stateless service does not store any user session data on the server. Every request contains all the information needed to process it:

Stateless servers:

[User Alex] --> [Load Balancer] --> [Server 1] (no session stored)
                                --> [Server 2] (no session stored)
                                --> [Server 3] (no session stored)

Session data stored externally:
  [Redis] or [Database] holds all session data.
  Any server can read it.

Now you can:

• Route any request to any server
• Add or remove servers without worrying about session data
• Replace crashed servers instantly

How to Make Services Stateless

1. Store sessions externally. Use Redis, Memcached, or a database for session data.
2. Use tokens instead of server sessions. JWT tokens carry the session information inside the token itself. The server does not need to store anything.
3. Store uploads externally. Use object storage (Amazon S3, Google Cloud Storage) instead of the local file system.

Stateless architecture:

[Client] --> [Load Balancer] --> [App Server 1] --> [Redis (sessions)]
                              --> [App Server 2] --> [PostgreSQL (data)]
                              --> [App Server 3] --> [S3 (files)]

Any server can handle any request.
All shared state lives in external stores.

Session Management in Distributed Systems

When you have multiple servers, managing user sessions becomes a real challenge. Here are three common approaches:

1. Sticky Sessions

The load balancer always sends a user to the same server. This is simple but fragile — if that server crashes, the user loses their session.

[User Alex] --> [Load Balancer] --always--> [Server 1]
[User Sam]  --> [Load Balancer] --always--> [Server 2]

2. Centralized Session Store

All servers share a session store (usually Redis). Any server can handle any request.

[Server 1] --> [Redis: sessions]
[Server 2] --> [Redis: sessions]
[Server 3] --> [Redis: sessions]

This is the most common approach in production systems.

3. Client-Side Sessions (JWT)

The server sends a signed token (JWT) to the client. The client includes it in every request. The server verifies the token without storing anything.

1. Client logs in --> Server creates JWT --> sends to client
2. Client sends JWT with every request --> Server verifies JWT
   No session stored on server!

JWT is great for stateless APIs but has trade-offs: tokens cannot be revoked easily (you need a blacklist), and large tokens add overhead to every request.

Real-World Examples

How Netflix Scales

Netflix serves over 200 million subscribers worldwide. They use:

• Horizontal scaling for their API servers — hundreds of instances handle requests
• AWS Auto Scaling — servers are added and removed based on traffic
• Microservices — over 700 microservices, each scaling independently
• CDN (Open Connect) — video content is cached on servers placed inside ISPs worldwide
• Cassandra — NoSQL database for horizontal scaling of user data

Netflix would not work with a single powerful server. The scale demands horizontal scaling at every layer.

How Uber Scales

Uber handles millions of ride requests per day across hundreds of cities. They use:

• Microservices — separate services for matching, pricing, payments, maps
• Geospatial sharding — data is partitioned by geographic region
• Real-time processing — Kafka for event streaming between services
• Horizontal scaling — each service runs on many instances, scaled by demand

During peak hours (Friday evenings, New Year’s Eve), Uber auto-scales to handle 10-50x normal traffic.

How Instagram Scales

Instagram started as a small team of 13 engineers serving 30 million users. They scaled by:

• PostgreSQL with sharding — user data split across many database machines
• Redis — caching for feeds, sessions, and counters
• Memcached — additional caching layer for hot data
• Cassandra — for the feed inbox (write-heavy workload)
• Horizontal scaling of web servers behind load balancers

Instagram was famously efficient — a small team handled massive scale because they made good system design decisions.

Auto-Scaling

Auto-scaling automatically adds or removes servers based on current demand. This is the modern way to handle varying traffic.

Auto-scaling rules example:

  IF average CPU usage > 70% for 5 minutes:
    Add 2 more server instances

  IF average CPU usage < 30% for 10 minutes:
    Remove 1 server instance

  Minimum instances: 2 (always running)
  Maximum instances: 50 (cost limit)

Cloud providers (AWS, Google Cloud, Azure) all offer auto-scaling. You define rules, and the platform handles the rest.

Common Auto-Scaling Metrics

• CPU usage — the most common metric. Scale up when CPUs are busy.
• Request count — scale based on incoming requests per second.
• Queue depth — scale based on how many messages are waiting to be processed.
• Response time — scale up when latency increases.
• Custom metrics — scale based on business metrics (active users, active orders).

Auto-Scaling Challenges

• Cold start time. New servers take time to boot up and become ready (30 seconds to several minutes). During this time, existing servers must handle the extra load.
• Scaling too aggressively. If your rules are too sensitive, you add and remove servers constantly (called “flapping”), which wastes money and causes instability.
• Database bottleneck. Auto-scaling your app servers is easy. Auto-scaling your database is hard. If your database cannot handle the increased connections from new servers, scaling does not help.

When to Scale Vertically vs Horizontally

Here is a decision framework:

Start with vertical scaling when:
  - You are a small team or early-stage startup
  - Your app is a monolith
  - You use a relational database (PostgreSQL, MySQL)
  - Your traffic is predictable
  - You want simplicity

Switch to horizontal scaling when:
  - Vertical scaling gets too expensive
  - You need fault tolerance (no single point of failure)
  - Your traffic is unpredictable or bursty
  - You serve users in multiple regions
  - You are moving to microservices

The practical approach: Start with vertical scaling (one server, make it bigger as needed). When you hit the limits of vertical scaling, switch to horizontal scaling for the components that need it most (usually web servers first, then databases).

Most production systems use both. You scale your database vertically (bigger machine) while scaling your web servers horizontally (more machines). This is a practical compromise.

Interview Tips

When discussing scalability in an interview:

1. Always mention both options. Show that you know the trade-offs.
2. Start simple. “First, I would use a single server and scale vertically. When we hit limits, I would add horizontal scaling.”
3. Explain stateless. Interviewers love hearing about stateless services and external session stores.
4. Use real numbers. “A single PostgreSQL instance can handle about 10,000-50,000 queries per second depending on complexity. Beyond that, we need read replicas or sharding.”
5. Mention auto-scaling. It shows you think about cost efficiency, not just raw capacity.

Related Articles

• System Design #1: What is System Design? — The fundamentals
• Docker Tutorial #5: Deploy with Docker — Deploying applications to production
• Go Tutorial #25: Microservice Project — Building a complete microservice

What’s Next?

In the next article, System Design #3: Load Balancers — How They Work, you will learn:

• What a load balancer is and why you need one
• Load balancing algorithms (round robin, least connections, IP hash)
• Layer 4 vs Layer 7 load balancing
• Health checks and failover
• Popular tools: Nginx, HAProxy, AWS ALB

This is part 2 of the System Design Tutorial series. Follow along to learn system design from scratch.