System Design #4: Caching — Redis, Memcached, CDN

In the previous article, you learned how load balancers distribute traffic across servers. But even with many servers, your system can be slow if every request hits the database.

This is where caching comes in. Caching is one of the most effective ways to speed up any system. It reduces database load, lowers latency, and saves money.

What is Caching?

Caching means storing a copy of data in a faster storage layer so future requests can be served quicker. Instead of fetching data from a slow source (like a database), you fetch it from a fast source (like memory).

Without caching:
  [Client] --> [Server] --> [Database]
  Response time: 200 ms (database query takes 180 ms)

With caching:
  [Client] --> [Server] --> [Cache] HIT --> return data
  Response time: 5 ms (cache lookup takes 3 ms)

  [Client] --> [Server] --> [Cache] MISS --> [Database] --> store in cache
  Response time: 210 ms (first time only)

The first request is slow because the data is not in the cache (a cache miss). But every subsequent request is fast because the data is already cached (a cache hit).

Cache Hit vs Cache Miss

These two terms are fundamental:

• Cache hit — the requested data is found in the cache. Fast response.
• Cache miss — the requested data is not in the cache. The system must fetch it from the original source (database, API, etc.).

The cache hit ratio measures how effective your cache is:

Cache hit ratio = cache hits / (cache hits + cache misses)

Example:
  1,000,000 requests
  950,000 cache hits
  50,000 cache misses

  Hit ratio = 950,000 / 1,000,000 = 95%

A 95% hit ratio means 95% of requests are served from the cache. This is a good target for most systems. Even a 90% hit ratio means your database handles only 10% of the traffic it would without caching.

Caching Strategies

There are four main caching strategies. Each one handles reads and writes differently.

1. Cache-Aside (Lazy Loading)

The application manages the cache manually. It checks the cache first. On a miss, it fetches from the database and stores the result in the cache.

Read flow:
  1. App checks cache for key "user:123"
  2. Cache HIT  --> return cached data
     Cache MISS --> query database --> store result in cache --> return data

Write flow:
  1. App writes to database
  2. App deletes key "user:123" from cache (invalidation)

This is the most common caching strategy. Here is how it looks in code:

func getUser(userID string) (*User, error) {
    // Step 1: Check cache
    cached, err := redis.Get("user:" + userID)
    if err == nil {
        // Cache HIT
        var user User
        json.Unmarshal([]byte(cached), &user)
        return &user, nil
    }

    // Step 2: Cache MISS — fetch from database
    user, err := db.QueryUser(userID)
    if err != nil {
        return nil, err
    }

    // Step 3: Store in cache for next time (TTL: 1 hour)
    data, _ := json.Marshal(user)
    redis.Set("user:"+userID, string(data), 1*time.Hour)

    return user, nil
}

Pros: Only caches data that is actually requested. Simple to implement. Cons: First request is always slow (cache miss). Data can become stale if the database changes.

2. Write-Through

Every write goes to both the cache and the database at the same time. The cache always has the latest data.

Write flow:
  1. App writes data to cache
  2. Cache writes data to database
  3. Return success

Read flow:
  1. App reads from cache (always a HIT because writes go through cache)

Pros: Cache is always consistent with the database. No stale data. Cons: Every write is slower because it goes to two places. Caches data that might never be read.

3. Write-Behind (Write-Back)

Writes go to the cache first, and the cache asynchronously writes to the database later. This is faster than write-through but riskier.

Write flow:
  1. App writes data to cache (fast)
  2. Cache queues the write
  3. Cache writes to database later (asynchronously)

Read flow:
  1. App reads from cache (always a HIT)

Pros: Very fast writes. Reduces database load because writes are batched. Cons: Risk of data loss if the cache crashes before writing to the database. More complex to implement.

4. Read-Through

Similar to cache-aside, but the cache itself handles fetching from the database on a miss. The application only talks to the cache.

Read flow:
  1. App asks cache for "user:123"
  2. Cache HIT  --> return data
     Cache MISS --> cache fetches from database --> stores it --> returns data

Pros: Simpler application code — the cache handles the logic. Cons: Requires a cache that supports read-through (some caches have this built in).

Strategy Comparison

Most used in practice: Cache-aside is the most common strategy. It is simple, safe, and works well for most applications.

Cache Eviction Policies

Caches have limited memory. When the cache is full and a new item needs to be stored, the cache must remove (evict) an existing item. The eviction policy decides which item to remove.

LRU (Least Recently Used)

Removes the item that has not been accessed for the longest time. The idea: if you have not used it recently, you probably will not use it soon.

Cache (capacity: 3):
  Access A -> [A]
  Access B -> [A, B]
  Access C -> [A, B, C]      (full)
  Access D -> [B, C, D]      (A evicted — least recently used)
  Access B -> [C, D, B]      (B moved to most recent)
  Access E -> [D, B, E]      (C evicted — least recently used)

LRU is the most popular eviction policy. Redis, Memcached, and most caching systems use LRU or a variant of it.

LFU (Least Frequently Used)

Removes the item that has been accessed the fewest times. The idea: popular items should stay in the cache.

Cache (capacity: 3):
  Access A (count: 1) -> [A:1]
  Access A (count: 2) -> [A:2]
  Access B (count: 1) -> [A:2, B:1]
  Access C (count: 1) -> [A:2, B:1, C:1]    (full)
  Access D            -> [A:2, D:1, C:1]    (B evicted — least frequent, tied with C but B is older)

Pros: Keeps popular items cached. Good when some items are much more popular than others. Cons: New items can be evicted quickly because they have low counts. Old popular items can stay forever even if they are no longer relevant.

FIFO (First In, First Out)

Removes the oldest item in the cache. Simple but not very smart.

TTL (Time to Live)

Each item has an expiration time. After the TTL expires, the item is automatically removed.

Set "user:123" with TTL = 1 hour
  --> After 1 hour, the item is automatically deleted
  --> Next request will be a cache miss

TTL is usually combined with another eviction policy (like LRU + TTL). It ensures data does not stay in the cache forever and become stale.

Redis vs Memcached

Redis and Memcached are the two most popular in-memory caching systems. Both are fast, but they have different strengths.

Redis

Redis is an in-memory data store that supports rich data structures. It is much more than a simple key-value cache.

Data structures Redis supports:

• Strings — simple key-value pairs
• Hashes — like a dictionary inside a key (useful for objects)
• Lists — ordered lists (useful for queues)
• Sets — unique collections (useful for tags, followers)
• Sorted Sets — sets with scores (useful for leaderboards, rankings)
• Streams — append-only log (useful for event streaming)

Extra features:

• Persistence — can save data to disk (survives restarts)
• Pub/Sub — publish and subscribe to channels
• Lua scripting — run scripts on the server
• Clustering — distribute data across multiple Redis nodes
• Transactions — group multiple commands

Memcached

Memcached is a simple, high-performance, distributed memory caching system. It does one thing well: key-value caching.

Features:

• Only key-value strings (no complex data structures)
• Multi-threaded (uses all CPU cores)
• No persistence (data is lost on restart)
• Simple protocol
• Excellent for pure caching workloads

When to Use Each

Choose Redis when: you need data structures beyond simple strings, persistence, pub/sub, or need the cache as a primary data store for some features (sessions, leaderboards).

Choose Memcached when: you need pure caching with maximum memory efficiency and multi-threaded performance. Memcached is simpler and uses less memory per key.

In practice, most teams choose Redis because its extra features are useful. Memcached is chosen when memory efficiency is critical at very large scale (like Facebook, which uses Memcached extensively).

CDN (Content Delivery Network)

A CDN caches static content (images, videos, CSS, JavaScript) on servers around the world. Users download content from the nearest CDN server instead of your origin server.

Without CDN:
  [User in Tokyo] --> [Your server in Frankfurt]
  Latency: 250 ms (cross-Pacific network trip)

With CDN:
  [User in Tokyo] --> [CDN edge server in Tokyo]
  Latency: 10 ms (local network trip)

How a CDN Works

First request (CDN does not have the file):
  1. User requests image.jpg
  2. CDN edge server does not have it (cache miss)
  3. CDN fetches image.jpg from your origin server
  4. CDN stores a copy and serves it to the user

Subsequent requests:
  1. User requests image.jpg
  2. CDN edge server has it (cache hit)
  3. CDN serves it directly — no origin server involved

Popular CDN Providers

• Cloudflare — free tier, global network, DDoS protection
• AWS CloudFront — integrated with AWS services
• Google Cloud CDN — integrated with Google Cloud
• Akamai — the largest CDN, enterprise-focused
• Fastly — edge computing capabilities

What to Cache on a CDN

• Images, videos, audio files
• CSS and JavaScript files
• Fonts
• Static HTML pages
• API responses that do not change often

Browser Caching

The browser itself is a cache. HTTP headers tell the browser how long to store files locally.

HTTP cache headers:

Cache-Control: max-age=3600
  --> Browser caches the file for 1 hour

Cache-Control: no-cache
  --> Browser must check with server before using cached version

Cache-Control: no-store
  --> Browser must NOT cache the file at all

ETag: "abc123"
  --> Browser sends this tag in the next request
  --> Server responds with 304 Not Modified if file has not changed

Browser caching is the fastest cache because the data never leaves the user’s device. No network request is needed at all.

Cache Invalidation

Cache invalidation is deciding when cached data is no longer valid and should be refreshed. Phil Karlton famously said:

“There are only two hard things in Computer Science: cache invalidation and naming things.”

Why Cache Invalidation is Hard

Imagine you cache a user’s profile for 1 hour. The user changes their name. For the next hour, other users see the old name. This is stale data.

Common invalidation strategies:

1. TTL-Based Invalidation

Set an expiration time on cached data. After the TTL expires, the cache automatically removes it.

Set "user:123" with TTL = 5 minutes
  --> After 5 minutes, the cached version expires
  --> Next request fetches fresh data from database

Simple but imprecise. The data could be stale for up to 5 minutes.

2. Event-Based Invalidation

When data changes, explicitly delete or update the cache entry.

User updates their profile:
  1. Update database
  2. Delete "user:123" from cache
  --> Next read fetches fresh data

More accurate but requires discipline. You must remember to invalidate the cache everywhere data changes.

3. Version-Based Invalidation

Include a version number in cache keys. When data changes, increment the version.

Version 1: cache key = "user:123:v1"
User updates profile:
  Version 2: cache key = "user:123:v2"
  Old key "user:123:v1" is ignored (eventually evicted by LRU)

The Thundering Herd Problem

The thundering herd problem happens when a popular cache entry expires and many requests hit the database at the same time.

Normal operation:
  1,000 requests/second for "popular-item"
  All served from cache (fast)

Cache entry expires:
  1,000 requests simultaneously see cache MISS
  1,000 requests hit the database at the same time
  Database overwhelmed --> slow responses or crash

Solutions

1. Lock-based approach: When a cache miss occurs, only one request fetches from the database. Other requests wait for the cache to be repopulated.

Request 1: Cache MISS --> acquire lock --> fetch from DB --> update cache
Request 2: Cache MISS --> lock taken --> wait...
Request 3: Cache MISS --> lock taken --> wait...
...
Request 1 finishes: cache updated --> release lock
Request 2, 3, ...: read from cache (HIT)

2. Stale-while-revalidate: Serve the stale cached data while fetching fresh data in the background. Users get fast responses (slightly stale) and the cache is refreshed without a thundering herd.

3. Early expiration (jitter): Instead of all cache entries expiring at exactly the same time, add a random offset. This spreads out the cache misses.

Instead of TTL = 60 minutes (all expire at the same time):
  Item 1: TTL = 58 minutes
  Item 2: TTL = 62 minutes
  Item 3: TTL = 56 minutes
  Item 4: TTL = 61 minutes

Caching at Every Layer

In a well-designed system, caching happens at multiple levels:

[Browser Cache] --> [CDN Cache] --> [Load Balancer]
                                    --> [App Server]
                                        --> [Application Cache (Redis)]
                                        --> [Database Query Cache]
                                        --> [Database Buffer Pool]

1. Browser cache — static files, API responses (fastest)
2. CDN cache — static content close to the user
3. Application cache (Redis) — computed results, database query results
4. Database cache — query cache, buffer pool (built into the database)

Each layer reduces the load on the layer below it.

Interview Tips

When discussing caching in a system design interview:

1. Always add caching. Almost every system design benefits from caching. Mention it early.
2. Specify the caching strategy. “I would use cache-aside with Redis and a 5-minute TTL.” Be specific.
3. Mention cache invalidation. It shows you think about data consistency, not just performance.
4. Know the numbers. Redis can handle 100,000+ operations per second. A database might handle 10,000. Caching gives you a 10x improvement.
5. Discuss the thundering herd. It is a common follow-up question. Know at least one solution.
6. Cache hit ratio. Mention that you would monitor the hit ratio and adjust TTL or caching strategy based on it.

Related Articles

• System Design #3: Load Balancers — How load balancers distribute traffic
• Go Tutorial #19: Database with sqlx — Working with databases in Go
• SQL Tutorial #6: Indexes and Performance — Database indexing and performance

What’s Next?

In the next article, System Design #5: Databases — SQL vs NoSQL, Sharding, Replication, you will learn:

• SQL vs NoSQL — when to use each
• ACID properties explained simply
• Database replication (leader-follower, leader-leader)
• Database sharding strategies
• Choosing the right database for your use case

This is part 4 of the System Design Tutorial series. Follow along to learn system design from scratch.