Architecture

📖 7 min read 📄 Part 3 of 10

Web Cache - System Architecture

High-Level Architecture

System Components

CLIENT LAYER 🌐 Browser 📱 Mobile Apps 🔌 API Clients 🌍 CDN Edge LOAD BALANCER LAYER ⚖️ Global Load Balancer (GeoDNS, Anycast, DDoS Protection) CACHE LAYER (L1) — Redis / Memcached ⚡ Node 1 Memory + SSD TTL: 300s ⚡ Node 2 Memory + SSD TTL: 300s ⚡ Node 3 Memory + SSD TTL: 300s ⚡ Node N Memory + SSD TTL: 300s Cache Miss ✗ Cache Hit ✓ 🖥️ Application Servers Cache-Aside Pattern: Read → Cache → Miss → DB → Write Cache ORIGIN / DATABASE LAYER 🗄️ Origin Servers (Web, App, Storage, Database) Invalidation: TTL Expiry • Event-driven Purge • Write-through • Write-behind
Web Cache — Multi-layer Caching Architecture

Cache Node Architecture

Internal Components

Cache Node Request Handler HTTP/HTTPS termination Request parsing Cache key generation Header processing ENTRY POINT Cache Lookup Engine L1: Memory Cache (128GB) • Hash table for O(1) lookup • LRU eviction policy • <1ms latency ⚡ FASTEST L2: SSD Cache (10TB) • B-tree index for fast lookup • LRU eviction policy • <5ms latency HIGH CAPACITY Origin Fetcher Connection pooling Request coalescing Circuit breaker Retry logic MISS PATH Cache Manager Eviction policy enforcement TTL management Invalidation handling Statistics collection
Cache Node Internal Components

Request Flow

Cache Hit Flow

1 Client Request 2 Load Balancer 3 Cache Node 4 L1 Memory Cache HIT! 5 Validate Freshness (TTL, ETag) 6 Return Response ⚡ <1ms total Route to nearest node Parse & generate key
Cache Hit Flow — Optimal Path

Cache Miss Flow

1 Client Request 2 Load Balancer 3 Cache Node 4 L1 Memory — MISS ✗ 5 L2 SSD — MISS ✗ 6 Origin Fetcher (coalescing check) 7 Fetch from Origin Server 8 Store in L2 SSD 9 Store in L1 Memory 10 Return Response ⏱ <100ms total
Cache Miss Flow — Full Origin Fetch Path

Conditional Request Flow

1 Client Request If-Modified-Since header 2 Cache Lookup Found cached content 3 Compare Not modified 4a 304 Not Modified No body transfer — fast! Modified 4b Content Modified • Fetch fresh content from origin • Update cache • Return 200 OK with new content Not Modified (304) — bandwidth saved Modified (200) — full fetch required
Conditional Request Flow — If-Modified-Since Handling

Cache Key Generation

Cache Key Components

Cache Key = hash(
  scheme +          // http or https
  host +            // www.example.com
  port +            // 80, 443
  path +            // /api/users
  query_string +    // ?id=123&sort=name
  vary_headers      // Accept-Encoding, Accept-Language
)

Examples:
1. Simple: "GET:example.com:/index.html"
2. With query: "GET:example.com:/api/users?id=123"
3. With vary: "GET:example.com:/page.html:gzip:en-US"
4. Custom: "GET:example.com:/api/data:user_id=456"

Vary Header Handling

Response Headers:
  Vary: Accept-Encoding, Accept-Language

Cache Keys Generated:
  - "GET:example.com:/page:gzip:en-US"
  - "GET:example.com:/page:gzip:es-ES"
  - "GET:example.com:/page:identity:en-US"
  - "GET:example.com:/page:identity:es-ES"

Each combination cached separately

Eviction Policies

LRU (Least Recently Used)

Implementation:
- Hash table for O(1) lookup
- Doubly linked list for O(1) eviction
- Move to head on access
- Evict from tail when full

Pros: Simple, effective for most workloads
Cons: One-time access can evict useful data

LFU (Least Frequently Used)

Implementation:
- Track access frequency per item
- Min-heap for O(log n) eviction
- Evict least frequently accessed

Pros: Better for skewed access patterns
Cons: Higher overhead, slow to adapt

TTL-Based Eviction

Implementation:
- Store expiration timestamp
- Background thread scans for expired items
- Lazy eviction on access

Pros: Respects cache-control headers
Cons: May keep stale data longer

Cache Invalidation

Invalidation Methods

1. TTL Expiration:
   - Automatic expiration after TTL
   - Lazy deletion on access
   - Background cleanup

2. Explicit Purge:
   - Purge specific URL
   - Purge by pattern/wildcard
   - Purge by tag
   - Purge all

3. Conditional Invalidation:
   - If-Modified-Since validation
   - ETag validation
   - Stale-while-revalidate

4. Event-Based:
   - Webhook triggers
   - Message queue events
   - Database change streams

Invalidation Propagation

🗑️ Invalidation Request PURGE /api/resource/* Invalidation Coordinator Fan-out to all cache nodes Cache Node 1 ✓ Purged Cache Node 2 ✓ Purged Cache Node 3 ✓ Purged Cache Node N ✓ Purged Propagation: <1 second
Invalidation Propagation — Coordinator Fan-out Pattern

Cache Stampede Prevention

Request Coalescing

Scenario: 1000 concurrent requests for same uncached URL

Without Coalescing:
- 1000 requests to origin
- Origin overload
- High latency

With Coalescing:
1. First request → Fetch from origin
2. Subsequent 999 requests → Wait for first
3. All 1000 requests served from single origin fetch

Implementation:
- Lock on cache key
- First request fetches
- Others wait on lock
- Broadcast result to all waiters

Stale-While-Revalidate

Cache-Control: max-age=3600, stale-while-revalidate=86400

Behavior:
- Content fresh for 1 hour
- After 1 hour, serve stale content
- Async refresh in background
- Next request gets fresh content

Benefits:
- No user-facing latency
- Origin load spread over time
- Always serve fast responses

Compression and Content Encoding

Compression Support

Supported Algorithms:
- Gzip: Good compression, wide support
- Brotli: Better compression, modern browsers
- Deflate: Legacy support

Compression Strategy:
1. Check Accept-Encoding header
2. Serve pre-compressed if available
3. Compress on-the-fly if needed
4. Cache both compressed and uncompressed

Storage:
- Store compressed version
- Decompress if client doesn't support
- Trade-off: CPU vs storage

Monitoring and Observability

Key Metrics

Performance:
- Cache hit rate (%)
- P50, P95, P99 latency
- Requests per second
- Bandwidth (in/out)

Resource:
- CPU utilization
- Memory usage
- Disk usage
- Network bandwidth

Cache:
- Cache size (items, bytes)
- Eviction rate
- Invalidation rate
- Miss rate by reason

Origin:
- Origin requests
- Origin latency
- Origin errors
- Origin bandwidth

This architecture provides a comprehensive foundation for building a high-performance, scalable distributed web caching system.