Architecture

📖 8 min read 📄 Part 3 of 10

API Rate Limiter - System Architecture

High-Level Architecture Overview

System Architecture Principles

  • Distributed Design: Rate limiting enforced across multiple servers
  • Low Latency: <5ms overhead for rate limit decisions
  • High Availability: 99.99% uptime with fault tolerance
  • Horizontal Scalability: Linear scaling with traffic growth
  • Eventual Consistency: Balance accuracy with performance
  • Fail-Safe Design: Configurable fail-open or fail-closed behavior

Core Architecture Components

CLIENT LAYER 🖥️ API Clients 🌐 Web Apps 📱 Mobile Apps 🌍 Load Balancer (Geographic Routing) 🔐 API Gateway (Auth + TLS) 🛡️ Rate Limiter Middleware Token Bucket · Sliding Window · Fixed Window ⚡ Rate Limit Engine Counter Check + Update ⚙️ Config Service Rules & Policies 📊 Metrics & Analytics Monitoring + Alerts 🗄️ Redis Cluster Token Bucket State 🗃️ PostgreSQL Rules + Audit Logs 🔄 Distributed Sync (Cross-Instance Coordination) → Backend Services
Rate Limiter System Architecture — Distributed rate limiting with multiple algorithms and Redis-backed state

Rate Limiting Gateway

Request Interception Flow

📨 Client Request Extract Identifiers User ID · IP Address · API Key · Endpoint Fetch Rate Limit Rules From Configuration Service ⚙️ Config Service Check Counter (Redis) Current usage vs. limit 🗄️ Redis Cluster Within Limit? YES ✓ Allow Request Increment Counter NO ✗ Reject Request Return 429 + Retry-After Header Response Headers: X-RateLimit-Remaining: 0 X-RateLimit-Reset: 1704723660
Request Interception Flow — Each request is evaluated against rate limit rules before reaching backend services

Gateway Components

  • Request Parser: Extract rate limiting identifiers from requests
  • Rule Matcher: Match request to applicable rate limiting rules
  • Counter Manager: Check and update rate limit counters
  • Response Handler: Add rate limit headers, return 429 when exceeded
  • Metrics Collector: Track rate limiting decisions and patterns
  • Cache Manager: Local cache for hot rules and counters

Rate Limiting Algorithms Implementation

Token Bucket Algorithm

Token Bucket Algorithm 🪣 Token Bucket ← Capacity: 100 ← Current: 75 +10 tokens/sec Parameters Capacity:100 tokens (max burst) Refill Rate:10 tokens/second Current:75 tokens available Algorithm Steps ❶ Calculate tokens to add since last refill ❷ Add tokens (capped at capacity) ❸ Check if enough tokens for request ❹ Yes → consume token, allow request ❺ No → reject with retry-after header 🗄️ Redis Data Structure key: user:12345 "tokens": 75 "last_refill": 1704723600 "capacity": 100 "refill_rate": 10 ⚡ Atomic Lua Script 1. GET tokens + last_refill 2. Calculate new token count 3. SET tokens + last_refill 4. Return allow/deny decision
Token Bucket Algorithm — Tokens refill at a steady rate, allowing controlled bursts up to bucket capacity

Sliding Window Counter Algorithm

Sliding Window Counter Algorithm Time Window Visualization Previous Window (10:00–11:00) 800 requests Current Window (11:00–12:00) 300 requests ▲ Now: 11:30 (50% into window) ← Sliding Window (1 hour lookback) → 📐 Weighted Count Calculation Formula: = Previous × (1 − overlap%) + Current Example: = 800 × (1 − 0.5) + 300 = 700 ✓ Decision 700 < 1000 (limit) ALLOW Remaining: 300 requests Window: 1 hour · Limit: 1000 🗄️ Redis Data Structure "user:12345:window:current" → 300 "user:12345:window:previous" → 800 "user:12345:window:start" → 1704722400
Sliding Window Counter — Approximates a true sliding window by weighting the previous window's count based on overlap

Fixed Window Counter Algorithm

Fixed Window Counter Algorithm Fixed Time Window (1 minute) LIMIT: 100 87 / 100 requests 11:00:00 11:00:59 Algorithm Steps ❶ Truncate timestamp → window key ❷ INCR counter for current window ❸ Check if count exceeds limit ❹ count ≤ limit → Allow Characteristics ✓ Simple implementation ✓ Low memory (1 counter per window) ✓ O(1) time complexity ✗ Boundary burst problem (see below) ⚠️ Boundary Burst Problem Window 1 (11:00–11:01) Window 2 (11:01–11:02) 60 + 60 = 120 requests in ~2 seconds! (exceeds 100/min limit at boundary) 🗄️ Redis: "user:12345:window:1704722400" → 87 TTL: 120s (auto-expire after 2 windows)
Fixed Window Counter — Simple and fast but vulnerable to burst traffic at window boundaries

Sliding Window Log Algorithm

Sliding Window Log Algorithm Window Size:1 hour Limit:1000 requests 🎯 Most Accurate Algorithm No boundary issues — true sliding window 📋 Request Log (Sorted Set of Timestamps) expired expired ... valid timestamps within window ... NEW now − 1hr now ← Sliding Window (count these) → Algorithm Steps ❶ ZREMRANGEBYSCORE — remove expired entries ❷ ZCARD — count remaining timestamps ❸ Check if count < limit ❹ Yes → ZADD new timestamp, allow Trade-offs ✓ Perfect accuracy ✓ No boundary issues ✗ High memory (stores all timestamps) ✗ O(n) cleanup per request 🗄️ Redis Sorted Set Commands ZADD user:12345:requests 1704720000 "req_abc" ← Add entry ZREMRANGEBYSCORE user:12345:requests 0 (now-3600) ← Prune old ZCARD user:12345:requests ← Get count
Sliding Window Log — Stores individual request timestamps for perfect accuracy at the cost of higher memory usage

Distributed Rate Limiting Architecture

Centralized Counter Approach

Rate Limiter Server 1 Rate Limiter Server 2 Rate Limiter Server 3 🗄️ Redis Cluster Centralized Counters ✓ Pros • Accurate counting across all servers • Simple implementation • Strong consistency • Single source of truth ✗ Cons • Single point of failure (mitigated by cluster) • Network latency for every request • Redis becomes bottleneck at high scale • Cross-region latency issues
Centralized Counter Approach — All rate limiter servers share a single Redis cluster for accurate, consistent counting

Local Counter with Synchronization

Rate Limiter 1 Local Counter Quota: 333/s Rate Limiter 2 Local Counter Quota: 333/s Rate Limiter 3 Local Counter Quota: 334/s 🔄 Sync Service Periodic Synchronization ✓ Pros • Low latency (local checks) • No single point of failure • Scales horizontally • Resilient to network partitions ✗ Cons • Less accurate (eventual consistency) • Complex quota distribution • Potential over-limit during sync • Rebalancing on server add/remove
Local Counter with Synchronization — Each server maintains local counters with periodic sync for low-latency decisions

Hybrid Approach (Recommended)

Rate Limiter Server 📋 Local Cache Hot Rules ⚡ Rate Limit Engine Algorithm Selection + Evaluation 🔢 Local Counter Fast Path 🔄 Sync Manager Background Sync Strategy 1. Check local counter (fast) 2. If near limit → check Redis 3. Sync to Redis every 1s 4. Redis for cross-server coord 5. Fail to Redis on cache miss 🗄️ Redis Cluster Global Counters ★ Recommended
Hybrid Approach (Recommended) — Combines local counters for speed with Redis for accuracy, syncing in the background

Configuration Service Architecture

Rule Management System

⚙️ Configuration Service ✓ Rule Validator Schema + Logic Checks 🗃️ Rule Store Database (PostgreSQL) 🧠 Rule Engine Matching + Priority ⚡ Rule Cache (Redis) Hot Rules + TTL Expiry 📡 Change Propagation Pub/Sub Notifications to All Instances → Distributes rules to all Rate Limiter instances RULE EXAMPLE: rule_id: "rl_user_api_v1" | priority: 100 | user_tier: "free" | endpoint: "/api/v1/*" | limit: 10 req/s | algo: token_bucket
Configuration Service Architecture — Manages rate limiting rules with validation, caching, and real-time propagation

Rule Matching and Priority

Request: GET /api/v1/users?user_id=12345&api_key=abc123

Rule Matching Process:
1. Extract identifiers: user_id, api_key, endpoint, IP
2. Fetch applicable rules (cached):
   - Global rule: 10000 req/s
   - Endpoint rule: 1000 req/s for /api/v1/*
   - User tier rule: 100 req/s for free tier
   - API key rule: 50 req/s for api_key=abc123
3. Sort by priority (highest first)
4. Apply most restrictive limit
5. Check counter against limit
6. Return decision

Rule Priority:
1. Blocklist (priority 1000) → Immediate reject
2. Allowlist (priority 900) → Bypass rate limiting
3. API Key specific (priority 800)
4. User specific (priority 700)
5. Endpoint specific (priority 600)
6. IP specific (priority 500)
7. Global limits (priority 100)

Metrics and Analytics Architecture

Real-time Metrics Pipeline

🛡️ Rate Limiter Decisions 📊 Metrics Collector ⚡ Stream Processor (Kafka) 🗄️ Time-Series DB (InfluxDB) 📈 Analytics Dashboard (Grafana) Metrics Collected: •Requests per second (by user, endpoint, region) •Throttled requests (by reason, rule) •Latency percentiles (P50, P95, P99) •Cache hit rate •Counter accuracy & Rule evaluation time Data Flow: Decisions → Collection → Processing → Storage → Visualization
Real-time Metrics Pipeline — Rate limiter decisions flow through Kafka to InfluxDB for Grafana dashboards

Alerting and Monitoring

Alerts:
1. High throttle rate (>10% of requests)
2. Latency spike (P99 >10ms)
3. Cache miss rate (>5%)
4. Redis connection failures
5. Configuration sync delays
6. Unusual traffic patterns (potential attack)

Monitoring Dashboards:
1. Real-time traffic overview
2. Per-user quota utilization
3. Per-endpoint throttle rates
4. System health metrics
5. Cost and capacity planning

Security and Abuse Prevention

Multi-Layer Defense

Multi-Layer Security Defense Defense-in-Depth — Requests pass through each layer sequentially Incoming Request Layer 1 IP Blocklist DDoS protection — Block known malicious IPs and botnets at the edge 🛑 Layer 2 API Key Validation Authentication — Verify API keys, tokens, and client identity 🔑 Layer 3 Rate Limiting (this system) Throttling — Enforce request quotas per user, IP, or API key ⚡ Layer 4 WAF Rules Input filtering — Block SQL injection, XSS, and malformed payloads 🛡️ Layer 5 Behavioral Analysis (ML-based) Anomaly detection — Identify credential stuffing, unusual patterns, distributed attacks 🧠 ✓ Allowed → Backend Services
Multi-Layer Security Defense — Five stacked security layers providing defense-in-depth from network edge to application

This architecture provides a robust, scalable, and accurate rate limiting system that can handle massive traffic while maintaining low latency and high availability.