Architecture

📖 8 min read 📄 Part 3 of 10

Distributed Unique ID Generator - System Architecture

High-Level Architecture Overview

System Architecture Principles

  • Coordination-Free Design: No inter-node communication required
  • Stateless Nodes: Each node operates independently
  • Time-Based Ordering: Leverage timestamps for sortability
  • Horizontal Scalability: Add nodes without coordination
  • High Availability: No single point of failure
  • Low Latency: Sub-millisecond ID generation

Core Architecture Components

👥 Client Applications (Web, Mobile, Microservices, DBs) ⚖️ Load Balancer (HAProxy, NGINX, Geographic Routing) 🔑 Generator 1 Worker ID: 1 ⏱️ Timestamp + Seq 🔑 Generator 2 Worker ID: 2 ⏱️ Timestamp + Seq 🔑 Generator 3 Worker ID: 3 ⏱️ Timestamp + Seq 🔑 Generator N Worker ID: N ⏱️ Timestamp + Seq 64-bit ID: [1 sign] [41 timestamp] [10 machine ID] [12 sequence] 🐘 Configuration Store (ZooKeeper / etcd / Consul) Dynamic Worker ID Assignment • Clock Sync • Leader Election 📊 Monitoring & Metrics (Prometheus, Grafana) 🕐 NTP Clock Sync Service
Unique ID Generator — Snowflake Architecture with ZooKeeper Coordination

Snowflake ID Format Design

64-Bit ID Structure

64-Bit Snowflake ID Structure S Timestamp 41 bits = 69 years DC 5 bits Worker 5 bits Sequence 12 bits 63 62 22 17 12 0 Bit Allocation: • Bit 63: Sign bit (always 0 for positive numbers) • Bits 62–22: Timestamp (ms since custom epoch) → 32 DCs • Bits 21–17: Datacenter ID (0–31) → 32 workers/DC • Bits 16–12: Worker ID (0–31) • Bits 11–0: Sequence (0–4095) → 4096 IDs/ms
64-Bit Snowflake ID — Bit allocation across sign, timestamp, datacenter, worker, and sequence fields

ID Generation Algorithm

class SnowflakeIDGenerator:
    def __init__(self, datacenter_id, worker_id, epoch=1609459200000):
        self.datacenter_id = datacenter_id  # 0-31
        self.worker_id = worker_id          # 0-31
        self.epoch = epoch                  # Custom epoch (2021-01-01)
        self.sequence = 0
        self.last_timestamp = -1
        
    def generate_id(self):
        timestamp = self.current_timestamp()
        
        # Handle clock moving backwards
        if timestamp < self.last_timestamp:
            raise Exception(f"Clock moved backwards. Refusing to generate ID")
        
        # Same millisecond - increment sequence
        if timestamp == self.last_timestamp:
            self.sequence = (self.sequence + 1) & 0xFFF  # 12-bit mask
            
            # Sequence overflow - wait for next millisecond
            if self.sequence == 0:
                timestamp = self.wait_next_millis(self.last_timestamp)
        else:
            # New millisecond - reset sequence
            self.sequence = 0
        
        self.last_timestamp = timestamp
        
        # Construct ID
        id = ((timestamp - self.epoch) << 22) | \
             (self.datacenter_id << 17) | \
             (self.worker_id << 12) | \
             self.sequence
        
        return id
    
    def current_timestamp(self):
        return int(time.time() * 1000)  # Milliseconds
    
    def wait_next_millis(self, last_timestamp):
        timestamp = self.current_timestamp()
        while timestamp <= last_timestamp:
            timestamp = self.current_timestamp()
        return timestamp

Alternative ID Generation Strategies

Instagram-Style IDs

Instagram-Style 64-Bit ID Timestamp 41 bits — ms since epoch Shard ID 13 bits — 8192 shards Sequence 10 bits — 1024/ms ✓ Advantages • Simpler structure (no DC/worker split) • More sequence space (23 bits combined) • Shard-aware for database partitioning ✗ Disadvantages • Less metadata embedded in ID • Harder to debug (no explicit worker ID)
Instagram-Style 64-Bit ID — Simplified two-segment layout with shard-aware sequencing

UUID v1 (Time-Based)

UUID v1 — 128-Bit Structure Time Low 32 bits Time Mid 16 bits Time Hi+Ver 16 bits Clock Sequence + Node (MAC) 64 bits ✓ Advantages • Standardized format (RFC 4122) • 128-bit space (virtually unlimited) • Includes MAC address for uniqueness ✗ Disadvantages • Larger storage (16 bytes vs 8 bytes) • Privacy concerns (MAC address exposure) • Not sortable in standard form
UUID v1 (Time-Based) — 128-bit RFC 4122 structure with embedded timestamp and MAC address

ULID (Universally Unique Lexicographically Sortable ID)

ULID — 128-Bit Lexicographically Sortable ID Timestamp 48 bits — ms since epoch Cryptographic Randomness 80 bits — secure random 01ARZ3NDEKTSV4RRFFQ69G5FAV ← Timestamp → ← Randomness → ✓ Advantages • Lexicographically sortable • No coordination • Case-insensitive base32 • 128-bit space ✗ Disadvantages • Larger storage than Snowflake (16 bytes) • Random component (no embedded metadata)
ULID — Universally Unique Lexicographically Sortable Identifier with 48-bit timestamp prefix

Multi-Datacenter Architecture

Geographic Distribution

Multi-Datacenter Architecture — Global Distribution 🇺🇸 US-East DC DC ID: 0 Workers 0–9 (10 nodes) ⚖️ Load Balancer ❤️ Health Checks 📊 Monitoring 🇪🇺 EU-West DC DC ID: 1 Workers 0–9 (10 nodes) ⚖️ Load Balancer ❤️ Health Checks 📊 Monitoring 🌏 APAC DC DC ID: 2 Workers 0–9 (10 nodes) ⚖️ Load Balancer ❤️ Health Checks 📊 Monitoring 🌐 Global Monitoring & Configuration Centralized metrics • Alerting • Config distribution
Multi-Datacenter Architecture — Independent datacenters with global monitoring overlay

Datacenter Failover Strategy

Normal Operation:
ClientGeoDNSNearest DCID Generator

Datacenter Failure:
ClientGeoDNSNext Nearest DCID Generator
                  (Automatic failover)

Benefits:
- No coordination between DCs
- Independent operation
- Automatic geographic routing
- Graceful degradation

Worker ID Management

Static Worker ID Assignment

Configuration File (config.yaml):
datacenter_id: 0
worker_id: 5
epoch: 1609459200000  # 2021-01-01 00:00:00 UTC
port: 8080

Advantages:
- Simple configuration
- No external dependencies
- Fast startup
- Predictable behavior

Disadvantages:
- Manual management
- Risk of conflicts
- Difficult to scale dynamically

Dynamic Worker ID Assignment

Dynamic Worker ID Assignment Flow 1. Node Startup Connect to ZooKeeper/etcd Request available worker ID Create ephemeral node: /workers/dc0/worker5 Start ID generation 2. Node Operation Maintain heartbeat to coordination service Renew ephemeral node lease Monitor for worker ID conflicts 3. Node Shutdown (Graceful) Graceful shutdown signal received Stop accepting new requests Complete in-flight requests Release worker ID (delete ephemeral node) Exit process 4. Node Failure (Ungraceful) Heartbeat timeout detected Ephemeral node deleted automatically Worker ID becomes available New node can claim ID after grace period 🐘 ZooKeeper Structure /id-generator/datacenters/dc0/workers/ worker0 (ephemeral) worker1 (ephemeral) worker2 (ephemeral) Each node = hostname, IP, port, started_at, version
Dynamic Worker ID Assignment — Lifecycle from startup through operation, shutdown, and failure recovery

Clock Synchronization and Time Management

NTP Synchronization Architecture

NTP Synchronization Architecture 🕐 NTP Server 1 Primary 🕐 NTP Server 2 Secondary 🕐 NTP Server 3 Tertiary 🔑 ID Generator Node • ntpd daemon • Clock monitor • Drift detection • Auto-correction Clock Monitoring Rules ✓ Check every 10s ⚠️ Alert if drift > 100ms ✗ Refuse if drift > 1s
NTP Synchronization — Multi-server time sync with drift detection and automatic correction

Handling Clock Regression

class ClockManager:
    def __init__(self):
        self.last_timestamp = 0
        self.clock_regression_count = 0
        
    def get_timestamp(self):
        current = int(time.time() * 1000)
        
        if current < self.last_timestamp:
            # Clock moved backwards
            regression = self.last_timestamp - current
            self.clock_regression_count += 1
            
            if regression < 5:  # Less than 5ms
                # Small regression - wait it out
                time.sleep(regression / 1000.0)
                return self.last_timestamp
            elif regression < 1000:  # Less than 1 second
                # Medium regression - use last timestamp
                logging.warning(f"Clock regression: {regression}ms")
                return self.last_timestamp
            else:
                # Large regression - refuse to generate
                raise ClockRegressionError(
                    f"Clock moved backwards by {regression}ms"
                )
        
        self.last_timestamp = current
        return current

API Design and Service Interface

REST API Endpoints

GET /api/v1/id
- Generate single ID
- Response: {"id": 1234567890123456789}

GET /api/v1/ids?count=100
- Generate multiple IDs
- Response: {"ids": [123..., 456..., 789...]}

GET /api/v1/parse?id=1234567890123456789
- Parse ID components
- Response: {
    "timestamp": "2024-01-03T19:30:00Z",
    "datacenter_id": 0,
    "worker_id": 5,
    "sequence": 42
  }

GET /api/v1/health
- Health check endpoint
- Response: {
    "status": "healthy",
    "worker_id": 5,
    "datacenter_id": 0,
    "uptime_seconds": 86400,
    "ids_generated": 1000000
  }

GET /api/v1/metrics
- Prometheus metrics endpoint
- Response: Prometheus format metrics

gRPC Service Definition

syntax = "proto3";

service IDGenerator {
  rpc GenerateID(GenerateIDRequest) returns (GenerateIDResponse);
  rpc GenerateBatch(GenerateBatchRequest) returns (GenerateBatchResponse);
  rpc ParseID(ParseIDRequest) returns (ParseIDResponse);
  rpc HealthCheck(HealthCheckRequest) returns (HealthCheckResponse);
}

message GenerateIDRequest {}

message GenerateIDResponse {
  int64 id = 1;
}

message GenerateBatchRequest {
  int32 count = 1;  // Number of IDs to generate
}

message GenerateBatchResponse {
  repeated int64 ids = 1;
}

message ParseIDRequest {
  int64 id = 1;
}

message ParseIDResponse {
  int64 timestamp_ms = 1;
  int32 datacenter_id = 2;
  int32 worker_id = 3;
  int32 sequence = 4;
}

message HealthCheckRequest {}

message HealthCheckResponse {
  string status = 1;
  int32 worker_id = 2;
  int32 datacenter_id = 3;
  int64 uptime_seconds = 4;
  int64 ids_generated = 5;
}

Monitoring and Observability

Key Metrics to Track

Performance Metrics:
- id_generation_latency_ms (histogram)
- id_generation_rate (counter)
- sequence_overflow_count (counter)
- clock_regression_count (counter)

Resource Metrics:
- cpu_usage_percent (gauge)
- memory_usage_bytes (gauge)
- goroutines_count (gauge)

Health Metrics:
- uptime_seconds (gauge)
- last_id_timestamp (gauge)
- clock_drift_ms (gauge)
- ntp_sync_status (gauge)

Business Metrics:
- total_ids_generated (counter)
- ids_per_second (gauge)
- error_rate (counter)

Alerting Rules

alerts:
  - name: HighClockDrift
    condition: clock_drift_ms > 100
    severity: warning
    action: Page on-call engineer
    
  - name: ClockRegression
    condition: clock_regression_count > 10 in 1m
    severity: critical
    action: Page on-call engineer
    
  - name: SequenceOverflow
    condition: sequence_overflow_count > 100 in 1m
    severity: warning
    action: Scale up workers
    
  - name: HighLatency
    condition: p99(id_generation_latency_ms) > 10
    severity: warning
    action: Investigate performance

This comprehensive architecture provides a robust, scalable, and efficient foundation for distributed unique ID generation across multiple datacenters with high availability and low latency.