Distributed Unique ID Generator - Scale and Constraints
Scale Estimation
Traffic Analysis
- Total Users: 1 billion registered users
- Active Users: 100 million daily active users
- ID Generation Rate: 10,000 IDs per second average
- Peak Traffic: 50,000 IDs per second (5x normal)
- Burst Traffic: 100,000 IDs per second for short periods
- Geographic Distribution: 40% Americas, 35% Asia, 25% Europe
ID Generation Patterns
Daily ID Generation:
- Average: 10,000 IDs/sec × 86,400 sec = 864 million IDs/day
- Peak hours: 50,000 IDs/sec × 3,600 sec = 180 million IDs/hour
- Burst periods: 100,000 IDs/sec × 60 sec = 6 million IDs/minute
Annual ID Generation:
- 864M IDs/day × 365 days = 315 billion IDs/year
- At this rate: 2^63 / 315B = ~29,000 years until exhaustionStorage and Memory Requirements
Per-Node Memory:
- Application code: 50MB
- Configuration: 1MB
- Runtime state: 10MB
- Monitoring/metrics: 20MB
- Buffer space: 19MB
Total per node: ~100MB
Cluster Memory (100 nodes):
- 100 nodes × 100MB = 10GB total
- Negligible compared to typical infrastructureCapacity Planning
Node Capacity Analysis
Single Node Capacity:
- Theoretical max: 4,096 IDs/millisecond = 4,096,000 IDs/sec
- Practical limit: 10,000 IDs/sec (accounting for overhead)
- CPU cores: 2-4 cores sufficient
- Memory: 100MB per node
- Network: 1Mbps bandwidth sufficient
Cluster Capacity:
- 10 nodes: 100,000 IDs/sec
- 50 nodes: 500,000 IDs/sec
- 100 nodes: 1,000,000 IDs/sec
- Provides 100x headroom for 10K IDs/sec averageWorker ID Space Allocation
Snowflake Format (64-bit):
- Timestamp: 41 bits = 69 years (with custom epoch)
- Datacenter ID: 5 bits = 32 datacenters
- Worker ID: 5 bits = 32 workers per datacenter
- Sequence: 12 bits = 4,096 IDs per millisecond
Total Capacity:
- 32 datacenters × 32 workers = 1,024 total workers
- 1,024 workers × 10,000 IDs/sec = 10.24M IDs/sec theoretical
- Provides 1,000x headroom for current loadDatacenter Distribution
Geographic Distribution:
- US-East: 10 workers (40% traffic)
- US-West: 8 workers (30% traffic)
- EU-West: 6 workers (20% traffic)
- Asia-Pacific: 8 workers (10% traffic)
Per-Datacenter Capacity:
- US-East: 10 workers × 10K IDs/sec = 100K IDs/sec
- US-West: 8 workers × 10K IDs/sec = 80K IDs/sec
- EU-West: 6 workers × 10K IDs/sec = 60K IDs/sec
- APAC: 8 workers × 10K IDs/sec = 80K IDs/secPerformance Constraints
Latency Breakdown
ID Generation Latency:
- Get current timestamp: 0.01ms
- Increment sequence: 0.01ms (atomic operation)
- Bit manipulation: 0.01ms
- Return ID: 0.01ms
Total: ~0.04ms per ID
Network Latency (REST API):
- Client to load balancer: 5ms
- Load balancer to node: 1ms
- ID generation: 0.04ms
- Response: 1ms
Total: ~7ms end-to-end
Batch Generation (100 IDs):
- Setup: 0.01ms
- Generate 100 IDs: 4ms
- Serialize response: 0.5ms
Total: ~4.5ms (0.045ms per ID)Throughput Constraints
Single-Threaded Performance:
- 1 / 0.04ms = 25,000 IDs/sec theoretical
- Practical: 10,000 IDs/sec (accounting for overhead)
Multi-Threaded Performance:
- 4 cores × 10,000 IDs/sec = 40,000 IDs/sec per node
- Lock-free atomic operations enable linear scaling
Sequence Number Limit:
- 4,096 IDs per millisecond per worker
- At 10,000 IDs/sec: 10 IDs/ms average
- Provides 400x headroom before sequence exhaustionResource Utilization
CPU Usage:
- Idle: 0.5%
- Normal load (10K IDs/sec): 3-5%
- Peak load (50K IDs/sec): 15-20%
- Burst load (100K IDs/sec): 30-40%
Memory Usage:
- Base: 50MB
- Per request: 1KB
- 1000 concurrent requests: 51MB
- Peak: 80MB
Network Bandwidth:
- Per ID: 8 bytes (64-bit integer)
- 10,000 IDs/sec: 80KB/sec = 0.64Mbps
- Peak 50K IDs/sec: 400KB/sec = 3.2Mbps
- Negligible network impactScalability Limits
Horizontal Scaling Limits
Worker ID Space:
- 5 bits = 32 workers per datacenter
- 5 bits = 32 datacenters
- Total: 1,024 unique workers globally
Scaling Beyond Limits:
- Reduce datacenter bits to 3 (8 datacenters)
- Increase worker bits to 7 (128 workers per DC)
- Total: 1,024 workers maintained
- Or use 128-bit UUIDs for unlimited scalingTime-Based Limits
Timestamp Exhaustion:
- 41 bits with epoch 2020-01-01
- 2^41 milliseconds = 69.7 years
- Exhaustion date: ~2089-09-07
- Solution: Update epoch in 2089 or migrate to 128-bit IDs
Sequence Overflow:
- 4,096 IDs per millisecond per worker
- At 10K IDs/sec: 10 IDs/ms (safe)
- At 100K IDs/sec: 100 IDs/ms (safe)
- At 1M IDs/sec: 1,000 IDs/ms (safe)
- At 5M IDs/sec: 5,000 IDs/ms (overflow risk)
- Solution: Add more workers or wait for next millisecondGeographic Scaling
Cross-Region Latency:
- Same region: 1-5ms
- Cross-region (US-EU): 80-120ms
- Cross-region (US-Asia): 150-250ms
- Impact: None (no coordination required)
Datacenter Addition:
- Current: 4 datacenters (2 bits used of 5 available)
- Capacity: 32 datacenters total
- Headroom: 28 additional datacenters
- Process: Update configuration, deploy new workersBottleneck Analysis
System Bottlenecks
1. Sequence Number Exhaustion:
- Limit: 4,096 IDs/ms per worker
- Current: 10 IDs/ms average
- Headroom: 400x
- Mitigation: Add workers, implement waiting
2. Clock Synchronization:
- Drift tolerance: 100ms
- NTP accuracy: ±10ms typical
- Headroom: 10x
- Mitigation: Multiple NTP sources, monitoring
3. Worker ID Allocation:
- Limit: 1,024 workers globally
- Current: 32 workers deployed
- Headroom: 32x
- Mitigation: Efficient worker ID reuse
4. Network Bandwidth:
- Per node: 3.2Mbps at peak
- Typical NIC: 1Gbps
- Headroom: 300x
- Mitigation: None neededPerformance Bottlenecks
1. Lock Contention:
- Solution: Lock-free atomic operations
- Impact: Eliminated
2. Memory Allocation:
- Solution: Pre-allocated buffers
- Impact: Minimal
3. Network I/O:
- Solution: Async I/O, connection pooling
- Impact: <1ms overhead
4. Serialization:
- Solution: Binary protocol (gRPC)
- Impact: <0.1ms overheadConstraint Trade-offs
ID Format Trade-offs
64-bit vs 128-bit:
- 64-bit: Database-friendly, limited worker space
- 128-bit: Unlimited scaling, larger storage
- Decision: 64-bit for most use cases, 128-bit for extreme scale
Timestamp Precision:
- Millisecond: 4,096 IDs/ms, 69 years lifetime
- Microsecond: 4 IDs/μs, 69,000 years lifetime
- Decision: Millisecond precision sufficient
Metadata Embedding:
- More metadata: Less sequence space
- Less metadata: More IDs per millisecond
- Decision: Balance based on requirementsConsistency Trade-offs
Clock Synchronization:
- Strict NTP: Better ordering, NTP dependency
- Loose sync: More resilient, less precise ordering
- Decision: Loose sync with monitoring
Worker ID Assignment:
- Static: Simple, manual management
- Dynamic: Complex, automatic management
- Decision: Static for small scale, dynamic for large scale
Uniqueness Guarantee:
- Coordination: 100% guarantee, performance impact
- No coordination: 99.9999% guarantee, high performance
- Decision: No coordination with proper worker ID managementAvailability Trade-offs
Redundancy:
- Single node: Simple, single point of failure
- Multiple nodes: Complex, high availability
- Decision: Multiple nodes per datacenter
Failover:
- Automatic: Fast recovery, risk of split-brain
- Manual: Slower recovery, safer
- Decision: Automatic with proper safeguards
Monitoring:
- Detailed: High overhead, better visibility
- Minimal: Low overhead, less visibility
- Decision: Detailed monitoring with samplingGrowth Projections
5-Year Growth Forecast
Year 1: 10K IDs/sec average
Year 2: 20K IDs/sec (2x growth)
Year 3: 40K IDs/sec (2x growth)
Year 4: 70K IDs/sec (1.75x growth)
Year 5: 100K IDs/sec (1.43x growth)
Infrastructure Needs:
Year 1: 10 nodes (current)
Year 2: 20 nodes
Year 3: 40 nodes
Year 4: 70 nodes
Year 5: 100 nodes
Cost Projection:
Year 1: $1,000/month (10 nodes × $100/node)
Year 2: $2,000/month
Year 3: $4,000/month
Year 4: $7,000/month
Year 5: $10,000/monthThis comprehensive scale and constraints analysis provides the foundation for capacity planning, performance optimization, and long-term scalability of the distributed unique ID generation system.