Scale & Constraints

📖 7 min read 📄 Part 2 of 10

Distributed Unique ID Generator - Scale and Constraints

Scale Estimation

Traffic Analysis

  • Total Users: 1 billion registered users
  • Active Users: 100 million daily active users
  • ID Generation Rate: 10,000 IDs per second average
  • Peak Traffic: 50,000 IDs per second (5x normal)
  • Burst Traffic: 100,000 IDs per second for short periods
  • Geographic Distribution: 40% Americas, 35% Asia, 25% Europe

ID Generation Patterns

Daily ID Generation:
- Average: 10,000 IDs/sec × 86,400 sec = 864 million IDs/day
- Peak hours: 50,000 IDs/sec × 3,600 sec = 180 million IDs/hour
- Burst periods: 100,000 IDs/sec × 60 sec = 6 million IDs/minute

Annual ID Generation:
- 864M IDs/day × 365 days = 315 billion IDs/year
- At this rate: 2^63 / 315B = ~29,000 years until exhaustion

Storage and Memory Requirements

Per-Node Memory:
- Application code: 50MB
- Configuration: 1MB
- Runtime state: 10MB
- Monitoring/metrics: 20MB
- Buffer space: 19MB
Total per node: ~100MB

Cluster Memory (100 nodes):
- 100 nodes × 100MB = 10GB total
- Negligible compared to typical infrastructure

Capacity Planning

Node Capacity Analysis

Single Node Capacity:
- Theoretical max: 4,096 IDs/millisecond = 4,096,000 IDs/sec
- Practical limit: 10,000 IDs/sec (accounting for overhead)
- CPU cores: 2-4 cores sufficient
- Memory: 100MB per node
- Network: 1Mbps bandwidth sufficient

Cluster Capacity:
- 10 nodes: 100,000 IDs/sec
- 50 nodes: 500,000 IDs/sec
- 100 nodes: 1,000,000 IDs/sec
- Provides 100x headroom for 10K IDs/sec average

Worker ID Space Allocation

Snowflake Format (64-bit):
- Timestamp: 41 bits = 69 years (with custom epoch)
- Datacenter ID: 5 bits = 32 datacenters
- Worker ID: 5 bits = 32 workers per datacenter
- Sequence: 12 bits = 4,096 IDs per millisecond

Total Capacity:
- 32 datacenters × 32 workers = 1,024 total workers
- 1,024 workers × 10,000 IDs/sec = 10.24M IDs/sec theoretical
- Provides 1,000x headroom for current load

Datacenter Distribution

Geographic Distribution:
- US-East: 10 workers (40% traffic)
- US-West: 8 workers (30% traffic)
- EU-West: 6 workers (20% traffic)
- Asia-Pacific: 8 workers (10% traffic)

Per-Datacenter Capacity:
- US-East: 10 workers × 10K IDs/sec = 100K IDs/sec
- US-West: 8 workers × 10K IDs/sec = 80K IDs/sec
- EU-West: 6 workers × 10K IDs/sec = 60K IDs/sec
- APAC: 8 workers × 10K IDs/sec = 80K IDs/sec

Performance Constraints

Latency Breakdown

ID Generation Latency:
- Get current timestamp: 0.01ms
- Increment sequence: 0.01ms (atomic operation)
- Bit manipulation: 0.01ms
- Return ID: 0.01ms
Total: ~0.04ms per ID

Network Latency (REST API):
- Client to load balancer: 5ms
- Load balancer to node: 1ms
- ID generation: 0.04ms
- Response: 1ms
Total: ~7ms end-to-end

Batch Generation (100 IDs):
- Setup: 0.01ms
- Generate 100 IDs: 4ms
- Serialize response: 0.5ms
Total: ~4.5ms (0.045ms per ID)

Throughput Constraints

Single-Threaded Performance:
- 1 / 0.04ms = 25,000 IDs/sec theoretical
- Practical: 10,000 IDs/sec (accounting for overhead)

Multi-Threaded Performance:
- 4 cores × 10,000 IDs/sec = 40,000 IDs/sec per node
- Lock-free atomic operations enable linear scaling

Sequence Number Limit:
- 4,096 IDs per millisecond per worker
- At 10,000 IDs/sec: 10 IDs/ms average
- Provides 400x headroom before sequence exhaustion

Resource Utilization

CPU Usage:
- Idle: 0.5%
- Normal load (10K IDs/sec): 3-5%
- Peak load (50K IDs/sec): 15-20%
- Burst load (100K IDs/sec): 30-40%

Memory Usage:
- Base: 50MB
- Per request: 1KB
- 1000 concurrent requests: 51MB
- Peak: 80MB

Network Bandwidth:
- Per ID: 8 bytes (64-bit integer)
- 10,000 IDs/sec: 80KB/sec = 0.64Mbps
- Peak 50K IDs/sec: 400KB/sec = 3.2Mbps
- Negligible network impact

Scalability Limits

Horizontal Scaling Limits

Worker ID Space:
- 5 bits = 32 workers per datacenter
- 5 bits = 32 datacenters
- Total: 1,024 unique workers globally

Scaling Beyond Limits:
- Reduce datacenter bits to 3 (8 datacenters)
- Increase worker bits to 7 (128 workers per DC)
- Total: 1,024 workers maintained
- Or use 128-bit UUIDs for unlimited scaling

Time-Based Limits

Timestamp Exhaustion:
- 41 bits with epoch 2020-01-01
- 2^41 milliseconds = 69.7 years
- Exhaustion date: ~2089-09-07
- Solution: Update epoch in 2089 or migrate to 128-bit IDs

Sequence Overflow:
- 4,096 IDs per millisecond per worker
- At 10K IDs/sec: 10 IDs/ms (safe)
- At 100K IDs/sec: 100 IDs/ms (safe)
- At 1M IDs/sec: 1,000 IDs/ms (safe)
- At 5M IDs/sec: 5,000 IDs/ms (overflow risk)
- Solution: Add more workers or wait for next millisecond

Geographic Scaling

Cross-Region Latency:
- Same region: 1-5ms
- Cross-region (US-EU): 80-120ms
- Cross-region (US-Asia): 150-250ms
- Impact: None (no coordination required)

Datacenter Addition:
- Current: 4 datacenters (2 bits used of 5 available)
- Capacity: 32 datacenters total
- Headroom: 28 additional datacenters
- Process: Update configuration, deploy new workers

Bottleneck Analysis

System Bottlenecks

1. Sequence Number Exhaustion:
   - Limit: 4,096 IDs/ms per worker
   - Current: 10 IDs/ms average
   - Headroom: 400x
   - Mitigation: Add workers, implement waiting

2. Clock Synchronization:
   - Drift tolerance: 100ms
   - NTP accuracy: ±10ms typical
   - Headroom: 10x
   - Mitigation: Multiple NTP sources, monitoring

3. Worker ID Allocation:
   - Limit: 1,024 workers globally
   - Current: 32 workers deployed
   - Headroom: 32x
   - Mitigation: Efficient worker ID reuse

4. Network Bandwidth:
   - Per node: 3.2Mbps at peak
   - Typical NIC: 1Gbps
   - Headroom: 300x
   - Mitigation: None needed

Performance Bottlenecks

1. Lock Contention:
   - Solution: Lock-free atomic operations
   - Impact: Eliminated

2. Memory Allocation:
   - Solution: Pre-allocated buffers
   - Impact: Minimal

3. Network I/O:
   - Solution: Async I/O, connection pooling
   - Impact: <1ms overhead

4. Serialization:
   - Solution: Binary protocol (gRPC)
   - Impact: <0.1ms overhead

Constraint Trade-offs

ID Format Trade-offs

64-bit vs 128-bit:
- 64-bit: Database-friendly, limited worker space
- 128-bit: Unlimited scaling, larger storage
- Decision: 64-bit for most use cases, 128-bit for extreme scale

Timestamp Precision:
- Millisecond: 4,096 IDs/ms, 69 years lifetime
- Microsecond: 4 IDs/μs, 69,000 years lifetime
- Decision: Millisecond precision sufficient

Metadata Embedding:
- More metadata: Less sequence space
- Less metadata: More IDs per millisecond
- Decision: Balance based on requirements

Consistency Trade-offs

Clock Synchronization:
- Strict NTP: Better ordering, NTP dependency
- Loose sync: More resilient, less precise ordering
- Decision: Loose sync with monitoring

Worker ID Assignment:
- Static: Simple, manual management
- Dynamic: Complex, automatic management
- Decision: Static for small scale, dynamic for large scale

Uniqueness Guarantee:
- Coordination: 100% guarantee, performance impact
- No coordination: 99.9999% guarantee, high performance
- Decision: No coordination with proper worker ID management

Availability Trade-offs

Redundancy:
- Single node: Simple, single point of failure
- Multiple nodes: Complex, high availability
- Decision: Multiple nodes per datacenter

Failover:
- Automatic: Fast recovery, risk of split-brain
- Manual: Slower recovery, safer
- Decision: Automatic with proper safeguards

Monitoring:
- Detailed: High overhead, better visibility
- Minimal: Low overhead, less visibility
- Decision: Detailed monitoring with sampling

Growth Projections

5-Year Growth Forecast

Year 1: 10K IDs/sec average
Year 2: 20K IDs/sec (2x growth)
Year 3: 40K IDs/sec (2x growth)
Year 4: 70K IDs/sec (1.75x growth)
Year 5: 100K IDs/sec (1.43x growth)

Infrastructure Needs:
Year 1: 10 nodes (current)
Year 2: 20 nodes
Year 3: 40 nodes
Year 4: 70 nodes
Year 5: 100 nodes

Cost Projection:
Year 1: $1,000/month (10 nodes × $100/node)
Year 2: $2,000/month
Year 3: $4,000/month
Year 4: $7,000/month
Year 5: $10,000/month

This comprehensive scale and constraints analysis provides the foundation for capacity planning, performance optimization, and long-term scalability of the distributed unique ID generation system.