Initial: 3 schedulers (1 leader, 2 followers)
After scaling: 5 schedulers (1 leader, 4 followers)

Benefits:
- Higher availability
- Faster failover
- Better read distribution
- More backup capacity

Process:
1. Add new scheduler instances
2. Join Raft cluster
3. Replicate state from leader
4. Ready to become leader if needed

Downtime: Zero (online scaling)

Initial: 100 workers, 1000 concurrent tasks
After scaling: 200 workers, 2000 concurrent tasks

Auto-Scaling Triggers:
- Queue depth > 1000 → Scale up
- Queue depth < 100 → Scale down
- Worker utilization > 80% → Scale up
- Worker utilization < 30% → Scale down

Scaling Strategy:
- Scale up: Add 10% capacity
- Scale down: Remove 5% capacity
- Cooldown: 5 minutes between scaling events
- Max workers: 10,000

Benefits:
- Linear scaling of execution capacity
- Handle traffic spikes
- Cost optimization during low traffic

Shard by Task ID:
- Shard 0: task_id % 10 = 0
- Shard 1: task_id % 10 = 1
- ...
- Shard 9: task_id % 10 = 9

Benefits:
- Distribute load across shards
- Parallel query execution
- Independent scaling
- Fault isolation

Challenges:
- Cross-shard queries
- Rebalancing complexity
- Transaction coordination

Hierarchical Time Wheels:

Level 1 (Seconds): 60 slots
- Precision: 1 second
- Range: 0-59 seconds
- Tasks: Immediate execution

Level 2 (Minutes): 60 slots
- Precision: 1 minute
- Range: 1-59 minutes
- Tasks: Near-term execution

Level 3 (Hours): 24 slots
- Precision: 1 hour
- Range: 1-23 hours
- Tasks: Today's execution

Level 4 (Days): 30 slots
- Precision: 1 day
- Range: 1-30 days
- Tasks: This month's execution

Benefits:
- O(1) insertion and deletion
- O(1) tick processing
- Memory efficient
- Handles millions of tasks

Task Scanning:
- Scan 1000 tasks at a time
- Process in parallel
- Reduce database load

Task Dispatching:
- Batch 100 tasks per queue operation
- Reduce network overhead
- Higher throughput

Status Updates:
- Buffer updates for 1 second
- Batch write to database
- Reduce write load by 100x

Multi-Level Cache:

L1 (In-Memory):
- Hot tasks (next 5 minutes)
- Size: 1 GB per scheduler
- Hit rate: 90%
- Latency: <1ms

L2 (Redis):
- Warm tasks (next 1 hour)
- Size: 10 GB cluster
- Hit rate: 95%
- Latency: <5ms

L3 (Database):
- All tasks
- Size: 1 TB
- Hit rate: 100%
- Latency: <50ms

Cache Warming:
- Pre-load upcoming tasks
- Refresh every minute
- Predictive loading

Regions:
- US-East (Primary)
- US-West (Secondary)
- EU-West (Primary)
- Asia-Pacific (Primary)

Task Placement:
- User tasks → User's region
- Global tasks → All regions
- Regional tasks → Specific region

Cross-Region Coordination:
- Raft consensus for leader election
- Async replication for task state
- Regional task execution
- Global monitoring

NTP Configuration:
- Sync interval: 60 seconds
- Accuracy: ±10ms
- Max drift: ±100ms

Clock Skew Handling:
- Monitor skew across servers
- Alert if skew > 1 second
- Reject tasks if skew > 5 seconds
- Use logical clocks for ordering

Leap Second Handling:
- Pause scheduling during leap second
- Resume after 1 second
- No task loss

Round Robin:
- Simple, even distribution
- No state required
- May cause hot spots

Least Loaded:
- Route to least busy worker
- Better load distribution
- Requires worker state tracking

Consistent Hashing:
- Route by task ID
- Same task → same worker
- Better cache locality
- Recommended approach

Weighted Distribution:
- Larger workers get more tasks
- Proportional to capacity
- Flexible scaling

Priority-Based Routing:
- High priority → Dedicated workers
- Medium priority → Shared workers
- Low priority → Best effort workers

Queue Depth Monitoring:
- Alert if depth > 1000
- Auto-scale workers
- Throttle task creation
- Prevent overload

Backpressure Handling:
- Reject new tasks if overloaded
- Return 503 Service Unavailable
- Client retry with backoff

Current (Year 1):
- 100M tasks
- 10K executions/second
- 100 workers
- $1M/month

Year 2 (2x growth):
- 200M tasks
- 20K executions/second
- 200 workers
- $2M/month

Year 3 (3x growth):
- 300M tasks
- 30K executions/second
- 300 workers
- $3M/month

Linear scaling with traffic

Per 1K executions/second:
- Workers: 10
- Schedulers: 1
- Database: 100 GB
- Memory: 30 GB
- Cost: $100K/month

Breakdown:
- Workers: 90% ($90K)
- Schedulers: 5% ($5K)
- Database: 3% ($3K)
- Other: 2% ($2K)

Scheduler Metrics:
- Tasks scheduled/second
- Scheduling latency (P50, P99)
- Queue depth
- Leader election count

Worker Metrics:
- Tasks executed/second
- Execution duration (P50, P99)
- Success rate
- Worker utilization

System Metrics:
- Total active tasks
- Overdue tasks
- Failed tasks
- Retry rate

Scale Up Workers:
- Queue depth > 1000 for 5 minutes
- Worker utilization > 80% for 5 minutes
- Execution latency P99 > 10s
- Failed tasks > 5%

Scale Down Workers:
- Queue depth < 100 for 15 minutes
- Worker utilization < 30% for 15 minutes
- All metrics healthy

Cooldown Periods:
- Scale up: 60 seconds
- Scale down: 300 seconds

Max/Min Limits:
- Min workers: 10
- Max workers: 10,000
- Scale increment: 10%

Over-Provisioned:
- 200 workers × $100 = $20K
- 40% utilization
- Wasted: $12K

Right-Sized:
- 120 workers × $100 = $12K
- 70% utilization
- Savings: $8K (40%)

Strategies:
- Monitor actual usage
- Scale down off-peak
- Use spot instances (70% savings)
- Reserved instances (40% savings)

Short Tasks (<1 minute):
- Use lightweight workers
- Fast startup time
- Lower cost per execution

Long Tasks (>10 minutes):
- Use dedicated workers
- Better resource utilization
- Batch similar tasks

Scheduled Optimization:
- Group tasks by time
- Batch execution
- Reduce overhead

Hot/Warm/Cold Tiering:
- Hot (Redis): Next 1 hour, $500/month
- Warm (SSD): Next 24 hours, $1K/month
- Cold (HDD): Older tasks, $500/month
- Archive (S3): Completed tasks, $100/month

Retention Policies:
- Delete completed one-time tasks after 30 days
- Archive execution history after 90 days
- Compress old logs (10:1 ratio)
- Reduce storage by 80%

1. Database Write Throughput:
   - 10K writes/second limit
   - Mitigation: Batch writes, sharding

2. Task Scanning:
   - O(n) scan of upcoming tasks
   - Mitigation: Time-based indexing, time wheel

3. Leader Election:
   - 5 second failover time
   - Mitigation: Faster consensus, more replicas

4. Worker Communication:
   - Network latency
   - Mitigation: Regional deployment, batching

5. Clock Synchronization:
   - Drift causes timing issues
   - Mitigation: NTP, monitoring, tolerance

Database Bottleneck:
- Implement write batching
- Add read replicas
- Shard by task ID
- Use time-series database for executions

Scanning Bottleneck:
- Use time wheel algorithm
- Index by next_execution_time
- Partition by time range
- Parallel scanning

Network Bottleneck:
- Deploy regionally
- Batch task distribution
- Compress payloads
- Use persistent connections