Scale & Constraints

📖 6 min read 📄 Part 2 of 10

Stock Trading Platform - Scale and Constraints

Trading Volume

Order Statistics

Daily Orders: 100 million orders
Orders per Second: 1,157 average, 10,000 peak
Order Types: 40% market, 40% limit, 20% stop orders
Order Size: Average $10,000 per order
Cancellation Rate: 30% of orders cancelled
Modification Rate: 10% of orders modified

Trade Statistics

Daily Trades: 50 million trades (50% fill rate)
Trades per Second: 578 average, 5,000 peak
Average Trade Value: $10,000
Daily Trading Volume: $500 billion
Peak Trading: First/last hour (3x normal)

Market Data Volume

Symbols: 10,000 tradable symbols
Ticks per Symbol: 1 million ticks/day
Total Ticks: 10 billion ticks/day
Tick Rate: 115K ticks/second average
Peak Tick Rate: 500K ticks/second
Data Size: 100 bytes per tick = 1 TB/day

Latency Requirements

Order Processing Latency

Order Placement → Acknowledgment: <100 microseconds
Order Matching: <500 microseconds
Trade Execution: <1 millisecond
Order Cancellation: <100 microseconds
Portfolio Update: <1 millisecond

Breakdown:
- Network: 50 microseconds
- Order validation: 20 microseconds
- Matching engine: 500 microseconds
- Database write: 200 microseconds
- Response: 50 microseconds
Total: <1 millisecond

Market Data Latency

Exchange → Platform: <50 microseconds
Platform → User: <50 microseconds
Total: <100 microseconds

Requirements:
- Co-location with exchanges
- Kernel bypass networking (DPDK)
- Zero-copy data transfer
- Lock-free data structures

Throughput Requirements

Order Processing

Sustained: 1 million orders/second
Peak: 10 million orders/second (market open/close)
Per Symbol: 100 orders/second average
Hot Symbols: 10,000 orders/second (AAPL, TSLA)

Processing Pipeline:
- Order validation: 1M ops/second
- Risk checks: 1M ops/second
- Matching engine: 1M matches/second
- Trade settlement: 500K trades/second

Market Data Distribution

Subscribers: 1 million concurrent users
Updates per Second: 115K ticks/second
Fan-out: 115K × 1M = 115 billion updates/second
Bandwidth: 115B × 100 bytes = 11.5 TB/second
Compression: 10x reduction = 1.15 TB/second

Storage Requirements

Hot Data (In-Memory)

Order Book (per symbol):
- 10,000 symbols × 1 MB = 10 GB
- Top 100 price levels per side
- Real-time updates

Active Orders:
- 10 million active orders × 200 bytes = 2 GB
- Pending limit orders
- Stop orders

User Sessions:
- 1 million users × 10 KB = 10 GB
- Portfolio, positions, balances

Total Hot Data: 25 GB (fits in RAM)

Warm Data (SSD)

Today's Trades:
- 50M trades × 500 bytes = 25 GB
- Intraday history

Recent Orders:
- 100M orders × 300 bytes = 30 GB
- Last 24 hours

Market Data:
- 10B ticks × 100 bytes = 1 TB
- Today's tick data

Total Warm Data: 1.1 TB

Cold Data (HDD/Archive)

Historical Trades:
- 50M trades/day × 365 days × 5 years = 91B trades
- 91B × 500 bytes = 45 TB

Historical Orders:
- 100M orders/day × 365 days × 5 years = 182B orders
- 182B × 300 bytes = 55 TB

Historical Market Data:
- 10B ticks/day × 365 days × 5 years = 18 trillion ticks
- 18T × 100 bytes = 1.8 PB (compressed to 180 TB)

Total Cold Data: 280 TB

Network Bandwidth

Inbound Traffic

Order Submissions:
- 1M orders/second × 500 bytes = 500 MB/s = 4 Gbps

Market Data from Exchanges:
- 115K ticks/second × 100 bytes = 11.5 MB/s = 92 Mbps

Total Inbound: 5 Gbps

Outbound Traffic

Market Data to Users:
- 1M users × 115K ticks/second × 100 bytes = 11.5 TB/s
- With compression (10x): 1.15 TB/s = 9.2 Tbps
- With selective subscription (10% active): 920 Gbps

Order Confirmations:
- 1M orders/second × 300 bytes = 300 MB/s = 2.4 Gbps

Total Outbound: 1 Tbps (with optimizations)

Compute Requirements

Matching Engine

Servers: 100 instances (64 cores, 256 GB RAM each)
- 10 symbols per instance
- Lock-free matching
- In-memory order books
- <500 microsecond latency

CPU: 6,400 cores
RAM: 25 TB
Network: 100 Gbps per server

Market Data Gateway

Servers: 50 instances (32 cores, 128 GB RAM each)
- Receive from exchanges
- Normalize and distribute
- <50 microsecond latency

CPU: 1,600 cores
RAM: 6.4 TB
Network: 200 Gbps per server

Risk Management

Servers: 20 instances (32 cores, 128 GB RAM each)
- Real-time risk calculations
- Position limits
- Margin requirements
- <100 microsecond checks

CPU: 640 cores
RAM: 2.6 TB

Cost Estimates

Infrastructure Costs (Monthly)

Compute:
- Matching engines: $200K/month
- Market data gateways: $100K/month
- Risk management: $40K/month
- Application servers: $60K/month
Total Compute: $400K/month

Storage:
- Hot data (RAM): $50K/month
- Warm data (SSD): $20K/month
- Cold data (HDD): $10K/month
Total Storage: $80K/month

Network:
- Co-location: $100K/month
- Bandwidth (1 Tbps): $200K/month
- Direct exchange connections: $50K/month
Total Network: $350K/month

Exchange Fees:
- Market data feeds: $500K/month
- Connectivity: $100K/month
Total Exchange: $600K/month

Grand Total: $1.43M/month = $17.2M/year

Cost per Trade

Infrastructure: $17.2M / 18B trades = $0.00096 per trade
Revenue: $2 per trade (commission)
Gross Margin: $1.999 per trade (99.95%)

Performance Bottlenecks

Critical Bottlenecks

  1. Matching Engine: 1M orders/second per instance
  2. Network Latency: 50 microseconds to exchange
  3. Memory Bandwidth: 100 GB/s for order book updates
  4. Lock Contention: Eliminated with lock-free structures
  5. Database Writes: 500K writes/second for trades

Mitigation Strategies

Matching Engine:
- Lock-free data structures
- CPU pinning
- NUMA optimization
- Kernel bypass

Network:
- Co-location with exchanges
- DPDK (Data Plane Development Kit)
- RDMA (Remote Direct Memory Access)
- 10/40/100 Gbps NICs

Memory:
- In-memory order books
- Zero-copy operations
- Huge pages (2 MB)
- Cache-line optimization

Database:
- Async writes
- Batch commits
- Write-ahead logging
- SSD with NVMe

Latency Budget

End-to-End Latency Breakdown

User → Platform:
- Network (internet): 10-50 ms
- Load balancer: 100 μs
- API gateway: 200 μs

Order Processing:
- Validation: 20 μs
- Risk check: 100 μs
- Matching engine: 500 μs
- Trade execution: 100 μs

Database:
- Write to WAL: 200 μs
- Async commit: (background)

Response:
- Serialize: 50 μs
- Network: 100 μs

Total: ~1 millisecond (platform)
Total with internet: 11-51 milliseconds

Availability Requirements

Uptime Targets

Trading Hours: 6.5 hours/day (9:30 AM - 4:00 PM ET)
Annual Trading Hours: 6.5 × 252 days = 1,638 hours
Target Uptime: 99.999%
Allowed Downtime: 5.9 minutes/year = 1.4 seconds/day

Disaster Recovery:
- RTO: 1 second
- RPO: 0 (zero data loss)
- Failover: Automatic, <100 milliseconds

This scale analysis demonstrates the extreme performance requirements of a high-frequency trading platform, requiring specialized hardware, optimized software, and co-location with exchanges to achieve microsecond latencies.