Horizontal Scaling:
- 10 API server instances
- Load balanced (round-robin)
- Stateless (can scale infinitely)
- Each handles 1,000 req/s

Optimization:
- Caching layer (80% hit rate)
- Request prioritization
- Rate limiting per user
- Connection pooling to etcd

Capacity:
- 10 instances × 1,000 req/s = 10K req/s
- Scale to 50 instances for 50K req/s

Vertical Scaling:
- Increase CPU (8 → 16 cores)
- Increase memory (16 → 32 GB)
- Faster SSD (NVMe)
- Dedicated network

Optimization:
- Compaction (remove old revisions)
- Defragmentation (reclaim space)
- Pagination (limit list size)
- Watch bookmarks (efficient resume)

Limits:
- Max writes: 1,000/second
- Max database size: 8 GB
- Max watch streams: 10,000

Workarounds:
- Separate etcd for events
- Aggregate updates
- Client-side caching (informers)

Horizontal Scaling:
- 5 scheduler instances
- Leader election (only leader schedules)
- Automatic failover

Optimization:
- Pre-filter nodes (remove infeasible)
- Parallel scoring
- Cache node information
- Batch scheduling

Performance:
- 1,000 pods/second
- <1 second per pod
- 5,000 node evaluations per pod

Cluster Autoscaler:
- Monitor resource utilization
- Add nodes when pods pending
- Remove nodes when underutilized

Scale-Up Triggers:
- Pending pods for >30 seconds
- CPU utilization > 80%
- Memory utilization > 80%

Scale-Down Triggers:
- Node utilization < 50% for 10 minutes
- All pods can be rescheduled
- No local storage

Limits:
- Max nodes: 5,000 per cluster
- Scale-up: 100 nodes at a time
- Scale-down: 10 nodes at a time

Pods per Node: 110 (default limit)

Factors:
- IP address availability (254 per /24)
- Kubelet overhead
- Container runtime overhead
- Resource fragmentation

Optimization:
- Larger node CIDR (/23 = 510 IPs)
- Efficient resource packing
- Reduce system overhead
- Increase to 250 pods/node

CoreDNS:
- 3 replicas for HA
- 10,000 queries/second per replica
- 30K total capacity

Optimization:
- Cache DNS responses (30-second TTL)
- Negative caching
- Parallel queries
- Horizontal scaling

Scaling:
- Add replicas based on query rate
- 1 replica per 10K queries/second
- Auto-scale with HPA

Challenge: 1,000 network policies across 150K pods

Implementation:
- eBPF for efficient filtering
- Compile policies to eBPF programs
- Kernel-level enforcement
- <1ms latency overhead

Optimization:
- Policy aggregation
- Namespace-level policies
- Avoid per-pod policies
- Cache policy decisions

Service Types:

ClusterIP (Internal):
- iptables/IPVS rules
- Kube-proxy on each node
- Local load balancing
- No external traffic

LoadBalancer (External):
- Cloud load balancer (ELB, GCE LB)
- External IP assigned
- Health checks
- SSL termination

Ingress:
- HTTP/HTTPS routing
- Path-based routing
- Host-based routing
- SSL termination
- Single entry point

Scaling:
- ClusterIP: Scales with nodes
- LoadBalancer: Cloud provider handles
- Ingress: Scale ingress controller pods

Dynamic Provisioning:
- Create volumes on demand
- Storage classes for different tiers
- Automatic binding
- Lifecycle management

Scaling:
- 50,000 volumes per cluster
- 100 provisions/minute
- Parallel provisioning
- Cloud provider APIs

Optimization:
- Pre-provision common sizes
- Volume pools
- Fast provisioners (local SSD)
- Async provisioning

Challenge: Attach 50K volumes to pods

CSI (Container Storage Interface):
- Standard interface
- Pluggable storage drivers
- Parallel operations
- Timeout handling

Performance:
- Attach: <10 seconds
- Detach: <5 seconds
- Mount: <5 seconds
- Concurrent: 100 operations

Metrics:
- CPU utilization
- Memory utilization
- Custom metrics (requests/second)
- External metrics (queue depth)

Algorithm:
desired_replicas = current_replicas × (current_metric / target_metric)

Example:
- Current: 3 replicas
- Current CPU: 80%
- Target CPU: 50%
- Desired: 3 × (80/50) = 4.8 → 5 replicas

Constraints:
- Min replicas: 2
- Max replicas: 10
- Scale-up: Max 2x per minute
- Scale-down: Max 50% per minute
- Stabilization: 5-minute window

Recommendations:
- Analyze historical usage
- Recommend requests/limits
- Update pods with new values

Modes:
- Off: Recommendations only
- Initial: Set on pod creation
- Auto: Update running pods

Challenges:
- Requires pod restart
- May cause disruption
- Coordinate with HPA

Algorithm:
1. Check for pending pods
2. Simulate scheduling on new node
3. If pods would schedule: Add node
4. Check for underutilized nodes
5. If safe to remove: Delete node

Scale-Up:
- Trigger: Pending pods for >30s
- Action: Add nodes
- Time: 2-5 minutes (cloud provider)

Scale-Down:
- Trigger: Node utilization <50% for 10 min
- Action: Drain and delete node
- Time: 5-10 minutes

Architecture:
- Control plane cluster
- 100 member clusters
- Federated resources
- Cross-cluster scheduling

Use Cases:
- Multi-region deployment
- Disaster recovery
- Burst capacity
- Compliance (data residency)

Challenges:
- Cross-cluster networking
- Consistent configuration
- Unified monitoring
- Cost management

Caching:
- In-memory cache (10 GB)
- Cache frequently accessed objects
- 80% cache hit rate
- Reduce etcd load

Request Prioritization:
- Critical: System components
- High: User operations
- Low: Batch operations
- Shed low-priority under load

Connection Pooling:
- Reuse connections to etcd
- 100 connections per API server
- Connection multiplexing

Pre-Filtering:
- Cache node information
- Filter infeasible nodes early
- Reduce scoring overhead

Parallel Scoring:
- Score nodes in parallel
- Use all CPU cores
- Reduce scheduling latency

Batch Scheduling:
- Schedule multiple pods together
- Amortize overhead
- Better bin packing

Prometheus:
- Scrape metrics from all components
- 15-second scrape interval
- 15-day retention
- Alerting rules

Key Metrics:
- API server latency (p50, p95, p99)
- etcd latency and throughput
- Scheduler latency
- Pod startup time
- Node resource utilization

Alerting:
- API server down
- etcd unhealthy
- Scheduler not scheduling
- Node not ready
- High error rate

Jaeger:
- Trace API requests
- Identify bottlenecks
- Debug performance issues

Sampling:
- 1% of requests
- 100% of errors
- 100% of slow requests (>1s)

Bin Packing:
- Pack pods efficiently on nodes
- Minimize wasted resources
- 70%+ utilization target

Right-Sizing:
- VPA recommendations
- Adjust requests/limits
- Reduce over-provisioning

Spot Instances:
- Use for fault-tolerant workloads
- 70% cost savings
- Handle interruptions gracefully

Namespaces:
- Logical isolation
- Resource quotas
- Network policies
- RBAC

Resource Quotas:
- Limit CPU/memory per namespace
- Prevent resource hogging
- Fair sharing

Benefits:
- Share cluster across teams
- Reduce infrastructure cost
- Better utilization