Phase 1: Requirements (5-10 min)
- Clarify scope (Kubernetes-like system)
- Understand scale (5K nodes, 150K pods)
- Identify key features (scheduling, networking, storage)

Phase 2: High-Level Design (10-15 min)
- Draw architecture: Control Plane + Data Plane
- Explain components (API server, etcd, scheduler, kubelet)
- Discuss data flow

Phase 3: Deep Dive (15-20 min)
- Focus on 2-3 components:
  * Scheduling algorithm
  * Service discovery and networking
  * State management with etcd

Phase 4: Tradeoffs (5-10 min)
- etcd vs other databases
- Overlay vs underlay networking
- Rolling vs blue-green deployments

Phase 5: Wrap-up (5 min)
- Security and RBAC
- Monitoring and operations
- Future improvements

✓ "Control plane has API server, etcd, scheduler, controller manager"
✓ "etcd uses Raft consensus for strong consistency"
✓ "API server is stateless and horizontally scalable"
✗ Avoid: "Single master node" (no HA)

Sample Answer:
"For the control plane, I'll use a distributed architecture:

API Server:
- 10 instances for HA
- Stateless (can scale horizontally)
- Load balanced
- Handles 10K requests/second

etcd:
- 5-node cluster
- Raft consensus for strong consistency
- Stores all cluster state
- 1K writes/second, 10K reads/second

Scheduler:
- 5 instances with leader election
- Only leader schedules
- Automatic failover
- 1,000 pods/second throughput

Controller Manager:
- 50+ controllers
- Reconciliation loops
- Maintain desired state
- Self-healing

This provides 99.99% availability for control plane."

✓ "Two-phase scheduling: filtering then scoring"
✓ "Consider resources, affinity, taints, and tolerations"
✓ "O(n log n) complexity where n = 5,000 nodes"
✗ Avoid: "Random node selection" (poor resource utilization)

Sample Answer:
"For scheduling, I'll use a two-phase algorithm:

Phase 1 - Filtering:
1. Check node resources (CPU, memory available)
2. Check node selector labels
3. Check taints and tolerations
4. Check affinity/anti-affinity rules
Result: Feasible nodes (typically 50-500 nodes)

Phase 2 - Scoring:
For each feasible node, calculate score (0-100):
- Resource balance: 30% weight
- Image locality: 20% weight
- Pod affinity: 25% weight
- Node affinity: 25% weight

Select highest-scoring node and bind pod.

Optimization:
- Cache node information
- Pre-filter early
- Parallel scoring
- <1 second per pod

This balances resource utilization, fault tolerance, and performance."

✓ "Use DNS for service discovery with CoreDNS"
✓ "Kube-proxy implements service load balancing with IPVS"
✓ "CNI plugin provides pod networking"
✗ Avoid: "Manual service registration" (doesn't scale)

Sample Answer:
"For networking, I'll implement three layers:

1. Pod Networking (CNI):
   - Every pod gets unique IP
   - Flat network (pods communicate without NAT)
   - CNI plugin (Calico, Cilium)
   - 10.0.0.0/8 cluster CIDR

2. Service Discovery (CoreDNS):
   - DNS-based service discovery
   - Service name → ClusterIP
   - Example: nginx-service.default.svc.cluster.local → 10.0.0.1
   - 30-second TTL, 10K queries/second

3. Load Balancing (Kube-proxy):
   - IPVS mode for performance
   - Service ClusterIP → Pod IPs
   - Round-robin load balancing
   - Health-based routing

Example:
Service: nginx-service (10.0.0.1:80)
Endpoints: [Pod1:8080, Pod2:8080, Pod3:8080]
Traffic: 10.0.0.1:80 → Random pod:8080

This provides transparent service discovery and load balancing."

✓ "Use etcd for distributed state with Raft consensus"
✓ "Strong consistency for critical operations"
✓ "Watch mechanism for real-time updates"
✗ Avoid: "Use MySQL for state" (no watch, no consensus)

Sample Answer:
"For state management, I'll use etcd:

Architecture:
- 5-node etcd cluster
- Raft consensus protocol
- Leader election
- Strong consistency (linearizable)

Data Model:
- Key-value store
- Hierarchical keys: /registry/pods/default/nginx
- Resource version for optimistic concurrency
- Watch mechanism for real-time updates

Operations:
- Writes: Go through leader, replicated to majority
- Reads: Can read from any node
- Watch: Efficient change notification

Performance:
- 1,000 writes/second
- 10,000 reads/second
- 10,000 watch streams
- <10ms write latency

Optimization:
- API server caching (80% hit rate)
- Client-side caching (informers)
- Pagination for large lists
- Compaction for old revisions

This provides strong consistency and real-time updates for cluster state."

❌ Bad: "Single API server"
✓ Good: "10 API servers for HA and scale"

❌ Bad: "No caching"
✓ Good: "Multi-level caching (80% hit rate)"

Key Point: 150K pods requires distributed architecture

❌ Bad: "Random node selection"
✓ Good: "Two-phase filtering and scoring"

❌ Bad: "No resource consideration"
✓ Good: "Consider CPU, memory, affinity, taints"

Key Point: Scheduling quality affects utilization and reliability

❌ Bad: "No authentication"
✓ Good: "RBAC with service accounts and certificates"

❌ Bad: "All pods can talk to all pods"
✓ Good: "Network policies for isolation"

Key Point: Multi-tenancy requires strong security

❌ Bad: "Single control plane node"
✓ Good: "Multi-master with etcd cluster"

❌ Bad: "No failover"
✓ Good: "Automatic failover in <1 minute"

Key Point: Production requires HA

"Kubernetes architecture separates control and data planes:

Control Plane (Brain):
- API Server: Central management interface
- etcd: Distributed state store
- Scheduler: Pod placement decisions
- Controllers: Maintain desired state

Data Plane (Muscle):
- Kubelet: Node agent
- Container Runtime: Run containers
- Kube-proxy: Service networking

This separation allows:
- Independent scaling
- Fault isolation
- Clear responsibilities
- Easier operations"

"At Kubernetes scale (5K nodes, 150K pods):

1. etcd: Bottleneck at 1K writes/second
   - Mitigation: Caching, pagination, separate events

2. API Server: 10K requests/second
   - Mitigation: Horizontal scaling, caching

3. Scheduler: 1K pods/second
   - Mitigation: Pre-filtering, parallel scoring

4. Networking: 10K services
   - Mitigation: IPVS instead of iptables

Every component must be designed for scale from day one."

"Beyond technical design:

1. Operations: Day 2 operations are harder than deployment
2. Upgrades: Zero-downtime upgrades are critical
3. Multi-tenancy: Isolation and fair resource sharing
4. Cost: Right-sizing and bin packing for efficiency
5. Security: RBAC, network policies, pod security

These operational concerns are as important as the architecture."

Answer:
"etcd uses Raft consensus algorithm:

Leader Election:
1. Nodes start as followers
2. If no heartbeat: Start election
3. Request votes from peers
4. Majority votes → Become leader
5. Leader sends heartbeats

Write Process:
1. Client sends write to leader
2. Leader appends to log
3. Leader replicates to followers
4. Majority acknowledge → Commit
5. Leader applies to state machine
6. Return success to client

Consistency: Linearizable (strongest consistency)

Benefits:
- Strong consistency
- Fault tolerance (survives minority failures)
- No split-brain
- Proven at scale

Tradeoff: Write latency (~10ms) vs eventual consistency (<1ms)"

Answer:
"Network policies at scale require efficient implementation:

Challenge: 1,000 policies across 150K pods

Solution: eBPF-based enforcement

1. Compile Policies:
   - Convert policies to eBPF programs
   - Compile to kernel bytecode
   - Load into kernel

2. Enforcement:
   - Kernel-level filtering
   - No iptables overhead
   - <1ms latency

3. Optimization:
   - Policy aggregation
   - Namespace-level policies
   - Cache policy decisions
   - Incremental updates

Performance:
- iptables: O(n) rule evaluation, slow at scale
- eBPF: O(1) lookup, fast at scale
- Improvement: 10x faster

This enables 10K services with network policies."

❌ "Use MySQL for state"
✓ "Use etcd with Raft consensus"

❌ "Single master node"
✓ "Multi-master with HA"

❌ "No resource limits"
✓ "Requests and limits for all pods"

❌ "All pods in one namespace"
✓ "Namespace isolation with RBAC"

❌ "No monitoring"
✓ "Prometheus for metrics, Jaeger for tracing"

"To summarize my container orchestration system:

1. Control Plane: API server + etcd + scheduler + controllers
2. Data Plane: Kubelet + container runtime + kube-proxy
3. Networking: CNI for pod networking, CoreDNS for discovery
4. Storage: CSI for persistent volumes
5. Scale: 5K nodes, 150K pods, 10K services

Key strengths:
- Highly available (99.99% uptime)
- Scalable (horizontal scaling)
- Self-healing (automatic recovery)
- Declarative (desired state)

Architecture decisions:
- etcd for strong consistency
- IPVS for service load balancing
- Rolling updates for zero downtime
- RBAC for security

Areas for improvement:
- Better multi-cluster management
- Improved scheduling efficiency
- Lower resource overhead
- Simpler operations

I'm happy to dive deeper into any component."