Trade-offs & Alternatives

📖 7 min read 📄 Part 7 of 10

Container Orchestration System - Tradeoffs and Alternatives

Orchestration Platform Tradeoffs

Kubernetes vs Docker Swarm vs Nomad

Chosen: Kubernetes

Kubernetes:

Advantages:
- Rich feature set
- Large ecosystem
- Industry standard
- Strong community

Disadvantages:
- Complex
- Steep learning curve
- Resource overhead
- Over-engineered for simple use cases

Use: Production at scale

Docker Swarm:

Advantages:
- Simple setup
- Easy to learn
- Built into Docker
- Lower overhead

Disadvantages:
- Limited features
- Smaller ecosystem
- Less flexible
- Declining adoption

Why not: Limited for complex requirements

Nomad:

Advantages:
- Simple and lightweight
- Multi-workload (containers, VMs, binaries)
- Easy operations
- Good performance

Disadvantages:
- Smaller ecosystem
- Fewer features
- Less mature
- Limited adoption

Why not: Kubernetes is industry standard

Scheduling Tradeoffs

Bin Packing vs Spread

Chosen: Configurable (Default: Spread)

Bin Packing:

Strategy: Pack pods tightly on fewer nodes

Advantages:
- Better resource utilization
- Fewer nodes needed
- Lower cost

Disadvantages:
- Single node failure affects more pods
- Less fault tolerance
- Harder to scale up

Use: Cost-sensitive workloads

Spread:

Strategy: Distribute pods across many nodes

Advantages:
- Better fault tolerance
- Easier to scale
- Isolated failures

Disadvantages:
- Lower utilization
- More nodes needed
- Higher cost

Use: High-availability workloads

Implementation:
- Pod anti-affinity rules
- Topology spread constraints
- Zone-aware scheduling

Hybrid Approach:

Configuration:
- Default: Spread across zones
- Allow: Bin packing within zone
- User choice: Affinity rules

Benefits: Balance cost and reliability

Static vs Dynamic Scheduling

Chosen: Dynamic Scheduling

Static:

Advantages:
- Predictable placement
- Simple implementation
- No rescheduling

Disadvantages:
- Inflexible
- Poor resource utilization
- Can't adapt to changes

Why not: Too rigid for dynamic workloads

Dynamic:

Advantages:
- Adapts to cluster state
- Better resource utilization
- Handles failures

Disadvantages:
- More complex
- Unpredictable placement
- Rescheduling overhead

Implementation:
- Continuous scheduling
- Rescheduling on node failure
- Preemption for priority

State Management Tradeoffs

etcd vs Other Databases

Chosen: etcd

etcd:

Advantages:
- Strong consistency (Raft)
- Watch mechanism
- Designed for Kubernetes
- Proven at scale

Disadvantages:
- Write throughput limit (1K/s)
- Database size limit (8 GB)
- Operational complexity

Use: Kubernetes state store

Alternatives:

Consul:

Advantages:
- Service discovery built-in
- Multi-datacenter support
- Key-value store

Disadvantages:
- Different consistency model
- Not optimized for Kubernetes
- Migration complexity

Why not: etcd is standard

ZooKeeper:

Advantages:
- Mature and proven
- Strong consistency
- High availability

Disadvantages:
- Java-based (higher overhead)
- More complex operations
- Older technology

Why not: etcd is more modern

Networking Tradeoffs

Overlay vs Underlay Networking

Chosen: Overlay (with underlay option)

Overlay (VXLAN):

Advantages:
- Works on any network
- Flexible
- Easy to set up
- Portable

Disadvantages:
- Encapsulation overhead (50 bytes)
- Higher latency (1-2ms)
- Lower throughput (10-20% overhead)

Use: Default for most clusters

Underlay (BGP):

Advantages:
- No encapsulation
- Lower latency
- Higher throughput
- Native routing

Disadvantages:
- Requires network control
- Complex setup
- Less portable

Use: Performance-critical workloads

Implementation (Calico):
- BGP routing between nodes
- Direct pod-to-pod communication
- No tunneling overhead

iptables vs IPVS vs eBPF

Chosen: IPVS (with eBPF for advanced use cases)

iptables:

Advantages:
- Simple
- Well-understood
- Universal support

Disadvantages:
- O(n) rule evaluation
- Slow at scale (>1K services)
- High CPU usage

Why not: Doesn't scale to 10K services

IPVS:

Advantages:
- O(1) lookup
- Kernel-level load balancing
- Supports 10K+ services
- Multiple algorithms (round-robin, least-conn)

Disadvantages:
- Requires kernel module
- More complex setup

Use: Default for large clusters

Performance:
- 10K services: <1ms latency
- 100K connections: <5% CPU

eBPF:

Advantages:
- Kernel bypass
- Lowest latency
- Most flexible
- Best performance

Disadvantages:
- Requires modern kernel (4.18+)
- Complex programming
- Limited tooling

Use: Advanced networking (Cilium)

Performance:
- 50% lower latency than IPVS
- 30% lower CPU usage
- Best for high-performance workloads

Storage Tradeoffs

Local vs Network Storage

Chosen: Both (Use Case Dependent)

Local Storage:

Advantages:
- Lowest latency (<1ms)
- Highest throughput (1M IOPS)
- No network overhead
- Cheapest

Disadvantages:
- Not portable (tied to node)
- Data loss on node failure
- Limited capacity

Use: Caches, temporary data, databases with replication

Network Storage (EBS, GCE PD):

Advantages:
- Portable (survives node failure)
- Durable (replicated)
- Unlimited capacity
- Managed by cloud

Disadvantages:
- Higher latency (5-10ms)
- Lower throughput (10K IOPS)
- Network overhead
- More expensive

Use: Stateful applications, databases

Implementation:
- CSI driver for cloud storage
- Dynamic provisioning
- Volume snapshots
- Cross-zone replication

Distributed Storage (Ceph, GlusterFS):

Advantages:
- Portable
- Scalable
- Self-managed
- Cost-effective

Disadvantages:
- Operational complexity
- Performance overhead
- Requires dedicated nodes

Use: On-premises clusters

Deployment Strategy Tradeoffs

Rolling Update vs Blue-Green vs Canary

Chosen: Rolling Update (Default), Others Available

Rolling Update:

Process:
1. Create new pod
2. Wait for ready
3. Delete old pod
4. Repeat

Advantages:
- Zero downtime
- Gradual rollout
- Easy rollback
- Resource efficient

Disadvantages:
- Mixed versions during update
- Slower rollout
- Potential compatibility issues

Configuration:
maxSurge: 1 (max 1 extra pod)
maxUnavailable: 0 (no downtime)

Blue-Green:

Process:
1. Deploy new version (green)
2. Test green environment
3. Switch traffic to green
4. Keep blue for rollback

Advantages:
- Instant switch
- Easy rollback
- No mixed versions

Disadvantages:
- 2x resources needed
- More complex
- Waste during stable periods

Use: Critical applications

Canary:

Process:
1. Deploy new version (10% traffic)
2. Monitor metrics
3. Gradually increase (25%, 50%, 100%)
4. Rollback if issues

Advantages:
- Risk mitigation
- Gradual validation
- Easy rollback

Disadvantages:
- Slower rollout
- Complex traffic splitting
- Requires service mesh

Use: High-risk changes

Implementation (Istio):
- VirtualService for traffic splitting
- DestinationRule for versions
- Metrics-based promotion

Resource Management Tradeoffs

Requests vs Limits

Chosen: Both (Requests for Scheduling, Limits for Enforcement)

Requests Only:

Advantages:
- Guaranteed resources
- Predictable performance
- Simple

Disadvantages:
- Wasted resources
- Lower utilization
- Higher cost

Why not: Inefficient resource usage

Limits Only:

Advantages:
- Higher utilization
- Lower cost
- Flexible

Disadvantages:
- No guarantees
- Unpredictable performance
- Noisy neighbor problems

Why not: No scheduling guarantees

Requests + Limits:

Implementation:
resources:
  requests:
    cpu: "100m"
    memory: "128Mi"
  limits:
    cpu: "200m"
    memory: "256Mi"

Benefits:
- Requests: Scheduling guarantee
- Limits: Prevent resource hogging
- Overcommit: Better utilization

QoS Classes:
- Guaranteed: requests = limits
- Burstable: requests < limits
- BestEffort: no requests/limits

High Availability Tradeoffs

Single vs Multi-Master

Chosen: Multi-Master (HA)

Single Master:

Advantages:
- Simple setup
- Lower cost
- No split-brain

Disadvantages:
- Single point of failure
- No HA
- Downtime during updates

Why not: Unacceptable for production

Multi-Master:

Implementation:
- 3 or 5 control plane nodes
- Load balanced API servers
- etcd cluster (Raft)
- Leader election for controllers

Advantages:
- High availability
- No single point of failure
- Rolling updates
- Fault tolerance

Disadvantages:
- More complex
- Higher cost
- Consensus overhead

Benefits: Worth it for production reliability

Monitoring Tradeoffs

Push vs Pull Metrics

Chosen: Pull (Prometheus)

Pull (Prometheus):

Advantages:
- Centralized control
- Service discovery
- Consistent scraping
- Detect down targets

Disadvantages:
- Firewall challenges
- Short-lived jobs difficult
- Network overhead

Use: Default for Kubernetes

Push (StatsD, Graphite):

Advantages:
- Works through firewalls
- Good for short-lived jobs
- Lower network overhead

Disadvantages:
- No service discovery
- Can't detect down targets
- Potential data loss

Use: Specific use cases only

This comprehensive tradeoff analysis demonstrates the complex decision-making required to build a container orchestration platform that balances performance, reliability, cost, and operational complexity.