Problem: A trace shows a span took 500ms, but WHY did it take 500ms?
Solution: Link trace spans to continuous profiling data (CPU, memory, allocations)

Architecture:
  Application
  ├── OpenTelemetry SDK (traces)
  └── Profiling Agent (pprof/async-profiler)
       ├── CPU profiles (sampled every 10ms)
       ├── Memory allocation profiles
       └── Lock contention profiles

Correlation Mechanism:
- Profile samples tagged with active span_id at sample time
- When viewing a slow span, query profiling backend for that span's time window
- Display flame graph filtered to that span's execution

Implementation:
1. Profiling agent runs continuously (1% CPU overhead)
2. Each profile sample records: timestamp, stack trace, span_id (if active)
3. Storage: profile data indexed by (service, time_range, span_id)
4. Query: "Show me the flame graph for span X" →
   Filter profiles where span_id = X, aggregate stack traces

Real-World Examples:
- Datadog Continuous Profiler (linked to APM traces)
- Pyroscope + Grafana Tempo integration
- Google Cloud Profiler + Cloud Trace

Interview Talking Point:
"Traces tell you WHERE time is spent across services.
 Profiles tell you WHY time is spent within a service.
 Together they provide complete performance visibility."

Problem: Developers switch between trace UI and log search to debug issues.
Solution: Embed trace context in logs, enable bidirectional navigation.

Implementation:

1. Inject trace context into log records:
   import logging
   from opentelemetry import trace

   class TraceContextFilter(logging.Filter):
       def filter(self, record):
           span = trace.get_current_span()
           ctx = span.get_span_context()
           record.trace_id = format(ctx.trace_id, '032x')
           record.span_id = format(ctx.span_id, '016x')
           return True

   # Log output:
   # 2024-01-15 10:30:00 [trace_id=5b8aa5a2... span_id=051581bf...]
   # ERROR: Payment declined for order 12345

2. Log storage indexes trace_id field:
   - Elasticsearch: trace_id as keyword field
   - CloudWatch: trace_id in structured JSON logs
   - Loki: trace_id as label

3. Bidirectional navigation:
   - From trace UI: "Show logs for this span" → query logs where
     trace_id = X AND timestamp BETWEEN span.start AND span.end
   - From log UI: "Show trace for this log" → link to trace viewer
     using trace_id from log record

4. Exemplars (metrics → traces):
   - When recording a metric (e.g., request_duration histogram),
     attach trace_id of an example request
   - From metric dashboard: "Show me a trace that contributed to this P99"
   - Prometheus exemplars: {trace_id="5b8aa5a2..."} on histogram buckets

Benefits:
- Single pane of glass for debugging
- Faster root cause analysis (trace → span → logs → code)
- Correlation across all three signals without manual searching

RED Metrics (Rate, Errors, Duration):
- Computed directly from span data
- No separate metrics instrumentation needed
- Consistent with trace-level observations

Derivation Pipeline:
  Spans → Stream Processor → Metrics Store (Prometheus/ClickHouse)

For each SERVER span, emit:
  - request_count{service, operation, status_code} += 1
  - request_duration{service, operation} = span.duration
  - error_count{service, operation} += 1 (if status = ERROR)

ClickHouse Materialized View:
  CREATE MATERIALIZED VIEW red_metrics
  ENGINE = SummingMergeTree()
  ORDER BY (service_name, operation_name, minute)
  AS SELECT
      service_name,
      operation_name,
      toStartOfMinute(start_time) AS minute,
      count() AS requests,
      countIf(status_code = 'ERROR') AS errors,
      avg(duration_us) AS avg_duration,
      quantile(0.5)(duration_us) AS p50,
      quantile(0.95)(duration_us) AS p95,
      quantile(0.99)(duration_us) AS p99
  FROM spans
  WHERE span_kind = 'SERVER'
  GROUP BY service_name, operation_name, minute;

Advantages over separate metrics:
+ Single source of truth (traces generate metrics)
+ No instrumentation drift (metrics always match traces)
+ Drill-down: from metric anomaly → find contributing traces
+ Reduced instrumentation burden on developers

Disadvantages:
- Sampling affects metric accuracy (1% sample ≠ true P99)
- Higher latency than direct metrics (processing pipeline delay)
- More expensive to compute than pre-aggregated counters

Solution: Use 100% sampling for metrics derivation at the collector,
even if only 1% of traces are stored. Count all spans, store few.

Problem: In event-driven systems, producer and consumer are decoupled.
Traditional parent-child spans don't capture the relationship.

Scenarios:
1. Message Queue (Kafka, SQS, RabbitMQ)
2. Event Bus (EventBridge, SNS)
3. Scheduled Jobs (triggered by earlier events)
4. Saga/Choreography patterns

Solution: Span Links + Context Propagation in Messages

Message Queue Tracing:
  Producer Service:
    with tracer.start_as_current_span("publish_order_event",
                                       kind=SpanKind.PRODUCER) as span:
        # Inject trace context into message headers
        headers = {}
        inject(headers)
        kafka_producer.send("orders", value=event, headers=headers)

  Consumer Service:
    def process_message(message):
        # Extract producer's trace context
        producer_context = extract(message.headers)
        
        # Create new trace with LINK to producer (not child)
        with tracer.start_as_current_span("process_order_event",
                                           kind=SpanKind.CONSUMER,
                                           links=[Link(producer_context)]) as span:
            # Process the event
            handle_order(message.value)

Trace Visualization:
  Trace A (Producer): [API Gateway] → [Order Service] → [Kafka Publish]
                                                              |
                                                         (link, not parent)
                                                              |
  Trace B (Consumer): [Kafka Consume] → [Inventory Service] → [DB Update]

Why Links Instead of Parent-Child:
- Consumer may process message hours after production
- One message may trigger multiple consumers (fan-out)
- Consumer trace has its own lifecycle and duration
- Parent-child implies synchronous call semantics

Batch Processing:
- One consumer span may process 100 messages
- Link to all 100 producer spans (or sample)
- Span event: "processed batch of 100 messages"

Problem: Request flows from Company A's service to Company B's API.
How do you maintain trace continuity across trust boundaries?

Challenges:
1. Security: Don't leak internal trace IDs to external parties
2. Privacy: Don't expose internal service topology
3. Trust: External trace context could be malicious (DoS via trace explosion)
4. Standards: Both parties must agree on propagation format

Solutions:

1. W3C Trace Context Standard (Recommended):
   - traceparent header: version-trace_id-parent_id-flags
   - tracestate header: vendor-specific key-value pairs
   - Standardized, supported by all major vendors
   
   Example:
   traceparent: 00-5b8aa5a2d2c872e8321cf37308d69df2-051581bf3cb55c13-01
   tracestate: company_a=internal_span_ref

2. Gateway Translation:
   - API Gateway at boundary creates new internal trace
   - Links new trace to external trace (preserves correlation)
   - Strips internal details from outgoing responses
   - Logs mapping: external_trace_id → internal_trace_id

3. Trust Policies:
   - Accept external trace context: only from whitelisted partners
   - Rate limit: max traces/sec from external sources
   - Validate: trace_id format, reasonable flags
   - Sanitize: strip tracestate from untrusted sources

4. Federated Trace Viewing:
   - Each organization stores their portion of the trace
   - Shared trace_id allows correlation
   - No direct access to partner's trace data
   - Agreed-upon SLA for trace availability

Problem: Backend traces start at API Gateway, missing client-side latency.
Users experience: DNS + TCP + TLS + Request + Server + Response + Rendering
Backend sees only: Server processing time

Solution: Client-side span generation

Mobile SDK Implementation:
  class NetworkTracer: URLSessionDelegate {
      func urlSession(_ session: URLSession, 
                      task: URLSessionTask,
                      didFinishCollecting metrics: URLSessionTaskMetrics) {
          
          let span = tracer.startSpan("HTTP \(task.request.httpMethod)")
          
          // Timing breakdown from OS-level metrics
          span.setAttribute("dns.duration_ms", metrics.domainLookupDuration)
          span.setAttribute("tcp.duration_ms", metrics.connectDuration)
          span.setAttribute("tls.duration_ms", metrics.secureConnectionDuration)
          span.setAttribute("ttfb.duration_ms", metrics.responseStartDuration)
          span.setAttribute("download.duration_ms", metrics.responseEndDuration)
          
          // Device context
          span.setAttribute("device.model", UIDevice.current.model)
          span.setAttribute("network.type", currentNetworkType())
          span.setAttribute("app.version", Bundle.main.appVersion)
          
          span.end()
      }
  }

Challenges:
1. Clock Skew: Mobile device clock may differ from server
   - Solution: Use relative durations, not absolute timestamps
   - NTP sync check before sending spans

2. Batching: Don't send spans on every request (battery/bandwidth)
   - Buffer spans locally, send in batches every 30 seconds
   - Compress payload (protobuf + gzip)
   - Respect low-power mode and connectivity

3. Sampling: Mobile generates many spans (UI interactions)
   - Sample at 1-5% for normal flows
   - 100% for errors and slow requests
   - User-triggered: "Report a problem" captures full trace

4. Privacy: Client spans may contain sensitive data
   - Strip PII before sending (URLs, headers)
   - Consent-based collection (GDPR)
   - No recording of screen content or user input

5. Volume: Millions of mobile devices × spans per session
   - Aggressive sampling (0.1% for normal, 100% for errors)
   - Edge collection (CDN-level span aggregation)
   - Client-side aggregation (send summaries, not individual spans)

Problem: Span shows "DB query took 500ms" but doesn't explain why.
Solution: Capture query execution details as span attributes/events.

Implementation Levels:

Level 1: Basic (auto-instrumentation default)
  span.name = "SELECT orders"
  span.attributes = {
      "db.system": "postgresql",
      "db.name": "orders_db",
      "db.statement": "SELECT * FROM orders WHERE user_id = $1",
      "db.operation": "SELECT"
  }

Level 2: Execution Plan
  span.events = [{
      "name": "query.plan",
      "attributes": {
          "plan.type": "Index Scan",
          "plan.rows_estimated": 1,
          "plan.rows_actual": 1,
          "plan.cost": 0.42,
          "plan.index_used": "idx_orders_user_id"
      }
  }]

Level 3: Connection Pool Metrics
  span.attributes = {
      "db.pool.wait_time_ms": 50,    // Time waiting for connection
      "db.pool.active": 18,           // Active connections
      "db.pool.idle": 2,              // Idle connections
      "db.pool.max": 20               // Pool max size
  }

Level 4: Query Fingerprinting and Aggregation
  - Normalize queries: "WHERE id = 123" → "WHERE id = ?"
  - Aggregate by fingerprint: avg duration, call count, error rate
  - Detect N+1 queries: same fingerprint repeated N times in one trace
  - Alert on slow query patterns

N+1 Query Detection:
  def detect_n_plus_one(trace):
      db_spans = [s for s in trace.spans if s.attributes.get("db.system")]
      fingerprints = Counter(normalize(s.attributes["db.statement"]) 
                            for s in db_spans)
      for query, count in fingerprints.items():
          if count > 10:
              alert(f"N+1 detected: {query} called {count} times")

Problem: Services on different hosts have slightly different clocks.
Impact: Spans may appear to start before their parent ends, or
        child spans may appear to have negative duration relative to parent.

Solutions:

1. NTP Synchronization:
   - All hosts sync to same NTP server
   - Typical accuracy: ±1-5ms with NTP
   - Sufficient for most tracing use cases
   - Monitor clock drift as infrastructure metric

2. Logical Ordering (Lamport Timestamps):
   - Use causal ordering: parent always starts before child
   - If child.start < parent.start, adjust child.start = parent.start
   - Preserves causality even with clock skew

3. Hybrid Approach (Google Spanner-style):
   - Use TrueTime-like confidence intervals
   - Each timestamp has uncertainty bound: [earliest, latest]
   - Display spans with uncertainty visualization

4. Client-Side Correction:
   - SDK records both wall-clock time and monotonic clock
   - Duration computed from monotonic clock (immune to NTP adjustments)
   - Start time from wall clock (for ordering across services)
   - Collector adjusts start times to maintain parent-child ordering

Production Reality:
- NTP gives ±5ms accuracy (good enough for >99% of spans)
- Only matters for very short spans (<10ms)
- UI should handle gracefully (don't show negative gaps)
- Jaeger UI adjusts child spans to not exceed parent bounds

Answer:

Standard: W3C Trace Context (supported by all OTel SDKs)
Headers:
  traceparent: 00-{trace_id}-{parent_span_id}-{flags}
  tracestate: {vendor_key}={vendor_value}

Propagation Across Protocols:
1. HTTP: traceparent/tracestate headers
2. gRPC: grpc-trace-bin binary header (more efficient)
3. Kafka: Message headers (key: "traceparent", value: header string)
4. SQS: Message attributes
5. RabbitMQ: Message headers
6. Redis: Not propagated (client span only, no server context)

Polyglot Challenges:
- Each language SDK must implement same propagation format
- Binary protocols (Thrift, Protobuf) need custom propagation
- Some frameworks strip unknown headers (must whitelist)
- Legacy services may not propagate (trace breaks)

Legacy Service Handling:
- If service doesn't propagate: trace splits into two traces
- Solution: Service mesh sidecar propagates even if app doesn't
- Alternative: Correlation by timing (heuristic trace stitching)

Answer:

Problem: Some operations generate massive traces:
- Batch jobs processing 10K items (one span per item)
- Fan-out requests to 500 shards
- Recursive operations (tree traversal)

Challenges:
1. Storage: Single trace = 1000 spans × 500 bytes = 500 KB
2. Assembly: Loading 1000 spans for display
3. Visualization: UI can't render 1000 spans readably
4. Query: Trace-by-ID returns massive payload

Solutions:

1. Span Limits:
   - SDK configuration: max_spans_per_trace = 1000
   - After limit: drop lowest-priority spans, increment counter
   - Record: span.attributes["otel.dropped_spans"] = 5000

2. Progressive Loading:
   - API returns trace summary first (root + direct children)
   - UI lazy-loads deeper spans on expand
   - Pagination: GET /traces/{id}?depth=2&offset=0&limit=50

3. Aggregated Spans:
   - Batch operations: single span with count attribute
   - Instead of 10K "process_item" spans:
     One span: "process_batch", attributes: {batch_size: 10000, 
     errors: 3, avg_duration_ms: 5}

4. Trace Summarization:
   - Compute critical path (longest sequential chain)
   - Show only critical path + error spans by default
   - Collapse repetitive patterns (N identical child spans → "×N")

5. Storage Optimization:
   - Large traces stored in separate "jumbo trace" table
   - Different TTL (shorter, since they're expensive)
   - Alert on services generating oversized traces

Answer:

Performance Budget:
- CPU overhead: <2% of application CPU
- Memory overhead: <50 MB per process
- Latency overhead: <1ms per request (for context propagation)
- Network overhead: <5 MB/sec per host

Techniques:

1. Async Processing:
   - Span creation is synchronous (must capture timing)
   - Span export is asynchronous (background thread/goroutine)
   - Never block application thread on span delivery

2. Bounded Buffers:
   - Fixed-size queue between creation and export
   - If queue full: drop spans (never block)
   - Monitor dropped span count as health metric

3. Efficient Serialization:
   - Protobuf over JSON (3-5x smaller, 10x faster to serialize)
   - Pre-allocate buffers for span creation
   - Object pooling for span objects (reduce GC pressure)

4. Sampling at Source:
   - Head-based sampling eliminates work for unsampled traces
   - Unsampled traces: only propagate context (no span creation)
   - Reduces CPU, memory, and network by sampling_rate factor

5. Benchmarking:
   - Measure with/without tracing in load tests
   - Acceptable: <2% latency increase at P99
   - If exceeded: reduce sampling rate or optimize SDK

Real-World Numbers (OpenTelemetry Java Agent):
- Startup overhead: +200ms (agent initialization)
- Per-request overhead: 5-50μs (span creation + context propagation)
- Memory: +30 MB (agent + buffer)
- Throughput impact: <3% reduction in max RPS

Answer:

Concept: Alert on patterns observed in traces, not just metrics.

Alert Types:
1. Latency Threshold: "Alert if P99 of service X > 500ms for 5 minutes"
2. Error Rate: "Alert if error rate of operation Y > 5%"
3. Dependency Failure: "Alert if service A cannot reach service B"
4. Pattern Match: "Alert if trace contains retry > 3 times"
5. Anomaly: "Alert if latency deviates >3σ from baseline"

Architecture:
  Spans → Kafka → Alert Evaluator → Alert Manager → PagerDuty/Slack

  Alert Evaluator:
  - Maintains sliding windows per (service, operation)
  - Computes metrics from spans in real-time
  - Evaluates alert rules against computed metrics
  - Fires alert with example trace_ids attached

Alert Rule Example:
  rules:
    - name: "Payment Service High Latency"
      condition: |
        percentile(duration, 0.99, 
          service="payment-service", 
          operation="POST /charge") > 2000ms
      for: 5m
      severity: critical
      annotations:
        example_traces: "{{$labels.trace_ids}}"

Key Advantage Over Metric-Based Alerting:
- Alert includes trace_ids → immediate debugging context
- No need to search for relevant traces during incident
- Alert → Click trace_id → See exactly what went wrong

Answer:

Challenges:
1. Cold starts: SDK initialization adds to cold start latency
2. Short-lived: Function may run for only 50ms
3. No persistent agent: Can't run sidecar/daemon
4. Concurrent executions: Many instances, each independent
5. Async invocations: SNS/SQS/EventBridge triggers

Solutions:

1. Lightweight SDK:
   - Minimal initialization (<10ms)
   - Lazy loading of exporters
   - Pre-compiled protobuf schemas

2. Extension-Based Collection:
   - AWS Lambda Extension (separate process in execution environment)
   - Receives spans via local UDP/HTTP
   - Batches and forwards to collector
   - Survives across warm invocations

3. Context Propagation:
   - API Gateway → Lambda: trace context in HTTP headers
   - SQS → Lambda: trace context in message attributes
   - SNS → Lambda: trace context in message attributes
   - Step Functions → Lambda: trace context in input payload

4. Cold Start Tracing:
   - Separate span for initialization vs handler execution
   - Attribute: faas.coldstart = true
   - Enables filtering cold start traces for analysis

5. Cost Optimization:
   - Sample aggressively (Lambda generates many short traces)
   - Use OTLP/HTTP (simpler than gRPC for short-lived processes)
   - Batch export at end of invocation (flush before freeze)

1. Observability is interconnected: Traces + Metrics + Logs form a triangle.
   Show you understand how they complement each other.

2. Scale drives design: At 10K spans/sec, anything works.
   At 10M spans/sec, every decision matters.

3. Sampling is the key tradeoff: Full fidelity vs cost.
   Tail-based sampling is the sophisticated answer.

4. Context propagation is the hard part: Getting trace context
   through every protocol, framework, and boundary.

5. Performance overhead must be invisible: Tracing that slows
   the application defeats its purpose.