At-Least-Once (Recommended):
✓ No missed deliveries
✓ Simpler implementation
✓ Higher throughput
✗ Possible duplicates
✗ Requires idempotent endpoints

Implementation:
- Retry on failure
- Idempotency keys
- Duplicate detection

Use Cases:
- Most webhook scenarios
- Payment notifications
- Order updates

Exactly-Once:
✓ No duplicates
✓ No missed deliveries
✗ Complex implementation
✗ Lower throughput
✗ Higher latency

Implementation:
- Distributed transactions
- Deduplication tracking
- Coordination overhead

Use Cases:
- Financial transactions
- Critical operations
- Strict requirements

At-Most-Once:
✓ No duplicates
✓ Highest throughput
✓ Simplest implementation
✗ Possible missed deliveries
✗ No retry on failure

Implementation:
- Fire and forget
- No acknowledgment
- No retry logic

Use Cases:
- Non-critical notifications
- Best-effort delivery
- High-volume, low-value events

Recommendation: At-least-once for most use cases

Exponential Backoff (Recommended):
✓ Fast recovery for transient errors
✓ Reduces load on failing endpoints
✓ Industry standard
✗ Longer recovery for persistent issues

Formula: delay = base * 2^attempt
Example: 1s, 2s, 4s, 8s, 16s

Use When:
- Transient failures expected
- Network issues
- Temporary overload

Fixed Delay:
✓ Predictable timing
✓ Simple implementation
✗ May be too aggressive
✗ May be too slow

Example: Retry every 60 seconds

Use When:
- Known recovery time
- Scheduled maintenance

Linear Backoff:
✓ Gradual increase
✓ More predictable than exponential
✗ Slower recovery
✗ Less common

Formula: delay = base * attempt
Example: 1s, 2s, 3s, 4s, 5s

Use When:
- Moderate failures
- Balanced approach

Recommendation: Exponential backoff with jitter

Architecture:
Service → Queue → Workers push to endpoints

Pros:
✓ Real-time delivery
✓ Lower latency
✓ Simpler for customers
✓ Standard webhook pattern

Cons:
✗ Requires customer endpoint
✗ Firewall configuration needed
✗ Customer must handle retries

Use Cases:
- Standard webhooks
- Real-time notifications
- Event-driven integrations

Latency: <1 second

Architecture:
Service stores events → Customers poll for events

Pros:
✓ No firewall issues
✓ Customer controls rate
✓ Simpler security
✓ No endpoint needed

Cons:
✗ Higher latency (polling interval)
✗ More customer complexity
✗ Polling overhead
✗ Not real-time

Use Cases:
- Batch processing
- Firewall restrictions
- Customer preference

Latency: Polling interval (e.g., 1 minute)

Recommendation: Push for real-time, pull as fallback

Flow:
Event → Immediate webhook delivery → Response

Pros:
✓ Immediate feedback
✓ Simpler error handling
✓ No queue needed
✗ Blocks event processing
✗ Lower throughput
✗ Cascading failures

Use When:
- Low volume (<100 events/second)
- Immediate confirmation needed
- Simple use cases

Latency: Event processing blocked until delivery complete

Flow:
Event → Queue → Background delivery → Response

Pros:
✓ High throughput
✓ Decoupled architecture
✓ Better fault tolerance
✓ Scalable
✗ No immediate feedback
✗ More complex
✗ Requires queue

Use When:
- High volume (>1000 events/second)
- Scalability needed
- Production systems

Latency: Event processing immediate, delivery in background

Recommendation: Asynchronous for production systems

PostgreSQL (SQL):
✓ ACID transactions
✓ Complex queries
✓ Strong consistency
✓ Mature ecosystem
✗ Harder to scale horizontally

Use For:
- Webhook configuration
- Delivery history
- Audit logs

Cassandra (NoSQL):
✓ Linear scalability
✓ High availability
✓ Multi-region
✗ Eventual consistency
✗ Limited queries

Use For:
- High-scale delivery logs
- Time-series data
- Multi-region deployment

MongoDB (NoSQL):
✓ Flexible schema
✓ Good query support
✓ Horizontal scaling
✗ Weaker consistency

Use For:
- Rapid development
- Flexible event schemas

Recommendation: PostgreSQL for most use cases

Kafka:
✓ Highest throughput (1M+ msg/s)
✓ Durable log storage
✓ Multiple consumers
✓ Replay capability
✗ More complex
✗ Higher resource usage

Use For:
- High-volume events
- Event sourcing
- Multiple consumers

RabbitMQ:
✓ Flexible routing
✓ Priority queues
✓ Easy to operate
✗ Lower throughput (100K msg/s)
✗ No replay

Use For:
- Complex routing
- Priority handling
- Moderate volume

AWS SQS:
✓ Fully managed
✓ Serverless
✓ Auto-scaling
✗ Vendor lock-in
✗ Higher latency
✗ Limited throughput

Use For:
- AWS-native applications
- Managed service preference
- Variable load

Recommendation: Kafka for high volume, RabbitMQ for simplicity

HMAC-SHA256 (Recommended):
✓ Simple and secure
✓ Industry standard
✓ Fast computation
✓ Small signature size
✗ Shared secret required

Implementation:
signature = HMAC-SHA256(secret, payload)
X-Signature: sha256={signature}

JWT:
✓ Self-contained
✓ Includes metadata
✓ Standard format
✗ Larger size
✗ More complex
✗ Overkill for webhooks

Implementation:
token = JWT.encode(payload, secret)
Authorization: Bearer {token}

Custom Signature:
✓ Flexible
✓ Optimized for use case
✗ Non-standard
✗ Customer confusion
✗ More support burden

Recommendation: HMAC-SHA256 for simplicity and security

Ordered Delivery:
✓ Maintains event sequence
✓ Easier to process
✓ Predictable behavior
✗ Lower throughput
✗ Head-of-line blocking
✗ Complex implementation

Implementation:
- Single queue per webhook
- Sequential processing
- Wait for acknowledgment

Use When:
- Order matters (state changes)
- Sequential processing required

Unordered Delivery (Recommended):
✓ Higher throughput
✓ Parallel processing
✓ Better scalability
✗ Out-of-order delivery
✗ Customer must handle ordering

Implementation:
- Multiple queues
- Parallel workers
- No ordering guarantee

Use When:
- Order doesn't matter
- Independent events
- High volume

Recommendation: Unordered with sequence numbers for customer-side ordering

Purpose: Protect failing endpoints

Pros:
✓ Automatic failure detection
✓ Fast recovery
✓ Prevents cascading failures
✗ May miss recovery window
✗ All-or-nothing approach

Use When:
- Endpoint completely down
- Persistent failures
- Protect system resources

Thresholds:
- Open: >50% failure rate
- Half-open: After 60 seconds
- Closed: >90% success rate

Purpose: Control delivery rate

Pros:
✓ Prevents overwhelming endpoints
✓ Gradual load increase
✓ Respects endpoint capacity
✗ May delay deliveries
✗ Requires configuration

Use When:
- Endpoint has rate limits
- Gradual rollout
- Protect endpoint capacity

Limits:
- 100 deliveries/minute per endpoint
- 1000 deliveries/hour per endpoint
- Configurable per webhook

Recommendation: Use both (circuit breaker + rate limiting)

Configuration:
- Exactly-once delivery
- Immediate retries
- Dedicated workers
- Premium instances
- Multi-region active-active

Cost: $500K/month
Throughput: 100K deliveries/second
Latency: <500ms
Reliability: 99.99%

Use When: Critical, high-value webhooks

Configuration:
- At-least-once delivery
- Exponential backoff retries
- Shared workers
- Standard instances
- Multi-region active-passive

Cost: $144K/month
Throughput: 10K deliveries/second
Latency: <1 second
Reliability: 99.9%

Use When: Most production workloads

Configuration:
- At-most-once delivery
- No retries
- Spot instances
- Single region

Cost: $30K/month
Throughput: 5K deliveries/second
Latency: <2 seconds
Reliability: 95%

Use When: Non-critical, high-volume webhooks

Custom Service (This Design):
✓ Full control
✓ Optimized for use case
✓ No vendor lock-in
✓ Cost-effective at scale
✗ Development effort
✗ Maintenance overhead
✗ Operational complexity

AWS EventBridge:
✓ Fully managed
✓ Serverless
✓ AWS integration
✗ Vendor lock-in
✗ Limited customization
✗ Higher cost at scale

Zapier/IFTTT:
✓ No-code solution
✓ Many integrations
✗ Not for production APIs
✗ Limited control
✗ High cost

Recommendation: Custom for large scale, managed for simplicity