Database Batch Auditing Service - System Architecture

High-Level Architecture

Batch Auditing — CDC Pipeline Architecture

Core Components

Change Data Capture (CDC)

Responsibilities:
- Monitor database logs
- Capture changes in real-time
- Minimal database overhead
- Reliable delivery

Technologies:
- Debezium (Kafka Connect)
- Maxwell (MySQL)
- Oracle GoldenGate
- SQL Server CDC

Audit Processor

Processing Pipeline:
1. Validate change records
2. Enrich with metadata
3. Detect anomalies
4. Apply retention policies
5. Store in audit database

Storage Layer

ClickHouse for Analytics:
- Fast columnar storage
- Efficient compression
- Real-time queries
- Aggregation support

S3 for Archival:
- Long-term retention
- Cost-effective
- Compliance storage

This architecture provides comprehensive, scalable database auditing.