System Design Interview: Payment Settlement Batch Processing

Design a batch processing system for end-of-day payment settlement at a payments company that processes 50 million transactions per day. The system must net merchant positions, calculate fees, and initiate fund transfers to merchant bank accounts within a strict bank cutoff window. Walk me through your design, covering reliability, scalability, and how you'd handle failures.

Key Takeaways

1. The outbox pattern is the canonical solution to the dual-write problem. Know it cold.
2. Partition keys define ordering and parallelism — choose with intention, usually around the natural aggregation boundary (here, merchant ID).
3. Throttling must be distributed when you have multiple workers — Redis-backed buckets or a sidecar.
4. Idempotency is non-negotiable in payments. Design for at-least-once and dedupe.
5. Retries are tiered: in-process for transient, delay-queue for slower-resolving, DLQ for terminal.
6. Backpressure beats dropping — use Kafka lag as your buffer when downstream is slow.
7. Reconciliation closes the loop — you don't know it worked until ground truth confirms.
8. Corrections are new events — never rewrite history in a financial system.