systemdesign.comData Reconciliation After Failures
Detect and repair inconsistencies after partial failures in distributed workflows.
After failures, data may exist in inconsistent states across services—for example, a payment succeeds but the order creation fails, or vice versa. Reconciliation identifies these gaps by comparing states using agreed timestamps, version identifiers, or checksums. One service acts as a source of truth and periodically checks dependent services for deviations.
Reconciliation often uses periodic background jobs or event-driven listeners. A service with partial success can publish incomplete status events; listeners compare against expected outcomes and trigger correction actions like re-sending a message, marking records as orphaned, or initiating manual review. Automating checks ensures consistency without manual intervention.
Design for reconciliation by storing causal context—such as request IDs or operation logs—on each operation. This helps trace how data diverged and ensures corrections are applied only when safe. Avoid automatic overwrites that might erase newer, legitimate updates; instead, merge, flag, or prompt for human review when ambiguity remains.
Key points
- Compare states across services using identifiers or checksums
- Use event logs or background jobs to detect mismatches
- Reconcile cautiously with causal context to avoid conflicts