The problem
An online merchant sends card payments through two payment processors. When payments fail or get stuck, the merchant loses sales. The operations team then rebuilds what happened from separate processor dashboards and logs, while the failing processor keeps taking traffic.
My approach
One system around that gap, kept to one merchant and two simulated processors so the problem stays concrete and every claim can be tested:
- One timeline per payment. Every state change is written once, in the same database transaction as the payment, and published from there. History and state cannot disagree.
- A plain-language explanation of each failure. A small local model writes it, and deterministic rules take over when the model is unavailable or disagrees with the rules.
- Automatic failover. Routing reads each processor’s recent error rate and sends new payments away from one that is failing.
- Operations depth. Incidents opened from processor health, analytics, and Prometheus and Grafana dashboards.
- For developers. Signed webhooks with retries, a documented API with a quickstart, and a Python SDK.
Results
In the scripted outage, success was 58% with static routing and 89% with Relay, on the same 300 synthetic payments. The run repeats exactly from its seed. A developer new to the project reached a first API call 43 seconds after cloning (images already cached) by following only the quickstart.
What did not work
I also tested whether learned (adaptive) routing could beat the rule-based router. I fixed the success measure, the seeds and the win rule before any run, and made two attempts. Neither beat the rules under that rule:
- a Thompson-sampling router: +1.8 points of success, but better on only 10 of 15 paired runs, with slower payments on calm days;
- a gated hybrid, tested on fresh seeds: +0.6 points, better on 8 of 15.
Both consistently helped when a processor was degraded but not down: about +2 to +2.7 points of success, with the slowest payments finishing in roughly a third to three-fifths of the time. They also sent far less traffic to the faulty processor. The full numbers, and why each attempt fell short, are in the repository.
About the data
Every payment, customer and outage in Relay is generated. Runs repeat exactly, so results can be checked. Nothing comes from a real merchant or card network.
All projects