Never double-charge: the UPDATE is the lock

I’ve spent a good chunk of the last decade building payment systems, and every one of them eventually grows a collections side - the part that wakes up each morning, looks at which instalments are due, and takes money from real people’s bank accounts. (You’d be genuinely surprised how far a simple cron job and a hundred-odd lines of code can get you here - rather further, in fact, than most people would believe if I put the number in writing 😉) Whatever the architecture, one directive outranks everything else: never double-charge. A charge that fails can be retried tomorrow, a charge that gets stuck can be investigated by a human, and a charge you skip entirely costs a day of float at worst. A double charge costs trust, refund admin, possibly a complaint to the regulator, and a customer who tells everyone they know. So these systems get designed around that asymmetry, and every ambiguous situation resolves towards not charging.

That’s easy to say and surprisingly hard to build, because the ways a second charge sneaks in are all boring. Two workers pick up the same instalment because a cron overlapped with a backfill. A process crashes after the charge went out but before the acknowledgement came back, and the retry charges again. A network timeout leaves you genuinely unable to say whether the first attempt happened. None of these are exotic, and all of them will occur eventually at any volume worth having.

Claim first, and make the claim atomic

The classic shape of the bug is check-then-act: SELECT the instalment, confirm it’s due, then UPDATE it to in-flight and call the payment gateway. Between the SELECT and the UPDATE there’s a window, and given enough traffic something will land in it. Locks and transactions can close the window, but they bring their own failure modes, and distributed locks in particular are where these bugs go to breed.

The fix is to collapse the check and the lock into a single statement:

UPDATE instalments
SET status = 'in_flight', in_flight_since = NOW()
WHERE id = :id
  AND status = 'due'
  AND outstanding_amount > 0
  AND has_open_chargeback = FALSE

Then you read the rows-affected count. One row means you own the instalment and may proceed to charge; zero rows means somebody else got there first, or it’s already paid, or a chargeback landed, and you walk away without ever calling the gateway. The UPDATE is the check and the lock in one operation, the database’s own row locking provides the mutual exclusion, and there is no window because there’s nothing between the check and the act. Two concurrent workers can race as hard as they like and exactly one will see a row count of one.

There’s a wrinkle worth knowing about before you ship this. If a worker claims the row and then dies before acknowledging, its retry will come back, see the row already in-flight, conclude that someone else owns it, and politely walk away - stranding the instalment forever. The fix is to record who claimed it - a claimed_by column holding the ID of the payment request doing the claiming - and to widen the claim predicate to also accept a row that’s in-flight and claimed by that same ID. Now when the dead worker’s retry comes back, it’s carrying the same request ID that’s written on the row, so the claim succeeds again and the work resumes where it stopped, whilst any other worker asking about that instalment still matches nothing and walks away. That one column turns a stuck-forever state into a self-recovering one.

The charges that already left the building

Claiming atomically protects you within your own database, but the nastier case is the charge whose fate you don’t know. The process died mid-call, or the gateway timed out after possibly taking the money. The first defence at that boundary is an idempotency key, derived deterministically from the same intent that keys everything else - this instalment, this attempt - and sent with the charge, so a retried request is recognised by the provider and answered with the original outcome rather than a fresh charge. Take that protection everywhere it’s offered, because it closes the retry-after-crash case at the one layer you don’t control. What it doesn’t do is close everything: keys expire, their scope varies between providers, and some rails don’t support them at all, so an attempt whose key has aged out or crossed a provider boundary still needs the fate of the money established the hard way. Before charging a claimed instalment, we reconcile: ask the provider about prior attempts. A previous attempt that confirmed with a matching amount and currency means we settle from that attempt and never charge again. A definitive decline means we’re clear to proceed. Anything else - anything ambiguous - means we stop, leave the instalment in-flight, and let a human or a later reconciliation pass work out what happened. The directive decides the default: when you can’t tell whether the money moved, you assume it did.

On money paths I treat adversarial review as part of the definition of done, because fail-closed logic has a failure mode that unit tests love: the allowlist. Sure enough, a blind audit of the reconciliation caught an iteration that classified a prior attempt as “unresolved” if its error code appeared on a list - codes like EXCEPTION and TIMEOUT. It looked reasonable and the tests passed, but no gateway I’ve ever integrated actually emits those codes; a real timeout-after-charge comes back as a provider-specific error with a retryable flag set. A genuinely ambiguous prior attempt would have fallen straight through the list and proceeded to a second charge, which is precisely the failure the entire control exists to prevent. The correct shape is always the inverse: enumerate the provably safe cases, and fail closed on everything else. If you write fail-closed code by listing the dangerous cases, what you’ve actually written is fail-open code with extra steps - and that’s exactly the class of bug the audit step is there to catch, because everyone writes it eventually.

The orchestrator refuses the duplicate

There’s a layer above the database worth having too. I build almost everything on Temporal these days, in no small part because durable execution hands you primitives that map beautifully onto payments. Every collection attempt runs as a workflow whose ID is derived deterministically from the work itself - this instalment, this collection cycle - rather than generated fresh per run, and Temporal enforces uniqueness of running workflow IDs at the server. When an overlapping cron or an eager retry tries to start a second collection for the same instalment, the start call is simply rejected: the duplicate never gets a process, never claims a row, and never reaches for the gateway, because the orchestrator refused to run it at all. Deduplicating by intent rather than by request is the difference between idempotency you implement and idempotency you inherit from the infrastructure.

Failure paths get the same treatment through SAGA-style compensation. A collection workflow that trips a safety gate partway - the claim comes back empty, or reconciliation declines to proceed - doesn’t just die and leave debris behind; each step registers its compensating action, so the pending payment request is released and the in-flight marker cleared on every terminal path, whether that’s success, a graceful skip, or a thrown gate. Zombie state is what turns a one-off incident into a support queue, and compensation is how a workflow guarantees it cleans up whichever door it exits through.

None of this retires the conditional UPDATE, and that’s deliberate. A workflow ID only deduplicates identical intent, so anything arriving by a different door - a backfill keyed differently, an admin retrying by hand, a bug in the key derivation itself - slips straight past it. Layers in a payments system aren’t allowed to trust each other: the orchestrator refuses duplicate workflows, the database refuses double claims, the gateway refuses replayed charges, and the reconciler refuses ambiguity, each one assuming the layer above it has already failed. It’s the same discipline that puts an interlock on a submarine airlock so both doors can never stand open at once, and then builds bulkheads behind the airlock anyway - no single mechanism, however sound, ever gets sole custody of the hull.

The toolkit is older than the problem

None of this is novel, and that’s rather the point - claim-by-conditional-write, idempotency keys, fail-closed defaults, and reconcile-before-retry have been the standard kit for exactly-once-ish systems since long before anyone was renting distributed lock managers, and durable execution mostly lets you buy the outermost layer instead of building it. The craft is in applying them without flinching. Phrase every claim as a single conditional write and let rows-affected be the verdict, because the database has been better at mutual exclusion than your application code for forty years. Decide the preference order over failures before the first line of code - the missed charge beats the double charge, always - so every ambiguous branch has its answer in advance rather than being improvised at 2am. And put safety-critical review effort into the default branch, because the else clause is where a system reveals whether it fails open or closed, and test suites only ever exercise the cases somebody thought to enumerate.