There is no exactly-once delivery, only idempotency

Every integration conversation in payments arrives at the same request eventually: we need exactly-once delivery. What the person means is reasonable - never double-charge, never double-settle, never pay a merchant twice for one instruction. What they are asking for, taken literally, is not on sale. Over an asynchronous network where either side can fail, a sender that receives no acknowledgement cannot distinguish a lost message from a lost reply. It has two options: send again, or do not. Those are at-least-once and at-most-once, and there is no third setting on the wire. Exactly-once is not a delivery guarantee anyone can hand you. It is a processing outcome you construct, and it is worth being precise about where it can hold.

It can hold inside a boundary, and Kafka's exactly-once semantics are a real example rather than a marketing one. The idempotent producer tags each batch with a producer ID and a per-partition sequence number, so the broker recognizes a retried batch and drops it instead of appending it twice. Transactions extend that: a consumer can write its output records and commit its input offsets atomically, so a read-process-write step does not reprocess after a crash or a rebalance, and readers on read_committed never see the aborted attempt. Look closely at why it works. The effect and the record of having performed it land in the same transactional store. That is the whole trick, and it is also the limit - the guarantee is a property of the boundary, not of the broker.

Cross that boundary and you are back to at-least-once, whatever the diagram says. Post to a card network, a core banking system, or a partner's REST endpoint and the effect lives on their side while the offset lives on yours. No transaction spans both, and two-phase commit across an organization you do not operate is not a plan. So we build for at-least-once and make a duplicate cheap. Every mutation that touches money carries an idempotency key supplied by the caller and derived from the business event - this instruction, this settlement leg - rather than generated at the retry site, because a retry that invents a fresh key defeats the mechanism entirely. On our side the handler writes the key, the outcome, and the effect in one local transaction, enforced by a unique constraint rather than a lookup first: check-then-act across two statements is a race that concurrency will find. A second arrival with the same key returns the stored response instead of doing the work again.

The same key travels downstream with the request, so the counterparty can run the same check on their side, and we record the attempt before making the call, so a process that dies mid-flight recovers by asking what happened to that key rather than firing a second instruction blind. Some operations resist all of this because they are defined as increments. “Add 100 to the balance” carries no identity of its own, so a redelivery is indistinguishable from a second genuine instruction. The fix is not a cleverer deduplication window - it is to give the effect an identity. Post a ledger entry keyed by the instruction and derive the balance from the entries, and applying the same instruction twice becomes a primary key violation instead of a hundred units of quiet loss.

Two operational details decide whether any of it holds under load. Keys need a retention window longer than the longest retry in the chain, your partners' included, because a key expired early is a duplicate accepted late. And every event for one account belongs on one partition, so ordering is preserved where it matters and the deduplication you are doing stays local rather than global. Get those right and the payoff is operational rather than architectural. In national settlement work the question is never whether the message arrived; it is whether we can prove the instruction was applied once. A system carrying millions of transactions a day across every bank in a country cannot be run by people who reason case by case about whether a redelivery is safe, and that is what makes replaying a partition at 3 a.m. a decision an engineer can take alone. Exactly-once was the wrong thing to ask for. A system that does not care how many times you deliver is the thing worth building.