The Payment That Charged Twice

A user taps "Pay". The spinner runs. Their connection drops somewhere between the phone and your server. The app retries. Two charges land on the card.

Nothing in that story requires a bug. Every component behaved exactly as designed. The phone retried because retrying is correct behaviour for an unacknowledged request. Your server processed both requests because, as far as it could tell, they were two people paying. The payment provider charged twice because you asked it to, twice.

This is the failure that teaches you what idempotency is actually for.

The word means less than people think

An operation is idempotent if doing it twice has the same effect as doing it once. DELETE /users/42 is naturally idempotent — the user is gone either way. POST /payments is not: each call creates something new.

Where teams go wrong is assuming that HTTP verb semantics do the work. They don't. PUT being "idempotent by spec" is a statement about what clients may assume, not a guarantee your handler provides. If your PUT appends to an audit table, increments a counter, or fires a webhook, it is not idempotent no matter what the RFC says. The property lives in your code, not in the method name.

So the real question is never "is this endpoint idempotent?" It is: what is the identity of this operation, and where do I store the fact that I already did it?

The naive version, and why it fails

Here is the implementation almost everyone writes first. The client sends a unique key; the server checks whether it has seen it.

async function createPayment(key: string, input: PaymentInput) {
  const existing = await db.payment.findUnique({ where: { idempotencyKey: key } });
  if (existing) return existing;                    // already done — return it
 
  const charge = await provider.charge(input);      // call the payment provider
  return db.payment.create({
    data: { idempotencyKey: key, chargeId: charge.id, amount: input.amount },
  });
}

Read that again with two requests running at the same instant, because that is exactly what a retry storm produces.

Request A checks: nothing found. Request B checks: nothing found — A hasn't written its row yet. Both proceed. Both call provider.charge. Two charges. The findUnique gave you a read, and a read tells you about the past, not about what another connection is doing right now.

This is a time-of-check-to-time-of-use race, and it is the single most common way idempotency is implemented incorrectly. The window is small — often single-digit milliseconds — which is precisely why it survives testing and shows up in production, where the retry arrives 40 ms after the original.

Let the database decide

The fix is to stop asking and start claiming. A unique constraint is the only participant in this system that can make an atomic decision across concurrent connections.

CREATE TABLE payment_attempts (
  idempotency_key  TEXT PRIMARY KEY,
  request_hash     TEXT        NOT NULL,
  status           TEXT        NOT NULL DEFAULT 'in_progress',
  response_body    JSONB,
  created_at       TIMESTAMPTZ NOT NULL DEFAULT now()
);

Now the first thing the handler does is insert. Whoever wins the insert owns the operation; everyone else is a duplicate by definition.

async function createPayment(key: string, input: PaymentInput) {
  const hash = sha256(JSON.stringify(input));
 
  // Claim the key. Exactly one concurrent caller can win this insert.
  const claimed = await db.$queryRaw`
    INSERT INTO payment_attempts (idempotency_key, request_hash)
    VALUES (${key}, ${hash})
    ON CONFLICT (idempotency_key) DO NOTHING
    RETURNING idempotency_key
  `;
 
  if (claimed.length === 0) {
    return handleReplay(key, hash);   // someone else owns it
  }
 
  const charge = await provider.charge(input, { idempotencyKey: key });
 
  await db.$executeRaw`
    UPDATE payment_attempts
       SET status = 'succeeded', response_body = ${JSON.stringify(charge)}::jsonb
     WHERE idempotency_key = ${key}
  `;
 
  return charge;
}

The race is gone, and it is gone for a structural reason: the decision is made by a single-writer index inside one transaction, not by application logic reading a value it hopes is still true.

The three cases people forget

Winning or losing the insert is the easy part. What you do on a replay is where correctness actually lives.

Same key, different body. A client reuses an idempotency key for a genuinely different request — usually a bug in their retry logic, occasionally an attack. If you blindly return the cached response, you have silently swallowed a payment the user meant to make. This is what request_hash is for:

async function handleReplay(key: string, hash: string) {
  const row = await db.paymentAttempt.findUnique({ where: { idempotencyKey: key } });
 
  if (row.requestHash !== hash) {
    throw new ConflictError('Idempotency key reused with a different payload');
  }
  if (row.status === 'in_progress') {
    throw new ConflictError('Original request still in flight — retry shortly');
  }
  return row.responseBody;
}

Returning 409 here is not unhelpful strictness. It is the only honest answer: you cannot serve a cached result for a request you never processed.

The still-in-flight replay. The retry arrives while the original is mid-call to the provider. There is no result to return yet. Returning 409 with a Retry-After is right; blocking the connection until the first one finishes is a good way to exhaust your connection pool during exactly the incident that caused the retries.

The crash between claim and completion. Your process dies after provider.charge succeeds but before the UPDATE. The row is now permanently in_progress, and every future retry gets a 409 for a payment that actually went through. This is the case that turns a small bug into a support ticket with a real charge attached to it.

The fix is a reconciliation job, not cleverer code in the request path:

// Runs every few minutes. Stale claims are asked about, never assumed.
const stale = await db.paymentAttempt.findMany({
  where: { status: 'in_progress', createdAt: { lt: minutesAgo(5) } },
});
 
for (const attempt of stale) {
  const remote = await provider.lookupByIdempotencyKey(attempt.idempotencyKey);
  await db.paymentAttempt.update({
    where: { idempotencyKey: attempt.idempotencyKey },
    data: remote
      ? { status: 'succeeded', responseBody: remote }
      : { status: 'failed' },
  });
}

Note what makes this work: the provider is the source of truth, and you queried it with the same key. Your idempotency key is only worth as much as the propagation of that key to every downstream system that can move money.

Where the key should come from

The client generates it, once, per user intent — not per HTTP attempt. A UUID created when the checkout screen mounts is right. A UUID created inside the retry wrapper is worthless, because every retry gets a fresh key and you are back to double charges.

That single sentence is, in my experience, the most commonly violated rule in this whole design. The server-side machinery is usually fine. The key is generated in the wrong place, so all of it is decoration.

Server-generated keys are the wrong shape too, though occasionally tempting. If you derive the key from (user_id, amount, minute), you have built a rule that a user may not legitimately buy the same thing twice in a minute — which is a product decision smuggled in as an infrastructure detail. Sometimes that is what you want. Usually it will surprise someone six months from now.

What to keep and for how long

Keep the attempt rows long enough to outlive any retry a client could plausibly send, and no longer. Twenty-four hours covers essentially all real retry behaviour, including a user who backgrounds the app and returns. Some providers use 24 hours; some use longer. Pick a number, write it in the API docs, and delete on a schedule — an unbounded table of keys becomes its own operational problem.

DELETE FROM payment_attempts WHERE created_at < now() - INTERVAL '24 hours';

Storing the response body matters more than it looks. A replay should return the same result — same charge ID, same timestamps — not a freshly computed equivalent. Clients diff these. If the second call returns a different created_at, someone's reconciliation will eventually flag it.

This post is about the request you send. The mirror image — the callback a provider sends you, and why deduplicating it needs a state machine rather than a unique index — is in Idempotency Is Not a Header. It Is a Contract.

The broader shape

Idempotency keys are a specific instance of a general move: when you cannot prevent duplicate work, make duplicate work detectable and cheap. The same shape appears in message consumers (dedupe by message ID), in webhook receivers (dedupe by event ID), and in job queues (a unique job key so the same task is not enqueued twice).

In all of them the discipline is identical. Name the operation with a stable identifier that survives retries. Claim it atomically. Record the outcome. Decide, explicitly, what a replay means.

Get those four right and duplicate delivery stops being a category of bug. It becomes a thing that happens several times a day and that nobody ever notices — which is what reliability actually looks like from the inside.