Idempotency keys for safe retries in HTTP APIs and job workers

TL;DR — Idempotency keys are a server-side contract: the client says "this retry is the same operation," and the server stores enough state to return the original result instead of doing the work twice. The most common correct implementation is: require an Idempotency-Key on side-effecting requests, atomically reserve it before executing, store the final HTTP status/body keyed by (tenant, route, key), and reject key reuse with a different payload. Reading time: ~7 min

What it is and where it sits

An idempotency key is not "deduplication magic." It is a narrow contract between a caller and the system handling a side-effecting operation: "If I send this same operation again with the same key, treat it as the same attempt and give me the same outcome."

In practice, it sits at the API boundary or job-consumer boundary, before the code that creates irreversible effects:

• charging a card
• creating an order
• provisioning an account
• sending an email/SMS
• enqueueing downstream work

What it replaces: brittle "did this already happen?" checks scattered through business logic, and unsafe client behavior like blind retrying POST /payments after a timeout.

What talks to it:

• mobile/web clients retrying after network errors
• API gateways or SDK retry middleware
• internal services calling other services
• queue consumers reprocessing the same message after visibility timeout / crash

Where it lives in a typical flow: usually in the application service, backed by a durable store like Postgres or Redis plus durable result storage. You can enforce it at the gateway, but the authoritative record needs to be close to the write transaction or you will still race.

Client/SDK
  |
  | POST /payments
  | Idempotency-Key: 7d6d...
  v
API server / handler
  |
  |-- reserve key in idempotency store
  |     (tenant, route, key, request_hash, status)
  |
  |-- execute business transaction
  |     create payment row
  |     call provider
  |
  |-- store final response for that key
  v
HTTP response

Retry with same key ---> same handler checks store ---> returns stored response

A useful mental model: the key names an operation instance from the caller's point of view. The server binds that name to exactly one result.

How it actually works

Walk one realistic example: POST /v1/orders from a checkout service. The user clicks Pay, the client sends the request, the server creates the order and charges a payment provider. The response is lost because the load balancer drops the connection after upstream completed. The client retries.

Step 1: client sends a key

The client generates a high-entropy key per logical operation, not per HTTP attempt. UUIDv4 is fine.

curl -i https://api.example.com/v1/orders \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer eyJ...' \
  -H 'Idempotency-Key: 8c1f6d7e-0a8b-4f2f-8f98-9f0f0d8c2b11' \
  --data '{"cart_id":"cart_123","amount":4999,"currency":"USD"}'

Step 2: server canonicalizes and reserves

Before doing any side effect, the handler computes a request fingerprint from the parts that define sameness. Usually:

• authenticated tenant/account ID
• HTTP method + route template
• canonical JSON body
• sometimes selected headers if they affect semantics

Do not key only on the raw idempotency key. tenant A + key X must not collide with tenant B + key X.

The server then performs an atomic insert like:

• status = in_progress
• request_hash = SHA-256 of canonical request
• locked_until = now + short lease

If the insert succeeds, this request owns execution.

If the row already exists:

• same request_hash and final response present → return stored response
• same request_hash but in_progress → return 409 Conflict or 425 Too Early style response and tell client to retry later
• different request_hash → return 409 Conflict; same key, different operation is a client bug

Typical conflict response shape:

HTTP/1.1 409 Conflict
Content-Type: application/json
Idempotency-Status: conflict

{"error":"idempotency_key_reused_with_different_parameters"}

Step 3: business transaction runs

The handler creates the order row and, if needed, records an outbox event in the same database transaction. If you call an external payment provider directly inside the request, you now have two systems to reconcile. The safer pattern is:

1. create order with status pending
2. insert outbox event charge_requested
3. commit
4. worker processes outbox and talks to provider with its own idempotency key

That gives you local atomicity and pushes external retries into a controlled worker.

Step 4: store the final response

After success, the server updates the idempotency row with:

• status = completed
• HTTP status code, e.g. 201
• response body bytes or a pointer to them
• resource ID, e.g. order_id
• expiry timestamp

If the first attempt dies after the order commit but before this update, you need recovery logic. Common options:

• wrap order creation and idempotency update in one DB transaction when the response can be derived from DB state
• or on retry, detect existing order_id by business key and backfill the idempotency record

Step 5: retry arrives

The client times out and retries with the same key.

curl -i https://api.example.com/v1/orders \
  -H 'Content-Type: application/json' \
  -H 'Authorization: Bearer eyJ...' \
  -H 'Idempotency-Key: 8c1f6d7e-0a8b-4f2f-8f98-9f0f0d8c2b11' \
  --data '{"cart_id":"cart_123","amount":4999,"currency":"USD"}'

Server sees completed for the same (tenant, route, key, hash) and returns the stored result without charging again.

Typical replay response:

HTTP/1.1 201 Created
Content-Type: application/json
Idempotency-Status: replayed
Idempotency-Key: 8c1f6d7e-0a8b-4f2f-8f98-9f0f0d8c2b11

{"order_id":"ord_987","status":"pending_payment"}

That is the whole point: retries become safe because the server remembers the first outcome.

When to use it (and when not to)

Use idempotency keys when the caller may retry and the operation has side effects that must not happen twice.

You probably do not need idempotency keys if all of these are true:

• duplicates are harmless or already tolerated
• the operation is naturally idempotent by resource identity, like PUT /users/123
• a unique constraint on a business key fully prevents duplicate state and you do not need to replay the original HTTP response

You probably do need them if any of these are true:

• clients retry on timeout or 5xx
• the operation triggers money movement, messages, or provisioning
• duplicate work is expensive or embarrassing
• you cannot tell from the client whether the first attempt committed

Trade-offs

Every benefit has a cost.

• Safe client retries without duplicate side effects

  • Cost: durable storage for keys and responses, plus expiry cleanup.

• Cleaner API contract for POST

  • Cost: clients must generate and persist keys across retries; SDKs need support.

• Fewer duplicate orders/payments

  • Cost: request canonicalization and payload-hash comparison logic, which is easy to get subtly wrong.

• Better resilience during network failures

  • Cost: more states to operate: in_progress, completed, expired, conflict, and recovery after partial failures.

• Replay exact original response

  • Cost: storing response bodies can be expensive; large payloads may need compression or storing a resource pointer instead.

• Works across horizontally scaled app servers

  • Cost: shared backing store becomes part of the write path; expect extra latency and contention on hot keys.

Operationally, the hard questions are:

• TTL: too short and legitimate retries miss; too long and storage grows. Common windows are 24 hours to 7 days depending on client behavior.
• Scope: key by tenant + route + method, not globally.
• Hashing: canonicalize JSON before hashing or semantically identical payloads with different key order will conflict.
• Concurrency: two identical requests can arrive at once. Use atomic insert/upsert, not read-then-insert.
• External side effects: your local idempotency record does not make a third-party API idempotent. Pass a corresponding key downstream if the provider supports it.

In practice

Postgres table and atomic reservation

CREATE TABLE api_idempotency_keys (
  tenant_id text NOT NULL,
  route text NOT NULL,
  idem_key text NOT NULL,
  request_hash bytea NOT NULL,
  status text NOT NULL CHECK (status IN ('in_progress', 'completed', 'failed')),
  http_status integer,
  response_body jsonb,
  resource_id text,
  locked_until timestamptz,
  expires_at timestamptz NOT NULL,
  created_at timestamptz NOT NULL DEFAULT now(),
  updated_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (tenant_id, route, idem_key)
);

INSERT INTO api_idempotency_keys (
  tenant_id, route, idem_key, request_hash, status, locked_until, expires_at
) VALUES (
  $1, $2, $3, $4, 'in_progress', now() + interval '30 seconds', now() + interval '24 hours'
)
ON CONFLICT DO NOTHING;

This reserves the key exactly once across all app instances. Gotcha: ON CONFLICT DO NOTHING only tells you whether you won the race; you still need a second SELECT to inspect existing request_hash, status, and stored response.

Express middleware with conflict/replay behavior

import crypto from 'node:crypto';
import express from 'express';
import pg from 'pg';

const app = express();
app.use(express.json());
const db = new pg.Pool({ connectionString: process.env.DATABASE_URL });

function canonicalJson(value) {
  if (Array.isArray(value)) return '[' + value.map(canonicalJson).join(',') + ']';
  if (value && typeof value === 'object') {
    return '{' + Object.keys(value).sort().map(k => JSON.stringify(k) + ':' + canonicalJson(value[k])).join(',') + '}';
  }
  return JSON.stringify(value);
}

app.post('/v1/orders', async (req, res) => {
  const idemKey = req.get('Idempotency-Key');
  if (!idemKey) return res.status(400).json({ error: 'missing_idempotency_key' });

  const tenantId = req.user.account_id;
  const route = 'POST:/v1/orders';
  const requestHash = crypto.createHash('sha256').update(canonicalJson(req.body)).digest();

  const insert = await db.query(
    `INSERT INTO api_idempotency_keys (tenant_id, route, idem_key, request_hash, status, locked_until, expires_at)
     VALUES ($1,$2,$3,$4,'in_progress', now() + interval '30 seconds', now() + interval '24 hours')
     ON CONFLICT DO NOTHING`,
    [tenantId, route, idemKey, requestHash]
  );

  if (insert.rowCount === 0) {
    const existing = await db.query(
      `SELECT request_hash, status, http_status, response_body
       FROM api_idempotency_keys WHERE tenant_id=$1 AND route=$2 AND idem_key=$3`,
      [tenantId, route, idemKey]
    );
    const row = existing.rows[0];
    if (!row.request_hash.equals(requestHash)) {
      return res.status(409).set('Idempotency-Status', 'conflict').json({ error: 'idempotency_key_reused_with_different_parameters' });
    }
    if (row.status === 'completed') {
      return res.status(row.http_status).set('Idempotency-Status', 'replayed').json(row.response_body);
    }
    return res.status(409).set('Retry-After', '2').json({ error: 'request_still_in_progress' });
  }

  const client = await db.connect();
  try {
    await client.query('BEGIN');
    const order = await client.query(
      `INSERT INTO orders (account_id, cart_id, amount, currency, status)
       VALUES ($1,$2,$3,$4,'pending_payment') RETURNING id, status`,
      [tenantId, req.body.cart_id, req.body.amount, req.body.currency]
    );
    const body = { order_id: order.rows[0].id, status: order.rows[0].status };
    await client.query(
      `UPDATE api_idempotency_keys
       SET status='completed', http_status=201, response_body=$4, resource_id=$5, updated_at=now()
       WHERE tenant_id=$1 AND route=$2 AND idem_key=$3`,
      [tenantId, route, idemKey, body, order.rows[0].id]
    );
    await client.query('COMMIT');
    return res.status(201).json(body);
  } catch (e) {
    await client.query('ROLLBACK');
    throw e;
  } finally {
    client.release();
  }
});

This is the core pattern most teams need: reserve, compare hash, replay completed response, otherwise execute once. Gotcha: if you call an external provider before updating the idempotency row, a crash can leave you with a real side effect but no stored response; use an outbox or pass the same key downstream.

Cleanup job for expired keys

⚠️ Deleting old idempotency records shortens your replay window. If clients may retry after mobile offline periods or long queue delays, increase expires_at before running aggressive cleanup.

psql "$DATABASE_URL" -c "DELETE FROM api_idempotency_keys WHERE expires_at < now() LIMIT 5000;"

Run this from cron or your scheduler every few minutes. Gotcha: on large tables, add an index on expires_at or cleanup will turn into a table scan.

Further reading

• RFC 9110 HTTP Semantics
• The "Idempotent methods" section of the MDN HTTP docs
• Stripe API docs: Idempotent requests
• Martin Kleppmann, "Designing Data-Intensive Applications"
• The "Transactional Outbox" pattern on microservices.io

This article was written by an AI system and published pending human review. Verify anything you intend to act on.