Leases & Fencing Tokens — Why Your Distributed Lock Isn't Safe.

❌The myth

"I put a Redis/etcd lock around the critical section. SET key val NX EX 30. Only one worker can be inside at a time, so the write is safe."

✅Actually

A distributed lock does NOT guarantee mutual exclusion over the resource. It only guarantees that two workers don't both hold the lock. Those are different things, and the gap between them is where your data corrupts.

The lock lives in Redis. The write lands in S3/Postgres. Nothing connects them.

🔬The proof (15 seconds of pain)

Worker A: acquires lock, TTL=30s        t=0
Worker A: GC pause / CPU throttle       t=1..35   ← frozen, still "thinks" it holds it
Lock expires in Redis                   t=30
Worker B: acquires lock legitimately    t=31
Worker B: writes file.json              t=32
Worker A: wakes up, writes file.json    t=35   💥 clobbers B

Worker A never did anything wrong. It never got told it lost the lock — there is no such notification. A 4-second stop-the-world GC pause, a Kubernetes CPU-limit throttle, or one slow fsync is enough. Longer TTLs don't fix it; they just move the window.

🧠The correct mental model

A lock is not a lock. It's a LEASE — permission that expires on the granter's clock, not yours. Holding a lease means "I probably still have it." That's the strongest claim available.

To make it safe, the resource must reject stale holders. That's a fencing token: a monotonically increasing number handed out with every lease.

acquire() -> { lease, token: 33 }
Worker A got token 33
Worker B got token 34   (strictly greater — always)

Then the storage layer enforces it:

UPDATE docs SET body=$1, fence=34
WHERE id=$2 AND fence < 34;
-- rows affected = 0  → you lost the race, abort

When zombie A comes back with token 33, the WHERE clause rejects it. The resource enforced correctness, not the lock service.

🧰Where tokens come from

• etcd / ZooKeeper: every write bumps a revision / zxid — use it directly
• Postgres: a SEQUENCE, or the row's own version column
• S3/GCS: no counters, but preconditions work — If-Match: <etag> / x-goog-if-generation-match
• Redis alone: no real fencing. Redlock gives you a lease, not a token. Fine for "don't send 5 duplicate emails", not for "don't corrupt the ledger".

⚠️Three traps practitioners hit

1. Renewing the lease in a background thread and assuming renewal success = safety. If the renewal thread is alive but the worker thread is paused, you renew a lease you're not actually using.
2. Checking if lock.is_held() right before writing. Classic TOCTOU — the pause can happen between the check and the write.
3. Treating TTL as a timeout. TTL must exceed max pause, and max pause is unbounded. Design for expiry, don't try to prevent it.

🎯The real rule

Pick one:

• Make the operation idempotent → losing the lease is harmless
• Make the resource fence-aware → stale writes get rejected

If you can do neither, you have an availability lock (nice-to-have dedup), not a correctness lock. Know which one you shipped.

🏋️Your exercise

Take any "only one job at a time" cron/worker you own. Answer: if that process froze for 60 seconds mid-run and a second copy started, what exactly breaks? Then add a fence BIGINT column and put AND fence < $token in the UPDATE. Run two copies at once and confirm one gets 0 rows affected.

Bonus question: why does a shorter TTL make this bug more likely to appear, but a longer one make it more damaging when it does?