Skip to content
←Resilience & Messaging

// Resilience & Messaging · 09

Distributed Lock

Two nodes, one jobleads toOne holder at a time

Only one process, across many machines, does this at a time.

How it fits together

Distributed Lock structureInstance Alease 60s, set if absentInstance Blease 60s, set if absentLock storeinvoice-job = ownerfence = incrementInvoices tablereject fence < lastSkipped run1 set if absent2 token 73 write token 71 set if absent2 heldno lock: skip
uses / callscreatesfails / error path· numbered steps replay in order
  • Instance A uses / calls Lock store (1 set if absent)
  • Lock store creates Instance A (2 token 7)
  • Instance A uses / calls Invoices table (3 write token 7)
  • Instance B uses / calls Lock store (1 set if absent)
  • Lock store fails / error path Instance B (2 held)
  • Instance B fails / error path Skipped run (no lock: skip)

Spot it

  • A scheduled job runs on every instance at once
  • Two workers process the same file
  • Races that only appear when scaled out

It costs

Another component that can fail, and leases make correctness depend on timing.

In code

public sealed class InvoiceJob(IDatabase redis, AppDb db){    private const string Key = "invoice-job";    private const string ReleaseScript = "if redis.call('get', KEYS[1]) == ARGV[1] then return redis.call('del', KEYS[1]) end return 0";​    public async Task<long?> TryAcquireAsync(string owner, CancellationToken ct)    {        var acquired = await redis.StringSetAsync(            Key,            owner,            TimeSpan.FromSeconds(60),            When.NotExists);        return acquired ? await redis.StringIncrementAsync($"{Key}:fence") : null;    }​    public Task ReleaseAsync(string owner) =>        redis.ScriptEvaluateAsync(ReleaseScript, [Key], [owner]);​    public async Task SaveAsync(long fence, IEnumerable<Invoice> invoices, CancellationToken ct)    {        var job = await db.JobFences.SingleAsync(f => f.Name == Key, ct);        if (job.LastFence >= fence) throw new InvalidOperationException("Stale fencing token.");        job.LastFence = fence;        db.Invoices.AddRange(invoices);        await db.SaveChangesAsync(ct);    }}
Go deeperThe full explanation and what to ask for · 1 min readExpand

A distributed lock is a lease, not a mutex: it expires so a crashed holder cannot block forever. That expiry is also the danger. A GC pause or network delay can outlast the lease, and the old holder carries on believing it owns the lock while a new holder works.

The classic mistake is a lock without fencing tokens. Each acquisition should get a monotonically increasing number, and the protected resource must reject writes carrying an older one. Release must check the value too, or a slow holder deletes a lock that is now someone else's.

Lean on locks for efficiency (avoiding duplicate work) more readily than for correctness. When correctness depends on it, prefer the database: optimistic concurrency, a unique constraint, or application locks inside the transaction (SQL Server's sp_getapplock, PostgreSQL advisory locks). An idempotent operation often removes the need for a lock entirely.

Ask for it

  • lease / TTL
  • fencing token
  • set-if-absent with expiry
  • lock renewal
  • leader election
  • Redlock

Run the nightly invoice job on one instance only: take a lock on the job key with a 60s expiry (set only if absent), renew every 20s, release with a compare-and-delete, and take an increasing fencing token that the invoices store rejects if it is older than the last one written.

Connects to