Job Scheduling System Design

Level: advanced
Read: 15 min
Concepts: 1

A job scheduler is a durable list of work with a claiming protocol on top, and nearly every design question in it resolves to "what happens if a worker dies holding this job". Jobs are persisted with a run-at time before anyone tries to execute them, so scheduling is a query over due work rather than an in-memory timer that a restart forgets. Workers claim jobs with a lease — a claim that expires — rather than a lock, because a crashed worker stops renewing and the job becomes claimable again with no operator involved and no stuck lock to clear. That single property is why leases win for work distribution, and it is also the source of the system's central constraint: a lease can expire while its holder is merely slow, and from outside a slow worker and a dead one are indistinguishable. So execution is at-least-once, idempotent handlers are mandatory rather than advisable, and the remedy lives in the handler, which is the only place that knows what "already done" means for this work. Leases are kept short and renewed by heartbeat rather than set long, since a long lease trades slow crash recovery for fewer duplicates and a short un-renewed one makes healthy workers lose jobs mid-run. Retries back off up to a maximum attempt count and then dead-letter, so a poison job is inspectable and replayable instead of retried forever or silently dropped. Recurrence emits one job per occurrence keyed by schedule and occurrence time, which stops two scheduler instances double-firing a nightly run and simultaneously deduplicates enqueue from an at-least-once producer. And monitoring watches the age of the oldest waiting job rather than the error rate, because a scheduler's characteristic failure is omission — no jobs run, no jobs fail, every dashboard is green, and the outage is reported by a customer asking where their report went.

This section

Batching, Fan Out and scheduling in System DesignHitesh Choudhary

What is true here

A lease is a claim with an expiry, so crash recovery is a property of the mechanism rather than of someone noticing.
The same expiry means a slow worker can lose a job it is still running — at-least-once is the guarantee, so handlers must be idempotent.
Renew leases by heartbeat instead of setting them long; a fixed lease shorter than the work makes healthy workers duplicate each other.
Dead-letter on attempt exhaustion so a poison job can be inspected, fixed and replayed rather than retried forever or dropped.
Alert on oldest-job age, not error rate — a stalled scheduler produces zero errors because it produces nothing at all.

What you will be able to do

Write a claim query that two workers cannot both win, and explain why the lease has an expiry rather than an explicit release
Choose a lease duration and renewal interval from the job's runtime distribution
Make a job handler idempotent against the specific duplication a lease expiry can cause
Design alerts that fire when scheduled work silently stops

The whole design, from one question

Durable jobs, leases, heartbeat renewal, retries, dead-lettering, recurrence keys and omission-aware monitoring.

Durable jobs, leases, retries, recurrence, dead-lettering and monitoring

coreadvanced

A job scheduler is a durable list of work with a claiming protocol on top, and almost every design question resolves to "what happens if a worker dies holding this job". Jobs are stored durably before anyone tries to run them, with a run-at time, so scheduling is a query — pick up everything due — rather than an in-memory timer that a restart forgets. Workers claim jobs with a lease rather than a lock: a lease is a claim with an expiry, so a worker that crashes stops renewing and the job becomes claimable again automatically, with no operator involved and no lock left dangling forever. That single mechanism is why leases are preferred to locks for work distribution, and it is also why jobs must be idempotent: a lease can expire while the original worker is merely slow, so the job can genuinely run twice. Retries have a backoff and a maximum attempt count, and when that count is exhausted the job goes to a dead-letter queue rather than being retried forever or dropped — a dead letter is a job you can inspect, fix and replay, which a discarded one is not. Recurrence is expressed as a schedule that produces a new job for each occurrence, keyed by occurrence so a scheduler running on two machines cannot produce two runs of the same nightly report. Deduplication is that same key applied to enqueue: an at-least-once producer can submit the same job twice, and a unique key on the occurrence makes the second submission a no-op. And monitoring watches queue depth, oldest-job age and dead-letter rate, because a scheduler fails quietly — nothing is broken, work just stops happening.

Think of it as

A pile of job cards and a set of pigeonholes. A worker takes a card and pins their name and a time to it: "mine until 10:15". If they finish, the card is filed as done. If they are still working at 10:14, they re-pin a later time. If they fall over, nobody has to notice — at 10:15 the card is unpinned automatically and someone else takes it. Every hard question about job systems becomes easy in this picture: a slow worker and a dead worker look identical from outside, so the card can be worked twice, so the work has to be safe to repeat.

sql

-- claim: one statement, so two workers cannot
-- both take the same job
UPDATE jobs
   SET leased_until = now() + interval '60 seconds',
       leased_by    = $1,
       attempts     = attempts + 1
 WHERE id = (
     SELECT id FROM jobs
      WHERE run_at <= now()
        AND (leased_until IS NULL OR leased_until < now())
      ORDER BY run_at
      FOR UPDATE SKIP LOCKED
      LIMIT 1)
RETURNING *;

What we're doing: Watch a worker die mid-job, then watch a slow worker lose its lease, and see why both end at the same requirement.

lease-expiry.txttext

Job 771: send the monthly invoice for account 4.
Lease duration 60s. Handler normally takes 8s.

CASE A -- the worker crashes
  10:00:00  worker-3 claims, leased_until 10:01:00
  10:00:04  worker-3 is killed (node terminated)
  10:01:00  lease expires; job returns to ready
  10:01:02  worker-7 claims and completes it
  Recovery took 62 seconds and no human.
  Nobody had to notice the crash.

CASE B -- the worker is merely slow
  10:00:00  worker-3 claims, leased_until 10:01:00
  10:00:04  the invoice PDF service is degraded;
            the handler is still waiting
  10:01:00  lease expires -- worker-3 is ALIVE
            and still working
  10:01:02  worker-7 claims the SAME job and
            starts running it
  10:01:40  worker-3 finishes: invoice sent
  10:01:48  worker-7 finishes: invoice sent AGAIN

Case B is not a bug in the lease. It is what a
lease means. From outside, a slow worker and a
dead worker are indistinguishable -- so the
handler must be idempotent, or the customer gets
two invoices.

The fix is in the handler, not the lease:
  INSERT INTO sent_invoices (account, period)
  ON CONFLICT DO NOTHING
  -- the second run sends nothing

9: This is the entire argument for leases over locks. A lock held by a terminated process needs something to detect the death and release it; an expiry needs nothing, so recovery is a property of the mechanism rather than of an operator being awake.
18: The same property that recovers case A creates case B. You cannot have automatic recovery from crashes without accepting that "crashed" and "slow" look identical, which is precisely why at-least-once is the guarantee on offer.
29: The remedy lives in the handler because it is the only place that knows what the work means. A unique constraint on (account, period) makes the second execution a no-op regardless of how many times the job runs.

Why this works: Lengthening the lease does not solve case B, it only makes it rarer while making case A slower to recover — the two are the same dial pulled in opposite directions. The design accepts at-least-once execution and puts correctness in the handler, which is the only place with enough context to define what "already done" means.

Setting a lease shorter than the job it covers

Wrong

text

lease = 30s
# handler routinely takes 45s
# Every job is reclaimed while its worker is
# still running it. Two workers finish the same
# job, then a third starts it. Throughput
# collapses as workers duplicate each other.

Better

text

lease = 30s, renewed every 10s by the handler
# a healthy worker keeps its claim indefinitely;
# a dead one stops renewing and the job is
# reclaimed 30s later, not 45+ minutes later

What you see: Jobs are executed several times each under normal conditions, worker CPU is spent on duplicate work, and the queue drains far more slowly than the worker count suggests it should — while every individual worker appears healthy.

Why: A fixed lease has to be longer than the slowest normal run, which for variable work means an uncomfortably long lease and correspondingly slow crash recovery. Heartbeat renewal decouples the two: the lease stays short, so a crash is detected quickly, while a healthy worker keeps extending its claim for as long as it genuinely needs.

One job's lifecycle, including the paths a worker crash takes

scheduled (run_at in the future)

start

ready (due, unclaimed)

leased (a worker holds it)

done

end

dead-lettered

end

Lease versus lock for distributing work

Lease versus lock for distributing work
Property	Lock	Lease
Held until	Explicitly released	An expiry time, unless renewed
Worker crashes	Lock is held until someone intervenes	Expires; the job is reclaimed automatically
Slow worker	Keeps the lock, blocks progress	Loses the claim unless it renews — the job may run twice
Requires idempotent work?	Less often	Always
Operational burden	Stuck locks need manual clearing	None — expiry is the recovery mechanism

Ten requirements, and where each one lives

Ten requirements, and where each one lives
Requirement	Mechanism	What it prevents
Scheduling	`run_at` column plus a due-work query	Work lost on restart
Durable storage	Persist before execution	Work that exists only in a worker's memory
Workers	Pull-based claim loop	Push-based assignment to a worker that is already dead
Leases	Claim with an expiry, renewed by heartbeat	A crash stranding a job forever
Retries	Backoff plus a max attempt count	A poison job retried indefinitely
Recurrence	One job per (schedule, occurrence)	Two scheduler instances double-firing
Deduplication	Unique constraint on the occurrence key	Duplicate enqueue from an at-least-once producer
Idempotency	Handler safe to run twice	Double effects when a lease expires on a slow worker
Dead-lettering	A separate queue for exhausted jobs	Silent loss, and infinite retry
Monitoring	Oldest-job age, queue depth, dead-letter rate	Work quietly stopping with no errors raised

Remember: Persist jobs with a `run_at` so scheduling is a query, and claim them with a lease — a claim that expires — so a crashed worker releases its job automatically. That same expiry means a slow worker can lose a job it is still running, so at-least-once execution is the guarantee and idempotent handlers are mandatory. Renew leases by heartbeat rather than setting them long, retry with backoff up to a cap, dead-letter on exhaustion so failures can be inspected and replayed, key recurrence and deduplication by occurrence, and alert on oldest-job age — because a scheduler that stops produces no errors at all.