DEV Community: delay.run

The transactional outbox, explained: how to save to the database and enqueue a job without losing either

delay.run — Mon, 09 Nov 2026 11:00:01 +0000

Here's a piece of code that looks completely reasonable:

def place_order(cart):
    order = db.insert_order(cart)      # 1. save to the database
    queue.enqueue("send_confirmation", order_id=order.id)  # 2. enqueue a job
    return order

And here's the problem. There are two separate systems involved: the database and the queue. Nothing makes those two steps happen together.

If the process crashes between step 1 and step 2, the order exists but the confirmation email is never sent.
If you swap the order, enqueue first and then save, a crash in between means the worker sends a confirmation for an order that doesn't exist.
If the database transaction rolls back after you enqueued inside it, same problem: a job for something that never happened.

This is sometimes called the dual-write problem. The transactional outbox pattern is the standard, boring and effective fix.

The idea in one sentence

Instead of writing to the database and the queue, write only to the database, including a record of the job you want to run, in the same transaction. A separate process then moves those records into the queue.

Step by step

1. Add an outbox table.

CREATE TABLE outbox (
  id           bigserial PRIMARY KEY,
  job_type     text        NOT NULL,
  payload      jsonb       NOT NULL,
  created_at   timestamptz NOT NULL DEFAULT now(),
  published_at timestamptz
);

2. Write the business change and the outbox row in one transaction.

def place_order(cart):
    with db.transaction():
        order = db.insert_order(cart)
        db.insert_outbox("send_confirmation", {"order_id": order.id})
    return order

Now the database guarantees it: either the order and its outbox row both exist, or neither does. There's no gap for a crash to fall into.

3. Run a relay that publishes outbox rows.

A small background process repeatedly picks up unpublished rows, sends them to the queue, and marks them published.

while True:
    with db.transaction():
        rows = db.query("""
            SELECT id, job_type, payload FROM outbox
            WHERE published_at IS NULL
            ORDER BY id
            LIMIT 100
            FOR UPDATE SKIP LOCKED
        """)
        for row in rows:
            queue.enqueue(row.job_type, **row.payload)
            db.execute("UPDATE outbox SET published_at = now() WHERE id = %s", row.id)
    if not rows:
        time.sleep(1)

FOR UPDATE SKIP LOCKED lets you run more than one relay without them grabbing the same rows.

The catch: at-least-once, not exactly-once

Look closely at the relay. If it enqueues a job and then crashes before marking the row published, the next run will enqueue it again.

That's not a bug in the pattern; it's the trade-off. The outbox converts "might lose the job" into "might deliver the job twice". Losing work silently is much worse than duplicating it, as long as the job can handle duplicates.

So the outbox pattern goes hand in hand with idempotent consumers: workers that check whether they've already handled a given outbox ID (or a business key, like "confirmation email for order 123") before acting. Pass the outbox row's ID along in the payload so consumers have something stable to deduplicate on.

Variations

Skip the queue entirely. If your workers can read from Postgres directly, the outbox table is the queue. Workers claim rows with SKIP LOCKED, do the work and mark them done. One fewer system to run.

Change data capture. Instead of polling, some setups read the database's write-ahead log and stream new outbox rows to a broker. Lower latency, more infrastructure.

Delayed jobs. Add a run_at column, and the relay only publishes rows whose time has come. Now scheduling a reminder for next week is part of the same transaction as creating the appointment.

Housekeeping

The outbox grows forever unless you clean it. Delete or archive published rows after a retention period, keeping them long enough to debug "did we ever send this?" questions. An index on unpublished rows keeps the relay's query fast:

CREATE INDEX outbox_unpublished ON outbox (id) WHERE published_at IS NULL;

And monitor two numbers: the count of unpublished rows and the age of the oldest one. If either keeps growing, the relay is stuck or too slow, and every job in your system is waiting behind it.

Ordering, and why you might not get it

Relays usually publish rows in ID order, but that doesn't guarantee consumers process them in that order. With several workers, a later job can easily finish before an earlier one. If order matters, for example "account created" must be handled before "account upgraded", either process jobs for the same entity on a single partition or worker, or make each consumer check the current state in the database rather than trusting the order of events. The second approach is usually simpler and more robust.

Testing it

The pattern is only as good as its failure behaviour, so test the failures directly. Kill the relay between enqueueing and marking a row published, and confirm the job is delivered again and handled safely. Roll back a transaction that wrote an outbox row, and confirm nothing is published. Stop the relay for ten minutes, and confirm your alert on the oldest unpublished row fires.

When you don't need it

If losing an occasional job is genuinely harmless, say a non-essential analytics event, the simple two-step code may be fine. And if your queue lives in the same database as your data, enqueuing inside the transaction already gives you the same guarantee.

The outbox earns its keep when the job matters: confirmations, payments, webhooks to customers, anything a person will notice if it never happens.

Why we care

"Did the job actually get scheduled?" is the question behind a surprising number of support tickets. Making scheduling part of the same commit as the change that caused it is one of the foundations we rely on at delay.run.

Should background workers ship with the web app or deploy separately? Let's argue about it

delay.run — Mon, 09 Nov 2026 10:57:36 +0000

Most apps with background jobs start the same way: the web app and the workers live in the same codebase, built into the same image, deployed together. The worker is just the app started with a different command.

At some point, someone suggests splitting them: separate deploys, maybe separate services, maybe separate teams. The argument that follows is one of the more interesting ones in backend architecture, because both sides are right about different things.

Here's my attempt at a fair summary. Then tell me where you land.

The case for deploying together

One version of the truth. When the web app enqueues a job, the worker that picks it up runs the same code. The job's payload format, the database schema and the business logic all match. A whole class of bugs, "the worker didn't understand the new job format", mostly disappears.

Simpler pipelines. One build, one artifact, one deploy. Rollbacks are one action. Fewer moving parts to monitor.

Shared code without ceremony. Models, validation and helpers are just imports. No shared library to version and publish.

Easier local development. One repo, one setup, everything runs together.

The case for deploying separately

Different reliability needs. A web request takes milliseconds; a job might run for twenty minutes. Restarting workers on every web deploy means interrupting long jobs constantly, or making every deploy wait for them to finish.

Different scaling. Web traffic and job volume rarely peak at the same time. Separate deployments make it natural to scale them independently.

Smaller blast radius. A bad change to a report-generation job shouldn't require redeploying the checkout page, and a web deploy shouldn't risk the payment retry worker.

Release cadence. Web code often ships many times a day. Workers that process money, emails or data imports may deserve a more careful pace.

The part both camps underestimate: jobs in flight

Whichever you choose, there's a moment that matters more than the architecture diagram: the deploy, with jobs still in the queue.

Consider a job enqueued by the old code and processed by the new code, or vice versa. This happens in both setups:

Deploying together, rolling deploys mean old and new versions run side by side for a while. A job enqueued by a new web instance can be picked up by an old worker.
Deploying separately, the gap is longer and more explicit. The web app might be days ahead of the workers.

And for scheduled jobs, the gap can be enormous. A job scheduled today to run in 30 days will be executed by whatever code is deployed in 30 days. Its payload was written by today's code.

So the real requirement, in both camps, is the same:

Job payloads must be backwards and forwards compatible, at least for a while. Add fields; don't rename or remove them in the same release that starts using the new ones.
Version your payloads when changes are unavoidable, and make workers handle every version that might still be in the queue.
Long-delayed jobs should carry minimal data: an ID to look up, rather than a snapshot of state that may be stale or in an old format by the time it runs.

{ "type": "send_renewal_reminder", "version": 2, "subscription_id": "sub_123" }

Deploying together reduces the window. It doesn't remove it.

Graceful worker restarts matter more than the topology

The other underestimated piece: what happens to a job that's running when a worker is told to stop? Workers need to stop taking new jobs, finish or safely abandon the current one within a timeout, and leave anything unfinished to be retried. If that isn't in place, deploy frequency, together or separate, translates directly into interrupted jobs.

What about database migrations?

Migrations sharpen the whole debate. If the web app and workers share a database, a schema change has to work with every version of both that might be running at once. The usual answer, expand first and contract later, applies in both camps: add the new column, deploy code that writes both, backfill, switch reads, and only then remove the old column in a later release. Separate deploys make that sequence longer, but also more visible, which some teams find is an advantage rather than a cost.

Where I land

For most small and medium teams: same repo, same image, separately scaled and separately restarted processes. Build once, deploy the same artifact to both, but don't force every web deploy to bounce long-running workers at the same moment. Treat job payloads as a public contract regardless.

Split into truly separate services when there's a real reason: different teams, very different scaling, or reliability requirements that justify their own release process. "It feels cleaner" isn't quite enough.

But I can see the opposite case clearly. If your workers handle money and your web app ships twenty times a day, a slower, separate release process for workers isn't bureaucracy. It's caution in the right place.

Questions to settle it for your team

Do any jobs run longer than your typical deploy takes?
Are job payloads versioned, and how long can a job sit in the queue?
Do web and job load peak together?
Who gets paged when a worker deploy goes wrong, and is it the same people?

Long-delayed jobs, whose payloads outlive several deploys, are the reason we think so much about this at delay.run: the code that schedules a job and the code that runs it are rarely the same version.

6 myths about calling rate-limited APIs from background jobs

delay.run — Sun, 08 Nov 2026 11:59:53 +0000

Background jobs and third-party APIs have a complicated relationship. Your queue can produce work as fast as your workers can pull it. The API on the other side has a limit: so many requests per second, per minute, or per day. And the moment those two numbers disagree, things get interesting.

Here are six beliefs about rate limits that tend to hold right up until the first incident.

Myth 1: "More workers means faster processing"

Up to the API's limit, yes. Beyond it, more workers just means more rejected requests.

If an email API allows a certain number of requests per second for your account, it doesn't matter whether you run 5 workers or 50. Above the limit, the extra requests come back as 429 Too Many Requests, consume retries, and often make total throughput worse, because every rejected request is wasted work that has to be redone.

Instead: size concurrency against the downstream limit, not against your queue depth. If the limit is the bottleneck, the number of workers calling that API should be set from the limit.

Myth 2: "The rate limit is per worker"

Every worker obeys "no more than 10 requests per second", so you're fine. Then you deploy 8 replicas, and the API sees 80 per second from your account.

Rate limits are almost always enforced per API key, account or IP, not per process. A limit enforced locally in each worker doesn't add up to a global limit.

Instead: coordinate the limit across workers. Common approaches:

A shared token bucket in Redis or your database that every worker takes from before calling the API.
A dedicated queue for that API with a fixed number of consumers, each with a local limit, so the total is known.
A single "dispatcher" that hands out permission at the right rate.

# a minimal shared limiter: N requests per window, across all workers (Redis)
import time, redis
r = redis.Redis()

def acquire(key="emailapi", limit=10, window=1.0):
    while True:
        slot = int(time.time() / window)
        k = f"rl:{key}:{slot}"
        n = r.incr(k)
        if n == 1:
            r.expire(k, int(window) + 1)
        if n <= limit:
            return
        time.sleep(window - (time.time() % window))

A fixed window like this is simple and good enough for many cases; it can allow bursts at window boundaries, so use a sliding window or token bucket if the API is strict.

Myth 3: "On a 429, retry immediately"

An immediate retry after a 429 is almost guaranteed to get another 429. Multiply that by every worker and you've built a loop that hammers the API at full speed while accomplishing nothing.

Instead: respect Retry-After if the API sends it (as seconds or an HTTP date). If it doesn't, use exponential backoff with jitter. And crucially, delay the job rather than sleeping in the worker. A worker sleeping for 30 seconds is a worker not processing anything else. Re-enqueue the job with a delay and let the worker move on.

def handle(job):
    resp = call_api(job.payload)
    if resp.status_code == 429:
        wait = parse_retry_after(resp.headers.get("Retry-After")) or backoff(job.attempt)
        queue.enqueue_in(wait, handle, job.payload, attempt=job.attempt + 1)
        return
    resp.raise_for_status()

Myth 4: "A 429 is a failure, so count it as one"

If 429s count towards the job's maximum attempts, a busy afternoon can push perfectly good jobs into the dead-letter queue. Nothing was wrong with them; the API was just busy.

Instead: treat rate-limit responses differently from real errors. A 429 means "not now", not "never". Many teams give rate-limited retries their own budget, measured in time ("give up after 6 hours") rather than attempts, and keep the attempt counter for genuine failures like 500s or timeouts.

Myth 5: "We're well under the limit, so we don't need to think about it"

Average rate and peak rate are different things. You might average 2 requests per second against a limit of 10, and still hit the limit every morning at 09:00 when the scheduled jobs all fire at once, or after an outage when the backlog drains at full speed.

The backlog case is the sneaky one. The API was down for 20 minutes, jobs piled up, it comes back, and your workers immediately try to process 20 minutes of work in 20 seconds.

Instead: rate-limit even when you're "comfortably under", so that bursts and backlog drains are smoothed automatically. Spread scheduled work rather than firing it all on the hour.

Myth 6: "The API's documented limit is the limit"

Documentation describes the policy. Reality includes limits per endpoint, per resource, daily quotas, concurrent connection limits, and limits that change by plan or under load. Some APIs tell you in response headers; the common pattern is something like X-RateLimit-Remaining and X-RateLimit-Reset, although names vary between providers.

Instead: read the headers the API actually sends, log them, and slow down proactively when the remaining budget gets low instead of waiting for a 429. And keep the configured limit somewhere you can change without a deploy, because you will need to.

The pattern underneath

All six myths come from treating a shared, external constraint as if it were local. The fix is always some version of the same thing: make the rate limit a first-class part of your job system, shared across workers, applied before the call, and paired with delayed retries rather than sleeping or spinning.

That last part, "try this again in 40 seconds, without holding a worker hostage", is the bread and butter of delayed job scheduling, and it's exactly the kind of thing we built delay.run to handle.

A quick checklist for your next integration

What is the limit, and is it per key, per account or per endpoint?
Is your limit enforced globally across all workers?
Do 429s re-enqueue with a delay instead of sleeping?
Do rate-limited retries have their own time budget?
What happens when a backlog drains after an outage?
Are you logging the API's rate-limit headers?

If you can answer all six, the next "the API is rate-limiting us" incident will be a slow afternoon rather than a fire.

Fixed rate, fixed delay or cron? Three ways to say 'every 5 minutes', and they behave very differently

delay.run — Sun, 08 Nov 2026 11:57:19 +0000

"Run the sync every five minutes" sounds like a complete specification. It isn't. It leaves out the question that decides how your system behaves on a bad day: five minutes from when?

From the start of the previous run?
From the end of the previous run?
On the clock: 10:00, 10:05, 10:10, no matter what happened before?

Those are three different scheduling models: fixed rate, fixed delay and wall-clock (cron). Most libraries offer at least two of them, often with confusingly similar names. Here's how they differ, with examples, and how to choose.

Fixed rate: every N from the start

The next run is scheduled N after the previous run started. The idea is a steady cadence.

Java's ScheduledExecutorService makes the choice explicit:

ScheduledExecutorService ex = Executors.newSingleThreadScheduledExecutor();
ex.scheduleAtFixedRate(this::sync, 0, 5, TimeUnit.MINUTES);

When runs are fast: starts at 10:00:00, 10:05:00, 10:10:00. Lovely.

When a run takes 7 minutes: this is where it gets interesting. The Java documentation says that if an execution takes longer than its period, subsequent executions may start late, but will not run concurrently. So the next run starts immediately after the slow one finishes, and the scheduler tries to "catch up" towards the original cadence. A sequence of slow runs produces back-to-back executions with no breathing room.

Good for: sampling and metrics, where a steady rhythm matters and runs are reliably short.

Risk: catch-up bursts after a slow period or a pause (a laptop sleeping, a GC pause, a busy host).

Fixed delay: N after the previous run ends

The next run is scheduled N after the previous run finished.

ex.scheduleWithFixedDelay(this::sync, 0, 5, TimeUnit.MINUTES);

Or the hand-rolled version every language has:

import time
while True:
    sync()
    time.sleep(300)

When runs are fast: 10:00:00, 10:05:02, 10:10:04... it drifts by the run duration every time.

When a run takes 7 minutes: next run at 10:12, then the one after at 10:17 or later. No overlap, no catch-up, guaranteed rest between runs.

Good for: polling and sync jobs where what matters is "not too often" and "never overlapping", not hitting exact times.

Risk: drift. "Every five minutes" quietly becomes "every five minutes plus however long it took". Over a day that adds up, and the job will never run at predictable times.

Wall-clock (cron): at fixed times

The schedule is defined against the clock: */5 * * * * means minute 0, 5, 10... of every hour.

When runs are fast: 10:00, 10:05, 10:10. Predictable, aligned, easy to reason about in logs.

When a run takes 7 minutes: classic cron doesn't care. It starts the 10:05 run while the 10:00 one is still going. Now you have two copies running at once, competing for the same rows, the same API quota, the same lock. If runs keep getting slower, copies pile up.

Many cron-like schedulers let you choose the behaviour. Kubernetes CronJob has concurrencyPolicy: Allow | Forbid | Replace. With plain cron, you add a lock yourself:

*/5 * * * * flock -n /tmp/sync.lock /usr/local/bin/sync

flock -n exits immediately if the lock is held, so an overlapping run is skipped instead of stacked.

Good for: jobs tied to human time (reports at 09:00, end-of-day batches), and jobs that need to line up with other systems.

Risk: overlap by default, and the "everyone runs at :00" thundering herd if many jobs use round times.

Side by side

Here's what happens in each model when one run takes 7 minutes in a 5-minute schedule:

	Fixed rate	Fixed delay	Cron (no lock)	Cron + skip-if-running
Runs overlap?	No (most libraries)	No	Yes	No
Next start after slow run	Immediately	5 min after it ends	On schedule, in parallel	Next free slot
Drift over time	Low (catches up)	Accumulates	None	None
Catch-up bursts	Possible	No	No	No
Predictable start times	Mostly	No	Yes	Yes (when not skipped)

Which one should you use?

Use fixed delay for "keep this in sync" jobs: polling an API, draining a table, refreshing a cache. You want a guaranteed gap and no overlap more than you want exact timing.

Use cron with a skip-if-running guard for anything humans or other systems expect at a specific time. Make the overlap policy explicit. Don't rely on "it usually finishes in time".

Use fixed rate only when the work is short and the rhythm itself matters, and make sure catch-up bursts are harmless.

Questions to ask before you pick

What happens if two copies run at once? If the answer is anything other than "nothing", you need fixed delay or a lock.
What happens if a run is skipped? If skipping is fine, "skip if running" is a great default. If every interval matters (billing periods, for example), you need to process the missed window explicitly rather than rely on timing.
Does anyone care what time it runs? If not, fixed delay is the simplest correct choice.
What happens after downtime? Does the scheduler run once, run every missed occurrence, or run nothing? Each library has a different answer, and it's worth reading the docs for yours.

The real lesson

"Every five minutes" is a requirement. Fixed rate, fixed delay and cron are implementations, and they disagree about what to do when reality doesn't fit the schedule. Pick one on purpose, write the choice next to the job, and test it with a run that's slower than the interval.

When we designed scheduling at delay.run, most of the hard questions were exactly these: overlap, drift and what to do about missed runs. They're worth answering before your job answers them for you.

Before a scheduled job emails 50,000 people: a 12-point checklist for bulk sends that can't be unsent

delay.run — Sat, 07 Nov 2026 13:59:11 +0000

Most scheduled jobs fail quietly. A report is late, a cache is stale, somebody notices on Tuesday.

A scheduled bulk email is the opposite. When it goes wrong, it goes wrong loudly, instantly, in thousands of inboxes, and there is no undo. "Hi {{first_name}}" to your whole customer list. The trial-ending email to people who already paid. The same newsletter, three times, because a retry fired after a timeout.

Here's the checklist I'd want before any job that sends email (or SMS, or push) to a large audience at a scheduled time.

Before it ships

1. The audience query is reviewed, not just the template

Template bugs are embarrassing. Audience bugs are expensive. Review the SQL or filter that selects recipients as carefully as any migration. Look for:

missing WHERE unsubscribed = false
joins that can duplicate a user (one row per order, not per customer)
time comparisons in the wrong time zone

Add a SELECT count(*) of the audience to the job's logs before anything is sent.

2. The audience is frozen at a known moment

If the job selects recipients and sends in a long loop, someone who signs up or unsubscribes mid-send gets inconsistent treatment. Better: snapshot the recipient IDs into a send table first, then send from that table.

INSERT INTO send_recipients (send_id, user_id)
SELECT :send_id, id FROM users
WHERE marketing_opt_in AND NOT unsubscribed
ON CONFLICT DO NOTHING;

This also gives you an exact record of who was meant to get it.

3. Every recipient has a "sent" marker, and the job checks it

The single most important line. If the job crashes at recipient 31,402 and restarts, it must not start again from 1.

for r in pending_recipients(send_id):          # WHERE sent_at IS NULL
    msg_id = provider.send(r.email, render(r), idempotency_key=f"{send_id}:{r.user_id}")
    mark_sent(send_id, r.user_id, msg_id)

If your provider supports idempotency keys, use one per recipient per send. If not, the sent_at marker is your only protection.

4. Retries are per recipient, not per job

A job-level retry ("the send failed, run it again") is how people get three copies. Retry the individual failed recipients with backoff; never re-run the whole batch blindly.

5. Missing personalisation fails closed

Decide what happens when first_name is empty. A fallback ("Hi there") is fine. Rendering the raw placeholder is not. Make the template renderer raise on undefined variables, and test it with a user who has every optional field empty.

6. There's a kill switch you can reach in under a minute

A feature flag, a row in a sends table with status = 'paused', an environment variable the loop checks every few hundred recipients. Something that doesn't require a deploy.

if i % 200 == 0 and send_is_paused(send_id):
    log.warning("send paused at %d", i)
    return

If you can't stop a send halfway, you can't limit the blast radius of a mistake.

Before it runs

7. Seed list first

Send to a small internal list (with real-looking test accounts across mail providers) through the same code path. Not the preview button: the actual job, with a filter.

8. Rate limits are respected on purpose

Your provider has sending limits, and receiving mail servers dislike sudden spikes from one sender. Throttle deliberately rather than letting the provider reject you halfway through.

9. The schedule is unambiguous

"Tuesday 9am" in whose time zone? Store the scheduled time as UTC plus the intended zone, and log both. If you're sending in each recipient's local time, test someone in UTC+13 and someone in UTC−10.

10. Nothing else big is running at the same time

A bulk send hits your database (audience query, rendering, marking sent) and your web servers (people clicking through at once). Check it doesn't coincide with backups, reindexing or another campaign.

While and after it runs

11. Progress is visible

Log sent/failed/remaining counts at intervals. Somebody should be able to answer "how far along is it?" without reading code.

12. You can answer "who got what" afterwards

When a customer says "I got this twice" or "I never got it", you need the send table: recipient, timestamp, provider message ID, result. Keep it.

Bonus: a dry-run mode that does everything except send

The cheapest safety net of all is a --dry-run flag that runs the real audience query, renders every message, writes the results to a file and stops short of calling the provider. Run it the day before. Then open the file and look: sort by length to find empty renders, grep for {{, count unique email addresses against the row count. Ten minutes with a dry-run output has caught more bad sends than any amount of template previewing, because it exercises the data you'll actually send to, not the one test account everyone uses.

The three that matter most

If you only do three: per-recipient sent markers (3), a kill switch (6), and a seed send through the real path (7). Those turn a disaster into an incident you can stop at 2%.

Scheduling the moment a job runs is the easy part; tools such as delay.run handle the "run this at 09:00 Tuesday" piece. What happens once it's running is still your code, and that's where this checklist lives.

Your background jobs need SLOs too. 'It eventually ran' isn't good enough

delay.run — Sat, 07 Nov 2026 13:58:54 +0000

Most teams can tell you their API's p95 latency. Few can tell you how late their background jobs run.

Ask "how long after a user signs up does the welcome email go out?" and you'll get guesses. "Pretty quickly." "A few seconds?" "Should be instant." Then one day the queue backs up, welcome emails arrive four hours late, password resets take twenty minutes, and nobody knows when it started, because nobody was measuring.

My opinion: background jobs deserve service level objectives just as much as your API does. Not because SLOs are fashionable, but because users experience job latency directly, and "it eventually ran" hides a lot of pain.

Users feel job latency, they just can't see the job

Look at what typically runs in the background:

password reset and magic-link emails
payment confirmations and receipts
file exports and report generation
notifications ("someone replied to you")
webhooks to your customers' systems

Every one of these has an implicit promise. A password reset that arrives in 30 seconds is fine; one that arrives in 15 minutes is a broken login flow. A webhook delivered an hour late may be worse than useless if the receiving system made decisions in the meantime.

From the user's perspective, there's no difference between "the API was slow" and "the job was slow". There's only "the product was slow".

The metric that matters: lateness

For background work, the most useful single number isn't how long a job takes to run. It's how long after it should have started it actually started:

lateness = started_at - scheduled_for

For immediate jobs, scheduled_for is when it was enqueued. For delayed jobs, it's the time it was due. Either way, lateness captures queue backlog, worker shortages, scheduler drift and crashed workers in one number.

If you store both timestamps (and you should), measuring it is a query. In Postgres:

SELECT
  job_type,
  count(*) AS jobs,
  percentile_cont(0.50) WITHIN GROUP (
    ORDER BY extract(epoch FROM started_at - scheduled_for)) AS p50_s,
  percentile_cont(0.95) WITHIN GROUP (
    ORDER BY extract(epoch FROM started_at - scheduled_for)) AS p95_s,
  max(extract(epoch FROM started_at - scheduled_for)) AS max_s
FROM jobs
WHERE started_at > now() - interval '24 hours'
GROUP BY job_type
ORDER BY p95_s DESC;

The first time you run this, something will surprise you. It always does.

What an SLO for jobs looks like

Keep it simple and tied to the user promise:

Password reset emails: 99% start within 10 seconds of being requested.
Outgoing webhooks: 99% of first attempts start within 30 seconds of the event.
Scheduled reminders: 99% start within 60 seconds of their due time.
Nightly exports: 100% complete by 06:00 local time.

Different job types get different targets because they carry different promises. That's the whole point: lumping everything into one queue with one vague "should be fast" expectation is how your marketing digest ends up delaying your password resets.

"But our jobs are async, latency doesn't matter"

Async means the user doesn't wait on the HTTP request. It doesn't mean they don't wait. They wait in their inbox, on the confirmation screen, in their own system for your webhook. Async moved the waiting; it didn't remove it.

"But it's extra work"

It's two timestamps and a query. If your job system doesn't store scheduled_for and started_at, add them; you'll want them during the next incident anyway. The alerting can start crude:

-- anything due more than 5 minutes ago that hasn't started?
SELECT job_type, count(*)
FROM jobs
WHERE started_at IS NULL
  AND scheduled_for < now() - interval '5 minutes'
GROUP BY job_type;

Run that every minute and alert if it returns rows for a critical job type. That single check would have caught most of the background-job incidents I've seen discussed in postmortems, usually hours earlier than the customer complaints did.

Don't forget the jobs that never start

Lateness only measures jobs that eventually ran. A job that was never enqueued at all (a bug in the code that schedules it, a cron entry deleted during a migration) produces no row and no lateness. For critical recurring work, pair the lateness query with a simple expectation: "at least one nightly export completed in the last 26 hours". Absence is a failure mode too.

What SLOs change in practice

Once lateness has a target, decisions get easier:

Separate queues for urgent and bulk work stop being a nice-to-have; the numbers tell you when one is starving the other.
Capacity planning becomes concrete: "at Monday 09:00 the p95 for reminders hits 4 minutes; we need more workers or spread-out scheduling".
Deploys can be judged: did lateness spike during the rollout because workers were restarted all at once?
Incidents get a clear start time from the graph rather than from the first support ticket.

Where we stand

We care about this a lot. Delivering work at the right time is the whole job of delay.run, and "did it run on time?" is the question we think every scheduled or background task should be able to answer. But you don't need any particular tool for this opinion to apply. Add the timestamps. Write the query. Pick a target for the three job types users would miss most.

"It eventually ran" was never really a reliability standard. It just felt like one because nobody was measuring.

How to build "Undo send" with a delayed job (and why it's harder than a timer)

delay.run — Fri, 06 Nov 2026 11:22:07 +0000

Gmail popularised it: you hit Send, and for a few seconds there's an "Undo" button. Click it and the email never leaves.

There's no magic. The email isn't sent when you click Send. A job is scheduled to send it a few seconds later, and Undo cancels that job. The same pattern works for deleting records, posting comments, publishing changes, sending invoices: anything where "oops" is common and expensive.

It sounds like a five-minute feature. Then you hit the edge cases. Let's build it properly.

Why not just use a timer?

The obvious version is an in-memory timer in your web process: wait five seconds, then send, unless cancelled. It works in a demo and fails in production:

Deploys and restarts kill the process, and pending sends vanish silently.
Multiple instances mean the Undo request may hit a different server from the one holding the timer.
No record exists of what was scheduled, so nobody can debug "my email never sent".

The fix is to store the pending action somewhere durable and let a worker execute it.

The data model

A single table is enough to start:

import sqlite3, time, json

db = sqlite3.connect("jobs.db", isolation_level=None)
db.execute("""CREATE TABLE IF NOT EXISTS jobs (
  id INTEGER PRIMARY KEY,
  kind TEXT NOT NULL,
  payload TEXT NOT NULL,
  run_at REAL NOT NULL,
  status TEXT NOT NULL DEFAULT 'pending'  -- pending | running | cancelled | done
)""")

UNDO_WINDOW = 5  # seconds

Every pending action is a row with a run_at time. The status column is the whole state machine: pending, then either cancelled (user clicked Undo) or running and done (worker picked it up).

Scheduling and undoing

def send_later(to, body):
    cur = db.execute(
        "INSERT INTO jobs (kind, payload, run_at) VALUES (?, ?, ?)",
        ("send_email", json.dumps({"to": to, "body": body}), time.time() + UNDO_WINDOW),
    )
    return cur.lastrowid

def undo(job_id):
    cur = db.execute(
        "UPDATE jobs SET status='cancelled' WHERE id=? AND status='pending'",
        (job_id,),
    )
    return cur.rowcount == 1  # False means it already ran (or was already undone)

The important detail is in undo(): the UPDATE only matches rows that are still pending. If the worker already claimed the job, zero rows change and undo() returns False, so the UI can say "Too late, it's already sent" instead of lying.

The worker

def claim_due():
    """Atomically move one due job from pending to running."""
    row = db.execute(
        "UPDATE jobs SET status='running' WHERE id = ("
        "  SELECT id FROM jobs WHERE status='pending' AND run_at <= ?"
        "  ORDER BY run_at LIMIT 1"
        ") RETURNING id, kind, payload",
        (time.time(),),
    ).fetchone()
    return row

def work_once():
    while (job := claim_due()):
        job_id, kind, payload = job
        data = json.loads(payload)
        print(f"[worker] sending job {job_id} to {data['to']}")
        db.execute("UPDATE jobs SET status='done' WHERE id=?", (job_id,))

The worker claims a job by flipping it from pending to running in a single statement. This is what makes Undo safe: either the user's UPDATE ... WHERE status='pending' wins, or the worker's does. They can't both succeed. (UPDATE ... RETURNING needs SQLite 3.35+; in Postgres you'd typically use SELECT ... FOR UPDATE SKIP LOCKED inside a transaction.)

In a real app, work_once() runs in a loop every second or so in a separate process.

Trying it out

import time
from undo import send_later, undo, work_once, db

a = send_later("ana@example.com", "See you at 3")
b = send_later("bo@example.com", "Oops, wrong person")
print("queued", a, b)

time.sleep(1)
print("undo b:", undo(b))

time.sleep(5)          # undo window passes
work_once()            # worker picks up whatever is still due

print("undo a (too late):", undo(a))
for row in db.execute("SELECT id, status FROM jobs"):
    print(row)

Output from our run:

queued 1 2
undo b: True
[worker] sending job 1 to ana@example.com
undo a (too late): False
(1, 'done')
(2, 'cancelled')

The cancelled email never reaches the worker, and a late Undo is correctly refused.

Edge cases worth handling

The worker's clock is the source of truth. If the browser shows a five-second countdown, the server should give a little extra margin: stop showing the button at 5 seconds, but schedule run_at at 6. That way a click in the final moment still lands before the worker.

Worker polling adds delay. If the worker polls every 2 seconds, a 5-second undo window means the email goes out somewhere between 5 and 7 seconds. That's usually fine. If it's not, poll more often, or use a scheduler that triggers at the exact time.

Crashes while running. If the worker crashes after claiming a job, it stays running forever. Add a claimed_at column and have a sweeper reset jobs that have been running for too long, and make the send itself idempotent so a retry doesn't double-send.

What the user sees meanwhile. During the undo window, the item should appear in a "sending" state in the UI. If the user refreshes the page, load pending jobs from the table so the Undo button survives the reload.

Undo is not the same as delete. Keep cancelled rows (at least for a while). When someone asks "why didn't my email send?", a row with status='cancelled' and a timestamp answers it.

Beyond email

Once you have this table and worker, you'll find more uses for it:

Soft-delete with a grace period: "Project deleted. Undo?" then hard-delete 30 seconds later.
Batching notifications: delay a "new comment" notification by a minute so three comments become one message.
Scheduled publishing: the same run_at column with a longer delay.

The core never changes: store the intent, delay the execution, allow a cancel, and make the worker's claim atomic.

When to stop building it yourself

The table-and-worker approach is great for a single app. It gets heavier as you add retries, exact-time triggers, visibility into what's pending, and webhooks to other services. That's the territory a hosted scheduler like delay.run covers: you schedule an HTTP call for later, cancel it if the user changes their mind, and see what ran and when. Whichever route you take, the design principles above apply.

Postmortem: the nightly sync ran green for 23 days while syncing nothing

delay.run — Fri, 06 Nov 2026 11:21:56 +0000

This is a composite postmortem. The pattern is real and common; the details are illustrative.

Summary

A nightly job synced customer orders from a partner's API into our database. For 23 days, it ran on schedule, exited with code 0, and logged "sync complete". It synced zero orders. The partner's API token had expired, and our code turned "unauthorised" into "no new orders".

We found out when a customer asked why their dashboard hadn't changed in three weeks.

The job

# sync_orders.py — runs nightly at 02:30
def fetch_new_orders(since):
    resp = requests.get(
        f"{PARTNER_API}/orders",
        params={"since": since.isoformat()},
        headers={"Authorization": f"Bearer {TOKEN}"},
        timeout=30,
    )
    data = resp.json()
    return data.get("orders", [])

def main():
    since = last_sync_time()
    orders = fetch_new_orders(since)
    for o in orders:
        upsert_order(o)
    set_last_sync_time(now())
    log.info("sync complete: %d orders", len(orders))

Read it carefully and you'll spot three problems.

What happened

The partner issued API tokens valid for one year. Ours was created by an engineer who'd since moved teams. Nobody had the expiry date written down.
On day 0, the token expired. The partner's API started returning 401 Unauthorized with a JSON body like {"error": "token_expired"}.
Our code never checked the status code. resp.json() parsed the error body fine. data.get("orders", []) returned an empty list.
Zero orders were processed. The log said sync complete: 0 orders.
Then the worst line ran: set_last_sync_time(now()). Every night, the "since" cursor moved forward, as if we'd successfully synced everything up to that moment.
Our job monitoring checked that the job ran and exited 0. It did both, every night.

Detection

Day 23: a customer emailed support. Their order dashboard had been unchanged for three weeks. Support escalated; an engineer ran the job by hand with debug logging and saw the 401 immediately.

Impact

23 days of partner orders missing for all customers using that integration.
Because the cursor had advanced, simply fixing the token wasn't enough. The next run would only fetch orders from that day onward.

Recovery

A new token was issued and stored.
The cursor was manually reset to the last known good sync (found from the last non-zero log line).
The backfill was run in daily chunks to respect the partner's rate limits.
Customers on the integration were told what happened and when data would be complete.

Root causes

HTTP errors weren't treated as errors. The code assumed any JSON response was a success.
"Nothing to do" and "couldn't check" looked identical. An empty list was a valid answer, and an error produced an empty list.
The cursor advanced on failure. The job recorded progress it hadn't made.
Monitoring checked the wrong thing. "Did it run?" isn't "did it work?"
Credentials had no owner or expiry tracking.

Fixes

Fail loudly on HTTP errors

def fetch_new_orders(since):
    resp = requests.get(
        f"{PARTNER_API}/orders",
        params={"since": since.isoformat()},
        headers={"Authorization": f"Bearer {TOKEN}"},
        timeout=30,
    )
    resp.raise_for_status()               # 401, 403, 429, 5xx → exception
    data = resp.json()
    if "orders" not in data:
        raise ValueError(f"unexpected response shape: {list(data)[:5]}")
    return data["orders"]

Only advance the cursor to what you actually processed

def main():
    since = last_sync_time()
    run_started = now()
    orders = fetch_new_orders(since)      # raises on failure; cursor untouched
    for o in orders:
        upsert_order(o)
    # Advance to the time we *queried at*, not "now" after processing
    set_last_sync_time(run_started)
    log.info("sync ok since=%s orders=%d", since.isoformat(), len(orders))

Alert on suspicious success

Zero results isn't always wrong, but zero results for several nights in a row, from a partner that normally sends hundreds, is almost certainly a problem:

recent = recent_run_counts(days=3)        # e.g. [0, 0, 0]
typical = median(recent_run_counts(days=30))
if typical > 20 and all(c == 0 for c in recent):
    alert(f"orders sync: 0 orders for 3 nights (typical {typical}). Check credentials and API.")

Track credentials like they're infrastructure

Every API token used by scheduled jobs now has an entry in a simple inventory: what it's for, who owns it, when it expires. A scheduled reminder fires 30 days and 7 days before each expiry. Whether you schedule those reminders with your own cron, a calendar, or a hosted scheduler like delay.run matters less than having them at all.

Where the API offers it, we also call a cheap authenticated endpoint at the start of each run (a "who am I" or similar), so credential problems fail fast with a clear message.

Add a test for the failure path

The original code had tests, but only for the happy path. We added one for the case that actually happened:

def test_sync_does_not_advance_cursor_on_401(monkeypatch, fake_api):
    fake_api.respond(status=401, json={"error": "token_expired"})
    before = last_sync_time()
    with pytest.raises(requests.HTTPError):
        main()
    assert last_sync_time() == before

It's a five-minute test that would have turned 23 silent days into one failed night.

What went well

The raw data still existed on the partner's side, so the backfill was complete.
Once someone looked, diagnosis took minutes.

Lessons

Exit code 0 is not a success metric. Measure what the job was supposed to produce.
Never let an error masquerade as "no data".
Don't advance a cursor past work you didn't do.
Every credential a scheduled job uses will expire or be revoked one day. Plan for that day.

Our nightly cleanup job deleted the wrong things. Here's how I'd build it today

delay.run — Thu, 05 Nov 2026 08:20:07 +0000

Every app eventually gets one: the cleanup job. Delete expired sessions. Purge soft-deleted rows after 30 days. Remove unconfirmed sign-ups. Trim old logs.

It's usually written in an afternoon and scheduled at 3am:

-- cleanup.sql, run nightly by cron
DELETE FROM sessions WHERE created_at < now() - interval '30 days';
DELETE FROM users WHERE confirmed = false AND created_at < now() - interval '7 days';
DELETE FROM events WHERE created_at < now() - interval '90 days';

It's the most boring code in the system, and it's the one scheduled job where a bug can't be fixed with a retry. A failed email job sends late. A failed report job reruns. A cleanup job that deletes the wrong thing is just... done.

Here's what I'd do differently, built from the classic ways these jobs go wrong.

1. Never write a bare DELETE in a scheduled job

The users line above looks fine until someone adds an "import users from CSV" feature that creates accounts with confirmed = false by default, for users who log in via a different flow. A week later, the cleanup job quietly removes every imported customer.

Today I'd select first, check, then delete by ID:

def cleanup_unconfirmed_users(db, dry_run=False, max_rows=500):
    rows = db.fetch_all("""
        SELECT id, email, created_at FROM users
        WHERE confirmed = false
          AND source = 'self_signup'          -- be explicit about WHICH users
          AND created_at < now() - interval '7 days'
        LIMIT %s
    """, [max_rows + 1])

    if len(rows) > max_rows:
        raise RuntimeError(
            f"Refusing to delete: more than {max_rows} candidates. "
            "Investigate before raising the limit."
        )

    ids = [r.id for r in rows]
    log.info("cleanup_unconfirmed_users candidates=%d dry_run=%s", len(ids), dry_run)
    if dry_run or not ids:
        return len(ids)

    db.execute("DELETE FROM users WHERE id = ANY(%s)", [ids])
    return len(ids)

Two changes that matter:

An explicit positive condition (source = 'self_signup') instead of "everything that isn't confirmed". Delete what you meant, not "whatever matches".
A circuit breaker. If the job suddenly wants to delete 10x more than usual, something changed. Stop and alert instead.

2. Soft delete first, hard delete later

The safest cleanup is two-stage:

-- Stage 1 (nightly): mark
UPDATE users SET pending_purge_at = now()
WHERE id = ANY(%s) AND pending_purge_at IS NULL;

-- Stage 2 (nightly, separate job): purge what has been marked for 7+ days
DELETE FROM users
WHERE pending_purge_at < now() - interval '7 days';

Now a bad rule gives you a week to notice and run UPDATE users SET pending_purge_at = NULL WHERE .... Stage 2 can also refuse to run if stage 1 marked an unusual number of rows.

3. Delete in batches, not one giant statement

DELETE FROM events WHERE created_at < now() - interval '90 days';

On a big table, this can hold locks for a long time, bloat the database, fill up replication lag and make the 3am job collide with the 6am traffic. Batch it:

while True:
    n = db.execute("""
        DELETE FROM events
        WHERE id IN (
            SELECT id FROM events
            WHERE created_at < now() - interval '90 days'
            ORDER BY id
            LIMIT 5000
        )
    """)
    if n == 0:
        break
    time.sleep(0.2)   # let replicas and other queries breathe

For very large, time-based tables, partitioning by month and dropping old partitions is usually much better than deleting rows at all.

4. Pin "now" at the start of the run

If the job runs for 40 minutes and every statement calls now() separately (or every batch computes its own cutoff), different parts of the job use different cutoffs. Compute the cutoff once and pass it everywhere:

cutoff = job_scheduled_time - timedelta(days=90)   # not "now" at each step

Using the scheduled time rather than the start time also makes reruns deterministic: if the job is retried two hours later, it deletes the same set it would have deleted originally.

5. Log what you deleted, not just that you deleted

"cleanup ok" in a log is useless during an incident. I want:

cleanup_unconfirmed_users cutoff=2026-10-27T03:00Z candidates=37 deleted=37 dry_run=false
cleanup_events cutoff=2026-08-05T03:00Z batches=12 deleted=58211

And for anything user-facing, I'd write the deleted IDs (not the data) to an audit table. When support asks "what happened to this account?", you have an answer.

6. Treat retention rules as product decisions, not code details

"Delete unconfirmed users after 7 days" is a business rule. So is "keep invoices for 7 years", which may be a legal requirement. The person writing the cron job shouldn't be the only person who knows these numbers.

Today I'd keep retention rules in one reviewed config file:

retention:
  sessions:          { keep_days: 30 }
  unconfirmed_users: { keep_days: 7, source: self_signup }
  events:            { keep_days: 90 }
  invoices:          { keep_days: 2555, note: "legal requirement, do not change without review" }

7. Make sure the job actually runs

The opposite failure is just as common: the cleanup job silently stopped months ago. The database grows, privacy promises ("we delete X after 30 days") quietly become false, and nobody notices until disk alerts fire.

Whatever runs it (cron, a cloud scheduler, or a hosted scheduler like delay.run), add a check that it ran and that it found a plausible number of rows. Zero deletions for a month is as suspicious as ten times the normal amount.

The checklist version

Explicit positive conditions, never "everything except".
A maximum-rows circuit breaker.
Soft delete, then hard delete after a grace period.
Batches, or partition drops for big tables.
One cutoff per run, based on the scheduled time.
Logs with counts and cutoffs, audit IDs for user data.
Retention rules in reviewed config.
Alerts for "didn't run" and "unusual count".

Retries and backoff FAQ: 11 questions about retrying failed jobs without making things worse

delay.run — Thu, 05 Nov 2026 08:19:44 +0000

Retries look like the easy part of background jobs. Something failed? Try again. But badly designed retries turn a small outage into a big one, hide real bugs, and occasionally charge a customer twice.

Here are the questions that come up most, with short answers and code.

1. Why not just retry immediately?

Because whatever caused the failure is probably still happening. If an API is overloaded and returns errors, every client retrying instantly makes it more overloaded. You get a retry storm: the service can't recover because the retries keep it busy.

Waiting before retrying gives the other side time to recover.

2. What is exponential backoff?

Each retry waits longer than the last, usually doubling:

attempt 1 fails → wait 1s
attempt 2 fails → wait 2s
attempt 3 fails → wait 4s
attempt 4 fails → wait 8s

In code:

def backoff(attempt, base=1.0, cap=300.0):
    return min(cap, base * 2 ** attempt)

The cap matters. Without it, attempt 20 waits about 12 days.

3. What is jitter, and why does everyone say to add it?

If a thousand jobs all fail at the same moment (because a dependency went down), plain exponential backoff makes them all retry at the same moments too: all at +1s, all at +2s, all at +4s. You've synchronised the storm.

Jitter adds randomness so retries spread out. The commonly recommended version is "full jitter": pick a random wait between zero and the backoff value.

import random

def backoff_with_jitter(attempt, base=1.0, cap=300.0):
    return random.uniform(0, min(cap, base * 2 ** attempt))

It looks strange (sometimes you wait almost nothing) but over many clients it smooths the load far better than fixed delays.

4. How many times should I retry?

It depends on what the job is and how long the result stays useful:

User is waiting (an API call during a request): 1–3 quick attempts, then show an error.
Background job, result needed soon (sending a notification): maybe 5–8 attempts over several minutes to an hour.
Background job, result needed eventually (syncing data, delivering a webhook): attempts spread over hours or a day.

A better way to think about it: decide the total time window you're willing to keep trying, then choose attempts and backoff to fit inside it.

5. Which errors should I retry?

Retry errors that might go away on their own. Don't retry errors that won't.

function isRetryable(err, res) {
  if (err) return ["ECONNRESET", "ETIMEDOUT", "ECONNREFUSED", "EAI_AGAIN"].includes(err.code);
  if (res.status === 429) return true;          // rate limited
  if (res.status >= 500) return true;           // server errors
  if (res.status === 408) return true;          // request timeout
  return false;                                  // 400, 401, 403, 404, 422...
}

A 400 "invalid request" will fail the same way every time. Retrying it ten times just delays the moment someone notices the bug.

6. What about `Retry-After`?

If a server responds with 429 or 503 and a Retry-After header, it's telling you when to come back. Use it instead of your own backoff, at least as a minimum:

function waitMs(res, attempt) {
  const ra = res.headers.get("retry-after");
  if (ra) {
    const secs = Number(ra);
    if (!Number.isNaN(secs)) return secs * 1000;
    const date = Date.parse(ra);
    if (!Number.isNaN(date)) return Math.max(0, date - Date.now());
  }
  return Math.random() * Math.min(300_000, 1000 * 2 ** attempt);
}

The header can be a number of seconds or an HTTP date, so handle both.

7. Is it safe to retry anything?

Only if doing it twice has the same effect as doing it once. Reading data: safe. Sending an email or charging a card: not safe unless you've made it so.

The usual approach is an idempotency key: a unique ID for the operation that the receiver uses to detect duplicates. Many payment APIs support this directly. For your own jobs, store "this job ID already completed" and check before doing the work.

The important case to think about is the ambiguous failure: your request timed out, but did the other side process it? Without idempotency, you can't safely tell.

8. Should I retry inside the job or reschedule it?

Two patterns:

In-process retry: loop with sleep inside the job. Simple, but the worker is busy waiting, and if the process restarts, the retry state is lost.
Reschedule: on failure, record the attempt count and schedule the job to run again later. Workers are free in between, and retries survive restarts and deploys.

For anything beyond a few seconds of waiting, reschedule. This is exactly what delayed-job queues and schedulers are for. Many queue libraries support it with a built-in attempts/backoff option, and an HTTP scheduler like delay.run can call your endpoint again at the next retry time.

9. What happens after the last attempt?

Something deliberate. Common options:

Move the job to a "failed" list for a human to look at.
Alert someone if it's important.
Mark the related record (order, user, sync) with a clear failed state.

The worst option is the default in many quick implementations: log an error and forget about it.

10. How do I know if retries are hiding a problem?

Track first-attempt success rate separately from final success rate. If final success is 99.9% but first-attempt success dropped from 99% to 70%, something is wrong and retries are covering for it. You're also doing far more work than you think.

metrics.increment("job.attempt", tags={"job": name, "attempt": attempt, "ok": ok})

One metric, tagged by attempt number, makes this visible.

11. What's a sensible default if I just need something?

For a typical background job calling an external API:

Retry on network errors, 408, 429 and 5xx. Don't retry other 4xx.
Respect Retry-After.
Full-jitter exponential backoff, base 1–5 seconds, cap 5–10 minutes.
6–8 attempts total.
Reschedule rather than sleep.
Idempotency key on anything with side effects.
A clear final failed state, with an alert for important jobs.

Then adjust based on what you see in the metrics.

Quick tip: your cron script 'succeeded' because bash ignored the error. Start every job with set -euo pipefail

delay.run — Wed, 04 Nov 2026 15:39:10 +0000

Here's a nightly backup script that looks perfectly reasonable:

#!/usr/bin/env bash
pg_dump "$DATABASE_URL" | gzip > /backups/db-$(date +%F).sql.gz
aws s3 cp /backups/db-$(date +%F).sql.gz s3://my-backups/
echo "backup done"

Suppose pg_dump fails: wrong password, database unreachable, disk full. What happens?

gzip happily compresses the empty output into a valid, tiny .gz file.
aws s3 cp uploads it successfully.
echo prints "backup done".
The script exits with status 0.

Your monitoring sees success. Your backup bucket fills with 20-byte files. You find out when you need a restore.

The tip: start every scheduled bash script with this line.

set -euo pipefail

What each flag does

`-e`: stop on errors

Without it, bash runs every line regardless of whether the previous one failed. With it, the script exits as soon as a command returns a non-zero status.

set -e
false            # script stops here, exit status 1
echo "never printed"

`-o pipefail`: pipelines fail if any part fails

This is the one that saves the backup script. By default, a pipeline's exit status is that of the last command. pg_dump | gzip "succeeds" because gzip succeeded. With pipefail, the pipeline fails if any command in it fails.

set -o pipefail
false | true     # pipeline now returns 1, and with -e the script stops

`-u`: unset variables are errors

A typo or a missing environment variable silently becomes an empty string without -u. Cron runs with a minimal environment, so missing variables are common.

set -u
rm -rf "/data/$TMPDIR_TYPO/"   # without -u: rm -rf "/data//" (!); with -u: error, script stops

The fixed script

#!/usr/bin/env bash
set -euo pipefail

: "${DATABASE_URL:?DATABASE_URL is not set}"   # explicit, readable error
file="/backups/db-$(date +%F).sql.gz"

pg_dump "$DATABASE_URL" | gzip > "$file"

# sanity check: refuse to upload a suspiciously small backup
size=$(stat -c %s "$file")
if [ "$size" -lt 1024 ]; then
  echo "backup too small ($size bytes), refusing to upload" >&2
  exit 1
fi

aws s3 cp "$file" s3://my-backups/
echo "backup done ($size bytes)"

Now a failing pg_dump stops the script with a non-zero exit, and your scheduler, cron's mail or your monitoring can see it.

Add a trap so failures are visible

set -e exits quietly. Add a trap to say where it failed:

trap 'echo "FAILED at line $LINENO: $BASH_COMMAND (exit $?)" >&2' ERR

In your logs:

FAILED at line 7: pg_dump "$DATABASE_URL" (exit 1)

That line is the difference between a two-minute fix and a half-hour investigation.

The gotchas

set -e has well-known quirks. Know these and you'll avoid most surprises:

1. Commands you expect to fail. grep returns 1 when it finds nothing, which isn't an error for you but stops the script:

count=$(grep -c "ERROR" app.log || true)   # "|| true" makes "no matches" OK

2. Conditions don't trigger -e. Commands inside if, while, && or || are exempt. That's usually what you want, but it means -e isn't a substitute for checking results you care about.

3. Functions called in conditions lose -e inside them. If you call my_func || handle_error, failures inside my_func won't stop it early. Keep important logic out of that pattern, or check explicitly.

4. Command substitution in local. local x=$(failing_cmd) succeeds, because local itself succeeded. Split it up:

local x
x=$(failing_cmd)    # now -e sees the failure

5. Optional variables need defaults with -u.

level="${LOG_LEVEL:-info}"   # default if unset

The same idea outside bash

Other languages have their own versions of "the error was swallowed and the job exited 0":

Python: a broad try/except Exception: log(...) around the whole job logs the error and then exits successfully. Re-raise, or call sys.exit(1) after logging.
Node.js: an unhandled promise rejection in older setups, or a .catch(console.error) at the top level, can leave the exit code at 0. End with process.exitCode = 1 in your error handler.
Subprocesses: subprocess.run(cmd) in Python doesn't raise on failure unless you pass check=True.

import subprocess, sys
try:
    subprocess.run(["pg_dump", "mydb"], check=True, stdout=open("/backups/db.sql", "wb"))
except subprocess.CalledProcessError as e:
    print(f"pg_dump failed with {e.returncode}", file=sys.stderr)
    sys.exit(1)

The rule is the same everywhere: if the work failed, the process must exit non-zero.

Lint it before cron runs it

ShellCheck catches a lot of what strict mode would trip over at 3 a.m.: unquoted variables, cd without error handling, and $? checks that never fire. Run it locally and in CI:

shellcheck scripts/*.sh

Treat its warnings as failing tests for anything that runs unattended. A clean ShellCheck run plus strict mode covers most of the silent-failure bugs in cron scripts.

Why it matters more for scheduled jobs

Interactive scripts have a human watching the output. Scheduled scripts don't. The exit status is the only signal your scheduler, heartbeat monitor or alerting gets. If the script lies about success, everything downstream believes it.

The same applies to HTTP-triggered jobs: whether a cron entry, a CI workflow or a scheduling service like delay.run calls your endpoint, the handler should return an error status when the work actually failed, not a cheerful 200.

The 30-second audit

# find scheduled scripts without strict mode
grep -L 'set -euo pipefail' /opt/jobs/*.sh

Every file it lists is a script that can fail silently. Add the line, add the trap, test each one by forcing a failure (wrong password, missing variable), and confirm it exits non-zero.

Case study: redesigning a failed-payment retry schedule that retried every card at 00:00 UTC

delay.run — Wed, 04 Nov 2026 15:38:39 +0000

This is a hypothetical case study. The company and details are invented to illustrate a common design problem; the patterns and code are real.

The starting point

"Tallyhub" is a small subscription SaaS. When a renewal charge fails (expired card, insufficient funds, a bank's fraud check), it retries. The original implementation was a single nightly job:

# runs at 00:00 UTC via cron
for invoice in Invoice.objects.filter(status="past_due"):
    if invoice.attempts < 4:
        charge(invoice)

It worked well enough until the team looked closely at three complaints.

1. Every retry happened at midnight UTC. For customers in Asia-Pacific that's mid-morning, which is fine. For customers on the US West Coast it's late afternoon. For some European customers it falls just after midnight, before many paydays have cleared. The retry timing was effectively random relative to when customers had money in their accounts.

2. Retries ignored when the failure happened. An invoice that failed at 23:58 was retried two minutes later. One that failed at 00:05 waited almost 24 hours. "Retry 1" meant anything from minutes to a day.

3. Overlapping runs double-charged twice in one quarter. A slow run (the payment API was degraded) was still going when an operator ran the job manually to "catch up". Both processes picked up the same invoices.

The redesign: one scheduled job per retry

The team replaced "scan everything nightly" with one scheduled job per invoice attempt. When a charge fails, the system decides when to retry that invoice and schedules exactly one job for that time.

Step 1: A retry policy relative to the failure

from datetime import timedelta

RETRY_OFFSETS = [timedelta(days=1), timedelta(days=3), timedelta(days=5), timedelta(days=7)]

Retry 1 is now always about a day after the failure, regardless of when it happened.

Step 2: Land retries at a sensible local time

Rather than an arbitrary UTC moment, retries are moved to mid-morning in the customer's billing timezone, with jitter so thousands of invoices don't all fire at 10:00:00.

import random
from datetime import datetime, time
from zoneinfo import ZoneInfo

def next_retry_at(failed_at_utc: datetime, attempt: int, tz_name: str) -> datetime:
    tz = ZoneInfo(tz_name or "UTC")
    target = (failed_at_utc + RETRY_OFFSETS[attempt]).astimezone(tz)
    local = datetime.combine(target.date(), time(10, 0), tzinfo=tz)
    local += timedelta(minutes=random.randint(0, 90))     # spread the load
    return local.astimezone(ZoneInfo("UTC"))

Whether mid-morning local time actually improves recovery is something to measure, not assume. The team logged attempt time and outcome so they could compare later (more on that below).

Step 3: Schedule exactly one job, with an idempotency key

def on_charge_failed(invoice, attempt):
    if attempt >= len(RETRY_OFFSETS):
        mark_uncollectible(invoice)
        return
    run_at = next_retry_at(now_utc(), attempt, invoice.customer.timezone)
    scheduler.schedule(
        run_at=run_at,
        url="https://api.tallyhub.example/internal/retry-charge",
        body={"invoice_id": invoice.id, "attempt": attempt + 1},
        idempotency_key=f"retry:{invoice.id}:{attempt + 1}",
    )

The scheduler can be a database table plus a worker, a job queue with delayed jobs, or an HTTP scheduling service such as delay.run. The important property is the same: one job, one time, one key.

Step 4: Make the handler safe to run twice

Even with a unique key, the handler assumes it might be called twice (retries, redeploys, operator mistakes). It takes a row lock and re-checks state before charging:

@transaction.atomic
def retry_charge(invoice_id: int, attempt: int):
    inv = Invoice.objects.select_for_update().get(id=invoice_id)
    if inv.status != "past_due" or inv.attempts >= attempt:
        return "skipped"                     # paid already, or this attempt already ran
    inv.attempts = attempt
    inv.save(update_fields=["attempts"])
    result = charge(inv, idempotency_key=f"charge:{inv.id}:{attempt}")
    ...

Two layers of protection: the database check stops duplicate work, and the payment provider's idempotency key stops duplicate charges if the same request is sent twice.

Step 5: Cancel retries when they're no longer needed

If the customer updates their card and pays manually, pending retries should disappear. Store the scheduled job's ID on the invoice and cancel it when the invoice is paid. The handler's state check is the backstop if cancellation is missed.

What changed (and what they measured)

Without inventing numbers, here's what the team set out to measure, and what you should measure too if you try this:

Recovery rate per attempt number, before and after.
Recovery rate by local hour of attempt, to test whether the 10:00 choice helps or whether another time is better for their customers.
Duplicate-charge incidents (target: zero).
Load shape on the payment API: one midnight spike versus a spread-out curve.

The design win doesn't depend on the numbers, though. The midnight batch made it impossible to even ask whether timing mattered, because every retry happened at the same moment. Per-invoice scheduling turned retry timing into something you can tune.

Testing the schedule before it touches real cards

A retry schedule is easy to get subtly wrong, so the team tested it without a payment provider in the loop. Three checks did most of the work:

Freeze the clock. The function that computes retry times takes now and the customer's timezone as arguments instead of reading the system clock. Unit tests then cover the awkward cases: a failure at 23:59 local time, a customer in UTC+13, and a retry that lands on a daylight-saving change.
Replay a real month. They exported last month's failed invoices (IDs, timestamps and timezones only, no card data) and ran them through the new scheduler in a dry-run mode that logs planned retry times instead of creating jobs. A histogram of the planned times showed the midnight spike was gone.
Fire every job twice in staging. The handler was called twice for each retry with the same idempotency key. The second call had to be a no-op, with one charge attempt and one log line saying "already handled".

None of this needs special tooling, and it caught two bugs before launch: one off-by-one-day error around month ends, and a missing timezone that defaulted to UTC.

Lessons that generalize

"Scan everything on a schedule" hides per-item timing. If each item has its own clock, give it its own job.
Relative delays beat fixed times for anything triggered by an event.
Every money-moving job needs two idempotency layers: your own state check and the provider's key.
Cancellation is a feature. Design it in, but don't rely on it alone.
Log the inputs you'll want to tune later, such as attempt time, local hour and outcome, from day one.

DEV Community: delay.run

The transactional outbox, explained: how to save to the database and enqueue a job without losing either

The idea in one sentence

Step by step

The catch: at-least-once, not exactly-once

Variations

Housekeeping

Ordering, and why you might not get it

Testing it

When you don't need it

Why we care

Related reading

Should background workers ship with the web app or deploy separately? Let's argue about it

The case for deploying together

The case for deploying separately

The part both camps underestimate: jobs in flight

Graceful worker restarts matter more than the topology

What about database migrations?

Where I land

Questions to settle it for your team

Related reading

6 myths about calling rate-limited APIs from background jobs

Myth 1: "More workers means faster processing"

Myth 2: "The rate limit is per worker"

Myth 3: "On a 429, retry immediately"

Myth 4: "A 429 is a failure, so count it as one"

Myth 5: "We're well under the limit, so we don't need to think about it"

Myth 6: "The API's documented limit is the limit"

The pattern underneath

A quick checklist for your next integration

Related reading

Fixed rate, fixed delay or cron? Three ways to say 'every 5 minutes', and they behave very differently

Fixed rate: every N from the start

Fixed delay: N after the previous run ends

Wall-clock (cron): at fixed times

Side by side

Which one should you use?

Questions to ask before you pick

The real lesson

Related reading

Before a scheduled job emails 50,000 people: a 12-point checklist for bulk sends that can't be unsent

Before it ships

1. The audience query is reviewed, not just the template

2. The audience is frozen at a known moment

3. Every recipient has a "sent" marker, and the job checks it

4. Retries are per recipient, not per job

5. Missing personalisation fails closed

6. There's a kill switch you can reach in under a minute

Before it runs

7. Seed list first

8. Rate limits are respected on purpose

9. The schedule is unambiguous

10. Nothing else big is running at the same time

While and after it runs

11. Progress is visible

12. You can answer "who got what" afterwards

Bonus: a dry-run mode that does everything except send

The three that matter most

Related reading

Your background jobs need SLOs too. 'It eventually ran' isn't good enough

Users feel job latency, they just can't see the job

The metric that matters: lateness

What an SLO for jobs looks like

"But our jobs are async, latency doesn't matter"

"But it's extra work"

Don't forget the jobs that never start

What SLOs change in practice

Where we stand

Related reading

How to build "Undo send" with a delayed job (and why it's harder than a timer)

Why not just use a timer?

The data model

Scheduling and undoing

The worker

Trying it out

Edge cases worth handling

Beyond email

When to stop building it yourself

Related reading

Postmortem: the nightly sync ran green for 23 days while syncing nothing

6. What about `Retry-After`?

`-e`: stop on errors

`-o pipefail`: pipelines fail if any part fails

`-u`: unset variables are errors