Designing Reliable Job Queues

Background jobs start as conveniences. Someone needs to send a welcome email without blocking the signup response, so they push a message to a queue. It works. Then notifications land in the queue, then payment webhooks, then search index updates, then AI inference calls. Six months later, the queue is load-bearing infrastructure and nobody designed it that way.

The failure modes of an under-designed queue are subtle and expensive. Jobs fail silently. They run twice and charge a customer twice. They pile up during a downstream outage and then all replay at once, spiking a database that just came back online. Workers crash mid-job and the work is lost with no record of what happened.

A reliable job queue is not just a broker. It is the set of rules around failure, duplication, and recovery, and most of those rules live in application code, not in the queue service itself.

The four properties above are the spine of this note. Choose the broker that fits your consistency needs, then design the worker code as if every job will run more than once.

Context#

This note applies to any system where application code pushes work into a queue to be processed asynchronously by one or more workers. The broker might be Redis (via Streams, Bull, or BullMQ), AWS SQS, RabbitMQ, Sidekiq, Celery, or a custom solution. The core design decisions are similar across all of them.

The two most common use cases where queues become critical infrastructure are:

1. Integrations with unreliable upstreams: sending email, calling payment processors, triggering webhooks, syncing with third-party APIs.
2. Decoupled compute: generating reports, running ML inference, indexing content, resizing images, sending batch notifications.

Both categories share the property that the work cannot be lost and must eventually complete even if the broker, the worker, or the upstream is temporarily unavailable.

What usually happens with a naive setup#

Teams add a queue with the simplest possible configuration. The broker gets a message. A worker picks it up, processes it, and the message disappears. This works until one of three things happens:

• The worker crashes mid-job. The message is gone. The work never completed. Nobody knows.
• The upstream is temporarily unavailable. The job fails. The worker drops it or retries immediately in a tight loop, hammering the already-struggling upstream.
• A job has a persistent bug. It fails every time. It goes back into the queue. Workers keep picking it up in a loop, consuming worker capacity while the real problem is hidden.

Each of these has a well-understood solution, but the solutions interact in non-obvious ways.

The core tradeoff is delivery guarantees versus processing complexity#

The root tension in queue design is delivery guarantees versus processing complexity. Before choosing, it helps to picture the job lifecycle a reliable queue has to support: a job moves through processing, and on failure it either retries with backoff or lands in a dead-letter queue.

The delivery guarantee you pick decides how often the Processing state can repeat for the same job.

• At-most-once delivery (fire-and-forget) is simple and fast, but drops messages on failure. Acceptable for non-critical analytics events.
• At-least-once delivery is the standard for critical background work. The broker redelivers unacknowledged messages. The tradeoff is that your worker may process the same message twice, so handlers must be idempotent.
• Exactly-once delivery is theoretically desirable but practically either impossible or extremely expensive to guarantee end-to-end. What most systems call "exactly-once" is actually at-least-once delivery with idempotent deduplication.

Design for at-least-once delivery and idempotent handlers. This is the correct combination for any job that has real side effects.

The recommended approach#

1. Use the visibility timeout correctly#

Most durable brokers implement some form of message lease or visibility timeout. In AWS SQS, the visibility timeout (default 30 seconds, max 12 hours) is the window during which a message is invisible to other consumers after a worker picks it up. If the worker does not delete the message before the timeout expires, SQS makes it visible again for another consumer to attempt.

Set the visibility timeout to slightly longer than your worst-case job processing time. A job that takes up to 2 minutes should have a visibility timeout of at least 3 minutes. If the job might take longer, extend the timeout programmatically during processing:

// AWS SDK v3 — extend visibility mid-job
import { SQSClient, ChangeMessageVisibilityCommand } from '@aws-sdk/client-sqs';

const sqs = new SQSClient({ region: 'us-east-1' });

async function extendVisibility(receiptHandle: string, additionalSeconds: number) {
  await sqs.send(new ChangeMessageVisibilityCommand({
    QueueUrl: process.env.SQS_QUEUE_URL,
    ReceiptHandle: receiptHandle,
    VisibilityTimeout: additionalSeconds,
  }));
}

Never delete a message before the job completes. Delete it only on successful completion.

2. Write idempotent handlers#

At-least-once delivery means your handler will run more than once for the same job under certain conditions: worker restart during processing, visibility timeout expiry, or broker-level redelivery. The handler must produce the same result whether it runs once or ten times.

The most reliable approach is to use the job's message ID as a deduplication key in a fast store:

async function processPaymentWebhook(message: SQSMessage) {
  const messageId = message.MessageId;
  const dedupeKey = `processed:payment:${messageId}`;

  // Atomic check-and-set: only one worker can set this key
  const isNew = await redis.set(dedupeKey, '1', 'NX', 'EX', 86400); // 24h TTL

  if (!isNew) {
    // Already processed — safe to acknowledge and skip
    console.log(`Duplicate message ${messageId}, skipping`);
    return;
  }

  // Process the job
  await applyPaymentToAccount(JSON.parse(message.Body));
}

The NX flag ensures only one concurrent worker wins the race. The TTL prevents the key from accumulating indefinitely. For operations with database-level deduplication requirements, use a unique constraint on the idempotency key column in a transaction instead:

-- Idempotency table approach
INSERT INTO processed_jobs (job_id, processed_at)
VALUES ($1, NOW())
ON CONFLICT (job_id) DO NOTHING
RETURNING job_id;
-- If no row returned, job was already processed — skip

Gunnar Morling's analysis of idempotency keys makes an important point: an idempotency key is only useful if the durable record of "this key was processed" is stored atomically with the side effect, or before it. Storing the key after the side effect still allows double-processing if the process crashes between the side effect and the key write.

3. Configure bounded retries with backoff#

Retry immediately on transient errors (network timeout, 503 from downstream). Wait longer on each subsequent failure. Stop and route to a dead-letter queue after a bounded number of attempts.

AWS SQS handles retry counting through maxReceiveCount in the redrive policy. When a message's receive count exceeds maxReceiveCount, SQS moves it to the configured dead-letter queue automatically.

For Redis-based queues like BullMQ, configure retry options per job:

import { Queue } from 'bullmq';

const emailQueue = new Queue('email', { connection: redisConnection });

await emailQueue.add('send-welcome', { userId: 42, email: 'user@example.com' }, {
  attempts: 5,
  backoff: {
    type: 'exponential',
    delay: 2000, // 2s, 4s, 8s, 16s, 32s
  },
  removeOnComplete: 1000,  // keep last 1000 completed jobs
  removeOnFail: 500,       // keep last 500 failed jobs
});

Add jitter to backoff to prevent synchronized retry storms:

function backoffWithJitter(attempt: number, baseDelayMs: number): number {
  const exponential = baseDelayMs * Math.pow(2, attempt);
  const jitter = Math.random() * baseDelayMs;
  return Math.min(exponential + jitter, 30_000); // cap at 30 seconds
}

4. Route failed jobs to a dead-letter queue#

A dead-letter queue (DLQ) is where unprocessable messages go after exhausting retries. It serves two purposes: it stops the retry loop from consuming worker capacity on broken jobs, and it gives operators a place to inspect, debug, and replay messages when the underlying problem is fixed. The DLQ sits at the edge of the normal path: producers enqueue, workers consume, failures loop back through retry, and only exhausted jobs spill into the DLQ.

Configure the DLQ with a longer message retention window than your main queue. AWS SQS supports up to 14 days of message retention and recommends setting the DLQ retention to the maximum to give teams enough time to investigate.

For Redis Streams, implement the DLQ pattern manually using the pending entries list. The Redis documentation on Streams describes the XPENDING command for listing messages that have been delivered but not acknowledged, which is the basis for detecting and rerouting stale messages.

# Check pending messages in a Redis Streams consumer group
XPENDING email-stream email-workers - + 100

# Reclaim a message that has been pending for more than 5 minutes
XAUTOCLAIM email-stream email-workers worker-1 300000 0-0 COUNT 10

Always set up a CloudWatch alarm (or equivalent) on DLQ message count. A growing DLQ is a silent signal that something is broken.

5. Implement backpressure#

Backpressure is the mechanism by which workers signal to producers that they cannot keep up. Without backpressure, a queue accumulates indefinitely during upstream outages or traffic spikes. Workers eventually fall behind so far that the queue takes hours to drain after the upstream recovers, with cascading effects on downstream services.

Simple backpressure mechanisms:

• Queue depth threshold: If the queue depth exceeds a threshold (e.g., 10,000 messages), reject new job submissions with a 503 or rate-limit the enqueue rate.
• Worker-side blocking: Workers pull jobs only when they have capacity rather than polling continuously. SQS long polling (up to 20 seconds) is a built-in form of this.
• Concurrency limits: Set an explicit maximum concurrency per worker process. BullMQ's concurrency setting, SQS's batch size, and Sidekiq's thread count all serve this purpose.

// BullMQ worker with concurrency limit and rate limiting
import { Worker } from 'bullmq';

const worker = new Worker('email', processEmailJob, {
  connection: redisConnection,
  concurrency: 5,       // max 5 jobs running in this worker process
  limiter: {
    max: 100,           // max 100 jobs per period
    duration: 60_000,   // per 60 seconds
  },
});

Edge cases that break the happy path#

Job ordering requirements. Most queues offer no ordering guarantees beyond FIFO within a single partition. If you need strict ordering (e.g., process events for user A in the order they arrived), use a FIFO queue (SQS FIFO, Kafka partitions, or Redis Streams with a single consumer) and route messages for the same entity to the same partition by key.

Long-running jobs and worker restart. If you deploy workers during active job processing, in-flight jobs may be interrupted. Use graceful shutdown: catch SIGTERM, stop accepting new jobs, wait for current jobs to complete (with a timeout), then exit. Cloud orchestrators (ECS, Kubernetes) send SIGTERM before SIGKILL with a configurable grace period.

Fan-out patterns. A single queue message that triggers work on behalf of thousands of users can create a thundering herd when it is processed. Add a small random delay per fan-out job, or spread the work across time windows rather than processing all items immediately.

Poison messages. A job that crashes the worker process itself (OOM, unhandled exception in native code) will keep being redelivered even if maxReceiveCount is set, because the worker never had a chance to increment the receive count. Implement a separate dead-letter mechanism based on wall-clock time in the DLQ rather than only on receive count.

Checklist#

Key Takeaways#

• Design every job handler assuming it will run more than once. At-least-once delivery is the norm for durable queues, not the exception.
• Idempotency is not just a nice-to-have. It is what separates a queue that is safe to retry from one that silently double-charges customers or sends duplicate emails.
• The visibility timeout is the mechanism that handles worker crashes. Set it correctly for your job's processing time, or messages become invisible forever.
• Dead-letter queues are the circuit breaker for broken jobs. Without them, a single buggy job can consume worker capacity indefinitely.
• Backoff with jitter prevents retry storms. Without jitter, every failed batch of jobs retries simultaneously, creating a thundering herd that makes the upstream problem worse.
• Observability is what turns a queue from a black box into a managed system. Job duration, queue depth, DLQ message count, and retry rates should all be instrumented and alerted on.

FAQ#

Should I use Redis or SQS for my job queue?

Redis (via BullMQ or Streams) is a good fit when you need sub-second scheduling precision, job deduplication, rate limiting, and rich job state visibility. SQS is a good fit when you want a fully managed, serverless-friendly broker with strong durability guarantees and native AWS integrations. Redis requires you to manage the Redis deployment (or pay for a managed Redis service). SQS has no operational overhead but has a maximum message size of 256KB and fewer job management features out of the box. For AI workloads that need priority queues and long job processing times, see system design for AI products for patterns that specifically address GPU job scheduling.

How many retries should I configure?

Three to five retries with exponential backoff covers the vast majority of transient failures. Most transient issues (network blip, temporary service unavailability) resolve within 30–60 seconds. Five retries with a starting delay of 2 seconds and exponential growth will span roughly 60 seconds of total retry time before routing to the DLQ. Beyond five retries, you are likely dealing with a persistent error that needs a human to investigate, not more retries.

Can a job queue replace a message broker like Kafka?

Job queues and event streaming systems solve different problems. A job queue is designed for unit-of-work dispatch: one message triggers one piece of work, is acknowledged when done, and is then gone. Kafka and similar systems are designed for ordered, replayable, multi-consumer event streams where many consumers can independently read the same messages at different offsets. If you need event sourcing, audit logs, or fan-out to multiple independent consumers, use Kafka or a similar streaming system. For background job dispatch, a job queue is simpler and sufficient. See Redis vs Kafka for event systems for a deeper comparison.

How do I handle a DLQ that has accumulated thousands of messages?

First, fix the underlying bug that caused the failures. Then replay messages from the DLQ back to the main queue in a controlled way, in batches and with rate limiting, to avoid a replay storm that overwhelms the downstream service that just recovered. Most queue systems have a redrive mechanism for this. SQS has a built-in "Start DLQ redrive" button in the console. For Redis, build a small redrive script that moves messages from the DLQ stream back to the main stream in batches with a delay between each batch.

How do I monitor queue health in production?

The minimum set of metrics to track: queue depth (current message count), DLQ message count (should alert at > 0 or > threshold), job processing duration (p50, p95, p99), retry rate (retried jobs / total jobs), and worker error rate. Emit these from your worker code as structured logs or metrics to your observability platform. Set up alerts on DLQ growth and on queue depth that stays above a threshold for more than 5 minutes, which signals that workers cannot keep up with the enqueue rate.