API Rate Limiting: System Design Notes

Every public API that ever went viral learned the same hard lesson: a well-intentioned client in a retry loop looks identical to an attacker. Without rate limiting, one runaway integration can exhaust your database, spike your costs, and deny service to everyone else. With a poorly designed rate limiter, you punish legitimate clients, confuse developers, and create retry storms that make the problem worse.

Rate limiting feels simple at first (just count requests) until you think about distributed servers, burst traffic, fair allocation across tenants, and what happens when your Redis node goes down. Each of those adds a decision point, and the wrong choice compounds.

Context#

Rate limiting applies at several layers of an API stack: the gateway (Nginx, Kong, AWS API Gateway), a middleware layer in the application, or a dedicated service. This note focuses on the application and architecture level: the decisions a backend team makes about which algorithm to use, how to store state, what to return to clients, and how to handle edge cases.

The HTTP status code for a rejected request is 429 Too Many Requests, defined in RFC 6585 (Section 4). The same RFC recommends including a Retry-After header indicating how long the client should wait before retrying.

What usually happens#

Most teams implement rate limiting in one of three ways, roughly in order of sophistication:

1. Nothing at all, until an incident.
2. A fixed-window counter per IP in Redis or a database, added quickly after the incident.
3. A token bucket or sliding window, added when the fixed-window approach causes complaints from legitimate clients.

The jump from step 2 to step 3 is where most design decisions live. Fixed windows are easy to reason about and cheap to store, but they create a predictable exploitation window at the edge of each reset period. A client that knows your window resets at the top of the minute can send 1,000 requests in the last second of one window and 1,000 more in the first second of the next: 2,000 requests in two seconds, with no 429 in sight.

Root cause and core tradeoff#

The core tradeoff in rate limiting is precision versus cost:

• More precise algorithms (sliding window log, token bucket) give fair, predictable limits and handle bursts correctly. They require more memory and more per-request computation.
• Less precise algorithms (fixed window, approximate sliding window) are cheaper and simpler. They allow edge-case exploitation and may be unfair to clients whose requests arrive at period boundaries.

A second tradeoff is centralized state versus distributed approximation:

• A single Redis node or cluster gives you accurate counts globally, but introduces a network round-trip on every request and a single point of failure.
• A distributed approximation (each server tracks its own counter, then gossips) is faster and more resilient but allows over-counting under bursty conditions.

For most APIs, accurate centralized state in Redis is the right default. The round-trip adds 1 to 3ms, which is acceptable for API use, and Redis availability is high in modern deployments.

Recommended approach#

Use a token bucket per client identifier (API key or authenticated user ID), stored in Redis, implemented with a Lua script for atomic execution.

The token bucket algorithm works as follows: a bucket holds up to capacity tokens. Tokens are added at a fixed refill_rate per second. Each request consumes one token. If the bucket is empty, the request is rejected with 429. The bucket never exceeds capacity, which is how burst size is bounded.

The Redis rate limiter documentation describes token bucket implementations using Lua scripts that execute atomically in Redis, preventing race conditions that would occur with separate GET and SET operations.

Compared to a leaky bucket (which enforces a constant output rate), a token bucket allows short bursts when the client has been idle. That is the correct behavior for API clients that make occasional spikes of work followed by silence.

Implementation details#

Redis Lua script for the token bucket#

Lua scripts execute atomically in Redis. This prevents the race condition where two parallel requests both read the same token count, both see tokens available, and both consume a token, resulting in over-serving.

-- token_bucket.lua
-- KEYS[1] = bucket key, e.g. "ratelimit:user:42"
-- ARGV[1] = capacity (max tokens)
-- ARGV[2] = refill_rate (tokens per second)
-- ARGV[3] = now (Unix timestamp in milliseconds)
-- ARGV[4] = requested_tokens (usually 1)
-- Returns: {allowed (0/1), tokens_remaining, retry_after_ms}

local key = KEYS[1]
local capacity = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local requested = tonumber(ARGV[4])

local bucket = redis.call('HMGET', key, 'tokens', 'last_refill')
local tokens = tonumber(bucket[1]) or capacity
local last_refill = tonumber(bucket[2]) or now

-- Refill tokens based on elapsed time
local elapsed_ms = math.max(0, now - last_refill)
local new_tokens = math.min(capacity, tokens + (elapsed_ms / 1000) * refill_rate)

if new_tokens >= requested then
  -- Allow the request
  redis.call('HMSET', key, 'tokens', new_tokens - requested, 'last_refill', now)
  redis.call('PEXPIRE', key, math.ceil((capacity / refill_rate) * 1000 * 2))
  return {1, math.floor(new_tokens - requested), 0}
else
  -- Reject the request, calculate wait time
  local tokens_needed = requested - new_tokens
  local retry_after_ms = math.ceil((tokens_needed / refill_rate) * 1000)
  redis.call('HMSET', key, 'tokens', new_tokens, 'last_refill', now)
  redis.call('PEXPIRE', key, math.ceil((capacity / refill_rate) * 1000 * 2))
  return {0, math.floor(new_tokens), retry_after_ms}
end

Express middleware example#

The middleware runs on every request: it builds a client key, evaluates the Lua script against Redis, sets the quota headers, and either passes the request through or returns a 429. The sequence below shows that round-trip.

import { createClient } from 'redis';
import { readFileSync } from 'fs';

const redisClient = createClient({ url: process.env.REDIS_URL });
const script = readFileSync('./token_bucket.lua', 'utf8');

async function rateLimitMiddleware(req, res, next) {
  const clientId = req.headers['x-api-key'] ?? req.ip;
  const key = `ratelimit:${clientId}`;

  const capacity = 100;    // max burst
  const refillRate = 10;   // tokens per second (600 rpm sustained)
  const now = Date.now();

  const [allowed, remaining, retryAfterMs] = await redisClient.eval(
    script, { keys: [key], arguments: [capacity, refillRate, now, 1].map(String) }
  );

  // Always set headers so clients can track their quota
  res.set('X-RateLimit-Limit', capacity);
  res.set('X-RateLimit-Remaining', remaining);
  res.set('X-RateLimit-Reset', Math.ceil((now + retryAfterMs) / 1000));

  if (!allowed) {
    res.set('Retry-After', Math.ceil(retryAfterMs / 1000));
    return res.status(429).json({
      error: 'rate_limit_exceeded',
      message: 'Too many requests. See Retry-After header.',
      retry_after_seconds: Math.ceil(retryAfterMs / 1000),
    });
  }

  next();
}

Response headers#

Return rate limit state on every successful response, not just on 429s. Stripe's rate limit documentation and GitHub's REST API rate limit docs both follow this convention with X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset. It lets well-behaved clients back off proactively before hitting the limit.

HTTP/1.1 200 OK
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 73
X-RateLimit-Reset: 1749394800
Content-Type: application/json

HTTP/1.1 429 Too Many Requests
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1749394812
Retry-After: 12
Content-Type: application/json

Edge cases#

What if Redis is unavailable? You have two options: fail open (allow all requests) or fail closed (reject all requests). For most APIs, fail open is correct. A brief window without rate limiting is preferable to taking the entire API down because Redis is restarting. Implement a circuit breaker and log the open period prominently.

Multi-tenant APIs with different limits per plan. Store the plan configuration in a fast lookup (Redis hash or an in-memory config map loaded at startup), look up the client's plan on each request, and parameterize the bucket capacity and refill rate accordingly. Avoid a database lookup per request on the hot path.

Distributed deployments without centralized Redis. If you truly cannot use a centralized store, use an approximate sliding window counter where each node tracks its own counts and periodically syncs. Accept that limits will be approximate: a client on a 10-node fleet might effectively get 10x the limit if requests spread evenly. This is usually acceptable for abuse protection but not for billing-accurate enforcement.

Bot traffic and IP spoofing. IP-based rate limiting is easy to bypass with rotating proxies. Rate limit on authenticated client IDs (API keys) as the primary identifier. For unauthenticated endpoints, combine IP-based limits with behavioral signals (request pattern analysis, device fingerprinting, or CAPTCHA challenges) rather than relying on IP alone.

Retry storms after a 429. If all clients retry immediately when they receive a 429, you get a thundering herd the instant the rate limit window resets. The Retry-After header helps, but only if clients respect it. Document and encourage exponential backoff with jitter in your SDK and API documentation. An example jitter formula: wait = base_delay * 2^attempt + random(0, base_delay).

Checklist#

Key Takeaways#

• Rate limiting is infrastructure, not an afterthought. Add it before you go public, not after an incident forces your hand.
• Token bucket is the right default algorithm for most APIs. It handles bursts naturally, is simple to understand, and maps well to Redis storage.
• Use atomic Lua scripts in Redis for distributed enforcement. Separate GET/SET operations create race conditions that allow over-serving under concurrent load.
• Always return rate limit headers on every response. Clients that can see their remaining quota do not need to guess, and well-implemented clients will back off before hitting the limit.
• The RFC 6585 standard defines 429 Too Many Requests and recommends Retry-After. Follow it so integrators can handle rejection the same way regardless of which API they are calling.
• Design for Redis failure. Fail open and log the incident rather than taking down your API when the rate-limit store becomes unavailable.
• IP-based limits are a floor, not a ceiling. Authenticate clients and rate limit on stable identifiers for anything that matters.

FAQ#

Should I use a fixed window or a sliding window?

A sliding window log is more accurate but expensive: it stores a timestamp for every request in the window. A sliding window counter (a weighted blend of the current and previous fixed windows) is a reasonable approximation at much lower cost. For most production APIs, a token bucket is preferable to either because it naturally handles bursts and requires only a small fixed-size structure per client, regardless of request history.

How do I choose capacity and refill rate?

Start with your p99 legitimate client behavior: what is the highest request rate a normal heavy user makes? Set the refill rate at 2 to 3x that. Set the capacity (burst) at roughly 5 to 10x the per-second refill rate to allow short spikes. Monitor 429 rates by client plan for the first two weeks and adjust. Incorrect limits will show up clearly in your 429 dashboards.

What if a client legitimately needs more than my limit?

Offer higher-tier plans with larger limits, or an enterprise tier with custom limits. The important architectural point is that limit configuration should be data-driven, not hardcoded. Look up the client's limits from config or a database, parameterize the bucket accordingly, and raising a plan limit becomes a config change, not a deployment.

Can I use rate limiting to prevent DDoS attacks?

Rate limiting helps with application-layer DDoS from a modest number of sources. For volumetric DDoS that saturates bandwidth or connection tables before your application code even runs, you need infrastructure-level protection (Cloudflare, AWS Shield, or similar). Think of rate limiting as protection for legitimate clients from each other, and infrastructure protection as defense against attackers who are never going to respect a 429.

What is the difference between rate limiting and throttling?

Rate limiting enforces a maximum request count in a period and rejects excess requests with 429. Throttling slows excess requests down rather than rejecting them, queuing or delaying them to smooth out spikes. Throttling is more user-friendly for batch operations but more complex to implement and can create latency spikes that are difficult to predict. Most REST APIs use rate limiting. Throttling is more common in streaming or job-submission APIs where delay is acceptable.