
Rate Limiting & Middleware for AI Endpoints
Designing Rate Limiting and Middleware for Production AI Endpoints
AI endpoints fail differently from conventional REST APIs. A request that reaches a typical CRUD service may consume a few milliseconds of CPU and a database connection. A request that reaches an LLM gateway can reserve a connection for minutes, consume a large and unpredictable number of input and output tokens, trigger multiple provider calls, and generate a bill that is not visible until the response completes.
That changes the purpose of middleware. Authentication is necessary, but it is not sufficient. A production AI middleware rate limiting architecture must control concurrency, request frequency, token consumption, provider quotas, and tenant cost. It must also make a decision quickly enough that the limiter itself does not become the bottleneck.
The central design principle is to enforce cheap, deterministic controls before expensive work begins. Edge middleware should identify the caller and reject obviously invalid traffic. A distributed limiter should reserve capacity before a model call. The application service should enforce request-specific budgets and release or reconcile capacity when the actual token usage is known. These layers address different failure modes and should not be collapsed into one global counter.
The Request Path and Its Control Points
A useful request path has four stages:
- Edge admission: Parse credentials, reject malformed requests, apply coarse IP and route controls, and attach a trusted identity context.
- Application authorization: Resolve the tenant, project, model policy, and permissions. This is where Next.js middleware authentication and server-side authorization meet.
- Distributed admission control: Atomically reserve request and token capacity in Redis or an equivalent low-latency store.
- Execution and reconciliation: Invoke the provider, enforce timeouts and concurrency limits, then record actual usage for billing and policy decisions.
Next.js middleware is a good location for stage one, but it should not be treated as the complete security boundary. Middleware can run in an edge runtime, may have limited access to Node.js APIs, and often executes before the request reaches the application region. It should validate a signed session or access token, not perform a database query for every request. A revoked credential or a changed tenant entitlement must still be checked by the origin service when the risk or policy requires it.
A middleware decision should be based on stable, low-cardinality facts:
- Route and HTTP method.
- Client IP, with careful handling of trusted proxy headers.
- Verified subject and tenant identifiers from a signed credential.
- A coarse product tier or policy version embedded in the token.
Do not trust a user-supplied tenant header, model name, or quota tier. Middleware can copy a verified subject into an internal header after removing any inbound version of that header, but the origin should authenticate the connection between the edge and the application as well.
A practical Next.js 15 middleware pattern looks like this:
import { NextRequest, NextResponse } from 'next/server'
import { verifyAccessToken } from './lib/auth-edge'
export async function middleware(request: NextRequest) {
const path = request.nextUrl.pathname
if (!path.startsWith('/api/ai/')) return NextResponse.next()
const authorization = request.headers.get('authorization')
const token = authorization?.startsWith('Bearer ')
? authorization.slice(7)
: null
const claims = token ? await verifyAccessToken(token) : null
if (!claims || !claims.sub || !claims.tenantId) {
return NextResponse.json({ error: 'unauthorized' }, { status: 401 })
}
const requestHeaders = new Headers(request.headers)
requestHeaders.delete('x-tenant-id')
requestHeaders.set('x-authenticated-subject', claims.sub)
requestHeaders.set('x-authenticated-tenant', claims.tenantId)
return NextResponse.next({ request: { headers: requestHeaders } })
}
export const config = {
matcher: ['/api/ai/:path*']
}
The example intentionally does not rate-limit in the Next.js middleware process itself. In a multi-region deployment, an in-memory counter creates one quota per instance, not one quota per tenant. It also disappears during a deploy and produces inconsistent behavior under autoscaling. Middleware can call a globally reachable limiter, but the latency and failure policy must be explicit. For most systems, coarse edge protection and an origin-side distributed limiter are a better split.
Choosing Between Token Bucket and Leaky Bucket
The token bucket algorithm in Node.js is usually the best default for interactive AI APIs. A bucket has a capacity and a refill rate. Each request consumes tokens, and a request is accepted if enough tokens are available. The bucket permits controlled bursts while enforcing a long-term average rate.
For example, a tenant may receive a bucket with a capacity of 20 requests and a refill rate of 2 requests per second. A quiet tenant can immediately issue 20 requests, but sustained traffic is limited to 2 requests per second. This is useful for chat and completion APIs where short bursts are normal.
A token bucket can also charge by estimated model tokens instead of requests. If the request declares a maximum output of 2,000 tokens and contains 1,000 input tokens, the admission cost can be 3,000 units. This prevents a client from bypassing a request-per-minute limit with large prompts. The estimate should be conservative because under-reserving capacity is a direct path to provider quota exhaustion.
A leaky bucket models a queue drained at a fixed rate. It smooths traffic rather than permitting large bursts. This is valuable when the downstream provider or an internal worker pool has a strict throughput ceiling. The queue must have a bounded size; an unbounded queue converts rate limiting into latency accumulation and eventually memory exhaustion.
The practical distinction is:
- Token bucket: Good for admission control, burst tolerance, and immediate 429 responses.
- Leaky bucket: Good for traffic shaping, fixed-rate workers, and smoothing provider calls.
- Concurrency limiter: Necessary for controlling in-flight work, because neither bucket directly limits requests that are already executing.
- Quota ledger: Necessary for monthly or billing-period consumption, where a short-lived bucket is not enough.
AI systems commonly need all four. A request may pass a per-second token bucket but still be rejected because the tenant has reached its monthly token quota or because the provider circuit is open. These are different decisions and should produce different metrics, even if the client receives a 429 or 503.
A basic local token bucket implementation is useful for tests and single-process workers, but it is not a distributed quota mechanism:
type Bucket = { tokens: number; updatedAt: number }
export function consume(
bucket: Bucket,
nowMs: number,
capacity: number,
refillPerSecond: number,
cost = 1
) {
const elapsed = Math.max(0, nowMs - bucket.updatedAt) / 1000
const refilled = Math.min(capacity, bucket.tokens + elapsed * refillPerSecond)
const allowed = refilled >= cost
bucket.tokens = allowed ? refilled - cost : refilled
bucket.updatedAt = nowMs
return { allowed, remaining: Math.floor(bucket.tokens) }
}
The important invariant is that refill and consume must be one atomic operation. Reading a Redis value, calculating locally, and writing it back allows concurrent requests to overwrite one another. Under load, that race systematically oversells capacity.
Distributed State, Atomicity, and Failure Semantics
Redis is a common limiter backend because it provides low-latency operations, TTLs, and Lua scripting. The key should include the policy scope and a stable identity, for example rl:v2:tenant:{tenantId}:route:{routeClass}. Versioning the key makes policy migrations safe and prevents old state from being interpreted under new units.
A Redis Lua script can calculate refill, test capacity, update the bucket, and set expiration atomically. Store the current token count and the last update timestamp in a hash. Use integer micro-units where possible to avoid floating-point drift. If refill is fractional, represent tokens as millionths or use a fixed-point rate.
local key = KEYS[1]
local now = tonumber(ARGV[1])
local capacity = tonumber(ARGV[2])
local refill = tonumber(ARGV[3])
local cost = tonumber(ARGV[4])
local ttl = tonumber(ARGV[5])
local state = redis.call('HMGET', key, 'tokens', 'updated')
local tokens = tonumber(state[1]) or capacity
local updated = tonumber(state[2]) or now
local elapsed = math.max(0, now - updated)
local available = math.min(capacity, tokens + elapsed * refill)
local allowed = available >= cost
local remaining = available
if allowed then
remaining = available - cost
end
redis.call('HSET', key, 'tokens', remaining, 'updated', now)
redis.call('EXPIRE', key, ttl)
return { allowed and 1 or 0, math.floor(remaining) }
The script should use a consistent time source. Redis server time avoids large differences between application hosts, although it adds a command or requires a carefully designed script interface. If application timestamps are used, monitor clock skew and make negative elapsed time resolve to zero, as the example does.
The failure mode for Redis deserves a deliberate policy. Fail-open protects availability but allows an outage to become an uncontrolled provider spend event. Fail-closed protects cost and provider quotas but can take down every AI request when the limiter is unavailable. A tiered policy is usually more defensible:
- Fail closed for anonymous traffic, expensive models, and high-risk administrative routes.
- Allow a small emergency budget for authenticated low-tier traffic, enforced locally with a conservative process bucket.
- Return 503 when the limiter dependency is unavailable and the request cannot be safely admitted.
- Emit a distinct
limiter_unavailablemetric; do not disguise it as an ordinary quota rejection.
A distributed limiter also introduces regional consistency trade-offs. A single global Redis cluster gives a more accurate quota but adds cross-region latency and creates a larger blast radius. Regional buckets reduce latency and improve fault isolation, but a tenant can temporarily exceed a global limit by using multiple regions. For strict billing or provider quotas, reserve from a home region or use a global ledger. For abuse prevention, approximate regional limits are often acceptable.
AI-Specific Controls: Tokens, Concurrency, and Streaming
Request rate is a weak proxy for AI cost. Ten one-token requests and ten 100,000-token requests are not equivalent. The admission layer should therefore combine multiple dimensions:
- Requests per second for connection and routing protection.
- Estimated input plus maximum output tokens for cost protection.
- Concurrent generations per tenant and model.
- Daily or monthly committed token quotas.
- Provider-specific requests-per-minute and tokens-per-minute limits.
A reservation is preferable to a post-hoc counter. Before invoking the provider, reserve the estimated cost. After completion, reconcile the reservation with actual usage. If the actual cost is lower, return the difference to the bucket. If it is higher, record the overage and apply a policy: charge the quota, cap output, or mark the tenant for review.
Streaming responses add another edge case. The HTTP request remains open while tokens are emitted, so a request-per-minute limiter alone does not protect connection pools. Apply a concurrency semaphore before the provider call and release it in a finally block. Set an idle timeout as well as an absolute timeout; a client that stops reading must not hold a provider connection forever.
const limit = pLimit(32)
export async function generate(input: Input, policy: Policy) {
const reservation = await reserveCapacity({
tenantId: policy.tenantId,
requestCost: 1,
estimatedTokens: estimateTokens(input),
model: input.model
})
if (!reservation.allowed) {
throw new HttpError(429, 'rate_limit_exceeded', {
retryAfter: reservation.retryAfterSeconds
})
}
try {
return await limit(async () => {
const response = await callProvider(input, {
timeoutMs: 45_000,
idleTimeoutMs: 15_000
})
await reconcileCapacity(reservation, response.usage)
return response
})
} catch (error) {
await releaseReservation(reservation)
throw error
}
}
Do not release a reservation blindly after a provider timeout. The provider may have accepted and processed the request even if the client connection failed. Reconciliation should distinguish not_sent, rejected_before_execution, executed_unknown, and completed. For the unknown case, retain a conservative charge or use provider request IDs to query usage asynchronously.
Production Protocols and Implementation Workflow
A robust implementation can be delivered in the following sequence:
- Define limits in cost units, not only request counts. Document burst capacity, sustained refill rate, concurrency, and billing-period quota for every plan.
- Create a policy service or versioned configuration file. Cache policies briefly, but include a policy version in limiter keys and logs.
- Implement authentication at the edge and authorization at the origin. Never infer tenant identity from a client-controlled header.
- Add atomic Redis reservation scripts and integration tests with concurrent callers. Verify that total accepted cost never exceeds the configured capacity within the test model.
- Add a local emergency limiter for dependency outages. Keep its capacity deliberately lower than the normal distributed allowance.
- Add response headers such as
Retry-After, a stable limit identifier, and remaining capacity where exposing that information does not create an abuse signal. - Make retries safe. Return an idempotency key for generation requests and ensure a client retry cannot create two billable provider calls.
- Separate client errors from infrastructure failures. Use 401 for missing credentials, 403 for insufficient permissions, 429 for exhausted policy capacity, and 503 for unavailable admission dependencies or providers.
- Load-test burst traffic, synchronized retries, long streams, Redis failover, clock skew, and hot tenants. Test many application instances, not only one local process.
Observability should cover both the limiter and the work it protects. Track accepted and rejected requests by tenant, route, model, region, and policy version. Track estimated versus actual tokens, reservation leakage, Redis latency, script errors, local fallback admissions, queue depth, active streams, provider latency, timeout rate, and circuit-breaker state.
Alert on invariants rather than arbitrary thresholds. Examples include accepted token cost exceeding configured capacity, reservations that remain unreconciled beyond a timeout, a rising ratio of estimated-to-actual usage, or provider tokens increasing while application admissions remain flat. These signals expose bypasses, accounting bugs, and provider-side behavior that ordinary HTTP metrics miss.
Conclusion
Rate limiting for AI endpoints is an admission-control system, not a counter attached to a route. The durable architecture combines Next.js middleware authentication for cheap identity checks, origin authorization for policy enforcement, an atomic distributed token bucket for burst-aware quotas, leaky-bucket or queue-based shaping for strict downstream throughput, and concurrency controls for long-lived generations.
The key invariants are simple: never trust client identity headers, never update shared quota state with a non-atomic read-modify-write sequence, never treat request count as a complete cost model, and never let limiter failure semantics remain implicit. Monitor accepted cost, actual provider usage, limiter availability, reservation reconciliation, active streams, and rejection reasons. Those metrics turn rate limiting from a reactive abuse patch into a predictable control plane for an AI product.