
Real-Time Streaming AI Responses with Next.js
Real-Time Streaming AI Responses with Next.js: Architecture, Protocols, and Production Reliability
A chat interface that renders an AI response only after the model has finished is functionally correct and operationally weak. Users experience the entire upstream inference time as a blank wait, while the application gives them no opportunity to observe progress, cancel work, or recover from a broken connection. For a low latency AI conversational agent, perceived responsiveness is a system property, not a CSS detail.
The production problem is more specific than adding a streaming flag to an SDK. A real-time streaming AI response Next.js implementation must coordinate a browser, a Next.js 15 request handler, an inference provider, persistence, authentication, cancellation, retries, and deployment infrastructure. Every boundary can buffer data, duplicate events, terminate idle connections, or produce a response that the client cannot safely reconcile.
The core design principle is to treat model output as an ordered event stream with an explicit lifecycle. Tokens are a rendering optimization. The authoritative unit is a message or generation identified by stable IDs, sequence numbers, and a terminal state. Once that contract exists, transport selection and UI implementation become engineering decisions rather than framework folklore.
The Streaming Architecture and Its Latency Budget
A typical request path contains five stages:
- The browser submits a user message and receives a generation identifier.
- The Next.js App Router endpoint authenticates the request and starts model inference.
- The inference adapter converts provider-specific chunks into a stable application event format.
- The transport delivers events to the browser as soon as they are available.
- The client reducer applies events idempotently and updates the rendered conversation.
The end-to-end time to first token can be modeled as:
TTFT = client upload + queueing + auth + retrieval + provider scheduling + model prefill + network delivery + browser commit
Streaming cannot eliminate provider scheduling or prompt prefill. It can reduce the time between the first available output and the first visible output. Measure both time to first byte and time to first token, because a route can flush headers while a proxy or model adapter still delays useful content.
A practical event envelope might look like this:
{
"type": "token",
"generationId": "gen_01HXYZ",
"messageId": "msg_01HXYZ",
"sequence": 17,
"data": { "text": "distributed" }
}
The lifecycle should include at least started, token, tool_call, usage, completed, cancelled, and error. Do not make the client infer completion from a closed TCP connection. A connection can close because of a mobile network transition, an idle timeout, or a load balancer restart after the provider has successfully completed the generation.
Persist the user message before beginning inference. Persist the assistant message as a placeholder with status running, then append or checkpoint generated content. At completion, atomically set the final status and usage metadata. This gives reconnecting clients a durable source of truth and prevents a transient stream from becoming the only copy of a valuable response.
For high-volume workloads, avoid writing one database row per token. Aggregate chunks in memory and checkpoint every 250 to 1,000 milliseconds, every several kilobytes, or at semantic boundaries. The exact policy depends on recovery objectives. A one-second checkpoint interval usually provides a reasonable balance between write amplification and recovery loss. Redis Streams, Kafka, or a dedicated generation service can decouple event production from web connections when generations outlive a server request.
WebSockets vs Server-Sent Events SSE
The choice between WebSockets vs Server-Sent Events SSE should follow communication requirements, not popularity. AI text generation is predominantly server-to-client traffic. The browser sends one request and receives an ordered stream, occasionally issuing cancellation or a tool approval. That shape strongly favors Server-Sent Events for many applications.
Server-Sent Events
SSE uses a long-lived HTTP response with text/event-stream. It is simple, observable, proxy-friendly, and natively supported by browsers through EventSource. Events can include IDs, allowing the client to reconnect with Last-Event-ID when the server maintains replayable history.
Advantages include:
- Unidirectional semantics that match token delivery.
- Normal HTTP authentication, routing, and infrastructure.
- Automatic browser reconnect behavior.
- Straightforward compatibility with serverless or edge HTTP handlers, subject to platform duration limits.
- Easy separation between command requests and event delivery.
Its limitations matter. The native EventSource API does not support arbitrary headers, so applications commonly use a cookie-based session, a short-lived signed stream token, or a fetch-based SSE client. SSE also does not provide a built-in bidirectional channel. Cancellation is normally implemented with AbortController on the client and a separate command or request teardown on the server.
A minimal Next.js App Router route can be written as follows:
// app/api/generations/[id]/stream/route.ts
import { NextRequest } from "next/server";
import { authenticate } from "@/lib/auth";
import { generationEvents } from "@/lib/generation-events";
export const runtime = "nodejs";
export const dynamic = "force-dynamic";
export async function GET(
request: NextRequest,
{ params }: { params: Promise<{ id: string }> }
) {
const user = await authenticate(request);
if (!user) return new Response("Unauthorized", { status: 401 });
const { id } = await params;
const encoder = new TextEncoder();
const stream = new ReadableStream({
start(controller) {
const send = (event: unknown, eventId?: string) => {
const idLine = eventId ? `id: ${eventId}\\n` : "";
controller.enqueue(
encoder.encode(`${idLine}data: ${JSON.stringify(event)}\\n\\n`)
);
};
const unsubscribe = generationEvents.subscribe(id, user.id, send);
const heartbeat = setInterval(() => {
controller.enqueue(encoder.encode(": heartbeat\\n\\n"));
}, 15000);
request.signal.addEventListener("abort", () => {
clearInterval(heartbeat);
unsubscribe();
controller.close();
});
}
});
return new Response(stream, {
headers: {
"Content-Type": "text/event-stream; charset=utf-8",
"Cache-Control": "no-cache, no-transform",
"Connection": "keep-alive",
"X-Accel-Buffering": "no"
}
});
}
This example is intentionally incomplete as a production event bus. An in-process subscription map works only when the producer and consumer share a process. In a horizontally scaled deployment, use a broker or route both operations to a generation worker. Do not assume that a process-local event emitter works behind a load balancer.
WebSockets
WebSockets are appropriate when the conversation is genuinely interactive in both directions: live voice, incremental tool approvals, typing signals, collaborative state, or frequent client commands. They avoid creating separate HTTP requests for cancellation and control messages, and they can reduce framing overhead for high-frequency events.
The costs are operational. WebSocket support varies across serverless platforms and proxies. Connections require explicit authentication, heartbeat, backpressure, reconnect, and connection draining behavior. A WebSocket also does not solve replay by itself. If a client disconnects after sequence 42, the server needs durable events or a reconstructed message to resume safely.
For text-only AI chat, a robust default is HTTP POST for commands plus SSE for output. Choose WebSockets when bidirectional session semantics justify the additional infrastructure. In both cases, define an application protocol with versioning, event IDs, and terminal states. Protocol simplicity is more valuable than marginal frame efficiency.
State Synchronization, Reconnects, and Cancellation
The client should process stream events through a reducer rather than concatenate strings inside an event callback. A reducer centralizes ordering, duplicate handling, terminal transitions, and recovery from reconnects.
type StreamEvent =
| { type: "started"; generationId: string; messageId: string; sequence: number }
| { type: "token"; generationId: string; messageId: string; sequence: number; data: { text: string } }
| { type: "completed"; generationId: string; messageId: string; sequence: number; data: { usage?: object } }
| { type: "error"; generationId: string; sequence: number; data: { code: string; retryable: boolean } };
type State = {
status: "idle" | "running" | "completed" | "failed";
text: string;
lastSequence: number;
};
function reduce(state: State, event: StreamEvent): State {
if (event.sequence <= state.lastSequence) return state;
switch (event.type) {
case "started":
return { ...state, status: "running", lastSequence: event.sequence };
case "token":
return {
...state,
status: "running",
text: state.text + event.data.text,
lastSequence: event.sequence
};
case "completed":
return { ...state, status: "completed", lastSequence: event.sequence };
case "error":
return { ...state, status: "failed", lastSequence: event.sequence };
}
}
The sequence check makes duplicate delivery harmless. It does not repair gaps. If sequence 17 arrives after sequence 15, the client should either request replay from 16 or refetch the canonical generation state. Never silently append an out-of-order token; corrupted conversation state is difficult to detect and harder to debug.
Reconnect behavior should be explicit:
- Keep the generation ID stable across reconnects.
- Send the last applied sequence in a query parameter or
Last-Event-ID. - Replay missed events when retained, otherwise send a snapshot followed by live events.
- Stop reconnecting after a terminal event or a non-retryable authorization error.
- Use exponential backoff with jitter to avoid a reconnect storm after an outage.
Cancellation also needs ownership semantics. When a user clicks Stop, mark the generation as cancellation-requested in durable state, abort the provider request if the worker supports it, and emit cancelled only after the worker observes or confirms cancellation. A browser disconnect is not equivalent to user cancellation. The user may simply have changed networks, and the generation may still need to finish for later retrieval.
Next.js App Router Performance and Production Protocols
Next.js 15 and the App Router provide useful primitives, but they do not automatically make streaming fast. Keep the streaming route dynamic and isolated from static rendering. Do not place a long-lived stream inside a page request that also performs expensive server component work.
Recommended implementation workflow:
- Use a short POST route to validate input, authorize the user, create the generation record, and return the generation ID quickly.
- Run model inference in a worker or dedicated service when execution can exceed platform request limits.
- Use an SSE route or WebSocket gateway only for event delivery.
- Disable response buffering with the correct content type and proxy-specific headers.
- Send heartbeats frequently enough to satisfy the shortest idle timeout in the path.
- Configure platform max duration, connection limits, and graceful shutdown explicitly.
- Use
AbortSignalthroughout the provider adapter and release subscriptions on every termination path. - Keep token rendering client-side and avoid triggering a full server component tree render for every chunk.
- Batch UI updates with
requestAnimationFrameor a small 20-to-50 millisecond window when providers emit very small chunks. - Virtualize long conversations and defer markdown syntax highlighting until a message is complete.
For example, the command route should enforce idempotency:
const idempotencyKey = request.headers.get("Idempotency-Key");
if (!idempotencyKey) {
return Response.json({ error: "Missing Idempotency-Key" }, { status: 400 });
}
const existing = await db.generation.findByUserAndKey(user.id, idempotencyKey);
if (existing) return Response.json(existing, { status: 200 });
const generation = await db.generation.create({
userId: user.id,
idempotencyKey,
status: "queued",
promptHash: hash(normalizedPrompt)
});
await queue.publish({ generationId: generation.id });
return Response.json(generation, { status: 202 });
This prevents mobile retries and proxy retries from launching duplicate paid generations. Add database uniqueness on (user_id, idempotency_key), not just an application-level lookup. For multi-region systems, choose a single write region or use a globally consistent idempotency store; eventual consistency can still create duplicates.
Security is part of the streaming protocol. Authorize access to the generation on every stream connection, scope signed stream tokens to one generation and user, limit prompt size, redact secrets from logs, and enforce tenant-level quotas. Do not log every token by default. Token content can contain personal or regulated data, while the operational value of raw token logs is usually low.
Observability should distinguish user-perceived latency from provider latency. Track:
- Request acceptance latency and queue wait time.
- Time to first byte and time to first token.
- Inter-token gap distribution, not only average throughput.
- Completion latency, cancellation rate, and provider error rate.
- Reconnect frequency, replay failures, sequence gaps, and duplicate events.
- Open streams by instance, memory per stream, broker lag, and heartbeat failures.
- Database checkpoint write rate and generation recovery loss.
Set alerts on p95 and p99 values. A healthy average can conceal a severe tail caused by overloaded model providers, exhausted connection pools, or a proxy buffering configuration. Load-test with slow clients, abrupt disconnects, duplicate POSTs, provider stalls, worker restarts, and broker partitions. Streaming systems fail in the gaps between components, so test the gaps directly.
Conclusion
A production real-time streaming AI response Next.js system is an event-processing system with a user interface attached. The durable generation record is the source of truth; the stream is an acceleration path. Stable IDs, monotonic sequence numbers, explicit terminal events, idempotent commands, and replay or snapshot recovery are the architectural invariants.
For conventional text chat, HTTP commands plus SSE usually provide the best ratio of simplicity to latency. WebSockets become justified when bidirectional, high-frequency interaction is central to the product. Next.js App Router performance depends on isolating long-lived connections, avoiding per-token server renders, and configuring every buffering and timeout layer in the deployment path.
Measure time to first token, inter-token gaps, completion latency, reconnects, replay correctness, broker lag, and resource usage under slow-client conditions. When those metrics are visible and the protocol is explicit, streaming stops being a fragile UI effect and becomes a reliable capability of the system.