← BACK_TO_LOGS

Running Local AI Models with Ollama & Node.js

•10 min read•KANAT NAZAROV
Running Local AI Models with Ollama & Node.js

Running Local AI Models with Ollama and Node.js: A Production Architecture

Running an LLM locally is easy to demonstrate and surprisingly difficult to operate correctly. A developer can install Ollama, download a model, send an HTTP request, and receive a useful answer within minutes. The production problem begins when that request becomes part of a multi-user web application: inference can monopolize CPU and memory, responses can take tens of seconds, clients disconnect, model state becomes inconsistent, and a seemingly harmless retry can execute an expensive generation twice.

This is the real problem behind Local AI models Ollama web integration. Ollama is an excellent local model runtime, but it is not an application architecture. Node.js, the database, the browser, and the model server still need explicit contracts for concurrency, cancellation, persistence, authentication, and failure recovery.

The architecture in this article treats inference as a bounded, observable distributed system. A Next.js 15 application or Node.js API owns product behavior. Ollama owns model execution. A database owns durable conversation state. Redis or a queue can own coordination when requests outgrow a single process. This separation makes local inference useful without confusing low infrastructure cost with unlimited capacity.

The System Boundary: Node.js Is the Control Plane, Ollama Is the Inference Plane

Ollama exposes a local HTTP API, commonly at

http://127.0.0.1:11434
. The Node.js service should call that API; browsers should not. Direct browser access creates several problems:

  • It exposes an internal service and its model inventory.
  • It makes authentication and tenant isolation difficult.
  • It allows users to consume all available inference capacity.
  • It couples the frontend to Ollama-specific request and response formats.
  • It creates an unsafe deployment assumption that Ollama is reachable from every client.

The backend should provide an application-level endpoint such as

POST /api/chat
. That endpoint authenticates the user, loads authorized context, applies token and message limits, forwards a normalized request to Ollama, and records the result. The model runtime remains replaceable: a future deployment can route the same interface to vLLM, an OpenAI-compatible provider, or a hosted model.

A minimal request to Ollama looks like this:

const response = await fetch("http://127.0.0.1:11434/api/chat", {
  method: "POST",
  headers: { "content-type": "application/json" },
  body: JSON.stringify({
    model: "llama3.1:8b",
    messages: [
      { role: "system", content: "Answer concisely and cite supplied context." },
      { role: "user", content: "Explain optimistic concurrency control." }
    ],
    stream: false,
    options: {
      temperature: 0.2,
      num_ctx: 8192,
      num_predict: 512
    }
  })
});

if (!response.ok) {
  throw new Error(`Ollama returned HTTP ${response.status}`);
}

const result = await response.json();
console.log(result.message.content);

The exact model name, context length, and quantization must be treated as deployment configuration, not hard-coded product logic. A quantized 8B model may be appropriate for a developer laptop or an on-premise CPU node, while a 70B model may require a dedicated GPU and materially different concurrency limits.

Streaming, Cancellation, and Backpressure in Node.js

Users perceive time to first token more strongly than total generation time. Streaming allows the UI to render partial output, but streaming does not make inference cheaper. It changes how the system handles a long-running operation.

Ollama returns newline-delimited JSON when

stream
is enabled. In a Node.js or Next.js route, parse the response body incrementally and emit Server-Sent Events (SSE). Do not buffer the entire response in a serverless function or proxy if the goal is low-latency interaction.

import { NextResponse } from "next/server";

export const runtime = "nodejs";

export async function POST(request) {
  const controller = new AbortController();
  request.signal.addEventListener("abort", () => controller.abort());

  const input = await request.json();
  const upstream = await fetch("http://127.0.0.1:11434/api/chat", {
    method: "POST",
    signal: controller.signal,
    headers: { "content-type": "application/json" },
    body: JSON.stringify({
      model: process.env.OLLAMA_MODEL ?? "llama3.1:8b",
      messages: input.messages.slice(-20),
      stream: true,
      options: {
        temperature: 0.2,
        num_ctx: 8192,
        num_predict: 600
      }
    })
  });

  if (!upstream.ok || !upstream.body) {
    return NextResponse.json({ error: "Inference unavailable" }, { status: 503 });
  }

  const reader = upstream.body.getReader();
  const decoder = new TextDecoder();
  const encoder = new TextEncoder();

  const stream = new ReadableStream({
    async pull(controller) {
      const { value, done } = await reader.read();
      if (done) {
        controller.close();
        return;
      }

      const lines = decoder.decode(value, { stream: true }).split("\n");
      for (const line of lines) {
        if (!line.trim()) continue;
        const chunk = JSON.parse(line);
        const token = chunk.message?.content ?? "";
        if (token) controller.enqueue(encoder.encode(`data: ${JSON.stringify({ token })}\n\n`));
        if (chunk.done) controller.enqueue(encoder.encode("data: [DONE]\n\n"));
      }
    },
    cancel() {
      controller.abort();
    }
  });

  return new Response(stream, {
    headers: {
      "content-type": "text/event-stream",
      "cache-control": "no-cache, no-transform",
      connection: "keep-alive"
    }
  });
}

Production parsing needs more care than this compact example. A TCP chunk can split a JSON line, so a real parser must retain an incomplete trailing fragment between reads. It must also handle malformed upstream data, client disconnects, and an Ollama process that stops responding. Set an explicit request deadline with

AbortSignal.timeout()
, but distinguish timeout from user cancellation in telemetry.

Backpressure matters. If the browser or reverse proxy reads slowly, the server should not accumulate unbounded tokens in memory. Web Streams provide a useful boundary, but the complete request path must support streaming: Nginx buffering, platform proxies, compression middleware, and CDN behavior can silently defeat it. Test with production-like infrastructure rather than relying on localhost behavior.

State Synchronization: Durable Conversations and Idempotent Generations

An LLM request is not a transaction. The model has no authoritative memory of your application conversation unless the application sends the relevant messages on every call. Therefore, the database—not Ollama—must be the source of truth.

A practical schema separates conversations, messages, and generation attempts:

CREATE TABLE conversations (
  id UUID PRIMARY KEY,
  user_id UUID NOT NULL,
  version INTEGER NOT NULL DEFAULT 0,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

CREATE TABLE messages (
  id UUID PRIMARY KEY,
  conversation_id UUID NOT NULL REFERENCES conversations(id),
  role TEXT NOT NULL CHECK (role IN ('system', 'user', 'assistant', 'tool')),
  content TEXT NOT NULL,
  sequence_no INTEGER NOT NULL,
  request_id UUID,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  UNIQUE (conversation_id, sequence_no)
);

When a user submits a prompt, the API should validate the conversation owner, read the current version, and append the user message using a request identifier. A unique constraint on

request_id
or an idempotency table prevents a network retry from creating duplicate messages. The generation can then run from a snapshot of messages.

Optimistic concurrency control prevents two browser tabs from corrupting ordering. The client submits

expectedVersion
; the database update succeeds only when the stored version matches. If it fails, return
409 Conflict
and let the client refresh. Do not silently merge two independent generations into one transcript.

There are two reasonable persistence strategies for streamed assistant output:

  • Buffer the generated text in the API and write one assistant message after completion. This is simpler, but a process crash loses partial output.
  • Write checkpoints periodically, such as every 1,000 characters or every two seconds. This improves recovery but creates more database writes and requires a message status such as
    generating
    ,
    completed
    ,
    cancelled
    , or
    failed
    .

For high-volume systems, keep the hot conversation index in PostgreSQL with an index on

(conversation_id, sequence_no)
and store large prompts or documents outside the primary row path. Do not load an unbounded conversation into every prompt. Use a rolling window, summarization, or retrieval over a vector index. Context-window overflow is a predictable capacity failure, not an exceptional model error.

Edge Computing vs Cloud Computing for Web Inference

The Edge computing vs Cloud computing web decision is a placement decision, not a branding exercise. Local Ollama inference can mean a developer workstation, an on-premise GPU, a private server, or an edge node near the user. Each location changes latency, privacy, utilization, and operational responsibility.

Local or edge inference is strongest when:

  • Sensitive prompts cannot leave a controlled environment.
  • The model is small enough for predictable hardware capacity.
  • Network round trips dominate the interaction.
  • Offline or intermittently connected operation is required.
  • Variable provider pricing is more expensive than owning capacity.

Cloud inference is stronger when:

  • Demand is bursty and GPU utilization would otherwise be poor.
  • Users need larger models or higher availability.
  • The team does not want to operate drivers, model files, and GPU nodes.
  • Global traffic requires regional capacity and managed failover.

A hybrid router is often the correct architecture. Route short, privacy-sensitive, low-complexity tasks to Ollama; route long-context or high-quality tasks to a hosted provider. The router must preserve policy boundaries. Never fall back from local to cloud if the prompt contains data classified as non-exportable.

This also clarifies the relationship to Fine-tuning OpenAI models web application projects. Fine-tuning a hosted OpenAI model and running an Ollama model locally solve different problems. Fine-tuning changes behavior using curated examples; retrieval, prompt design, and tool constraints often solve knowledge and workflow problems more cheaply. A locally deployed model may be adapted through supported Modelfiles, adapters, or a separately trained model, but the resulting artifact still needs evaluation, versioning, rollback, and license review. Fine-tuning is not a substitute for an inference capacity plan.

Production Protocols & Implementation

A reliable deployment should establish the following controls before exposing the endpoint to users.

  • Bound concurrency. Use a semaphore per Ollama worker or host. If the machine supports two concurrent generations, queue the third instead of allowing memory pressure to decide the outcome.
  • Set deadlines. Use separate limits for queue wait, model startup, first token, and total generation. A single 60-second timeout hides which subsystem is failing.
  • Limit inputs. Enforce maximum message count, UTF-8 byte size, estimated input tokens, output tokens, and attachment size at the API boundary.
  • Authenticate and authorize. Check user, tenant, conversation ownership, model entitlement, and data residency before constructing the Ollama request.
  • Use idempotency keys. Persist a request key with the generation status. A retry should return the existing result or status rather than launch another generation.
  • Handle model readiness. Pull models during deployment or image preparation. Do not let the first customer request trigger an unbounded model download.
  • Isolate the runtime. Run Ollama on a private network, use a dedicated service account, restrict filesystem access, and expose only the application gateway.
  • Treat output as untrusted. Escape rendered Markdown, constrain tool calls with schemas, and require explicit authorization for actions such as database writes or email delivery.
  • Design queue semantics. For asynchronous jobs, use a durable queue with visibility timeouts and a dead-letter path. A worker must renew leases while generating and mark jobs terminally on cancellation.
  • Version prompts and models. Store model name, digest or image version, system prompt version, sampling options, and retrieval configuration with every generation.

A simple in-process semaphore is sufficient for one Node.js instance:

class Semaphore {
  constructor(limit) {
    this.limit = limit;
    this.active = 0;
    this.waiters = [];
  }

  acquire() {
    if (this.active < this.limit) {
      this.active++;
      return Promise.resolve(() => this.release());
    }
    return new Promise(resolve => this.waiters.push(resolve));
  }

  release() {
    const next = this.waiters.shift();
    if (next) next(() => this.release());
    else this.active--;
  }
}

In a horizontally scaled Node.js deployment, this semaphore is only local to a process. Use a queue, a centralized lease, or route requests to dedicated inference workers. A database row lock can coordinate low throughput, but Redis-based leases or a purpose-built queue are generally better for sustained workloads. Never assume a process-local limit protects a shared GPU.

Observability must cover the entire request lifecycle. Record structured events containing a correlation ID, tenant, model version, queue wait, prompt token estimate, time to first token, total latency, generated token count, cancellation reason, and terminal status. Avoid logging raw prompts by default; they frequently contain credentials or personal data.

The minimum operational dashboard should include:

  • Request rate, success rate, cancellation rate, and error classes.
  • Queue depth, queue wait p50/p95/p99, and active generations.
  • Time to first token and time per output token.
  • CPU, RAM, VRAM, GPU utilization, thermal throttling, and model load time.
  • Context-window rejection rate and average input/output token counts.
  • Database latency, connection pool saturation, and failed state transitions.
  • Cloud fallback rate and cost per completed generation in a hybrid system.

Load testing must use realistic prompts and concurrency. A benchmark with one request at a time measures model speed, not service capacity. Test cold starts, repeated prompts, cancellation during generation, database failover, Ollama restarts, client reconnects, and a full queue. Verify that every failure leaves a recoverable state rather than an indefinitely generating message.

Conclusion

A successful Ollama web integration is defined by boundaries and invariants: the browser never controls the model runtime, the database remains the source of truth, every generation has an idempotent identity, concurrency is explicitly bounded, and cancellation propagates through every layer. Node.js and Next.js 15 can provide low-latency streaming, but only when proxies, queues, persistence, and model capacity are designed as one system.

Choose local, edge, cloud, or hybrid inference based on privacy, workload shape, latency, quality, and operational ownership. Monitor queue wait, time to first token, tokens per second, resource saturation, state-transition failures, and fallback cost. The model is only one component; production reliability comes from the protocol around it.

Leave a comment

Kanat Nazarov

Kanat Nazarov

Systems & Full-Stack Engineer
Kanat Nazarov is a product-minded Full-Stack Engineer. He is the co-founder of GetFusionChat, an enterprise-grade multi-tenant communication SaaS platform. Beyond architecting low-latency infrastructures and advanced AI systems, Kanat is a dedicated student of peak performance—optimizing his lifestyle through rigorous fitness, precise nutrition, and creative production in music. He operates on the core philosophy that breakthrough technical results stem from a high-functioning, deeply disciplined life.