AI Chat Service

An AI chat service is a ChatGPT-style assistant: a user sends a message, the system runs it through a large language model, and streams the answer back token by token — while remembering the conversation and, often, grounding answers in the user's own documents. The design challenges are modern and distinctive: streaming long responses, managing context, scaling expensive GPU inference, and doing RAG well. It combines Real-time Updates, Managing Long-Running Tasks, and Vector Databases.

1. Requirements

Functional:

• User sends a message; the assistant streams a response back incrementally.
• Conversations have memory — the model sees prior turns.
• (Common) ground answers in the user's documents via retrieval (RAG).

Non-functional:

• Responsive streaming — first token fast; tokens flow smoothly (perceived latency beats total latency).
• Expensive, scarce compute — GPU inference is the dominant cost and bottleneck.
• Scale — many concurrent conversations; long-running generations.
• Context limits — the model has a bounded context window.

2. Streaming is the defining UX

A full answer can take many seconds, so you never make the user wait for the whole thing. The model generates token by token, and you stream each token to the client as it's produced — the answer appears to "type out," so perceived latency is just time-to-first-token, not total generation time. The right transport is Server-Sent Events (SSE): one-directional server→client streaming, simple and HTTP-native — exactly the real-time pattern for this shape (you don't need WebSocket's bidirectionality; the request is one message, the response is a stream).

user message → server → model streams tokens → SSE → client renders token by token

3. The request path

client → API → build prompt (system + history + retrieved context + user msg)
       → inference service (GPU) → stream tokens back via SSE
       → persist the completed turn to conversation history

The API assembles the prompt from several pieces (below), submits it to the inference tier, relays the token stream to the client, and saves the finished exchange.

4. Managing context

The model only "remembers" what's in its prompt, and the context window is finite, so conversation memory is prompt construction, not model state:

• Short history: include recent turns verbatim.
• Long conversations: they overflow the window — summarize older turns into a compact running summary, keeping recent turns verbatim. So memory = recent turns + a rolling summary of the rest.
• Store full conversation history durably (keyed by conversation) for display and for rebuilding context.

5. RAG: grounding in your data

To answer from the user's own documents (not just the model's training), add a retrieval step (Vector Databases):

user question → embed → search vector DB for relevant doc chunks
             → inject those chunks into the prompt as context → model answers from them

Documents are chunked, embedded, and indexed in a vector store ahead of time; at query time you embed the question, retrieve the top-K relevant chunks, and prepend them to the prompt. This grounds answers in real data and reduces hallucination. (See Vector Databases for the ANN mechanics.)

6. Scaling inference — the real bottleneck

GPU inference is expensive and capacity is limited, so the inference tier, not your web servers, is the constraint:

• Queue + autoscaling GPU workers: submit generation requests to a queue fronting a pool of GPU workers; scale the pool to demand. The queue absorbs bursts and lets you prioritize (paid vs. free).
• Batching: GPUs are most efficient processing multiple requests together — dynamic batching of concurrent requests raises throughput dramatically.
• Caching: cache responses to identical/similar prompts; cache embeddings; reuse computation where possible to save GPU time and cost.
• Model tiering: route simple queries to a smaller/cheaper model and only hard ones to the big model.

7. Deep dives

• Cost control: tokens cost money — cap context length, summarize aggressively, cache, and tier models. Cost is a first-class design concern here.
• Cancellation: a user stopping a generation should free the GPU immediately — propagate cancellation to the worker.
• Safety/moderation: filter inputs and outputs; run moderation around the generation.
• Rate limiting / abuse: per-user limits (Rate Limiter) protect scarce GPU capacity.
• Multi-tenancy & privacy: isolate conversations and retrieved data strictly per user.

What to emphasise

Lead with streaming, context-as-prompt, RAG, and the GPU bottleneck:

"The defining UX is streaming — the model generates token by token, and I stream each token over SSE so perceived latency is time-to-first-token, not the full generation. Memory isn't model state; it's prompt construction — recent turns verbatim plus a rolling summary of older ones to fit the context window. For grounding I add RAG: chunk and embed documents into a vector DB ahead of time, then at query time embed the question, retrieve the top-K relevant chunks, and inject them into the prompt. The real bottleneck is GPU inference, so generation requests go through a queue to an autoscaling pool of GPU workers with dynamic batching for throughput, plus response and embedding caches and model tiering — cost and capacity are first-class concerns, so I cap context, summarize, and rate-limit per user."

Next up: Flash Sale — extreme contention under a traffic spike.