OpenAI Responses Became Slow in Production

Problem Summary

OpenAI completion latency is measured in two ways: Time to First Byte (TTFB) how long until the first token of the response arrives and total response time how long until the full response is complete. Most production latency complaints are about total response time because the application waits for the entire response before showing anything to the user. Implementing streaming eliminates perceived latency by showing output as it generates, but several other factors model choice, system prompt size, prompt token count, and network path affect even TTFB.

Symptoms

• Chat responses take 10–30 seconds end-to-end before anything appears in the UI
• API latency varies widely: some requests complete in 2 seconds, others take 20+
• Server logs show the OpenAI call blocking the entire request handler thread
• Users report the UI 'freezing' or 'hanging' while waiting for AI output
• Latency increased after adding more context to the system prompt
• gpt-4o requests are significantly slower than gpt-4o-mini for the same user prompt
• Parallel API calls (e.g., generating multiple suggestions) run sequentially instead of concurrently
• TTFB to the OpenAI API is fast but response arrives all at once after a long wait

Business Impact

Users perceive AI responses as 'thinking' when they see streaming text, but a blank screen for 15 seconds before text appears triggers frustration and abandonment. Research on web performance consistently shows that perceived performance matters more than actual performance. A 15-second streaming response that starts showing text in 800ms is experienced as faster than a 5-second response that shows nothing then appears all at once.

Root Cause

The most impactful root cause for perceived latency is the absence of streaming. Without stream: true, the Node.js server waits for the entire response, then forwards it to the browser in a single payload. The user sees nothing for the full generation time which for a 500-token response on gpt-4o at ~40 tokens/second is about 12 seconds. With streaming, the first token appears in the browser in approximately the same time as the TTFB from OpenAI to your server (~300–800ms).

Diagnosis

Step 1: Measure TTFB vs Total Response Time Separately

import OpenAI from "openai";

const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

export async function measureLatency(messages: OpenAI.ChatCompletionMessageParam[]) {
  const start = performance.now();
  let ttfb: number | null = null;
  let tokenCount = 0;

  const stream = await client.chat.completions.create({
    model: "gpt-4o",
    messages,
    stream: true,
  });

  for await (const chunk of stream) {
    if (ttfb === null) {
      ttfb = performance.now() - start;
      console.log(`TTFB: ${ttfb.toFixed(0)}ms`);
    }
    const delta = chunk.choices[0]?.delta?.content;
    if (delta) tokenCount++;
  }

  const total = performance.now() - start;
  console.log(`Total time: ${total.toFixed(0)}ms for ~${tokenCount} tokens`);
  console.log(`Generation speed: ${(tokenCount / (total / 1000)).toFixed(1)} tokens/sec`);
}

Step 2: Implement Server-Sent Events Streaming Endpoint

Pipe the OpenAI stream directly to the HTTP response using Server-Sent Events. This lets tokens appear in the browser as fast as OpenAI generates them.

import express from "express";
import OpenAI from "openai";

const app = express();
const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

app.post("/api/chat/stream", express.json(), async (req, res) => {
  const { messages } = req.body;

  // Set SSE headers
  res.setHeader("Content-Type", "text/event-stream");
  res.setHeader("Cache-Control", "no-cache");
  res.setHeader("Connection", "keep-alive");
  res.flushHeaders(); // Send headers immediately so the browser knows to start reading

  try {
    const stream = await client.chat.completions.create({
      model: "gpt-4o-mini",
      messages,
      stream: true,
      max_tokens: 1024,
    });

    for await (const chunk of stream) {
      const delta = chunk.choices[0]?.delta?.content;
      if (delta) {
        // SSE format: "data: <payload>\n\n"
        res.write(`data: ${JSON.stringify({ content: delta })}\n\n`);
      }

      if (chunk.choices[0]?.finish_reason) {
        res.write(`data: ${JSON.stringify({ done: true })}\n\n`);
        break;
      }
    }
  } catch (err) {
    res.write(`data: ${JSON.stringify({ error: "Stream failed" })}\n\n`);
  } finally {
    res.end();
  }
});

Step 3: Consume the Stream in the Browser

export async function streamChat(
  messages: { role: string; content: string }[],
  onToken: (token: string) => void,
  onDone: () => void
): Promise<void> {
  const response = await fetch("/api/chat/stream", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ messages }),
  });

  const reader = response.body!.getReader();
  const decoder = new TextDecoder();

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;

    const text = decoder.decode(value);
    const lines = text.split("\n").filter((l) => l.startsWith("data: "));

    for (const line of lines) {
      const payload = JSON.parse(line.slice(6));
      if (payload.content) onToken(payload.content);
      if (payload.done) { onDone(); return; }
    }
  }
}

Step 4: Run Independent AI Calls in Parallel

If a page requires multiple independent AI completions (e.g., a title, a summary, and suggested tags), await them sequentially adds their latencies. Use Promise.all to run them concurrently.

import OpenAI from "openai";

const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

async function generateDocumentMetadata(document: string) {
  // BAD: sequential  total time = t1 + t2 + t3
  // const title = await generateTitle(document);
  // const summary = await generateSummary(document);
  // const tags = await generateTags(document);

  // GOOD: parallel  total time = max(t1, t2, t3)
  const [title, summary, tags] = await Promise.all([
    client.chat.completions.create({
      model: "gpt-4o-mini",
      messages: [
        { role: "system", content: "Generate a concise title. Respond with the title only." },
        { role: "user", content: document },
      ],
      max_tokens: 30,
    }),
    client.chat.completions.create({
      model: "gpt-4o-mini",
      messages: [
        { role: "system", content: "Summarize in 2 sentences." },
        { role: "user", content: document },
      ],
      max_tokens: 100,
    }),
    client.chat.completions.create({
      model: "gpt-4o-mini",
      messages: [
        { role: "system", content: "Return 5 tags as a JSON array of strings." },
        { role: "user", content: document },
      ],
      max_tokens: 60,
      response_format: { type: "json_object" },
    }),
  ]);

  return {
    title: title.choices[0].message.content,
    summary: summary.choices[0].message.content,
    tags: JSON.parse(tags.choices[0].message.content!),
  };
}

Step 5: Reduce System Prompt Tokens to Lower TTFB

TTFB from OpenAI correlates with prompt token count. Longer prompts take more time to process before the first output token generates. Audit your system prompt and cut it to the minimum necessary instructions.

// BEFORE: verbose system prompt (~800 tokens)
const verboseSystemPrompt = `
You are a helpful, friendly, knowledgeable, and professional AI assistant
for our company. You should always be polite and courteous to users.
You should respond in a helpful manner and try to answer questions to
the best of your ability. If you don't know something, say so clearly.
You are an expert in customer support and should help users solve their problems...
[continues for 20 more paragraphs]
`;

// AFTER: minimal system prompt (~60 tokens)
const conciseSystemPrompt = `
Customer support assistant. Answer questions about our product clearly and concisely.
Escalate billing issues to support@example.com. Do not discuss competitors.
`;

Production Failure Scenarios

Scenario 1: Node.js Event Loop Blocked by Synchronous Token Counting

A tiktoken encoding operation on a 50,000-character document was running synchronously on the main Node.js thread before every API call, blocking the event loop for 80–120ms per request. Under concurrent load this serialized all requests through the encoder. Fix: run CPU-intensive token counting in a worker thread or use the async tiktoken API.

Scenario 2: Streaming Response Buffered by Proxy

An nginx reverse proxy had proxy_buffering on (the default), which buffered the entire streaming response before forwarding to the client. The browser received everything at once after the full generation time identical behavior to no streaming. Fix: add proxy_buffering off and X-Accel-Buffering: no headers for streaming endpoints.

Scenario 3: gpt-4o Used for All Tasks Including Classification

All API calls were hardcoded to 'gpt-4o' regardless of task complexity. Intent classification (which is a single-token output task) was experiencing the same 8–15 second latency as long-form generation. Switching classification, tagging, and extraction tasks to gpt-4o-mini reduced their P95 latency from 12 seconds to under 2 seconds.

Prevention Checklist

• Implement streaming (stream: true) for all user-facing chat completions
• Measure TTFB and total generation time separately in production monitoring
• Use gpt-4o-mini for classification, extraction, and short-form generation tasks
• Run independent AI calls with Promise.all instead of sequential awaits
• Disable proxy buffering for streaming endpoints in nginx and other reverse proxies
• Keep system prompts under 200 tokens unless the task genuinely requires more
• Run CPU-intensive preprocessing (token counting, text cleaning) in worker threads
• Add P50, P95, and P99 latency metrics to your monitoring dashboard by model and feature

Resolution Summary

The highest-impact fix for slow OpenAI responses is almost always implementing streaming, which reduces perceived latency from full generation time to TTFB. After streaming, the next gains come from model right-sizing (gpt-4o-mini for simple tasks), parallelizing independent calls, and trimming system prompt size. Together these changes can reduce perceived latency from 15 seconds to under 1 second for most user-facing features.

Lessons Learned

• Streaming reduces perceived latency by 10x or more it should be the default for all chat features
• Reverse proxies buffer streaming responses by default and must be explicitly configured for SSE
• TTFB is the metric that matters for user experience, not total generation time
• Sequential AI calls for independent tasks multiply latency unnecessarily
• gpt-4o-mini generates tokens 3x faster than gpt-4o and should be the default model

Conclusion

Slow AI responses are a UX problem before they are a performance problem. Users who see text appearing immediately even slowly stay engaged. Users who stare at a loading spinner for 15 seconds leave. Streaming is the single most impactful change you can make to AI response latency, and it requires fewer than 50 lines of code to implement end to end. Do it first, then optimize model selection and parallelism for the remaining gains.