Why Your OpenAI API Bill Increased Overnight

Problem Summary

OpenAI API costs are driven by token consumption: every input token and every output token costs money, and the price varies by model by a factor of up to 60x. A bill increase without a corresponding traffic increase almost always traces back to one of four causes: prompts that grew silently (conversation history, injected documents, verbose system prompts), missing caching for repeated identical prompts, wrong model selection (using gpt-4o for tasks gpt-4o-mini handles equally well), or output tokens that are longer than necessary because max_tokens was never set.

Symptoms

• Monthly OpenAI spend increased 2–10x without a corresponding increase in user traffic
• Usage dashboard shows input tokens growing faster than request count
• Identical user queries generating different token counts across requests
• Single API calls consuming 8,000+ tokens when the user message is under 100 words
• gpt-4o appearing in cost breakdowns for features that only do classification or extraction
• Embedding costs spiking because full PDF pages are sent instead of cleaned text chunks
• Output token count near the max_tokens cap, suggesting responses are being truncated
• No cache hit metrics because prompt caching was never enabled

Business Impact

At scale, unoptimized token usage can make a product economically unviable. A SaaS product billing users $29/month cannot absorb $15/month per user in API costs. Cost spikes also obscure real usage trends in analytics a doubling of spend that is purely driven by prompt bloat looks identical to a real 2x growth event in revenue dashboards, leading to incorrect business decisions.

Root Cause

The most common single root cause is unbounded conversation history injection. Developers append every user turn and assistant turn to the messages array without trimming. After 20 conversation turns with verbose responses, the history alone can add 15,000–30,000 tokens to every subsequent request tokens the model barely uses because useful information decays with distance from the current query.

Diagnosis

Step 1: Log Token Usage on Every Request

The OpenAI SDK returns a usage object on every non-streaming completion. Log it with a label identifying which feature generated the call so you can attribute cost by feature.

import OpenAI from "openai";

const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

const MODEL_COSTS: Record<string, { input: number; output: number }> = {
  "gpt-4o":          { input: 2.50,  output: 10.00 }, // per 1M tokens
  "gpt-4o-mini":     { input: 0.15,  output: 0.60  },
  "gpt-4-turbo":     { input: 10.00, output: 30.00 },
};

function estimateCost(model: string, usage: OpenAI.CompletionUsage): number {
  const pricing = MODEL_COSTS[model];
  if (!pricing) return 0;
  return (
    (usage.prompt_tokens / 1_000_000) * pricing.input +
    (usage.completion_tokens / 1_000_000) * pricing.output
  );
}

export async function trackedCompletion(
  feature: string,
  params: OpenAI.ChatCompletionCreateParamsNonStreaming
) {
  const response = await client.chat.completions.create(params);
  const usage = response.usage!;
  const cost = estimateCost(params.model as string, usage);

  console.log(JSON.stringify({
    feature,
    model: params.model,
    prompt_tokens: usage.prompt_tokens,
    completion_tokens: usage.completion_tokens,
    total_tokens: usage.total_tokens,
    estimated_cost_usd: cost.toFixed(6),
    timestamp: new Date().toISOString(),
  }));

  return response;
}

Step 2: Audit System Prompt Size

System prompts are paid for on every single request. A 2,000-token system prompt on 100,000 daily requests costs 200 million tokens per day in system prompt alone. Measure yours.

import { encoding_for_model } from "tiktoken";

export function auditSystemPrompt(systemPrompt: string): void {
  const enc = encoding_for_model("gpt-4o");
  const tokens = enc.encode(systemPrompt).length;
  enc.free();

  const dailyRequests = 100_000;
  const tokensPerDay = tokens * dailyRequests;
  const gpt4oCostPerDay = (tokensPerDay / 1_000_000) * 2.50;
  const miniCostPerDay = (tokensPerDay / 1_000_000) * 0.15;

  console.log(`System prompt size: ${tokens} tokens`);
  console.log(`Daily cost (gpt-4o): $${gpt4oCostPerDay.toFixed(2)}`);
  console.log(`Daily cost (gpt-4o-mini): $${miniCostPerDay.toFixed(2)}`);
  console.log(`Potential saving by switching to mini: $${(gpt4oCostPerDay - miniCostPerDay).toFixed(2)}/day`);
}

Step 3: Implement OpenAI Prompt Caching

OpenAI automatically caches prompt prefixes longer than 1,024 tokens. Cached input tokens cost 50% less. Structure your prompts so the static portion (system prompt + few-shot examples) comes first and the dynamic portion (user query, retrieved documents) comes last. Check the cached_tokens field in usage to verify caching is working.

import OpenAI from "openai";

const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

// WRONG: dynamic content before static content  cache miss every time
const badMessages = [
  { role: "system" as const, content: `Today is ${new Date().toISOString()}. ${STATIC_SYSTEM_PROMPT}` },
  { role: "user" as const, content: userQuery },
];

// RIGHT: static prefix first, dynamic suffix last
const goodMessages = [
  {
    role: "system" as const,
    // Static instructions first  this prefix gets cached after the first request
    content: STATIC_SYSTEM_PROMPT, // e.g., 2000-token instructions block
  },
  { role: "user" as const, content: userQuery }, // dynamic content last
];

const response = await client.chat.completions.create({
  model: "gpt-4o",
  messages: goodMessages,
});

// Check cache hit
const cached = (response.usage as any)?.prompt_tokens_details?.cached_tokens ?? 0;
console.log(`Cached tokens: ${cached} / ${response.usage?.prompt_tokens}`);

Step 4: Trim Conversation History with Sliding Window

import { encoding_for_model } from "tiktoken";
import OpenAI from "openai";

type Message = { role: "user" | "assistant" | "system"; content: string };

export function trimHistory(
  messages: Message[],
  model: string,
  maxHistoryTokens = 4000
): Message[] {
  const enc = encoding_for_model(model as any);

  // Always keep the system message
  const [system, ...rest] = messages;
  const trimmed: Message[] = [];
  let tokenCount = 0;

  // Walk backwards from newest to oldest, keeping until budget is full
  for (let i = rest.length - 1; i >= 0; i--) {
    const msgTokens = enc.encode(rest[i].content).length + 4;
    if (tokenCount + msgTokens > maxHistoryTokens) break;
    trimmed.unshift(rest[i]);
    tokenCount += msgTokens;
  }

  enc.free();
  return system ? [system, ...trimmed] : trimmed;
}

Step 5: Right-size max_tokens to Avoid Paying for Unnecessary Output

Without max_tokens, gpt-4o will generate up to 4,096 tokens of output even for tasks that need 50. Set conservative max_tokens per feature based on observed typical output length. For structured JSON output, set it to 150% of the largest expected response.

// Classification task: answer is one of 5 categories
const classifyResponse = await client.chat.completions.create({
  model: "gpt-4o-mini",
  messages,
  max_tokens: 20,       // More than enough for a category label
  temperature: 0,
});

// Structured extraction: known JSON schema with ~10 fields
const extractResponse = await client.chat.completions.create({
  model: "gpt-4o-mini",
  messages,
  max_tokens: 512,      // JSON with 10 fields rarely exceeds 300 tokens
  response_format: { type: "json_object" },
  temperature: 0,
});

// Long-form summary: deliberately generous
const summaryResponse = await client.chat.completions.create({
  model: "gpt-4o",
  messages,
  max_tokens: 1024,     // Cap summaries to keep them tight
});

Production Failure Scenarios

Scenario 1: Embedding Full PDFs Instead of Chunks

A document ingestion pipeline sent each PDF page (avg 800 words, ~1,100 tokens) to text-embedding-3-small one page at a time. At $0.02 per million tokens across 50,000 pages per day, this was $1/day until a customer uploaded a 500-page manual, which hit the 8,191-token limit per embedding request and failed silently. Fix: split documents into 512-token chunks with 50-token overlap before embedding. Smaller chunks also improve retrieval precision.

Scenario 2: System Prompt with Entire Knowledge Base

A developer injected a 15,000-token FAQ document into every system prompt to avoid building a retrieval system. With gpt-4o at $2.50/M input tokens and 50,000 daily requests, the FAQ alone cost $1,875/day in input tokens. Fix: build a vector search index over the FAQ and inject only the top 3 relevant chunks (~500 tokens) per request.

Scenario 3: Using gpt-4o for Boolean Classification

A content moderation pipeline used gpt-4o to classify user messages as safe or unsafe (binary output). The task requires zero reasoning and gpt-4o-mini achieves near-identical accuracy. Switching models reduced classification cost from $2.50 to $0.15 per million input tokens a 16x reduction on the highest-volume call in the system.

Prevention Checklist

• Log prompt_tokens, completion_tokens, and estimated cost on every API call
• Set max_tokens on every request based on the expected output length of that specific task
• Use gpt-4o-mini as the default model and only upgrade to gpt-4o when accuracy requires it
• Structure prompts with static content first to maximize OpenAI's automatic prefix caching
• Implement a conversation history sliding window capped at 4,000–6,000 tokens
• Chunk documents to 512 tokens before embedding; never embed entire pages or files
• Remove preamble from system prompts ('You are a helpful assistant that...') and keep only instructions
• Set up spend alerts in the OpenAI dashboard at $X/day thresholds
• Review the usage.prompt_tokens_details.cached_tokens field to verify caching is working

Resolution Summary

Cost spikes trace back to token bloat, wrong model selection, or absent caching. Fix them in order of impact: first audit your system prompt size and trim it, then add per-feature max_tokens caps, then switch classification and extraction tasks to gpt-4o-mini, then structure prompts for prefix caching. Together these changes typically reduce token spend by 60–85% without measurable quality loss for most features.

Lessons Learned

• The most expensive line in your OpenAI bill is almost always the system prompt multiplied by request count
• gpt-4o-mini handles classification, extraction, and simple Q&A as well as gpt-4o at 1/16 the price
• Prompt caching only works when static content precedes dynamic content in the message array
• max_tokens is not optional without it you pay for output the user never sees
• Logging usage.prompt_tokens per feature is the only way to find cost regressions before they hit the bill

Conclusion

OpenAI API costs are entirely predictable and controllable once you instrument token usage at the feature level. The optimization hierarchy is simple: right-size max_tokens first, then choose the cheapest model that meets your quality bar, then structure prompts for caching, then trim dynamic context to the minimum necessary. Done in that order, most teams can cut their OpenAI spend by more than half without touching their product quality.