RAG System Returning Incorrect AI Responses

Problem Summary

Retrieval-Augmented Generation (RAG) systems fail in a distinct pattern: the model gives a confident, fluent answer that is factually wrong or irrelevant to the user's question. This is rarely a model problem gpt-4o faithfully synthesizes whatever context it receives. The failure is in the retrieval pipeline: wrong chunks are retrieved, too few chunks are retrieved, or so many chunks are injected that the relevant signal is buried in noise. Diagnosing RAG failures requires instrumenting each stage of the pipeline independently.

Symptoms

• Model returns answers that contradict the source documents
• Model says 'I don't have information about that' when the answer exists in the knowledge base
• Retrieved chunks are semantically related but contextually wrong (e.g., retrieves pricing from the wrong product tier)
• Answers are correct for simple queries but wrong for multi-hop questions requiring two related chunks
• Quality degraded after re-embedding the knowledge base with a different model
• Adding more documents to the knowledge base made answers worse, not better
• Retrieval returns the same chunk repeatedly for different queries
• Long documents always return their first chunk regardless of query relevance

Business Impact

A RAG system that returns wrong answers is worse than no AI at all. Users who trust the system and act on incorrect information misquoted prices, wrong compliance requirements, incorrect technical specifications generate expensive support escalations and erode product trust. For regulated industries (legal, healthcare, finance), incorrect AI-generated answers sourced from authoritative documents can create liability.

Root Cause

RAG failures cluster into three root causes. First: chunking strategy is wrong chunks are too large (embedding averages semantics, reducing precision), too small (chunks lack the context needed to be meaningful), or split at arbitrary character boundaries that break sentences. Second: embedding model mismatch documents were embedded with text-embedding-ada-002 but queries are embedded with text-embedding-3-small, making cosine similarity scores meaningless. Third: similarity threshold is too low irrelevant chunks pass the retrieval filter and are injected into the context, confusing the model.

Diagnosis

Step 1: Log Retrieved Chunks and Similarity Scores

Before debugging anything else, log the actual chunks being retrieved and their cosine similarity scores. This single step reveals whether the retrieval problem is 'retrieving wrong chunks' or 'model ignoring correct chunks'.

import OpenAI from "openai";
import { Pool } from "pg";

const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const db = new Pool({ connectionString: process.env.DATABASE_URL });

export async function debugRetrieval(query: string, topK = 5) {
  // Embed the query
  const embeddingResponse = await client.embeddings.create({
    model: "text-embedding-3-small",
    input: query,
  });
  const queryVector = embeddingResponse.data[0].embedding;

  // Retrieve with similarity score
  const result = await db.query(
    `SELECT
       id,
       content,
       metadata,
       1 - (embedding <=> $1::vector) AS cosine_similarity
     FROM document_chunks
     ORDER BY embedding <=> $1::vector
     LIMIT $2`,
    [`[${queryVector.join(",")}]`, topK]
  );

  console.log("\n=== Retrieved Chunks for Query:", query, "===");
  for (const row of result.rows) {
    console.log({
      id: row.id,
      similarity: row.cosine_similarity.toFixed(4),
      preview: row.content.slice(0, 200),
      metadata: row.metadata,
    });
  }

  // Flag potentially bad retrieval
  const avgSimilarity =
    result.rows.reduce((sum: number, r: any) => sum + parseFloat(r.cosine_similarity), 0) /
    result.rows.length;
  if (avgSimilarity < 0.75) {
    console.warn(`Low average similarity: ${avgSimilarity.toFixed(3)}  chunks may be irrelevant`);
  }

  return result.rows;
}

Step 2: Validate Chunk Size Strategy

The optimal chunk size for most RAG applications is 512 tokens with 50–100 token overlap between adjacent chunks. Chunks under 100 tokens lack context. Chunks over 1,000 tokens dilute the embedding's semantic signal.

import { encoding_for_model } from "tiktoken";

export function chunkDocument(
  text: string,
  chunkSize = 512,
  overlapSize = 64
): string[] {
  const enc = encoding_for_model("gpt-4o");
  const tokens = enc.encode(text);
  const chunks: string[] = [];
  const decoder = new TextDecoder();

  let start = 0;
  while (start < tokens.length) {
    const end = Math.min(start + chunkSize, tokens.length);
    const chunkTokens = tokens.slice(start, end);

    // Decode tokens back to text
    const chunkText = decoder.decode(enc.decode(chunkTokens));
    chunks.push(chunkText.trim());

    if (end >= tokens.length) break;
    start = end - overlapSize; // Move back by overlap to maintain context
  }

  enc.free();
  return chunks;
}

// Prefer splitting on paragraph boundaries when possible
export function semanticChunk(text: string, maxTokens = 512): string[] {
  const enc = encoding_for_model("gpt-4o");
  const paragraphs = text.split(/\n\n+/);
  const chunks: string[] = [];
  let currentChunk = "";
  let currentTokens = 0;

  for (const para of paragraphs) {
    const paraTokens = enc.encode(para).length;
    if (currentTokens + paraTokens > maxTokens && currentChunk) {
      chunks.push(currentChunk.trim());
      currentChunk = para;
      currentTokens = paraTokens;
    } else {
      currentChunk += (currentChunk ? "\n\n" : "") + para;
      currentTokens += paraTokens;
    }
  }

  if (currentChunk) chunks.push(currentChunk.trim());
  enc.free();
  return chunks;
}

Step 3: Verify Embedding Model Consistency

import { Pool } from "pg";

const db = new Pool({ connectionString: process.env.DATABASE_URL });

export async function verifyEmbeddingConsistency() {
  // Check what embedding model was used for stored documents
  const result = await db.query(`
    SELECT
      metadata->>'embedding_model' AS model,
      COUNT(*) AS chunk_count,
      MIN(created_at) AS oldest,
      MAX(created_at) AS newest
    FROM document_chunks
    GROUP BY metadata->>'embedding_model'
    ORDER BY chunk_count DESC
  `);

  console.log("Embedding model distribution:");
  console.table(result.rows);

  if (result.rows.length > 1) {
    console.error(
      "CRITICAL: Multiple embedding models detected in the index! " +
      "Re-embed all documents with a single model before querying."
    );
  }
}

// Always store the embedding model in chunk metadata
export async function embedAndStore(
  text: string,
  documentId: string,
  chunkIndex: number
) {
  const EMBEDDING_MODEL = "text-embedding-3-small";

  const response = await openAIClient.embeddings.create({
    model: EMBEDDING_MODEL,
    input: text,
  });

  await db.query(
    "INSERT INTO document_chunks (content, embedding, metadata) VALUES ($1, $2, $3)",
    [
      text,
      `[${response.data[0].embedding.join(",")}]`,
      JSON.stringify({
        document_id: documentId,
        chunk_index: chunkIndex,
        embedding_model: EMBEDDING_MODEL, // Always record the model
        embedded_at: new Date().toISOString(),
      }),
    ]
  );
}

Step 4: Tune the Similarity Threshold

A cosine similarity threshold below 0.75 typically retrieves noise. Raise the threshold incrementally and measure the recall/precision tradeoff on a test set of known queries.

export async function retrieveWithThreshold(
  queryEmbedding: number[],
  topK = 5,
  minSimilarity = 0.78  // Do not retrieve chunks below this score
) {
  const result = await db.query(
    `SELECT
       content,
       metadata,
       1 - (embedding <=> $1::vector) AS cosine_similarity
     FROM document_chunks
     WHERE 1 - (embedding <=> $1::vector) >= $3
     ORDER BY embedding <=> $1::vector
     LIMIT $2`,
    [`[${queryEmbedding.join(",")}]`, topK, minSimilarity]
  );

  if (result.rows.length === 0) {
    // No chunks passed the threshold  signal to the model explicitly
    return { chunks: [], message: "No relevant documents found above similarity threshold" };
  }

  return { chunks: result.rows };
}

Step 5: Check for Context Window Overflow

Injecting 10 chunks of 512 tokens each adds 5,120 tokens to the prompt. Combined with the system prompt, conversation history, and user query, this can push the total over gpt-4o's working context limit. When the context is too long, the model truncates or ignores tail content often the most recently retrieved chunks.

import { encoding_for_model } from "tiktoken";

export function assembleContext(
  chunks: { content: string; cosine_similarity: number }[],
  maxContextTokens = 3000
): string {
  const enc = encoding_for_model("gpt-4o");

  // Sort by relevance  highest similarity first
  const sorted = [...chunks].sort((a, b) => b.cosine_similarity - a.cosine_similarity);

  let context = "";
  let tokenCount = 0;

  for (const chunk of sorted) {
    const chunkTokens = enc.encode(chunk.content).length;
    if (tokenCount + chunkTokens > maxContextTokens) {
      console.warn(
        `Stopping context assembly at ${tokenCount} tokens  ${sorted.length - sorted.indexOf(chunk)} chunks omitted`
      );
      break;
    }
    context += chunk.content + "\n\n---\n\n";
    tokenCount += chunkTokens;
  }

  enc.free();
  return context.trim();
}

Production Failure Scenarios

Scenario 1: Re-embedding with a Different Model After Schema Migration

A team migrated from text-embedding-ada-002 (1,536 dimensions) to text-embedding-3-small (1,536 dimensions by default, but configurable). They updated the query embedding but forgot to re-embed existing documents. Both models output 1,536-dimensional vectors so there were no schema errors but similarity scores were wrong. Queries returned the highest-scoring noise. Fix: store the embedding model in chunk metadata and assert consistency at query time.

Scenario 2: Retrieval k Too Small for Multi-Hop Questions

A question like 'What is the refund policy for Enterprise plan customers?' requires one chunk about the Enterprise plan and one chunk about the refund policy. With k=3, only the top 3 chunks are retrieved often not both required chunks. Fix: increase k to 8–10, then filter by the similarity threshold to remove noise before assembling context.

Scenario 3: PDF Parsed Without Structure Preservation

A PDF parser extracted text by reading characters left-to-right across columns, merging unrelated table cells and paragraph text into nonsensical strings. When chunked, the resulting chunks contained semantically incoherent text that embedded to meaningless vectors. Fix: use a structure-aware PDF parser (pdfplumber, Adobe PDF Extract API) that preserves headings, tables, and reading order.

Scenario 4: Model Ignoring Retrieved Context Due to Instruction Conflict

A system prompt said 'Answer from your general knowledge' but the RAG prompt template said 'Use only the provided context'. The contradicting instructions caused gpt-4o to default to its training knowledge and ignore the retrieved chunks entirely. Fix: make the RAG instruction authoritative and unambiguous: 'Answer ONLY using the context provided below. If the answer is not in the context, say so explicitly.'

Prevention Checklist

• Log retrieved chunks and similarity scores for every query in development
• Store the embedding model name in chunk metadata and assert consistency at query time
• Use semantic paragraph-boundary chunking with 512-token target and 64-token overlap
• Set a minimum cosine similarity threshold of 0.75–0.80 and return a 'not found' signal below it
• Retrieve k=8–10 chunks but assemble context from only the top N that fit within the token budget
• Re-embed the entire knowledge base whenever you change the embedding model
• Use unambiguous RAG instructions: 'Answer ONLY using the context below'
• Add a citation requirement: ask the model to quote the source chunk it used
• Validate PDF and document parsing output quality before ingesting into the vector index

Resolution Summary

RAG failures are retrieval failures. Fix them by instrumenting the retrieval pipeline with similarity score logging, enforcing embedding model consistency, tuning chunk size to 512 tokens with overlap, raising the similarity threshold, and assembling context with a token budget cap. Once retrieval quality is confirmed by inspecting logged chunks, model accuracy follows automatically the LLM is rarely the problem.

Lessons Learned

• Log retrieved chunks in production you cannot debug RAG without seeing what the model sees
• Embedding model mismatches produce plausible-looking cosine scores that are completely wrong
• Chunk size is the single most impactful parameter for retrieval quality
• A RAG system with no similarity threshold retrieves noise confidently
• The model follows its retrieval context faithfully garbage in, garbage out
• PDF parsing quality determines the ceiling of your RAG system's accuracy

Conclusion

A RAG system is only as good as its retrieval pipeline. When the model returns wrong answers, resist the impulse to switch to a more powerful model or add more context start by logging what is actually being retrieved and why. Most RAG failures resolve at the chunking, embedding consistency, or similarity threshold layer, not at the generation layer. Invest in retrieval observability first, and the model will take care of the rest.