AI Agent Tool Calls Randomly Failing in Production

Problem Summary

OpenAI's tool calling (function calling) allows models to request structured function invocations during a completion. When the feature works correctly, the model returns a tool_calls array with valid JSON arguments matching your schema. When it fails, the model either hallucinates a tool name that does not exist, generates malformed JSON arguments, silently ignores tools that should be called, or most dangerously the application receives tool results but never feeds them back to the model, leaving the agent in an infinite loop.

Symptoms

• Model calls a tool with a name that is not in your tools array
• tool_calls[0].function.arguments contains invalid JSON or is truncated mid-string
• Model returns a text response instead of a tool call when a tool call is expected
• Parallel tool calls (multiple tools in one response) fail but sequential calls work
• Tool results are sent back but the model generates another tool call instead of a final answer
• finish_reason is 'length' on a tool call response max_tokens is cutting off the JSON
• Required parameters are missing from tool call arguments despite being defined as required
• Agent loops indefinitely calling the same tool with identical arguments

Business Impact

A broken tool call in an agent pipeline fails silently from the user's perspective the agent appears to 'think' indefinitely, returns a generic error, or produces an answer without performing the required action (e.g., claiming to have sent an email it never sent). In automated workflows where the agent takes real actions (database writes, API calls, order processing), a malformed tool call that your application fails to validate can trigger partial or duplicate operations.

Root Cause

The most common root cause is an invalid or inconsistent JSON schema in the tool definition. OpenAI's tool calling uses JSON Schema to constrain the arguments the model generates. A schema with an incorrect type ('number' where 'string' is expected), a missing 'required' array, or properties that use unsupported JSON Schema keywords causes the model to generate arguments that do not conform to the schema or to skip the tool call entirely because the schema is ambiguous.

Diagnosis

Step 1: Log the Raw Tool Call Response

Before parsing tool call arguments, log the raw response object. This reveals whether the model is returning a tool call at all, whether finish_reason is 'tool_calls' or something else, and whether the arguments JSON is complete.

import OpenAI from "openai";

const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

export async function debugToolCall(
  messages: OpenAI.ChatCompletionMessageParam[],
  tools: OpenAI.ChatCompletionTool[]
) {
  const response = await client.chat.completions.create({
    model: "gpt-4o",
    messages,
    tools,
    tool_choice: "auto",
  });

  const choice = response.choices[0];

  console.log("finish_reason:", choice.finish_reason);
  // Expected: "tool_calls" when model wants to call a tool
  // Unexpected: "stop" (no tool call), "length" (truncated)

  if (choice.finish_reason === "length") {
    console.error("CRITICAL: Response truncated by max_tokens. Tool call JSON may be incomplete.");
  }

  if (choice.message.tool_calls) {
    for (const tc of choice.message.tool_calls) {
      console.log({
        id: tc.id,
        function_name: tc.function.name,
        raw_arguments: tc.function.arguments, // Log BEFORE parsing
        arguments_valid_json: (() => {
          try {
            JSON.parse(tc.function.arguments);
            return true;
          } catch {
            return false;
          }
        })(),
      });
    }
  } else {
    console.warn("No tool calls in response. Model content:", choice.message.content);
  }

  return response;
}

Step 2: Validate Tool Definitions with Zod-to-JSON-Schema

Define tools using Zod schemas and convert them to JSON Schema automatically. This eliminates hand-written JSON Schema errors and keeps your TypeScript types in sync with the tool definition.

import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";
import OpenAI from "openai";

// Define the tool arguments as a Zod schema
const SearchOrdersSchema = z.object({
  customer_id: z.string().describe("The unique customer ID to search orders for"),
  status: z
    .enum(["pending", "shipped", "delivered", "cancelled"])
    .optional()
    .describe("Filter orders by status. Omit to return all statuses."),
  limit: z
    .number()
    .int()
    .min(1)
    .max(50)
    .default(10)
    .describe("Maximum number of orders to return (1-50)"),
});

type SearchOrdersArgs = z.infer<typeof SearchOrdersSchema>;

// Convert to OpenAI tool format
const searchOrdersTool: OpenAI.ChatCompletionTool = {
  type: "function",
  function: {
    name: "search_orders",
    description:
      "Search for customer orders by customer ID and optional status filter. " +
      "Use this when the user asks about order status, history, or specific orders.",
    parameters: zodToJsonSchema(SearchOrdersSchema, { target: "openAi" }),
  },
};

// Parse and validate arguments safely
export function parseToolArgs(rawArgs: string): SearchOrdersArgs {
  let parsed: unknown;
  try {
    parsed = JSON.parse(rawArgs);
  } catch (e) {
    throw new Error(`Tool call arguments are not valid JSON: ${rawArgs}`);
  }

  const result = SearchOrdersSchema.safeParse(parsed);
  if (!result.success) {
    throw new Error(
      `Tool call arguments failed schema validation: ${JSON.stringify(result.error.issues)}`
    );
  }

  return result.data;
}

Step 3: Implement the Full Tool Call Loop Correctly

The most common agent loop bug is failing to append the assistant's tool call message AND the tool result message back to the conversation before the next completion call. Both are required.

import OpenAI from "openai";

const client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

type ToolHandler = (name: string, args: Record<string, unknown>) => Promise<string>;

export async function runAgentLoop(
  initialMessages: OpenAI.ChatCompletionMessageParam[],
  tools: OpenAI.ChatCompletionTool[],
  handleTool: ToolHandler,
  maxIterations = 10
): Promise<string> {
  const messages: OpenAI.ChatCompletionMessageParam[] = [...initialMessages];
  let iterations = 0;

  while (iterations < maxIterations) {
    iterations++;

    const response = await client.chat.completions.create({
      model: "gpt-4o",
      messages,
      tools,
      tool_choice: "auto",
      // Do NOT set max_tokens for agent loops  let the model complete naturally
    });

    const choice = response.choices[0];

    if (choice.finish_reason === "stop") {
      // Model is done  return the final text response
      return choice.message.content ?? "";
    }

    if (choice.finish_reason === "tool_calls") {
      // Step 1: Append the assistant message with tool_calls to history
      messages.push(choice.message);

      // Step 2: Execute each tool call and append results
      const toolResults: OpenAI.ChatCompletionToolMessageParam[] = [];

      for (const tc of choice.message.tool_calls!) {
        let result: string;
        try {
          const args = JSON.parse(tc.function.arguments) as Record<string, unknown>;
          result = await handleTool(tc.function.name, args);
        } catch (err) {
          result = `Error executing tool ${tc.function.name}: ${(err as Error).message}`;
        }

        toolResults.push({
          role: "tool",
          tool_call_id: tc.id, // Must match the tool_call id from the assistant message
          content: result,
        });
      }

      // Step 3: Append ALL tool results before the next completion call
      messages.push(...toolResults);
      continue;
    }

    if (choice.finish_reason === "length") {
      throw new Error("Agent response truncated  increase max_tokens or reduce context");
    }
  }

  throw new Error(`Agent exceeded maximum iterations (${maxIterations})`);
}

Step 4: Handle Parallel Tool Calls Correctly

When gpt-4o returns multiple tool calls in a single response (parallel tool calls), all must be executed and all results must be appended before the next call. Appending only the first result causes an API error because the conversation history references tool_call IDs that have no corresponding tool result.

// WRONG: Only handling the first tool call
if (choice.message.tool_calls) {
  const firstTc = choice.message.tool_calls[0]; // BUG: ignores remaining calls
  const result = await handleTool(firstTc.function.name, JSON.parse(firstTc.function.arguments));
  messages.push({ role: "tool", tool_call_id: firstTc.id, content: result });
}

// RIGHT: Handle ALL tool calls in parallel
if (choice.message.tool_calls) {
  messages.push(choice.message); // Append full assistant message first

  const results = await Promise.all(
    choice.message.tool_calls.map(async (tc) => {
      const args = JSON.parse(tc.function.arguments) as Record<string, unknown>;
      const result = await handleTool(tc.function.name, args);
      return {
        role: "tool" as const,
        tool_call_id: tc.id,
        content: result,
      };
    })
  );

  messages.push(...results); // Append ALL results
}

Step 5: Force Tool Use with tool_choice

If the model is returning text instead of a tool call when you expect one, use tool_choice to force it. For required tools, specify the exact tool name.

// Force the model to call a specific tool
const response = await client.chat.completions.create({
  model: "gpt-4o",
  messages,
  tools,
  tool_choice: {
    type: "function",
    function: { name: "search_orders" }, // Force this specific tool
  },
});

// Force the model to call ANY tool (never return text)
const response2 = await client.chat.completions.create({
  model: "gpt-4o",
  messages,
  tools,
  tool_choice: "required", // Must call at least one tool
});

// Allow the model to decide (default)
const response3 = await client.chat.completions.create({
  model: "gpt-4o",
  messages,
  tools,
  tool_choice: "auto",
});

// Disable all tools for this call
const response4 = await client.chat.completions.create({
  model: "gpt-4o",
  messages,
  tools,
  tool_choice: "none",
});

Production Failure Scenarios

Scenario 1: max_tokens Truncating Tool Call JSON

An agent had max_tokens: 256 set globally. Tool call arguments for a complex search query with 8 parameters generated ~300 tokens of JSON. The response was cut off mid-JSON, JSON.parse threw a SyntaxError, and the agent crashed. Fix: remove max_tokens from agent loops or set it to at least 1,024. Only set max_tokens on leaf completion calls where you control the expected output size.

Scenario 2: Tool Name Mismatch Between Definition and Handler

The tool was defined with name: 'search_orders' but the handler switch statement had case 'searchOrders'. The tool was never executed; the handler fell through to the default case and returned an empty string. The model received an empty tool result and hallucinated an answer. Fix: use the tool name constant in both the definition and handler, or generate handlers from the tool definitions.

Scenario 3: Tool Result Not Returned to Model

A developer executed the tool successfully but forgot to append the tool result to the messages array before the next completion call. The API threw a 400 error: 'An assistant message with tool_calls must be followed by tool messages'. This error was caught and swallowed by a global error handler, so the agent silently returned an empty response. Fix: always handle API errors explicitly in agent loops and log the full error body.

Scenario 4: Model Hallucinating Tool Names

With 15 tools in the tools array, the model occasionally called 'send_email_notification' when the correct tool was 'send_notification_email'. The names were too similar and the model confused them under load. Fix: use distinct, unambiguous tool names. Remove tools from the array that are not relevant to the current task the model performs better with fewer, well-named tools.

Prevention Checklist

• Define tool schemas with Zod and convert to JSON Schema with zod-to-json-schema
• Validate tool call arguments against the schema before executing the handler
• Log finish_reason on every completion and alert if 'length' appears in an agent loop
• Never set max_tokens on agent completion calls let the model stop naturally
• Handle ALL parallel tool calls in the tool_calls array, not just the first
• Always append both the assistant tool call message AND tool result messages before the next completion
• Use distinct, unambiguous tool names avoid names that differ only by word order
• Only include tools relevant to the current task to reduce hallucinated tool names
• Return tool errors back to the model as tool result messages so it can recover
• Implement a max_iterations guard to prevent infinite agent loops

Resolution Summary

Tool calling failures have three failure modes: invalid schema definitions that confuse the model, implementation bugs in the agent loop (missing messages, partial parallel call handling), and configuration errors like max_tokens truncating the JSON. Fix them in that order: validate your JSON schemas with Zod, implement the full message-append loop correctly, and remove max_tokens from agent completion calls. Add finish_reason logging and argument validation on every tool call to catch regressions before they reach users.

Lessons Learned

• Both the assistant tool call message AND tool result must be appended before the next completion
• max_tokens cuts off tool call JSON silently finish_reason: 'length' is the only warning
• Parallel tool calls require all results to be appended, not just the first
• Fewer, well-named tools perform better than many similar-sounding tools
• Tool errors should be returned to the model as tool results so it can self-correct
• Validate tool arguments with Zod before executing to catch schema bugs early

Conclusion

AI agent tool calling is a powerful primitive that, when implemented correctly, enables models to take reliable, structured actions. When it fails, the failures are almost always deterministic bugs in schema definitions, message handling, or configuration not model unpredictability. Instrument every step of the loop, validate every argument, handle every tool call in a parallel batch, and always feed results back to the model. With those patterns in place, tool calling becomes the most reliable component of your agent architecture.