Understand LLM Tool Calling Before You Build an Agent

Send a message to an agent that says "check my calendar and draft a Slack reply," and at least three tool calls fire before a word lands in your Slack channel. Most engineers know the surface: the model calls a function, gets a result, carries on. The machinery underneath is stranger and more consequential than that.

What the model actually does when it calls a tool

A large language model is a text prediction engine. It can plan actions - but it cannot execute anything. This is the foundational constraint that tool calling works around.

When you send a tool schema to an LLM, you're not "registering" a function. You're injecting a JSON blob into the model's context window, formatted as a system message. The model is then fine-tuned to output a special token sequence that signals a tool call. This is why the schema counts against your token budget - it's literally part of the prompt.

The model receives the user prompt and the tool catalog, decides whether a tool is needed, then returns a structured tool call with typed arguments. Your code runs the tool and feeds the result back. That loop - model emits call, your runtime executes it, result goes back as a new message - repeats until the model decides to return a final text answer or call another tool.

Function calling (also called "tool calling" by some providers) is a structured output feature where the LLM returns a JSON object specifying which function to call and what arguments to pass - instead of generating natural language text.

The naming is a minor mess. Function calling is also known as tool calling, and both terms coexist for historic reasons. First, LLMs only had access to a small selection of functions. Later, models became able to handle larger collections of external APIs, hence the name "tool" (as in "toolbox") was established. Today the two terms are largely interchangeable in the OpenAI and Anthropic APIs.

The token cost nobody budgets for

Here is the part most tutorials bury in a footnote.

Under the hood, functions are injected into the system message in a syntax the model has been trained on. This means callable function definitions count against the model's context limit and are billed as input tokens.

Each tool definition adds 100-300 input tokens, so a system with 15 tools adds roughly 1,500-4,500 tokens to every request. At GPT-4o rates, 4,500 extra input tokens per call is not catastrophic on its own - but in an agentic loop that makes 20 round-trips per task, those 15 tool definitions add up to 90,000 tokens of pure overhead across the chain. That is before any user message or retrieved context.

If you run into token limits, limiting the number of tools loaded upfront, shortening descriptions where possible, or using tool search so deferred tools are loaded only when needed are the primary mitigations.

Parallel calls, strict mode, and the trade-off between them

OpenAI ships parallel function calling as a default behavior. When the model determines that multiple functions are needed, it can emit multiple tool-call objects in a single response. The parallel_tool_calls parameter (default true) controls this behavior.

The LLMCompiler paper (ICML 2024) showed that parallel tool calls reduce end-to-end latency by up to 3.7x compared to sequential execution. A concrete example: if a user asks "compare the weather in Tokyo, London, and New York," a model that can parallelize fires three get_weather calls in one response instead of three sequential round-trips.

Most tutorials show one function call. In production, models can emit multiple tool calls in a single turn. If your loop doesn't handle that, you'll silently drop requests and corrupt state.

The other lever is strict mode. Setting strict to true will ensure function calls reliably adhere to the function schema, instead of being best effort. OpenAI recommends always enabling strict mode. Under the hood, strict mode works by leveraging the structured outputs feature. The practical cost: it requires additionalProperties to be set to false for each object in the parameters, and all fields in properties must be marked as required.

There's also a subtler constraint: OpenAI's strict: true mode guarantees schema-valid outputs at the cost of parallel call support. Anthropic takes a different approach - Claude 4 models offer built-in token-efficient tool calling and handle the parallelism differently at the architecture level.

Where tool calling actually breaks in production

A tool call goes wrong four ways: the model invokes the wrong tool; it invokes the right tool with malformed arguments; the tool succeeds but returns an unexpected payload, which the model misinterprets; or the tool fails, returns an error, and the model - trained to be helpful - papers over it with a confident invented answer.

This last failure is the silent one: clean response, successful LLM call in the logs, and only an integration test comparing the model's claim to the tool's actual return reveals the lie.

Tool argument spoofing is a recognized hallucination sub-type: the model invents arguments, IDs, or natural language descriptions when calling a tool, leading to silent corruption inside an agent loop.

The JSON schema you define is not just documentation - it's the only thing the model sees. If your descriptions are vague, the model will hallucinate arguments. The description field on each parameter does real work. "A city name" produces different behavior than "A city name, e.g. 'Tokyo' or 'London', never a country name or abbreviation."

Schema drift is a slower failure. Fields rename, enums add values, required becomes optional. An agent prompted on the old schema keeps calling tools the old way, and modern API gateways forgive it by silently coercing or ignoring fields. The agent looks correct; the system is wrong.

Tool calling reliability varies by provider. Testing shows correct tool selection at 90-95% for some providers, lower than OpenAI (97-99%) and Claude (96-99%). The main failure mode is argument hallucination - the model generates plausible but incorrect parameter values.

A concrete walk-through pulls the whole loop together. Imagine a Slack bot that can query your company's order system. The schema registers one tool: get_order_status(order_id: string). The user types: "where's order ORD-4821?"

1. The API call includes the user message and the tool schema, injected into the system prompt.
2. The model sees the schema, recognizes the intent, and responds with {"name": "get_order_status", "arguments": {"order_id": "ORD-4821"}} - no plain text, just that JSON.
3. Your runtime executes get_order_status("ORD-4821"), gets back {"status": "in transit", "eta": "Aug 2"}.
4. That result is added to the message history as a tool role message and sent back to the model.
5. The model reads the result and writes the final reply: "Order ORD-4821 is in transit - expected August 2."

When the model calls a function, you must execute it and return the result. Since model responses can include zero, one, or multiple calls, it is best practice to assume there are several.

LLM tool calling: common questions

What is the difference between tool calling and function calling?

Effectively nothing today. OpenAI originally called the feature "function calling" when it launched in June 2023 and later renamed the API parameter to tools. Anthropic has always called it "tool use." The underlying mechanism - schema in, JSON call out, result back in - is identical across providers.

Does the tool schema cost tokens on every call?

Yes. Functions are injected into the system message in a syntax the model has been trained on. Callable function definitions count against the model's context limit and are billed as input tokens. With a large tool catalog, this adds up fast across multi-turn agent loops.

Can the model run a tool itself, or does my code have to do it?

Your code always does the execution. A large language model is just a text prediction engine. It cannot execute anything. If you tell it to "send an email," it can generate a perfect JSON body for that email, but it won't actually send it. The model decides what to call; your runtime decides whether and how to run it.

What is parallel tool calling and when should I use it?

Parallel tool calling lets the model emit multiple tool-call objects in a single response, which your runtime then executes concurrently. Use it when tasks are independent - fetching data from three APIs simultaneously is a good fit. Avoid it if you need strict output validation via OpenAI's strict mode, since the two features conflict.

What causes argument hallucination in tool calls?

Argument hallucination (also called "tool argument spoofing") occurs when the model invents arguments, IDs, or descriptions when calling a tool, leading to silent corruption inside an agent loop. Vague parameter descriptions and missing enum constraints are the most common triggers. The fix is tighter schemas, strict mode where supported, and integration tests that compare the model's tool arguments to expected values - not just the final text output.