LLM Tool Calling Doesn't Call Anything - Your Code Does

The most common misconception about tool calling is baked into the name. LLMs do not call functions. Regardless of what you have been told, they simply don't. If your LLM API supports function calling, there is a layer of software wrapped around the model that takes care of invoking the actual function. The model produces a JSON blob and stops. Everything else is your orchestration layer's problem.

That gap - between what the model emits and what actually runs - is where most real-world tool-calling bugs live.

What LLM tool calling actually is

LLM tool calling is a mechanism that allows an AI model to generate structured requests - typically in JSON - to invoke external functions or APIs. Instead of the model guessing information it doesn't have, it recognizes a gap in its capability and requests to use a specific tool to bridge it.

The way this works under the hood: an LLM analyzes a natural language input, extracts the user's intent, and generates a structured output containing the function name and the necessary arguments to invoke that function. It's important to emphasize that the LLM itself does not execute the function. Instead, it identifies the appropriate function, gathers all required parameters, and provides the information in a structured JSON format. That JSON output can then be deserialized into a function call in Python (or any other programming language) and executed within the program's runtime environment.

A concrete walk-through:

1. User asks: "What's our current churn rate?"
2. The model sees a tool definition for query_metrics(metric_name: string, date_range: string) in its context
3. It outputs: {"tool": "query_metrics", "arguments": {"metric_name": "churn_rate", "date_range": "last_30_days"}}
4. Your application reads that blob, calls your metrics API, gets back 4.2%
5. That result goes back into the conversation as a new message
6. The model reads the result and composes a plain-English answer

The model never touched the metrics API. It wrote an address on an envelope; your code delivered the letter.

The "function calling vs tool calling" naming mess

OpenAI calls it "function calling" while Anthropic calls it "tool use," but the implementation is nearly identical. Both use JSON schemas to define tools and return structured outputs.

The distinction that does matter: function calling originally referred to matching a specific JSON signature, while tool calling builds upon this idea and supports a wider range of capabilities including provider-built tools, such as code interpreters, web browsing, and RAG.

So functions are a subset. A "tool" can be a custom function you wrote, but it can also be a hosted capability - like OpenAI's built-in web search - where the provider handles execution entirely. In practice most engineering teams mean "tool calling" when they say either term.

The reliability improvement over the years is dramatic. Function calling in older approaches achieved roughly 86% schema compliance. Structured outputs achieve 100% compliance. OpenAI reached that ceiling using constrained decoding: they took a deterministic, engineering-based approach to constrain the model's outputs, based on a technique known as constrained sampling or constrained decoding. By default, when models are sampled to produce outputs, they are entirely unconstrained and can select any token from the vocabulary as the next output. Constrained decoding eliminates that freedom specifically for the JSON structure - the model is still free to reason about which tool to call and what arguments to fill, but the brackets and field names come out clean.

A model that scores at the 90th percentile on MT-Bench can still fail 12% of the time on complex nested tool schemas. MMLU P90 doesn't tell you that the model produces 12% malformed JSON on complex multi-tool schemas. Run BFCL evals against your specific tool schemas before picking a model for an agent workload.

The hidden cost: tool definitions eat tokens too

Here's what most docs don't emphasize: tool definitions count as input tokens. If you define 20 complex tools with strict JSON schemas, descriptions, and parameter types, that might be 2,000 tokens of overhead. If you send that overhead with every single message, and the user just says "Hello," you are paying a premium tax on a basic greeting.

This matters more than it sounds. Consider the math: an agent handling 50,000 conversations a day, each averaging 10 turns, with a 20-tool catalog adds up to 1 billion extra input tokens per day from tool definitions alone - before any actual work gets done. At even a cheap model's rates, that's real money.

The practical implication: don't dump your entire tool catalog into every request. Load tools contextually - route the conversation to a smaller, relevant set first, then expose only those tools. Think of it as lazy loading for your agent's capabilities.

Structured function calling uses a dedicated API mechanism where the model returns validated JSON matching the tool schema. Schema validation happens at the API level, not via text parsing - and it's far more reliable for production use. But that validation requires the schema to be in the context window, which is why schema size discipline matters.

Parallel tool calls: the performance gain with a catch

Tool execution accounts for 35-60% of total agent latency depending on the workload - coding tasks sit at the high end, deep research tasks in the middle. Running independent calls simultaneously is the obvious optimization.

GPT-4o, Claude 3.5, and Gemini support parallel function calling. If you ask "What's the weather in London and New York?", the LLM can return two function calls in one response. This reduces latency - both calls execute in parallel instead of sequentially.

The non-obvious problem: tools that work reliably in sequential order silently break when they run concurrently. The behavior that was stable turns unpredictable, and often the failure produces no error - just a wrong answer returned with full confidence. Parallel tool calling is not primarily a performance feature. It is an involuntary architectural audit.

When you switch a working sequential agent to parallel execution, any hidden state dependency between tools becomes a bug. Tool A updates a record while Tool B reads it - in sequential order, fine; in parallel, you get a race. The model can't see that dependency unless you tell it explicitly.

The execution mechanics are simple: when the model emits N tool calls in a single turn, your runner dispatches all N simultaneously, waits for all to complete, and returns the full batch of results before continuing inference. The complexity lives in your tool design, not the model.

Speculative execution research published in March 2026 takes this further: PASTE (Pattern-Aware Speculative Tool Execution) from Shanghai Jiao Tong University and Microsoft Research engineered parallel execution for production deployment. Its key insight is that agent tool-call sequences are not random - they follow recurring patterns. An agent that looks up a user's profile will almost always check their permissions next. A coding agent that reads a file will usually edit it afterward. By pre-executing predicted next calls while the model reasons, PASTE achieved a 48.5% average reduction in task completion time and a 1.8x improvement in tool execution throughput.

LLM tool calling: common questions

What is the difference between function calling and tool calling in LLMs?

The terms are nearly synonymous today. Tool calling is the more general and modern term, referring to a broader set of capabilities that LLMs can use to interact with the outside world - including calling custom functions, built-in code interpreters, and retrieval mechanisms for accessing data from uploaded files or connected databases. Function calling was the original narrower term from OpenAI's first implementation.

Does the model actually execute the function, or does my code?

Your code. The model outputs a JSON object describing which function to call and with what arguments. When using function calling, the model itself does not run the functions and interact with external systems. Instead, it generates parameters for potential function calls. Your application then decides how to handle these parameters, maintaining full control over whether to call the suggested function or take another action.

How do tool definitions affect my token costs?

Every tool definition you include gets tokenized as part of the input. Tool definitions count as input tokens. If you define 20 complex tools with strict JSON schemas, descriptions, and parameter types, that might be 2,000 tokens of overhead. Load only the tools relevant to the current context, not your entire catalog every turn.

What's the risk of a model calling the wrong tool or hallucinating arguments?

It's real and measurable. LLMs will hallucinate arguments for function calls, and this also applies to the arguments it specifies for the parameters of the function call. Code accordingly. The failure mode is usually a wrong parameter value rather than a wrong tool - validate inputs server-side before executing anything with side effects.

When should I use parallel tool calls?

Sequential execution is the safe default, but it has a cost. When tools don't depend on each other's outputs, serializing them is pure latency with no benefit - so you can call tools in parallel. The test is simple: if Tool B needs Tool A's output, they're sequential. If both can run on what's already known, fire them together. Start sequential, profile your latency, then parallelize only the tools you've verified are independent.