The model does not call your function. It never touches your database, fires your API, or runs a line of code. What it does is write a small JSON note - name, arguments - and stop. Your application reads that note and decides what to do next.
That distinction matters more than it sounds, and it explains most of the surprising failures people hit when they build their first tool-calling agent.
OpenAI originally shipped the feature in June 2023 under the name "function calling," using a functions parameter in the Chat Completions API.
The terminology then shifted:
in late 2023, OpenAI renamed it to use tools and tool_choice to accommodate a broader range of capability types beyond custom functions - web search, file search, computer use, and MCP server connections.
Today the two terms coexist.
"Tool calling" is the more general and modern term; it refers to a broader set of capabilities that LLMs can use to interact with the outside world.
For this post the terms are interchangeable.
What the model actually sees
Before the model can decide to use a tool, it has to know the tool exists. When you send a tool schema to an LLM, you are not "registering" a function - you are injecting a JSON blob into the model's context window, formatted as a system message.
Under the hood, functions are injected into the system message in a syntax the model has been trained on, which means callable function definitions count against the model's context limit and are billed as input tokens.
So if you define five tools with verbose descriptions, those definitions eat into the same budget as your user's message and the conversation history. This is not a footnote - every tool definition is part of your prompt and consumes input tokens on every request, and the more tools you give the model, the more likely it is to confuse similar-sounding ones.
The schema itself is what the model reasons from. The JSON schema you define is not just documentation - it is the only thing the model sees. If your descriptions are vague, the model will hallucinate arguments.
The decision loop, step by step
Once the model has the schema in context, here is what happens on a single turn:
- The user sends a message - "What's the current inventory for SKU-4892?"
The model's first job is to decide whether a tool is needed and, if so, which one. That decision is still probabilistic; you shape it with prompts, schema, and tool_choice, but you don't program it like a deterministic rules engine.
3. The model, having been fine-tuned to do so, outputs a special token sequence that signals a tool call.
It generates a tool_calls field in the response; under the hood, the model outputs a JSON string inside the arguments field.
4. The LLM itself does not execute the function. Instead, it identifies the appropriate function, gathers all required parameters, and provides the information in a structured JSON format. That JSON output can then be deserialized into a function call in Python or any other language and executed within the program's runtime environment.
- Your application runs the actual lookup, gets a result, and sends that result back to the model as a new message with role
tool.
The LLM then returns a response by either specifying another function to call or returning a response that can be sent to the user.
This loop repeats - the model can call multiple tools in sequence or even plan a chain of tool calls - until it decides no further tools are needed.
How the JSON stays valid: constrained decoding
Here is where it gets mechanically interesting. Left to its own devices, a language model can generate malformed JSON - an unescaped newline, a missing bracket, a field name it invented. The standard approach to forcing valid outputs is constrained sampling, or constrained decoding. By default, when models are sampled to produce outputs, they are entirely unconstrained and can select any token from the vocabulary. This flexibility is what allows models to make mistakes - they are free to sample a curly brace token at any time, even when that would not produce valid JSON. Constrained decoding forces the model to only select tokens that would be valid according to the supplied schema.
In practice: when an LLM generates text, it predicts the probability of each possible next token. With constrained decoding, those logits are modified in real time to remove any tokens that would violate the defined structure. The model can then only generate valid continuations, guaranteeing the final output always follows the schema.
In August 2024, OpenAI launched Structured Outputs with Strict Mode, using constrained decoding to guarantee schema compliance.
Setting strict to true ensures function calls reliably adhere to the function schema instead of being best effort. OpenAI recommends always enabling strict mode.
Anthropic's Claude works slightly differently at the wire level.
Claude outputs a special XML-like tag <function_calls> and then a JSON block
, rather than OpenAI's tool_calls field - same idea, different syntax. Both providers, along with Google,
now support native structured output
, and the ecosystem has largely converged on constrained decoding as the reliable production approach.
You still need semantic validation - a syntactically valid JSON string can still contain a wrong order ID or a hallucinated confidence score. The schema is a floor, not a ceiling.
What can go wrong, and why tool descriptions are load-bearing
Two classes of failure show up in production more than any other.
The first is argument hallucination.
Function calling is not a built-in ability of the LLM in a hardware sense. It is the result of fine-tuning on synthetic examples where the correct answer is JSON, not text. Providers generate thousands of examples of prompt → tool call with chain-of-thought reasoning, where the model learns not just to generate JSON but to justify why a specific tool is needed in a specific context.
When the description you write does not match the examples the model trained on, it extrapolates - sometimes badly. Vague parameter names like date instead of date_iso8601_utc invite wrong formats.
The second is over-calling. A recent paper from arXiv found that while tool calling helps in aggregate, optimal tool calling can achieve much better performance than always calling tools, with significantly fewer calls - and self-decisions made by models today are far from optimal in terms of accuracy. In other words, models sometimes reach for a tool when they could answer from context, which wastes latency and tokens.
Most tutorials show one function call. In production, models can emit multiple tool calls in a single turn. If your loop doesn't handle that, you'll silently drop requests and corrupt state. A teammate like Beagle running inside Slack encounters this every time a single user message requires checking a calendar, a project tracker, and a document store simultaneously - three parallel tool calls, not three sequential ones.
For large tool catalogs, the token cost of loading every schema upfront becomes a real constraint. Anthropic's tool search cuts tool-definition tokens by roughly 85%, scaling to thousands of registered functions.
OpenAI added tool search with GPT-5.4. Google supports hundreds of tools per request natively. For fewer than 20 tools, passing them all directly is fine. Beyond that, tool search becomes essential to avoid wasting context tokens on irrelevant schemas.
The one-sentence version
Tool calling is a trained behavior, not a runtime capability: the model produces a structured JSON description of an action it wants taken, your code executes that action, and the result goes back into the conversation so the model can decide what to do next. The model is the planner; your application is the executor. Every agent in production, from the simplest Slack bot to a multi-step coding assistant, is just an implementation of that loop.
The quality of your tool descriptions, the strictness of your schema validation, and how carefully you handle multi-call responses determine whether that loop is reliable or erratic. The model's side of this bargain has improved dramatically since June 2023. The application-side discipline is still mostly up to you.