Read the LLM Tool Calling Loop Before You Ship an Agent

Every tool schema you pass to a model costs tokens-every single request. A realistic 5-tool setup adds roughly 346 tokens of overhead per call on average, and that number multiplies across every step an agent takes. Most engineers building on top of function calling have no idea the schema is literally injected into the context window each time, because the API makes it look like a configuration option. It is not. It is part of the prompt.

This post walks through exactly what happens when an LLM calls a tool: the wire format, the decision mechanism, the decoding trick that enforces JSON, and the parallel-call wrinkle that breaks naive agent loops. No hand-waving. Concrete examples throughout.

What LLM tool calling actually is

Rather than executing code directly, the LLM outputs a JSON object describing which function to call and what arguments to pass. Your application runs that function, then sends the result back to the model so it can finish answering. That round-trip is the tool calling loop.

In a regular API call, your code decides which endpoint to hit deterministically. With function calling, the LLM decides which function to invoke based on natural language input and outputs structured JSON, while your code handles execution.

The reason this matters: without a way to interact with the world, LLMs are essentially locked behind a glass wall-they have enough knowledge to explain a refund policy in perfect detail but lack the hands to actually trigger one. For developers, this disconnect between reasoning and action is what separates sophisticated chatbots from production-grade agents.

Here is the five-step loop in plain terms:

1. You send the user's message plus the tool schema(s) to the API
2. The model reads both and decides whether it needs a tool
3. If yes, it returns a tool_calls object (not a text reply) with the function name and arguments as JSON
4. Your code runs the function and gets the result
5. You send the result back; the model does a final pass and writes the actual reply

With the new data in its context window, the LLM performs a final inference pass. It uses the tool output to answer the user's original query or determines if a second, sequential tool call is necessary to finish the job.

The part the API hides: schemas are tokens in your prompt

This is the thing most documentation glosses over. When you pass a tools array to the API, you are not registering a function with the model. You're injecting a JSON blob into the model's context window, formatted as a system message. The model is then fine-tuned to output a special token sequence that signals a tool call. This is why the schema counts against your token budget - it's literally part of the prompt.

Tool definitions (function schemas, MCP servers, etc.) add tokens to the context. The cost is not trivial. A single tool definition in an MCP schema typically runs between 100 and 500 tokens, depending on how verbose the descriptions are. A server with 10 well-documented tools could cost 1,500-3,000 tokens per turn.

Real measured numbers: heavy tools each cost ~1,000 tokens to define; light tools cost ~100 tokens. The 10× delta is entirely in JSON Schema size - parameter descriptions, type definitions, and nested object structures.

The compounding problem is in agentic loops. In a heavy setup with 10 MCP servers × 5 tools each × 200 tokens average = 10,000 tokens. Tool definitions reload every step. 10,000 tokens × 15 steps = 150,000 tokens just for tool definitions. Before any actual work is done.

How the model decides to call a tool (and how JSON is enforced)

Two questions people conflate: how does the model choose whether to call a tool, and how does it produce valid JSON when it does?

The choice is learned. The model was fine-tuned with examples of structured calling. It uses tool schemas like JSON Schema to validate or infer argument types. The schema description you write is the primary signal - it tells the model when to reach for that tool. The JSON schema you define is not just documentation - it's the only thing the model sees. If your descriptions are vague, the model will hallucinate arguments.

The JSON validity is not a matter of prompting. It is enforced at the decoding layer through a technique called constrained decoding. Constrained decoding is the inference-time technique that forces an LLM's output to conform to a grammar, schema, or regex by masking the next-token distribution at every step. Tokens that would make the partial output invalid are set to logit negative infinity before sampling, leaving only legal continuations. Because the constraint is enforced during generation rather than after, the output is guaranteed valid - you never get a parse error, never have to retry, never need a fallback parser.

The mechanism in detail: constrained decoding ensures that every token an LLM generates follows predefined grammar rules, similar to how a compiler checks code validity. The grammar guides generation by producing a per-token mask: valid tokens are kept, while invalid ones are set to −∞ logits and excluded from sampling.

The decoder maintains a state machine (a regex compiled to DFA, a grammar compiled to a pushdown automaton, or a JSON-schema walker). At each step, the state machine returns the set of token IDs that can legally follow the current state. The decoder masks all other tokens to logit -infinity before sampling.

This is why strict: true in OpenAI's function calling API is significant. OpenAI's structured outputs strict mode guarantees the model's output JSON exactly matches the declared schema - no extra fields, no missing required fields, no type mismatches. This eliminates the entire category of argument-construction errors at the schema-validation step, at the cost of restricting the schema to a subset of JSON Schema (no anyOf, limited $ref use).

Anthropic's tool use also validates argument structure against the schema before returning the tool_use block. Both guarantee that if the model calls a tool, the arguments will be syntactically valid - semantic correctness (is the query actually useful?) remains the model's responsibility.

Parallel tool calls and the loop that silently breaks

A single user question can trigger multiple tool calls in one model turn. This is parallel tool calling, and it is a production failure mode that naive agent implementations miss.

In production, models can emit multiple tool calls in a single turn. If your loop doesn't handle that, you'll silently drop requests and corrupt state.

The model's returned message contains a tool_calls array, with each invocation including a unique id, tool name, and arguments JSON string. OpenAI's core advantage is native support for parallel function calling - the model can issue multiple tool invocation requests simultaneously in a single response, with the application layer executing them in parallel and returning all results at once.

The tool_choice parameter provides fine-grained control: "auto" lets the model decide autonomously, "required" forces the model to invoke a tool, "none" prohibits tool invocation, or a specific tool name can be specified to force invocation.

One concrete incompatibility worth knowing: Structured Outputs is not compatible with parallel function calls. When a parallel function call is generated, it may not match supplied schemas. Set parallel_tool_calls: false to disable parallel function calling when you need strict schema enforcement on every argument simultaneously.

The wire format looks like this when a parallel call fires. The content field is empty - the model is not generating a reply, it is requesting two actions. When the model wants to call a tool, content is empty. The model isn't talking to the user - it's requesting an action. The finish_reason is "tool_calls" instead of the usual "stop".

Tool calls are returned in choices[].message.tool_calls[] with each item containing a function.name and JSON-stringified function.arguments. After executing a tool, append a new message with role: tool, the matching tool_call_id, and the tool result in content.

Keeping tool overhead in check

Once you understand that schemas are tokens billed per request, the optimization options become obvious:

• Write tight descriptions. A description should answer: "When would the model choose this tool over other tools?" Everything beyond that is overhead. Keep descriptions under 15 words wherever the function name is already self-explanatory.

• Use tool search for large sets. If your application has many functions or large schemas, you can pair function calling with tool search to defer rarely used tools and load them only when the model needs them. Only gpt-5.4 and later models support tool_search.

• Put schemas first in your prompt. Caching kicks in on prompts above 1,024 tokens and applies to the longest shared prefix. Put your stable system prompt and tools definition first and the user-specific content last. That way prompt caching absorbs most of the schema cost on repeated calls.

• Enable strict: true. Adding strict: true to your function definition forces the model to generate arguments that exactly match your parameter schema. Without strict mode, the model occasionally produces arguments with wrong types or missing required fields (2-5% of calls in testing).

• Handle parallel calls explicitly. Check the length of tool_calls before assuming there's one. Execute them concurrently, collect all results, and return all tool messages in a single follow-up before the next model turn.

LLM tool calling: common questions

What is LLM tool calling and how does it differ from regular function calls?

LLM tool calling is a protocol where the model outputs structured JSON describing which function to run and with what arguments, rather than running the function itself. Your application executes the function and sends the result back. The model decides which tool to use; your code decides how to run it. A regular function call is deterministic - tool calling is decided by the model at inference time.

Does the tool schema count against my token limit?

Yes, always. The schema is injected into the model's context window on every request, formatted as part of the system prompt. Each tool definition adds approximately 50-100 tokens to the request. With 5 tools defined, expect 250-500 tokens of overhead per call. In multi-step agent loops, that overhead repeats on every step.

How does the model guarantee the tool arguments are valid JSON?

Through constrained decoding, not prompting. At every token step, a state machine masks invalid tokens to logit −∞ before sampling. Because the constraint is enforced during generation rather than after, the output is guaranteed valid - you never get a parse error, never have to retry, never need a fallback parser. Enabling strict: true adds schema-level enforcement on top of that.

What happens when a model fires parallel tool calls?

The response comes back with an empty content field and a tool_calls array containing multiple entries, each with its own unique ID. The application layer intercepts the model's request. It must validate that the generated JSON matches the expected schema and transform that data into the specific format required by the target API endpoint. You run all calls (ideally in parallel), then return each result as a role: tool message matching its tool_call_id, before making the next model call.

Is "function calling" the same as "tool calling"?

Mostly yes. Different AI providers use different names for the same idea. OpenAI calls it "function calling" while Anthropic calls it "tool use," but the implementation is nearly identical. Both use JSON schemas to define tools and return structured outputs. OpenAI's current API surface uses the tools parameter and refers to the broader capability as "tool calling"; "function calling" is the older name for the same thing.