How LLM Tool Calling Works, From Schema to Execution

A modest tool definition JSON file with about 300 lines of code adds approximately 1,300 tokens per query - regardless of whether the tool is ever called. Most developers discover this on their first real bill, not in the documentation. That single fact tells you more about how tool calling actually works than most tutorials do.

What tool calling actually is

Tool calling is the mechanism that lets an AI agent interact with external systems. Instead of only generating text, the LLM outputs a structured JSON object specifying which function to call and what arguments to pass. A runtime layer then executes that function and feeds the result back to the model for further reasoning.

LLMs do not call functions. Regardless of what you have been told, they simply don't. If your LLM API or SDK calls functions for you, there is a layer of software wrapped around it that is taking care of this and invoking the function.

That distinction matters. The model is a text predictor that has been fine-tuned to emit a special token sequence when it decides a tool is needed. Everything else - the actual HTTP request, the database query, the file read - happens in your code.

The terms "tool calling" and "function calling" refer to the same thing with different branding. OpenAI originally called it "function calling." Anthropic calls it "tool use." Google uses both. The industry has since converged on "tool calling" as the broader term, since tools can be much more than simple functions - they include APIs, code interpreters, browser actions, and entire MCP servers.

The loop, step by step

Tool calling lets an LLM invoke external functions rather than generate an answer from training data alone. The model receives a set of tool definitions, reasons about which to call, and returns a structured invocation. The calling application executes the function and feeds the result back into context.

Here is the complete cycle, mapped to what is actually passing between your application and the API:

1. Define - You write a JSON schema: a name, a description, and a typed parameter list.
2. Send - Your application sends the user message plus all tool schemas to the model API.
3. Model decides - The model reads the schemas (now part of its context window), determines whether a tool is needed, and if so, returns a tool_calls object with a function name and populated arguments.
4. Execute - Your application calls the actual function. The model has not touched it.
5. Return - You append the function result to the message history and call the API again.
6. Respond - The model uses the tool result to formulate its final response to the user, or decides to call additional tools. This loop can repeat multiple times within a single conversation turn - a model might call a search tool, analyze the results, then call a database query tool, then synthesize everything into a final answer.

The part your schema description is doing

When you send a tool schema to an LLM, you are not "registering" a function. You are injecting a JSON blob into the model's context window, formatted as a system message. The model is then fine-tuned to output a special token sequence that signals a tool call. This is why the schema counts against your token budget - it is literally part of the prompt.

That has a concrete consequence. Under the hood, functions are injected into the system message in a syntax the model has been trained on. Callable function definitions count against the model's context limit and are billed as input tokens. If you run into token limits, limiting the number of functions loaded up front, shortening descriptions where possible, or using tool search to defer rarely-used tools all help.

The description field is where most of the tool-selection signal lives. The JSON schema you define is not just documentation - it is the only thing the model sees. If your descriptions are vague, the model will hallucinate arguments. Naming matters too: start_date outperforms sd1 because the model is reading natural language to decide which tool to pick, not evaluating a function signature.

Gorilla research and ToolAlpaca experiments both confirm that precise descriptions and enum constraints can improve parameter generation accuracy by over 30%.

Parallel calls, strict mode, and where things break

When the model identifies multiple independent tool calls that can execute simultaneously, it returns them all in a single response. The application executes them in parallel, collects all results, and passes them back in one batch. Parallel tool calls reduce total latency for multi-tool workflows significantly.

In practice, this is not guaranteed. Some models are conservative and call tools sequentially even when parallel execution is safe. If latency matters, test with your specific model and prompt.

The other gap worth knowing: OpenAI and Anthropic handle structured output differently under the hood. OpenAI's implementation appends the tool definitions to the system message before tokenization. The model then generates a tool_calls field in the response. Under the hood, the model outputs a JSON string inside the arguments field, and the API then parses this JSON - but if the model outputs malformed JSON, the API returns an error.

Anthropic's Claude uses a different approach: it outputs a special XML-like <function_calls> tag and then a JSON block.

Setting strict to true ensures function calls reliably adhere to the function schema, instead of being best effort. OpenAI recommends always enabling strict mode. Under the hood, strict mode works by leveraging structured outputs, which introduces a requirement that additionalProperties must be set to false for each object in the parameters.

When tools multiply, model accuracy drops. Above 100 tools, it becomes essentially mandatory to use tool search - the model's ability to select the right tool degrades significantly when presented with too many options at once.

Here is how the major providers compare on the dimensions that matter in production:

Accuracy ranges from TokenMix.ai testing; overhead figures from provider docs and community benchmarks.

The compounding cost nobody budgets for

One more thing the tutorials skip. Multi-turn conversations, agent loops, and chain-of-thought patterns turn the last turn's output into the next turn's input. The cost compounds. A five-step agent that emits 1,000 tokens per step is paying input on 1,000, then 2,000, then 3,000, then 4,000 tokens of accumulated context.

Add a tool schema that costs 1,300 tokens on every turn and you are paying for it five times in a five-step loop - even if the tool is only used once. The practical fix: put your tool definitions early in the prompt so they sit in the cached prefix, and caching kicks in on prompts above 1,024 tokens and applies to the longest shared prefix. Put your stable system prompt and tool definitions first and the user-specific content last. Cached input is roughly 90% cheaper.

A teammate like Beagle - running inside Slack or Teams - uses this loop for every lookup it handles: the tool schemas are fixed per workspace, they sit in a stable cached prefix, and the per-call cost stays low even when the loop runs several steps.

How LLM tool calling works: common questions

What is the difference between function calling and tool calling?

They are the same capability with different names. OpenAI launched it as "function calling" in June 2023, Anthropic uses "tool use," and the industry has since standardized on "tool calling" as the broader term. Both describe the same mechanism: the model outputs a structured JSON intent, and your application executes the underlying code.

Does the model actually execute the function?

No. The LLM never executes code directly - a separate runtime layer handles all execution. The model outputs a name and arguments; your application runs the function, collects the result, and sends it back. The model then uses that result to reason further or produce a final answer.

How do tool schemas affect my token bill?

Every schema you attach is injected into the model's context on every API call and billed as input tokens. A 300-line schema adds roughly 1,300 tokens per request - whether or not the tool is ever invoked. With five tools across a five-turn agent loop, that overhead hits 25 times. Cache the prefix to reduce it by ~90%.

What happens when I give the model too many tools?

With fewer than 20-30 tools, loading them all upfront is fine. Above 100, it becomes essentially mandatory to use a tool search or retrieval layer - the model's ability to select the right tool degrades significantly when it is presented with too many options at once. Good descriptions and namespacing by domain help across any tool count.

How does strict mode prevent bad arguments?

Strict mode uses constrained decoding: the API compiles your JSON schema into a grammar and uses it to mask invalid tokens at generation time. Unlike prompting the model to "please return valid JSON," structured outputs compile your schema into a grammar and actively restrict token generation during inference. The model literally cannot produce tokens that would violate your schema.