Understand LLM Tool Calling Before You Build Around It

Send a message to a modern AI agent asking it to pull your Q3 churn number, and two API calls happen before you see any text. Most developers know this in the abstract. Fewer have looked at what those calls actually contain - and the detail that bites you in production is hiding right there.

Tool calling (also called function calling, depending on which vendor's docs you're in) is the mechanism that turns a language model from a text generator into something that can query a database, call a Slack API, or look up a live price. Understanding how it works at the token level changes how you design schemas, budget costs, and debug failures.

What tool calling actually is, mechanically

LLMs excel at generating coherent text, but they also need to interact with external systems. Tool calling lets them construct those calls - though the LLM does not execute them directly. Instead, it creates a data structure describing the call and passes that to a separate program for execution.

The exchange is a loop with four beats:

1. Your app sends the user's message plus a catalog of available tools, each described as a JSON Schema object
2. The model reads all of it, decides whether a tool is warranted, and emits a structured tool_calls response
3. Your app executes the function and returns the result
4. The model reads that result and either answers or calls another tool

The model doesn't switch between "text mode" and "function mode" as a binary state. During token generation, the model predicts the next token from its vocabulary, which includes special tokens that signal function call intent. When the model generates a function call token, subsequent tokens are constrained to produce valid JSON matching the provided schemas.

That last sentence matters. The output is not a special API response type - it is text, structured as JSON, that your application then parses and acts on.

The schema is part of the prompt - and you pay for it every time

Here is the part most tutorials skip: when you send a tool schema to an LLM, you're not "registering" a function. You're injecting a JSON blob into the model's context window, formatted as a system message. The model is then fine-tuned to output a special token sequence that signals a tool call. This is why the schema counts against your token budget - it's literally part of the prompt.

Under the hood, functions are injected into the system message in a syntax the model has been trained on. This means callable function definitions count against the model's context limit and are billed as input tokens.

The cost compounds fast. A simple function like get_weather(location: string) might cost 15 tokens. But a realistic production tool might require a detailed description, multiple parameters with descriptions, required fields, and constraints - easily 40-80 tokens per function. Multiply that by 20 tools in your system, and you've added 800-1,600 tokens to every request before the model generates a single output token.

At scale this is not a rounding error. Every single API call sends the full JSON schema for every tool you have defined - whether it is used or not. A production agent with 20 tools and standard schema definitions carries 5,000 to 8,000 tokens of schema overhead per call. At Opus 4.8 input pricing, that's $1.4 million per year in schema tokens alone on 100,000 daily calls. It does not appear as a line item in any billing dashboard.

And caching won't save you here. Function-calling schemas are not cached by prompt caching - only the message history is cached; tools are resent on every request, making schema consolidation your only optimization path for high-volume applications.

Parallel tool calls and the message flow

Parallel tool calls let an agent request multiple tool executions in a single LLM turn, dispatched concurrently by the runtime. OpenAI's Responses API (March 2025) made parallel tool calls the default, reducing round-trips for independent operations from N sequential turns to one concurrent dispatch.

In practice: you ask "what's my MRR and how does it compare to last quarter?" The model can emit two tool calls in a single response - get_mrr(period="current") and get_mrr(period="last_quarter") - your app runs both in parallel, and both results land in the next turn.

The response has an array of tool_calls, each with an id (used later to submit the function result) and a function containing a name and JSON-encoded arguments. You must return one result message per call, matched by that ID, before the model continues.

Multi-turn context accumulates through message history. Each function call and its result become part of the conversation context. The model sees the full sequence: user message, function call request, function result, assistant response.

This accumulation is where context budgets get eaten fastest in agentic workflows - not by the user's messages, but by the growing stack of tool results.

Strict mode and schema enforcement

Modern LLM providers now offer schema-enforced structured output that works by modifying the model's generation process at the token level. Before each token is generated, tokens that would violate the schema are masked out of the probability distribution. The model physically cannot generate output that doesn't conform to your Pydantic or Zod schema.

OpenAI's "Strict Mode," introduced in mid-2024, uses this approach. When you set strict: true on a tool definition, the arguments field is guaranteed to be valid JSON that matches your schema. No post-parse validation step needed.

The trade-off is that strict mode requires additionalProperties to be set to false for each object in the parameters, and all fields in properties must be marked as required. Optional fields have to be typed as nullable rather than simply omitted.

LLM tool calling: common questions

What is the difference between tool calling and function calling?

The terms describe the same mechanism. The underlying architecture is identical - the model outputs structured JSON specifying which external function to invoke and what parameters to pass. OpenAI historically said "function calling," Anthropic says "tool use," and most of the industry has converged on "tool calling" as the generic term.

Does the model actually run the function?

No. Function calling allows the model to construct calls to external systems, but the LLM does not execute these calls directly - it creates a data structure that describes the call, passing that to a separate program for execution and further processing. Your application holds all execution authority.

How many tokens do tool schemas actually consume?

Unlike regular completions, tools are not cached - they're sent with every request. A moderately complex tool schema (5 tools with detailed descriptions and parameters) adds 800-1,200 tokens per request. At 10,000 requests per day, that's 8-12 million tokens per day of pure schema overhead.

What happens if the model calls a tool with wrong arguments?

Without strict mode, the model outputs a JSON string inside the arguments field. The API parses this JSON - but if the model outputs malformed JSON, the API returns an error. Models can output unescaped newlines inside strings, which breaks the parser. With strict mode enabled, this class of failure is eliminated at the generation level.

Can a model call more than one tool per turn?

Yes. When you send a prompt to an LLM with tools defined, the model can decide to call zero, one, or multiple tools in a single response. OpenAI returns all of them at once in the tool_calls list, not one at a time. Your app runs them concurrently and submits all results before the next model turn. For agents connected to Beagle's Slack integration, this means a single question can fan out to multiple data sources and reassemble before the draft surfaces in-channel.