How LLM Tool Calling Works Under the Hood

A Slack bot that can answer a question is mildly useful. One that can pull a live Jira count, post a summary, and update a Notion doc in the same turn is a different product. The gap between those two things is tool calling-and most explanations of it stop at "the model can use tools," which tells you nothing about why it sometimes breaks and costs more than you expect.

This is the full mechanical picture.

What the model actually sees

When you send a tool schema to an LLM, you are not "registering" a function. You are injecting a JSON blob into the model's context window, formatted as a system message. The model is then fine-tuned to output a special token sequence that signals a tool call-and because the schema is literally part of the prompt, it counts against your token budget.

OpenAI appears to be converting those JSON schema definitions into TypeScript type definitions internally before tokenization , which is why naively counting schema characters gives you the wrong number. Function definitions or JSON schema in system prompts can add 100+ tokens silently. With a realistic agent catalog-say, 10 tools with descriptions, parameter types, and enums-you are looking at 500 to 1,000 hidden tokens on every single request before a user has typed one word.

That matters at scale. Over a million requests, unoptimized prompts cost thousands. Including a 2,000-token system prompt on every request versus caching it saves roughly $3 per 1,000 requests. A bloated tool catalog has the same economics.

The model receives the user prompt and the tool catalog, decides whether a tool is needed, then returns a structured tool call with typed arguments. It does not execute anything. The execution is always your code.

How valid JSON comes out of a token stream

This is the part that surprises most people. The LLM still generates tokens one by one. It does not "natively" return a Python dictionary.

So how does it reliably produce parseable JSON? The API applies constrained decoding-also called "guided generation." It constrains the LLM's token generation so that at each step, only tokens that would produce valid JSON matching your schema are allowed. For an enum field with values like ["job_search", "spam"], the model can only pick from those exact values. For a field typed as "number", only numeric tokens are valid at that position.

This is fundamentally different from asking "please reply in JSON" in the prompt-which is fragile, because the model might start with "I think this is in the job_search category because..." and break your parser.

The model still makes the decision about which tool to call and what arguments to fill in. Constrained decoding only guarantees the output is structurally valid-not semantically correct. The JSON schema you define is not just documentation; it is the only thing the model sees. If your descriptions are vague, the model will hallucinate arguments.

The round-trip: one full tool call, step by step

Here is what actually happens when a user asks "How many open P1 tickets do we have right now?":

1. Your application sends the user message plus a get_ticket_count tool schema to the model API
2. The model decides a tool call is needed; constrained decoding produces {"status": "open", "priority": "P1"} as the arguments

Your code runs the tool and feeds the result back as a new message in the conversation with role tool 4. The model uses the result to continue the conversation-or to call another tool

Step 3 is where most naive implementations crack. Most tutorials show one function call. In production, models can emit multiple tool calls in a single turn. If your loop does not handle that, you will silently drop requests and corrupt state.

Why sandboxing is not optional once agents run code

Tool calling can return data-or it can execute code. When a coding agent or data-analysis agent writes Python and runs it, the execution environment becomes a security boundary, not just an infrastructure choice.

Standard containers are not sufficient for AI-generated code because they share the host kernel.

All containers on a host share the same Linux kernel, and a kernel vulnerability provides a direct path to the host. That is not a theoretical concern: November 2025 brought three runc vulnerabilities (CVE-2025-31133, CVE-2025-52565, CVE-2025-52881) affecting Docker, Kubernetes, containerd, and CRI-O simultaneously.

The stronger alternative is a microVM. Firecracker launches a minimal virtual machine with its own kernel in under 150ms with less than 5MB of memory overhead. It uses KVM for hardware virtualization, providing the same isolation guarantees as a full VM at near-container performance.

E2B builds its entire platform around this: every code execution gets its own Firecracker microVM. Cold starts hit around 150 milliseconds. They went from 40,000 sandboxes per month in early 2024 to over 15 million per month by early 2025.

The comparison across isolation approaches looks like this:

The sandbox sits between the AI agent and the actual infrastructure. It is the enforcement layer that turns "the LLM decided to run rm -rf /" from a catastrophe into a denied operation.

One non-obvious detail: multi-turn agent sessions need filesystem state that persists across turns. An agent working on a Python project across 10 turns has installed packages, written files, and accumulated intermediate outputs. Full sandbox re-initialization on every turn wastes 200-500ms on environment setup. Firecracker's snapshot-restore mechanism lets you pause a sandbox, preserve its memory and filesystem state, and resume it in 5-30ms.

A teammate like Beagle, operating inside Slack, works inside a much narrower tool surface-read a doc, call an API, post a reply-so the isolation boundary is simpler. But the same principle applies: every tool execution should be scoped to the minimum capability needed.

AI agent tool calling: common questions

What is LLM tool calling?

Tool calling (also called function calling) is how a large language model triggers external actions. The model does not run anything itself-it returns a structured JSON intent specifying which tool to call and with what arguments. Your application runs the tool and feeds the result back for the model to use.

Does the tool schema use tokens?

Yes, and more than most developers expect. When you send a tool schema, you are injecting a JSON blob into the model's context window as a system message. A single well-described tool with several parameters can consume 50-100 tokens; a catalog of 10 tools easily adds 500-1,000 tokens to every request, billed at full input-token rates.

How does the model return valid JSON without hallucinating structure?

Through constrained decoding. The API constrains token generation at each step so only tokens matching your schema are allowed-numeric fields can only receive numeric tokens, enum fields can only receive declared values. This is enforced at the inference layer, not by prompt instruction.

Is a Docker container enough to sandbox an AI agent?

Standard containers are not sufficient for AI-generated code because they share the host kernel. For agents that execute arbitrary code, microVMs like Firecracker provide hardware-level isolation with a separate kernel per workload-the approach used by AWS Lambda and platforms like E2B. Plain Docker is acceptable only for fully trusted, pre-reviewed code.

What happens if an agent needs to call three tools?

The model can emit multiple tool calls in a single turn, and each one requires its own round-trip: your code executes the tool and returns the result before the model continues. Three tools means at least six API calls total-two per tool, plus the final synthesis step. Parallel tool calls can reduce this if your orchestration layer fires them concurrently, but not all frameworks handle that by default.