Trace an AI Tool Call From Prompt to API and Back

Most people who build with AI agents think of tool calling as a feature you turn on. It is not. It is a five-step round-trip that begins before the model sees a single word from the user, and every leg of that trip has a real cost and a real failure mode.

Here is exactly what happens.

What tool calling actually is (and isn't)

Tool calling is a structured protocol between your application and an LLM. Rather than executing code directly, the LLM outputs a JSON object describing which function to call and what arguments to pass. Your code handles execution, then sends the result back for interpretation. The model never touches your database, your API, or your file system. It only ever writes a note describing what it wants done.

This matters because it means your application is always the executor. In a regular API call, your code decides which endpoint to hit deterministically. With function calling, the LLM decides which function to invoke based on natural language input and outputs structured JSON, while your code handles execution. The intelligence lives in the routing decision; the trust boundary stays with you.

The five steps, in order

Step 1: Schema injection (this is where the cost starts)

When you send a tool schema to an LLM, you are not "registering" a function. You are injecting a JSON blob into the model's context window, formatted as a system message. The model is then fine-tuned to output a special token sequence that signals a tool call. This is why the schema counts against your token budget - it is literally part of the prompt.

The system prepares the environment by gathering the prompt, user history, and the tool definitions. You must provide the LLM with a JSON schema that describes each tool and its parameters, allowing the model to understand which tools are available and what inputs they expect before the first token is even generated.

The token overhead here is not trivial. Anthropic has published figures showing a typical multi-server setup (GitHub, Slack, Sentry, Grafana) can burn roughly 55,000 tokens on tool descriptions alone before a task starts. At standard rates on a mid-tier model, that is overhead billed on every single call - even the ones that never invoke a tool.

Step 2: Intent recognition

The user sends a query. The LLM analyzes the query and recognizes it needs external data or an action to fulfill the request. The model maps the user's language against the tool descriptions you provided. This is where description quality becomes load-bearing. The JSON schema you define is not just documentation - it is the only thing the model sees. If your descriptions are vague, the model will hallucinate arguments.

Step 3: The tool call output (not execution - a request)

The LLM analyzes the request against its available tools. If it determines a tool is needed - for instance, to check a real-time stock price or query a database - instead of a text response, it returns a tool_call object containing the tool name and the arguments it has generated.

The model generates a tool_calls field in the response. Under the hood, the model outputs a JSON string inside the arguments field. The API then parses this JSON for you - but if the model outputs malformed JSON, the API returns an error.

Step 4: Constrained decoding (the part most explanations skip)

How does a stochastic token generator reliably produce valid JSON? The answer is constrained decoding. Constrained decoding is the inference-time technique that forces an LLM's output to conform to a grammar, schema, or regex by masking the next-token distribution at every step. Tokens that would make the partial output invalid are set to logit negative infinity before sampling, leaving only legal continuations. Because the constraint is enforced during generation rather than after, the output is guaranteed valid - you never get a parse error, never have to retry, never need a fallback parser.

Constrained decoding ensures that every token an LLM generates follows predefined grammar rules, similar to how a compiler checks code validity. The grammar guides generation by producing a per-token mask: valid tokens are kept, while invalid ones are set to −∞ logits and excluded from sampling.

LLMs are autoregressive. At each step, the model reads the entire token sequence so far (prompt plus all previously generated tokens) and outputs a score for every token in its vocabulary. This score vector is called the logit vector. A typical vocabulary has 32,000 to 100,000 tokens. At each step, the model produces a vector of that size: one float per token. Constrained decoding prunes that distribution at every single step, keeping only the tokens that would produce a syntactically valid continuation of the JSON being built.

There is a cost to this. When evaluating XGrammar on Meta-Llama-3-8B, constrained decoding exhibits up to 37.5% higher latency at batch size 512 compared to unconstrained decoding. That gap closes at small batch sizes, but it is real in high-throughput production.

Step 5: Result re-injection and final answer

The application layer or an orchestration platform intercepts the model's request. Your code runs the actual function, gets the result, and sends it back to the model as a new message in the conversation. The model then reads the result and produces its final response to the user. That round-trip is a second inference call, billed at full input-token rates for everything that came before.

A concrete example: "What's our Q2 churn rate?"

Someone types that into a Slack channel. Here is the actual call trace:

1. Schema loaded: your orchestrator injects a query_database tool definition into the system prompt. The schema describes the function name, a required sql parameter typed as a string, and an optional database enum. Those descriptions consume tokens before the model sees the question.

2. Model decides: the model reads the user query and the schema. It returns finish_reason: tool_calls with {"name": "query_database", "arguments": {"sql": "SELECT churn_rate FROM metrics WHERE quarter = 'Q2'"}}.

3. Your code runs: your application receives that JSON, validates the SQL against an allowlist, runs it against your data warehouse, and gets back {"churn_rate": 0.042}.

4. Result re-injected: {"role": "tool", "content": "{\"churn_rate\": 0.042}"} goes back into the conversation as a new message turn.

5. Final answer: the model reads the result and replies "Q2 churn was 4.2%."

The user sees one response. The system made two inference calls, each billed on the full accumulated context.

The non-obvious trade-off: tool calling vs. structured output

These are often confused because both produce JSON. They are different mechanisms with different use cases.

Tool calling is best when the model needs agency - deciding whether and which action to take. JSON Schema output is best when you need deterministic extraction.

Structured tool calling introduces a cognitive trade-off that can impair performance on domain-specific tasks. This view is consistent with the literature on format constraints in LLMs. The limitation is not that LLMs cannot follow JSON schemas; modern models are strong at code generation. Rather, following a schema appears to redirect the model's representational resources away from the primary task, so format requirements compete with task instructions for cognitive bandwidth.

The practical implication: if you only need to extract fields from a document, skip tool calling entirely and use structured output mode. Fewer round-trips, less overhead, and the model puts more cognitive capacity toward the extraction task itself.

Parallel calls and the loop that breaks them

Most tutorials show one function call. In production, models can emit multiple tool calls in a single turn. If your loop does not handle that, you will silently drop requests and corrupt state.

When a model returns two simultaneous tool calls - say, looking up a user record while also fetching their subscription status - your orchestration layer must:

• Detect that finish_reason is tool_calls (not stop)
• Execute both calls, potentially in parallel
• Collect both results
• Append both as tool messages before the next inference call

Reused input dominates token bills because every API call resends the full conversation history, so later messages in a thread cost far more than the first one, even when nothing new is added. A three-tool-call agent loop that takes four inference passes resends the growing conversation on every pass. The last pass can easily be 5-10× the token cost of the first.

AI tool calling: common questions

What is the difference between tool calling and function calling?

The terms are used interchangeably. OpenAI originally called the feature "function calling" when it launched in 2023; most providers now use "tool calling" as the umbrella term, since tools can include functions, code interpreters, and retrieval systems. The underlying mechanism - JSON schema in, structured call out - is the same.

Does the model actually run the function?

No. The LLM outputs a JSON object describing which function to call and what arguments to pass. Your code handles execution, then sends the result back for interpretation. The model never has direct access to your infrastructure.

Why does my agent's token bill keep growing mid-session?

System prompts and function/tool schemas are sent with every request and count as input tokens. A verbose 2,000-token system prompt multiplied across 100,000 daily calls adds up to 200 million input tokens - potentially thousands of dollars per month. In multi-turn agent sessions, the full conversation history resends on every call. Tool results from earlier turns accumulate in context and are billed again on every subsequent pass.

What stops the model from outputting invalid JSON?

Constrained decoding. At each decoding step, the set of tokens that are valid continuations of the current partial output is computed, everything else is masked to −∞, and the model samples from the restricted distribution. Invalid tokens literally cannot be sampled. Without constrained decoding, malformed JSON is a real failure mode - especially with longer or nested argument schemas.

When should I not use tool calling?

When the task is read-only transformation: summarization, classification, extraction from a document already in the prompt. Use structured output when the transformation is read-only - the model analyzes and extracts, it does not act. The signal is: input → extraction → done. Tool calling adds latency and cost that only pay off when the model genuinely needs to decide whether and which external action to take.