Tool execution alone accounts for 35-61% of total agent request time. Build an agent that calls the wrong tool-or passes a bad argument-and that time is not just wasted; it poisons every step that follows. The problem is that most teams only discover this in production, where fixing one failure creates others. Evals exist to surface that before users do.
The word "evals" gets used loosely. It means different things depending on whether you are testing a chat model on a fixed dataset or running a multi-step agent through a live environment. For agents that call tools, the gap between those two worlds is where most surprises live.
What an eval actually measures for a tool-calling agent
An eval is a test for an AI system: give an AI an input, then apply grading logic to its output to measure success. For a simple text model, that grading logic is usually a string match or a rubric. For a tool-calling agent, it is three things at once:
- Tool selection - did the model pick the right function from the set available?
- Argument accuracy - did it fill the parameters correctly?
- Task completion - did the full run accomplish what the user actually asked?
An agent can call every tool correctly and still fail the task. That is the thing that trips teams up. Tool-level accuracy is a necessary condition, not a sufficient one. An agent that flawlessly queries your Salesforce API and your Snowflake table can still produce a wrong answer if it synthesizes the results badly or misunderstands the user's original goal.
Instead of scoring a single model on a fixed dataset, you evaluate a dynamic system that plans, retrieves information, calls functions, adjusts based on feedback, and may follow multiple valid trajectories toward a solution. That last part is what makes grading hard: there is often more than one correct path.
The three-layer evaluation stack
Think of evals for tool-calling agents as three concentric rings, from innermost to outermost.
Layer 1 - schema validation (deterministic)
With function calling, an LLM analyzes a natural language input, extracts the user's intent, and generates a structured output containing the function name and the necessary arguments. The LLM itself does not execute the function - it identifies the appropriate function, gathers all required parameters, and provides the information in a structured JSON format. At this layer you are checking whether that JSON is valid: right function name, right types, no hallucinated fields.
The Berkeley Function Calling Leaderboard (BFCL) is a comprehensive benchmark designed to evaluate function calling capabilities in a wide range of real-world settings. It evaluates serial and parallel function calls across various programming languages using a novel Abstract Syntax Tree (AST) evaluation method that can easily scale to thousands of functions. AST matching is fast and reproducible - it does not need a judge model and does not introduce variance.
Layer 2 - trajectory grading (LLM-as-judge)
Once you know the agent picked the right tools with the right arguments, you need to assess the reasoning between steps. Agent quality spans dimensions no single metric captures: whether responses are grounded in what the tools actually returned, whether the agent called the right tools with the right parameters, and whether the final output is coherent and useful to the person asking.
LLM-as-judge methods achieve 80-90% agreement with human judgment at 500-5000x lower cost. That cost delta is what makes automated evals viable at scale. The tradeoff is that LLM judges introduce their own biases - position bias (preferring the first answer they see) and a tendency to prefer verbose responses - so you still want humans reviewing flagged cases.
Layer 3 - end-to-end task completion
Task completion, also known as task success or goal accuracy, measures whether an LLM agent completed a user-given task. The definition of "done" depends on the task. For tool-calling, planning, and autonomous agents, this metric is often the clearest end-to-end signal of whether the agent actually worked.
This is the layer most teams skip until something breaks. It is also the hardest to automate. For a coding agent you can run unit tests against the output. For a research agent you might check factual groundedness against source documents. For a computer-use agent the question is whether the system state actually changed.
Beagle in action#product-ops, 2:17pmThe hidden token cost of your tool schemas
Here is the number most teams do not budget for: each tool definition consumes input tokens. A typical tool with a medium-complexity schema costs 100-300 tokens. A system with 20 tools adds 2,000-6,000 tokens per request - roughly $0.01-$0.06 at current pricing on GPT-4.1.
At low volume that is invisible. At a million calls a day it is $10,000-$60,000 per month just in schema overhead - before you count the actual inference.
Strategies: remove unused tools, consolidate related tools into one with an action enum parameter, and cache system prompts (Anthropic prompt caching reduces repeated tool definition costs by ~90%).
The same logic applies to evals: if you run nightly eval suites over a 20-tool agent, your evaluation costs scale with tool count, not just test-case count. Teams running BFCL-style harnesses internally should account for this in their eval budget.
Parallel tool calls: the latency win with a schema trap
Parallel tool calling is a pattern where an LLM identifies independent operations, requests them all in a single response, and your infrastructure executes those calls concurrently. Instead of waiting for each tool to finish before starting the next, independent calls run at the same time. Total latency drops from the sum of every tool call to the duration of the slowest one.
Parallel tool calls are the single highest-impact performance optimization for agentic systems. The LLMCompiler paper (ICML 2024) demonstrated up to 3.7x latency reduction by executing independent tool calls concurrently rather than waiting for each to complete before starting the next.
The catch: when the model outputs multiple function calls via parallel function calling, model outputs may not match strict schemas supplied in tools. If you are running evals with strict JSON schema enforcement, parallel mode can cause your harness to flag valid behavior as a failure. This is a gap between benchmark conditions and production conditions that is easy to miss.
| Eval layer | What it checks | Grader type | Speed |
|---|---|---|---|
| Schema validation | Right tool, right args, valid JSON | Deterministic (AST) | Fast |
| Trajectory grading | Reasoning between steps, groundedness | LLM-as-judge | Moderate |
| Task completion | Did the job get done | Unit tests or LLM judge | Slow |
Most eval frameworks - DeepEval, MLflow, Langfuse, W&B Weave - let you mix these. Code-based evaluators offer fast, reproducible results but penalize valid variations in approach. LLM-as-judge evaluators provide nuanced assessment at the cost of additional inference and careful prompt design. Most effective evaluation strategies combine both approaches.
A teammate like Beagle operates within a draft-and-approve loop - every response goes through a human before it posts - which is itself a lightweight form of online evaluation. But offline evals before you change a prompt or swap a model are what catch systematic failures, not just one-off surprises.
AI agent evals: common questions
What is the difference between a tool call eval and a task completion eval?
A tool call eval checks whether the model picked the right function and passed valid arguments - deterministic, fast, and automatable with schema matching. A task completion eval checks whether the full run achieved the user's goal. An agent can pass tool evals and still fail task completion if it synthesizes results incorrectly or misunderstands the request.
How does LLM-as-judge work for agent evaluation?
You send the agent's full output - and sometimes its reasoning trace - to a second model with a rubric. That model scores whether the response is grounded, coherent, and correct. LLM-as-judge methods achieve 80-90% agreement with human judgment at 500-5000x lower cost , making them practical for nightly eval runs. The main failure modes are position bias and a preference for longer answers.
What is the Berkeley Function Calling Leaderboard (BFCL)?
The Berkeley Function Calling Leaderboard (BFCL) has emerged as the de facto standard for evaluating function calls. Developed by UC Berkeley researchers, BFCL was the first comprehensive benchmark designed to evaluate function calling capabilities across real-world settings. It tests single-turn, parallel, and multi-turn tool calls using AST matching rather than execution, which keeps it fast and reproducible.
Should I run offline or online evals?
Run both, at different cadences. The key challenge with offline eval is ensuring your test dataset is comprehensive and stays relevant - the agent might perform well on a fixed test set but encounter very different queries in production. You should keep test sets updated with new edge cases and examples that reflect real-world scenarios. Online evals monitor live traffic and catch distribution shift that offline sets miss.
How many tools should my agent expose at once?
As few as possible. Research from 2024 found that dynamically limiting the set of available tools yields dramatic hardware gains: execution time dropped by up to 70% and power consumption by ~40% on edge devices. Even on cloud inference, fewer tools means fewer tokens per request, cleaner decision-making, and easier evals. Consolidate related tools into a single function with an action parameter before you add a new one.