Tool Evaluation

​

Overview

Confidence Scoring evaluates an agent’s work with tools through two judged modules, Tool Usage and Tractability. Tool Usage asks whether the agent made good use of the tools it had: whether it called the right ones, and whether it passed them correct arguments. Tractability asks whether the task was a good fit for the agent in the first place. The Tractability judgment weighs the kind of reasoning the task demands and the amount of work it requires, and it counts the agent’s tools as a way to take on part of both. Through that last factor, Tractability shows whether the tool set covered what the task needed. Like every judged module, each one rates a step on a scale from 0 to 10 and attaches a written explanation of the rating. See the Confidence Scoring concept guide for how judged modules work in general. Tool Usage scores how the agent played the hand it was dealt. Tractability scores the hand.

​

Why the Two Are Separate

Consider a request that an agent answers poorly. In one case, the agent had a search tool that could have found the answer and never called it. In another, none of the agent’s tools could reach the data the request needed, and the agent answered as well as it could without it. Both turns end with a poor answer, but the first calls for a change to the agent, such as clearer instructions, while the second calls for a new or extended tool. A single score for how the agent handled its tools would rate both turns low without indicating which change to make. Each module is therefore scoped to leave the other’s question alone. Tool Usage does not penalize an agent for a gap in its tool set that it could not have closed, and Tractability does not grade how well the agent used the tools it had. Read together, the two scores show what is driving an agent’s performance, whether a turn went well or badly.

​

When the Modules Run

Both modules evaluate each of an agent’s turns. A turn is one user message and everything the agent does in response to it, including every tool call across the steps that lead to its final answer. When the agent finishes a turn, the platform dispatches one scoring job for each enabled module, so a turn produces at most one Tool Usage score and one Tractability score. The modules see different parts of the turn, matching the questions they answer:

• Tool Usage sees the user’s request, the tools available to the agent with their descriptions and parameter schemas, and every tool call the agent made in the turn with the arguments it passed. It does not see what the tools returned or the agent’s final answer.
• Tractability sees the user’s request, the same available tools, and the agent’s final answer.

The available tools are those the agent could use in that turn, after any execution policy has been applied.

​

Tool Usage

Tool Usage asks whether the agent made good use of the tools it had. It evaluates all of an agent’s tool calls in a turn together, which lets the evaluator recognize calls that duplicate each other’s work. It scores every turn in which the agent had tools available, including turns in which the agent called none.

​

What It Evaluates

The evaluator looks at two aspects of each call:

• Tool choice: whether the tool suited what the request needed, whether a better-suited tool was available and went unused, and whether the call was unnecessary or repeated another.
• Arguments: whether the values passed fit the tool that was called. The evaluator checks that required parameters were supplied and useful optional ones were not left out, that each value is valid for its parameter, that the values carry the request’s constraints without dropping or inventing any, and whether a more precise value, such as a narrower query, would have served the request better.

A call is sound when both its tool choice and its arguments hold up, and it is rated by whichever of the two is worse. A well-chosen tool called with badly wrong arguments is an unsound call. A high score means the turn’s calls were sound and suited to the request. A low score means a substantial part of the tool use was unsound, and the score’s explanation goes through the choice and the arguments of every call, so it can be traced to the calls responsible and to whether the choice or the arguments fell short.

​

Several Calls in One Turn

When a turn has several calls, the score reflects how much of the tool use served the request, and a single unsound call does not set the score on its own. One unsound call among several sound ones is a partial failure, and how much it lowers the score depends on what it cost the request. A mistake the agent corrected in a later call counts for less than one it left uncorrected. The lowest scores are reserved for turns in which none of the calls served the request.

​

Turns with No Tool Calls

When tools were available and the agent called none, Tool Usage still scores the turn, because calling nothing can be the right decision. If none of the available tools would have helped, or the request needed no tool, as with a greeting or a clarifying question, then calling nothing was correct and the turn scores highly. If an available tool would have materially helped and the agent answered without it, the agent responded without a source it had been given. Tool Usage treats that as a serious tool-choice failure, and the more the tool could have benefited the response, the lower the score. Argument quality does not apply to these turns, since no calls were made, so the score rests on the choice alone. Tool Usage reports a turn as not_run in one case only: when the agent had no tools available. With no tools there is no choice to evaluate, and a score of zero would penalize the agent for a decision it was never offered.

​

What It Leaves to Other Modules

• Limitations of a tool. If a tool lacks a parameter the request would benefit from, that is a property of the tool set, which Tractability evaluates.
• Tools that do not exist. The agent is judged against the tools it was given. Whether that set was adequate is also Tractability’s question.
• Results and answers. What the tools returned, the quality of the final answer, instruction following, and coherence are evaluated by other modules.

​

Rubric

The evaluator writes its assessment of the turn first, then assigns a score consistent with that assessment from the following bands.

​

Tractability

Tractability asks whether a task was a good fit for what an agent can do reliably. It scores every turn, including turns of agents with no tools, because a task can be a poor fit for a language model whether or not tools are involved. The judgment combines three factors: the kind of reasoning the task demands, the amount of work it requires, and the tools available to take on part of either.

​

What It Evaluates

Language models are most reliable at interpretive work, such as summarizing or generalizing from examples, where several answers can be acceptable. They are less reliable at rule-bound work with a single correct result, such as exact arithmetic or applying strict rules, and at any work repeated across a large volume of material. Tools take on exactly this kind of work: a calculator handles arithmetic, a query tool handles exact lookups over large data, and a code execution tool handles formal computation. A task that would be a poor fit for the model alone can therefore be a good fit with the right tools, and Tractability scores each task as the agent’s tools make it. A high score means the task, with the tools available, sits within what the agent can do reliably. A low score means part of the task depends on work the agent is likely to get wrong, and its explanation identifies which part.

​

How the Tool Set Affects the Score

The evaluator judges the tools as a set. A task that no single tool can handle may be covered by several tools used together. The set falls short when no combination of the available tools can do what the task needs, or when the tool that fits lacks a parameter the task requires. A gap in the tool set lowers the score even though the agent could do nothing about it. That is what makes the score useful to the agent’s author: a low Tractability score whose explanation cites a missing capability points to a tool to add or extend.

​

Signs of Strain

Alongside the task itself, the evaluator reads the agent’s final answer for signs that the model was strained: quality that drops over the length of the answer, steps skipped where precision was required, or rules applied inconsistently across many items. The evaluator uses these signs as evidence for its view of the task’s difficulty and does not grade them as errors in their own right. Strain where the task was expected to be hard confirms a poor fit, and its absence suggests the agent handled the task comfortably. For tool evaluation, strain is most telling where it appears in work that no available tool took on, such as arithmetic done by hand or rows counted one by one, because there it supports a finding that the tool set left too much to the model. The explanation points to the strained work with examples from the answer, which indicates what a new or extended tool would need to handle. Where a tool did take on the difficult part, an answer free of strain is evidence that the tool is serving its purpose.

​

What It Leaves to Other Modules

• Reasoning and accuracy. Whether the reasoning holds together is evaluated by Coherence, whether the claims are true by Correctness, and whether the answer stays faithful to its sources by Faithfulness.
• Instructions. Whether the agent followed the instructions it was given is evaluated by Instruction Following.
• How the agent used its tools. Whether the agent called the tools it should have, and with what arguments, is Tool Usage’s question.

​

Rubric

As with Tool Usage, the evaluator writes its assessment first, then assigns a score from these bands.

​

Reading the Two Together

Each combination of the two scores points to a different place to look for improvement. Because both modules consider the same tool set, their scores tend to move together, and a good tool set raises both. The two are complementary diagnostics, so agreement between them should not be read as two independent confirmations of the same conclusion. A single Tool Usage score explains one turn: whether the agent reached for the right tool when it mattered, and whether it used that tool properly. A single Tractability score explains whether the agent was well positioned to answer that turn’s request at all. Across many turns, the scores answer broader questions about an agent and its tools. A low average Tool Usage score singles out the agents that use their tools worst. Low Tool Usage scores whose explanations repeatedly name the same tool suggest that the tool’s description misleads the agent. Requests of one kind that consistently score low on Tractability, with explanations that cite a missing capability, mark a gap in the tool set. Low Tool Usage scores concentrated in agents whose Tractability is also low suggest that the tooling, more than the agents, is the cause.

​

Choosing Which to Enable

Tool Usage is most informative for an agent with a non-trivial tool set. It earns its place where several tools have overlapping purposes, where arguments are expressive enough to get wrong, such as query strings, filters, or modes chosen from a fixed list, or where a wrong call is expensive. Because it also scores turns in which the agent called nothing, it catches an agent that ignored a useful tool. As a result, an agent with tools produces a Tool Usage score on nearly every turn. Tractability gives early warning about which requests to trust. It suits an agent that receives a wide range of requests, some of them a poor fit for a language model, and an agent whose tool set is still taking shape, where its explanations show what the tools fail to cover. It produces a score on every turn, including for agents with no tools. Enabling both is most useful while an agent’s tools are under active development, since the combinations in the table above show whether the next change belongs in the tool set or in the agent.

​

Module Reference

The modules are enabled by identifier in the confidence_configs list of an agent definition, which is set when the agent is created with Create Agent or changed with Update Agent. Each turn’s result is recorded as a scoring job, returned individually by Get a scoring job and as a filtered list by List scoring jobs. A job carries these fields for both modules: