Confidence Scoring

​

Overview

Meibel’s confidence scoring system gives clear evaluations of the quality of an AI workflow at each step of the pipeline. It works by running configurable combinations of 15 individual modules, each built and tuned by Meibel to evaluate a particular type of performance or quality of output. Once enabled on an agent, these modules run asynchronously as the agent performs its work. Confidence Scoring evaluates the agent’s own outputs, both its direct responses and its intermediate reasoning, as well as the results of supporting steps, including tool calls, data retrieval, and extraction. Each module produces an independent score for the step it evaluates. Those scores can be read on their own or aggregated and summarized across steps and over time. Both views matter. An individual score speaks to one specific step, while aggregates reveal broader patterns in an agent’s performance. Because the modules measure different things, a faithfulness result stays separate from an OCR result, which makes it possible to locate where and how quality changed.

​

Confidence Scoring Modules

Confidence Scoring modules fall into two families that differ in how they arrive at a score: judged modules and statistical modules. Judged modules use an internal AI agent as an expert evaluator. Each one receives the relevant input and output and assesses a specific quality dimension, working through a structured rubric before assigning a rating on a Likert scale from 0 to 10. The rubric sorts an output into one of a handful of discrete quality bands, and an integer from 0 to 10 captures that granularity without implying false precision. Because an AI agent performs the evaluation, every judged score comes with a written explanation of why it was assigned, so the reasoning behind the number travels with it. Judged modules are best suited to qualitative dimensions that require interpretation, such as whether an answer is helpful, coherent, or faithful to its sources. Statistical modules compute a continuous score from the data itself using statistical methods, instead of producing a rubric-based rating. They report on a scale from 0.0 to 1.0, derived from calculations such as the consistency of repeated generations, the confidence of an OCR engine, or the overlap between an answer and its retrieved context, and that continuous value preserves the underlying measurement directly. Statistical modules are deterministic in nature and grounded in observable quantities, which makes them well-suited to dimensions where a measurement is more applicable than a judgment. The following modules are available.

​

Scorable Steps and Outputs

Confidence Scoring evaluates the individual outputs an agent produces as it works, and different kinds of output support different modules. Those outputs include:

• agent responses: the responses an agent generates, including its final answers and intermediate reasoning steps.
• Tool calls: an agent’s choice of which tool to invoke and the result it returns, along with the performance of internal tools and their intermediate inputs and outputs, such as a generated SQL query and the rows that query returns.
• Data retrieval: the data an agent retrieves to work with, whether context gathered from its datasources to ground a response or input data fetched for the task itself.
• Structured extraction: the values a model extracts from source documents into structured artifacts according to a defined schema, together with the document parsing those values are built on.

The pairing of outputs to modules follows from what each output is. Scoring a SQL query for readability or a chat response for OCR confidence would measure nothing meaningful, so the platform applies only the modules that fit a given output.

​

Configuration

Confidence Scoring is configured declaratively. An agent definition lists the modules enabled for an agent, and the platform treats that list as the set of modules to run. At execution time it narrows the list to those that apply to the step the agent is performing, skipping any that are not applicable at each step. The platform already knows which modules fit which steps, for example that OCR Confidence applies to retrieval over parsed documents while Tool Selection applies to an agent’s tool-using turns, so the mapping of modules to steps is not something an author maintains by hand. The result is configuration that stays declarative and scoring that runs only where it produces a meaningful result. The logic behind this sits entirely within the platform. It determines which scores are supported by which processes and tool calls, what counts as an intermediate output that can be scored, and when and how a given output is scored. The platform also maintains the tuning behind each module, including its evaluator prompts, model choices, sampling parameters, and calibration, so enabling a module brings a well-tuned evaluator into play with no further setup. See the agents concept guide for more information on agents and agent definitions.

​

How Scoring Runs

Confidence Scoring runs asynchronously, in independent background jobs that sit outside an agent’s execution path. When an agent completes a scorable step, the platform dispatches a scoring job for each applicable module. The agent does not wait for these jobs, so enabling Confidence Scoring does not slow down the responses an agent returns. Because the jobs run independently, they can outlive the turn that triggered them and post each score back as it finishes, including for modules that take longer to compute.

​

Reading the Results

As individual confidence scores are computed, they become available for retrieval and inspection. Each score carries a numerical value on its module’s scale, and judged scores also carry the written explanation of why that score was assigned. A scoring job records the input and output it evaluated along with its status, so the full provenance of a score is traceable by the platform. The platform makes individual and aggregated scores available for analysis. An individual score helps explain a specific step or account for a particular result. Aggregated and summarized scores are just as important, whether across the steps of a single agent execution or across an agent’s performance over time. Because judged and statistical scores measure different things on different scales, the platform aggregates them separately rather than collapsing them into one figure, which preserves the meaning of each category. The platform provides built-in aggregations and summaries of results for common needs:

• Per-execution summaries: the scores produced across all the steps of a single agent run, which show how quality held up turn by turn within one conversation.
• Per-module breakdowns: aggregated scores for each module, which surface the dimension, such as faithfulness or relevance, where performance is weakest.

Confidence scores also support longer-range analysis, including how a module’s scores move over time and how agents or workflows compare against one another, or against their own past behavior across different periods. Together these views form a foundation for understanding typical performance over many runs, A/B testing different configurations or inputs, and detecting and correcting performance drift.

​

Using Confidence Scoring in Practice

Confidence Scoring supports several distinct uses, and each is suited to a different situation. Reading individual scores alongside their written explanations is most useful while diagnosing a specific result or developing an agent, when the question is why a particular step scored the way it did. Aggregated and summarized views are better suited to questions about an agent as a whole, such as establishing what typical performance looks like, comparing two configurations or inputs through A/B testing, and watching for drift once an agent is running in production. Which modules to enable is also worth revisiting over time. As an agent runs, it becomes clearer which quality dimensions matter most for it, and the enabled set can be narrowed to focus on them.