← All posts·How to

How to Build Conversation Evals for AI Agents: An End-to-End Guide

Noor· November 4, 2026 · 48 min read

How do you know your AI agent is ready for real customers?

For most teams, the honest answer is that someone called it a few times and it sounded fine. I get why. It's quick, and a good agent does sound good on a test call.

The trouble is that sounding good and doing the job are two different things. An agent can be polite and natural and still discuss a debt with the wrong person. It can tell a caller their payment went through when it didn't. It can set up a payment plan for the wrong amount and confirm it cheerfully.

You won't hear any of that on a demo call. You'll hear about it from a customer, unless you test for it first. That's what conversation evals are for.

This guide covers the whole process, from working out what to test to blocking a release when a test fails. I'll use one agent as the example all the way through: a voice agent at a collection agency that collects overdue payments for a lender. It calls customers, and it answers when they call back. It's a useful one because it has most of what makes evals hard. Money moves, the rules are strict, callers are often upset, and the payment system doesn't always answer.

What Are Conversation Evals?

A conversation eval tests your agent the way a caller actually uses it: across a whole multi-turn call, not one prompt and one reply. A simulated caller plays a scenario you've defined (or you replay a real recorded call), and when the call ends, the eval checks two things against criteria you wrote down beforehand. Did the right thing happen in your systems? And did the agent behave correctly along the way?

Here's what that looks like for our payment agent. Say the person who picks up says, "I'm his wife. How much does he owe?" The eval passes only if the agent follows its verification path, doesn't reveal the debt, and the call ends with the right outcome logged in your CRM. A polite, natural-sounding call that mentions the balance still fails.

What you get from each eval run

Each run should leave you with four things. Here's what each one looks like for the "I'm his wife" scenario:

Each one answers a different question. The verdict tells you whether to ship. The breakdown tells you what to fix first. The evidence lets an engineer reproduce the problem instead of guessing at it. And the baseline tells you whether your fix actually helped.

The baseline is also the only one you can't create after the fact. Say you tighten the verification prompt on Tuesday. The "I'm his wife" scenario now passes, but task completion across your other scenarios drops from 92% to 85%. If you saved last week's run, you can see which scenarios got worse and trace it to the prompt change. If you didn't, you can't tell whether the prompt caused the drop or something else changed.

The lifecycle, start to finish

The whole process runs as a loop:

Discover requirements → define metrics → build scenarios → execute conversations → grade results → debug → regress → monitor.

It's a loop because the last step feeds the first. Say monitoring turns up a real call where someone says "I'm his sister" and the agent handles it differently from the wife scenario. That call becomes a new scenario and goes back through the same steps.

The rest of this guide works through that loop.

Start With the Customer's Workflow

Before you write a single test, get to know the customer's business. What are they trying to get done? How do their human agents handle these calls today? Which rules do they have to follow?

It's tempting to skip this and start writing scenarios. I wouldn't. When you start from the customer's real workflows, your tests end up looking like the calls your agent will actually get. You're also testing against what the customer expects, which is the standard your agent will be judged on anyway.

Here's what to map, how to gather it and what it gives you. The last column shows what each one looks like for the payment agent:

That last row matters more than it looks. A call can end without a payment and still be handled perfectly. If your evals only count payments, they'll mark the correct dispute handoff as a failure.

Every step on the main path is a decision point, and every decision point needs a test. Take the plan step. You need one scenario where the caller accepts the first plan offered, one where they ask for less than the minimum, and one where they ask for something outside policy.

Look at real calls before you write synthetic ones

When you invent test cases from scratch, you mostly end up testing the problems you already expected. Real calls show you the ones you didn't.

So start with whatever already exists: call recordings, chat logs and support tickets. If the agent isn't live yet, use recordings of the customer's human agents doing the same job.

Then compare what the agent said with what actually happened in the business systems. Say the agent told a caller "your payment is confirmed". Is there a payment in billing, for that amount, on that account? If there isn't, you've found a failure that no amount of reading transcripts would catch.

How do you go through the calls without drowning in them? Hamel Husain, an independent consultant who helps teams build evals, describes a method that works well. Read at least 30 calls yourself and write a short note on what went wrong in each. Focus on the first thing that went wrong, because an early mistake often causes the later ones. For a voice agent, that means listening as well as reading. Talking over the caller and long silences don't show up in a transcript.

Then group your notes into failure types and count them. Hamel calls this the most important step, because the counts tell you which tests to write first. Keep reading new calls until they stop showing you new kinds of failure.

For the payment agent, your groups might be "mentioned the balance before verification", "missed a dispute", "read the account number back wrong" and "said the payment worked after a tool error". It helps to file each group under a broader layer too: understanding, reasoning, policy, tool execution, conversational behaviour or infrastructure. That tells you who'll fix it. Your first pass won't be perfect, and that's fine. The groups will change as you read more.

Keep confirmed incidents apart from hypothetical risks. A confirmed incident comes with a recording and a real cost. A hypothetical risk is still a guess. Both deserve tests, but you'll prioritise them differently.

Write a short spec for each workflow

For every important workflow, write down the business goal and what success means. Write success as a state you can check in a system, so there's no argument later about whether a run passed. Add the behaviours that are required and the ones that are banned, the system state the call starts from, and the failures you already know about, with how serious each one is. Then note what you'll check against as ground truth, and who signs off on the criteria.

For the payment workflow, it could look like this:

• Business goal: collect the payment or set up an approved plan.
• Success: the payment or plan is recorded in the billing system, for the correct amount.
• Required: verify the caller's identity and give the required disclosure.
• Prohibited: discussing the debt with anyone who isn't authorised, or promising terms outside policy.
• Starting state: an account with an open balance, and a caller who hasn't been verified yet.
• Known failure: a duplicate charge after a payment timeout. Severity: critical.
• Ground truth: the billing system record and the call recording.
• Sign-off: the compliance lead and the collections operations manager.

Decide What to Measure

I'd split your metrics into four groups instead of keeping one long list of voice AI KPIs.

Each group answers its own question. Goal metrics tell you whether the agent did the job. Guardrail metrics tell you whether it broke a rule. Operational metrics tell you whether it performed well enough on real calls. And diagnostic metrics tell you why it failed.

Mix them into one list and those questions blur together. A call where the agent took the payment but mentioned the balance before checking who it was talking to would score well on the goal and badly on a guardrail. In one list, the good score hides the bad one.

Goal metrics

Goal metrics measure the outcome that matters to the user or the business.

There's one rule here that's easy to get wrong. A task only counts as done if it's done in your systems. The payment agent succeeds when the payment exists in billing, for the right amount, on the right account. The agent saying "your payment is confirmed" proves nothing.

So what does checking it in your systems look like? For one payment call, you'd check:

• The agent's logs. Did the payment tool actually run, and what did it return?
• The billing system. Is there one payment, for the agreed amount, on the right account?
• The CRM. Is the call outcome logged correctly, for example "paid in full" or "plan agreed"?
• The follow-ups. Did the receipt or confirmation go out, and did any task the call should create get created?

Until those records agree, you don't know the payment happened.

Treat these targets as starting points. There's no industry standard for them, so agree real numbers with your customer for each workflow.

Containment needs a warning label. A caller who gives up and hangs up was "contained" too, so a high containment rate can hide a lot of unhappy people. When Gartner surveyed 5,728 customers in 2024, only 14% of customer service issues were fully resolved in self-service. Even for issues customers described as very simple, it was just 36%.

That survey looked at self-service in general, so don't read it as a voice agent benchmark. It's still a good reason to always read containment next to resolution, and to count the calls that ended because the caller hung up separately from the calls that ended because the job was done.

Escalation is why that metric is split into two rows. A high escalation rate isn't automatically bad. If the caller disputes the debt or asks for their lawyer, handing off is exactly what the agent should do, and missing that handoff is far worse than making one too many. One overall rate can't tell those two apart. Two rows can.

Guardrail metrics

Guardrails are the things your agent must never do, even when the task succeeds.

Usually there's no target or critical rate for these. A guardrail has to hold every time, because a single breach can be a compliance violation.

These are the guardrails for our payment agent:

The first three rows come from the US debt collection rules in Regulation F. In short, the collector has to say in its first communication that it's collecting a debt, and it can only discuss the debt with a short list of people, starting with the consumer. So your agent has to know who it's talking to before it mentions the debt at all.

One detail catches people out: for that rule, "consumer" includes the consumer's spouse. That's why the "I'm his wife" scenario needs its own verification path, and why you'll want separate tests for a spouse and for someone like a sister. Your guardrails will come from whatever rules apply to your customer's industry, and this isn't legal advice, so check the exact requirements with their compliance team.

Operational metrics

These tell you whether your agent holds up on realistic calls. A quiet demo call won't.

Look at percentiles as well as the average. If most turns come back quickly but one in twenty takes several seconds, the average still looks fine. The caller who waited doesn't think so, and on a payment call a long silence is often when they start to wonder whether the payment went through.

Concurrency is the one teams skip. An agent can be great on one test call and struggle when 100 come in at once. Test at the load you expect, then above it.

Debug and diagnostic metrics

Diagnostic metrics help you find where a failure came from, so you know which part of the agent to fix. I'd track four kinds.

Speech recognition errors and critical entities. These mostly come from your ASR: words inserted, deleted or swapped for the wrong word. Give extra weight to the words that matter most, like account numbers, amounts and dates. One wrong digit in an account number does far more damage than a dropped "um".

This is harder than it sounds, even for the best systems. Sierra's τ-Entity benchmark gives voice agents short calls where a cooperative caller supplies one value, such as a name, phone number, address or email. The top system captured the value exactly right in only 60.3% of regular calls across six languages (74.0% in English), and on its leaderboard it succeeded in just 34% of tasks under all three audio conditions tested. The paper behind it found the errors add up. When each field was right 75% of the time on its own, a call that needed all three fields right succeeded only 53% of the time. Agents that went back to check a value fixed just 24–37% of their mistakes. And having the agent ask for the value, ask the caller to spell it, then read it back raised success by 14–31 points, but added 21–28 seconds to the call. For the payment agent, that's a trade-off worth testing directly: run the account number scenarios with and without a read-back step, and see whether the accuracy gain is worth the extra time.

Context retention. This one's mainly about your LLM. Does the agent remember what the caller said three turns ago, or does it ask again? If the caller gave their date of birth during verification, the agent shouldn't ask for it again before taking the payment.

Tool calls. Count the failed tool calls and break them down by cause. Sometimes the agent called the wrong tool. Sometimes it called the right tool with the wrong parameters. And sometimes the tool itself failed and the agent handled the error badly. Each of those needs a different fix.

Turns, reprompts and needless clarifications. Count the turns in each call, how often the agent had to reprompt, and how often it asked for clarification it didn't need. Also watch for calls that end with an invalid disposition or that trigger the wrong workflow.

Component-level evals

Component evals test each part of your system on its own: speech recognition, speech synthesis, tool integrations and turn detection.

If you're already testing end to end, why bother? Because when an end-to-end test fails, the component scores show you where to look.

It doesn't work the other way round, though. Great component scores don't guarantee a good conversation, so you need both. τ-Entity shows how wide that gap can be. In the paper, a text agent given the same tasks passed every one, while the voice setups got the value exactly right, under every audio condition, on only 14–41% of them. The tasks were the same. What changed was that the details had to survive a phone call.

A tool call returning HTTP 200 only tells you the request got through. A declined payment can still come back as a 200, which is why the goal metrics check the outcome in billing.

A word on UTMOS. It was built and tested on synthetic speech from academic challenges, so compare its scores with real listeners on your own voices before you lean on it.

WER needs similar care. A low overall WER can still hide errors on the words that matter. If the agent hears "four hundred" as "fourteen hundred", that's one wrong word out of hundreds, and a very expensive one. So measure accuracy on amounts, dates and account numbers separately from the overall rate.

Build Your Eval Scenarios

You know what you want to measure. Now you need the scenarios that will produce those measurements.

What goes into a scenario

Every scenario needs eight parts.

The first four set the scene. The ID and purpose gives the scenario a stable ID and names the behaviour you're testing. The initial state covers the records, permissions, balances or bookings that exist when the call starts. The user persona describes what the simulated caller knows, wants and is willing to share. And the interaction instructions say how that caller behaves and when they reveal things.

The other four decide the result. Expected outcomes describe the final business state and any conversational behaviour you need to see. Guardrails list the banned actions and the required policy steps. Judges are the checks that decide whether the run passes. And severity and tags record the impact, the workflow, the failure mode and who owns it.

Here are eight scenarios for the payment agent in that format:

You'll notice the Judges column mixes code, Jev, LLM and human checks. Jev is a newer kind of judge model that returns typed answers, such as a category or a probability, instead of text. I'll cover all four and how to choose between them in the next section.

Remember the spouse detail from the guardrails? Scenario 2 is where it shows up. Run it once with a caller who says they're a sibling and once with a caller who says they're a spouse, because the right answer may not be the same.

How many scenarios do you need? Microsoft's evaluation guidance for its Copilot agents suggests 20–50 test cases for a prototype, 50–100 before production and 100 or more once you're live. It wasn't written with voice agents in mind, but it's a reasonable place to start.

Make each scenario give the same answer twice

A scenario is only useful if a failure means the agent got something wrong, not that the test setup changed. A few details decide that.

Fix the clock and the time zone. If the agent says "your payment is due tomorrow", the right answer depends on today's date. Set the date and the caller's time zone in the scenario's initial state, so a test that passed on Monday doesn't fail on Friday for no reason.

Fake the tools, and make the fakes fail on purpose. Your scenarios shouldn't hit the real billing system, so you'll use a stand-in payment tool that returns whatever the scenario needs. Make sure it can fail in the nasty ways too. The one that matters most for this agent: the payment goes through, but the response times out, so the agent never hears that it worked. That's exactly the setup scenarios 4 and 6 need, and a fake that can only return "success" or "error" can't test it.

Write tests for what must not happen. Most scenarios check that the agent did something. Some should check that it didn't. A caller who only asks for the office address shouldn't trigger a payment call or a balance lookup. If you only test for actions, an agent that does too much will look fine.

Use audio that sounds like a phone call. If the simulated caller's audio is clean studio speech, your ASR will look better in testing than it does on real lines. Pass the test audio through the same phone codec your callers use, and add background noise to some of the scenarios.

Know where a replayed call stops being valid. Replaying a real recorded call is a good way to reproduce a failure, but it only works up to a point. The caller's recorded words were replies to what the old agent said. Once the new agent says something different, the next recorded line may not make sense as a reply. Treat a replay as valid up to the first turn where the agent's response changes, and use a simulated caller from there.

Go beyond the happy path

Your golden workflows come first. These are the must-have journeys that have to pass before any version goes live. For the payment agent, that's a cooperative caller who pays in full.

Most real failures happen off that path, though, so your dataset needs other kinds of scenario too.

Some callers won't give you enough to work with. They don't know their account number, or they can't remember which card they used. Others change direction halfway through, starting to pay in full and then asking about a plan. Those are your missing-information and branching cases.

Then there are recovery cases, where a tool fails, sends back something unexpected or needs a retry. And there are adversarial and policy cases: conflicting requests, attempts to get something the agent isn't allowed to give, and deliberate attempts to get around the rules. Put profanity and every other kind of hostile caller here as well.

The last group is the long tail. Think industry jargon, strong accents, background noise, odd phrasing, and situations that are rare but expensive when they go wrong. Callers also say a lot that has nothing to do with the task: "say that again", "hold on a second", "who am I speaking to?". Each of those needs a defined behaviour, and a test.

One warning about the simulated callers you'll use for all of this. The model playing the caller affects the result. In Lost in Simulation, an ACL 2026 paper, swapping only the model that played the user moved agent success rates by up to 9 points. Simulated users also made the hardest tasks look harder than they were, and moderately difficult ones look easier. They were also best at imitating speakers of Standard American English, and worst at imitating speakers of African American English and Indian English. If your callers are in India, that one matters.

So treat the simulator as part of the test setup, not as a neutral stand-in for a caller:

• Pin the simulator. Record which model and prompt played the caller, and don't change them between the runs you're comparing. Otherwise a 5-point drop could be the simulator, not your agent.
• Run important suites with two simulator models. If both show the same failure, it's probably the agent. If only one does, look at the simulator first.
• Use the simulator's script as ground truth. The simulated caller knows the account number it was told to give. Compare what the agent captured with that value, rather than with a transcript that could have its own errors.
• Check simulated results against real calls. Simulation is best for catching regressions between versions. It isn't a reliable estimate of how many real calls will succeed.

Choosing Your Judge: Code, LLM, Jev or Human

A lot of guides treat automated grading as one thing, usually LLM-as-a-judge.

I think that's a mistake. Each kind of judge is good at different things, and picking the right one for each check is one of the biggest decisions you'll make.

The four judges at a glance

Here's how they compare:

Code-based judges

Use code whenever there's a right answer you can check exactly.

Take the payment agent. A short Python script can read the transcript and event log, find the "caller verified" event, find the Mini-Miranda wording, and check that they happened in that order. There's nothing to interpret. The order is either right or wrong.

Code is also the right judge for questions like these. Did the billing system record the payment for the correct amount? Did the plan get saved with the agreed dates? Were the required tool arguments passed? Was an action blocked when authorisation was missing? Did turn latency go over your limit? And where the policy sets out exact wording, did that wording appear?

Code is cheap and fast, and it gives the same answer every time. Its limit is that it can only check what you can spell out precisely. So a good habit is to ask of every check, "could code do this?" before you hand it to a model.

Jev-as-a-judge

Jev is a newer kind of model from TypeSafe AI. The company announced it on 15 September 2026 and calls it the first of its "System One" models.

It works differently from an LLM. Jev doesn't write an explanation. You give it a state (the conversation, or just the part you care about) and a set of typed questions. It sends back typed answers with probabilities.

There are three types of question. A choice question picks one option from a set you define. A noul question is a yes/no question that returns a probability. A score question rates the state on an ordered scale.

Here's how each one could work in a real eval.

Example 1: classify the failure (choice). Ask, "What best describes the failure in this conversation?" and give it these options:

• misunderstood_user
• policy_violation
• tool_execution_failure
• incomplete_task
• no_failure

The answer can go straight into your failure dashboards and issue routing.

Example 2: did the agent follow the caller's latest intent? (noul). Ask, "Did the agent respond to the user's latest correction, instead of carrying on with the previous request?" You get back a probability. Send the low-confidence cases for a closer look, and compare the results with a sample your team has checked by hand.

Example 3: score frustration (score). Ask, "How frustrated does the user seem by the end of the interaction?" and give it a defined scale, such as 1–5. Add up the scores across conversations. Then dig into the calls where frustration climbs alongside repeated reprompts or failed tasks.

These are examples of how you could use it. You'll still need to write your own question definitions and input format, and test them in your Jev integration.

Does it work? It's early. LangChain added Jev as a judge in LangSmith in September 2026 and tested it against three LLM judges on one of its agents. Jev matched a human reviewer on every decision. It took 0.44 seconds per call on average and cost $0.00035 per call.

That's one test on one agent. It's a promising sign, so keep watching it.

Langfuse's documentation has a practical tip: keep the state lean. Jev gets less accurate when the state includes things the questions don't need. Pass in only the fields you want judged, not the whole trace. For the "I'm his wife" scenario, that might be just the turns up to the first mention of the account, plus the verification events.

So where does Jev fit? It suits high-volume classification of traces, and yes/no or category questions that are hard to turn into exact code rules. It's also handy when you want several separate labels or scores from the same conversation. And it fits production scoring, where cost and speed decide how much of your traffic you can afford to evaluate.

Jev can do more than grade, too. We've written about using it as a deterministic safety layer in a live agent: Architecting Reliable Conversation AI.

LLM-as-a-judge

Use an LLM judge when a knowledgeable reviewer would find the requirement obvious, but you can't write it as code.

Clarification is a good example. When the caller gave too little information, did the agent ask a sensible question, or did it guess? Grounding is another. Was the agent's explanation correct and complete, and did it match the tool result and the policy?

Recovery fits too. After the caller corrected the agent, did it acknowledge the correction and act on it, or drift back to the old request? So does policy adherence across a whole call, where the agent might follow the policy perfectly while wording things differently from your reference example.

LLM judges can be good at this. A well-known 2023 study found that strong LLM judges agreed with human preferences more than 80% of the time. That's about as often as humans agree with each other.

The same study also found real biases. LLM judges can favour an answer because of where it appears, prefer longer answers, and rate their own outputs more highly. So check your judge against examples your team has labelled before you trust its scores.

How do you set one up so it's worth trusting? Hamel Husain recommends a pass/fail verdict for each criterion rather than a 1–5 score, because it's much easier to agree on what "fail" means than on the difference between a 3 and a 4. A few more things help:

• One judge per criterion. Have one judge decide "did the agent reveal the debt before verification?" and a separate one decide "was the dispute handled correctly?". A single judge scoring everything at once is harder to check and harder to fix.
• Give it what a human reviewer would have. For the payment agent, that's the transcript plus the relevant policy text and the tool results. A judge can't tell whether "your payment is confirmed" was true if it can't see what the payment tool returned.
• Let it say it can't tell. Add an "unknown" option, and send those calls to a human. Otherwise the judge has to guess, and guesses look like verdicts.
• Make it quote the turn. Ask the judge to cite the exact turn behind its verdict. It makes each verdict quick to check, and it shows you straight away when the judge has misread the call.
• Put tricky calls in the labelled set. Include calls that look fine but aren't. For example, the agent says "I can't discuss the $340 balance until I've verified you", which is polite, follows the script, and has just revealed the balance. A judge that only sees a refusal will pass it.
• Pin the model version. Use an exact model version, not an alias that updates on its own, so the judge doesn't change between two runs you're comparing.

Then measure the judge before you rely on it. Hamel's method for validating a judge uses about 100 calls labelled pass or fail by a domain expert, roughly half of each. Use a few of them as examples in the judge's prompt, tune the prompt on about half of the rest, and keep the remaining calls aside for the final check.

On that final set, look at two numbers rather than overall accuracy. How often did the judge pass a call the expert passed? And how often did it fail a call the expert failed? He suggests aiming for above 90% on both, and treating 80% as the minimum.

The second number matters most for guardrails. Say the agent gets the "I'm his wife" scenario right 98% of the time. A judge that passes every call scores 98% accuracy and never catches a single disclosure. Only the second number shows that.

Human evaluation

Humans are slow and expensive, so you won't put them on every call. You can't leave them out either, because they're how you know your other judges are right.

Your reviewers set the first definition of what acceptable behaviour looks like. After that, they handle the ambiguous, high-impact and low-confidence cases. They also look into any gap between what the judges said and what the product team expected.

Their labels become the trusted set you calibrate the automated judges against. And they should repeat those reviews on a regular schedule, because a judge that was right last month may not be right now. Expect the rubric to change during the first few rounds, too. Reviewers often only work out what "handled correctly" means once they've seen enough calls that weren't.

Hosted judge models can change underneath you. Vinay Karumuri, a staff SDET working on agent quality, suggests an anchor set: a small, fixed set of items with frozen human labels that your current judge re-scores every week. If the anchors move while production stays steady, the judge drifted. If production moves and the anchors don't, the agent changed.

Use more than one judge per scenario

You don't have to pick a single judge for each scenario. Most of the important ones need several.

Take an appointment-booking agent. Code checks that the correct appointment exists in the backend. Jev classifies whether the caller's latest correction was handled. An LLM judge assesses whether the conversation was clear and appropriate. And a human reviews a sample, especially where the automated judges disagree.

The payment agent works the same way. In scenario 6, code counts the successful transactions and Jev classifies how the transaction was handled. An LLM judge assesses the recovery conversation, and a human looks at any case where they disagree.

Run the Evals and Read the Results Properly

With scenarios and judges in place, you need to turn them into a run you can repeat and compare from one version to the next.

How a run works

Every run should follow the same seven steps:

1. Load the scenario and set up an isolated system state.
2. Start the agent and the simulated caller.
3. Run the conversation, capturing messages, audio, events and tool activity.
4. Check that the test run itself was valid.
5. Run the selected judges against the right evidence.
6. Save the raw results, the judge outputs and the final business outcome.
7. Reset or clean up the test state.

Teams often skip step 4. If the simulated caller went off-script or the payment sandbox was down, the run tells you nothing about your agent. Mark it invalid instead of counting it as a fail.

Run each scenario more than once

Your agent won't behave exactly the same way twice. A scenario that passes once might not pass the next time.

So run your stochastic and high-impact cases several times. Report valid and invalid runs separately, and work out the pass rate across all the trials. Track how consistent the agent is, as well as its best result. When you compare two versions, run both on the same scenarios and compare them scenario by scenario, so a difference in which scenarios ran doesn't look like a difference in the agent. And if you only have a handful of runs, say so, because a small sample can't support a strong conclusion.

The τ-bench researchers showed how big this gap can be. They measured pass^k, the chance that all k attempts at the same task succeed. In their 2024 tests, GPT-4o succeeded on about 61% of retail tasks on a single try. When it had to succeed on all eight of eight tries, that fell below 25%.

For the payment agent, the eight-out-of-eight number is the one to watch. Every scenario runs thousands of times in production, once for each caller who brings it. An agent that gets the account number right most of the time still gets it wrong for a lot of real people, and pass^k shows you how often "most of the time" breaks.

Microsoft's guidance points the same way. It recommends running evaluations several times and says core regression tests should get close to 100% consistency.

Decide what "pass" means before you run

Write the pass rules down first. Otherwise you'll end up arguing about each result after the fact.

Critical guardrails have to pass, and the required business outcome has to be confirmed in your systems. Non-critical quality checks can each have their own threshold. Judge operational limits against the use case, because a slow answer hurts more on a live call than in a chat. And where it helps, show partial success openly.

Partial success matters more than it sounds. An agent that verifies the caller correctly but fails to take the payment is in a very different place from one that ignored the request altogether. Both fail the task, but your failure labels should keep the difference.

Be careful about what a clean run proves, too. If the "I'm his wife" scenario passes 50 times out of 50, it's tempting to say the agent never leaks the balance. It doesn't prove that. A quick rule of thumb, the "rule of three", says that with zero failures in n runs, the real failure rate could still be as high as about 3 ÷ n. So 50 clean runs is still consistent with the agent failing about 6% of the time. If you need to show the rate is below 1%, you need around 300 clean runs, not 50.

Report at the right level

One overall score hides too much. Break your results down five ways:

• Per criterion, such as compliance pass rate, task success or latency.
• Per scenario, so you know whether each one meets its release bar.
• Per workflow, across the set of workflows that matter.
• Per agent version, against the baseline.
• Per business-critical category, so your high-risk workflows don't disappear into an average.

Keep guardrail breaches out of the averages and report them on their own.

Say the payment agent scores 97% overall. Looks great. But if the missing 3% includes one call where it discussed a debt with a third party, that's the only number your compliance team will care about.

Turn Failed Evals Into Fixes

Finding a failure is half the job. The other half is giving the engineer who has to fix it something they can act on.

Sort each failure first

Put every failure into one bucket. Did the agent misunderstand a requirement or policy? Did it lose track of where the conversation was, or what the caller wanted? Was it a tool problem, like the wrong tool, the wrong arguments or a broken integration? Was it speech recognition, turn detection or TTS? Or was it timing, concurrency or infrastructure?

There's one more bucket, and it's easy to forget. Sometimes the agent is fine and the test is broken. The simulation, the test setup or the judge itself can be wrong.

Find the cause before you touch the agent

When an eval fails, it's tempting to go straight to the prompt. Hold off for a bit.

Start with the original conversation and its event timeline. Compare the state you expected with the state you got, and look through the model, tool and runtime traces. If you can, replay the failing component on its own. Check whether the judge got it wrong. Then reproduce the failure before you change the prompt or the code.

For a voice agent, one quick test splits the problem in half. Take a corrected transcript of what the caller actually said (not the ASR's version, which already contains any mishearings) and replay it to the agent as text. If the agent gets it right in text, the problem is in the speech layer: something was misheard, cut off or mistimed. If it gets it wrong in text too, the problem is in the reasoning, the prompt or the tools, and you can debug it there, where it's faster and cheaper.

Here's why finding the cause matters. Say scenario 6 fails and the agent charged the caller twice. The obvious fix is to add "never retry payments" to the prompt.

Look at the trace first, though. If the payment tool timed out and the agent had no way to check the transaction status, the problem is in the tool. A prompt change would only hide it.

Some fixes don't belong in the prompt at all. If a rule must hold every time, enforce it in code. The account lookup can refuse to return the balance until the verification step has succeeded, so the agent can't reveal what it doesn't have. And each payment request can carry an idempotency key, a unique ID for that payment, so the payment system ignores a second request with the same key. With those in place, the tools themselves block the two failures behind the "I'm his wife" call and the "just try it again" call, so a model mistake can't cause them through the tools. Keep the tests anyway. They now check that the safeguards still work.

Make every failure a test

This is the habit that makes the rest of this guide pay off. Each failure you find becomes a test that stops it coming back:

1. Find a failure through testing, manual QA or production monitoring.
2. Log an issue with the expected behaviour, what actually happened, the severity and the evidence.
3. Write a scenario that reproduces it.
4. Confirm the new eval fails while the old behaviour is still there.
5. Fix the agent and rerun the scenario.
6. Run the wider regression suite to check for side effects.

Step 4 gets skipped a lot, and it shouldn't. If the new eval passes before you've fixed anything, it isn't testing the problem.

Microsoft's agent evaluation guidance describes the same loop. Triage the user report, add it as a test case, fix the issue, and keep the case in your regression set so it stays fixed.

Keep fixed failures fixed

Once something's fixed, its test stays. Keep your important regression cases under version control. Don't retire an old scenario just because the agent has got better at it. Change or remove a test only when the requirement behind it actually changes.

Keep an eye on how often fixed failure types come back, too. And when you ship a fix, note whether it solved one case or a whole category of behaviour. They're worth very different amounts.

Evaluate Before Every Release

Evals only pay off when they're part of how you ship. If you run them when you remember to, they won't catch much.

Pick the right suite for each change

You don't need to run everything on every change. Match the suite to what changed.

Relevant code or config changes need fast component tests. Your core journeys and guardrails need a critical regression suite. Release candidates get the larger end-to-end suite. Voice-specific changes need targeted audio and full-duplex tests. And anything that could affect capacity or latency needs load and operational tests.

It also helps to agree what counts as a change, because plenty of them don't touch your code. For a voice agent, the list includes:

• A prompt edit.
• A new model, or a new version of the same model.
• A different ASR or TTS provider, or a new voice.
• A change to a tool's inputs or outputs.
• An update to the knowledge base or a business rule, such as a new minimum plan amount.
• A telephony or turn-detection setting.
• A new version of the judge or the simulated caller.

The last one is easy to miss. If you change the judge and the agent at the same time, you can't tell which one moved the score.

Keep a holdout set as well: a group of scenarios you never look at while you tune the prompt. If you keep adjusting the agent until the regression suite passes, it can end up good at those exact scenarios and not much better at anything else. The holdout set shows you whether the improvement is real.

When should evals run? Three triggers fit together well. Run them on demand while you're building. Run them automatically on every change through CI/CD, or on sampled production traffic. And run them on a schedule, to catch quality that slowly drifts as user behaviour and models change.

Decide what blocks a release

Decide in advance what stops a release, so nobody has to argue about it on release day.

Critical policy or safety failures should block it outright. Set a minimum level for your goal metrics, and reject changes that break your operational limits. Compare against a fixed baseline. Agree up front how you'll treat flaky or invalid runs. And name the high-impact exceptions that need a person to approve them.

For the payment agent, a single failure on scenario 2, the third-party disclosure, blocks the release. No average makes up for it.

Make the results usable

A red build on its own doesn't tell anyone much. Each failure should link to its conversation, trace and audio. It should show which criteria got worse and how this version compares with the baseline. Show the spread of reliability and latency results as well as the averages. And name an owner and a next step.

That's what CI/CD for evals is really for. Automating the test runner is the easy part. The goal is to make the evidence part of the release decision.

Tools for Building Conversation Evals

You don't need any particular tool to do this. You do need something for each job in this guide: scenarios, simulated callers, judges, a way to run it all again and again, and a way to learn from real calls. Here are three options that cover most voice teams, and where each one fits.

1. RubricHQ

Full disclosure: this is what we build. RubricHQ is a testing and evaluation platform for voice and chat agents, and it covers the whole loop in this guide in one place.

• Scenarios and callers. Generate test scenarios from your agent's prompt, and run them against a library of caller personas, including angry, confused, elderly, multilingual and adversarial callers.
• Running evals. Run them as real phone calls, web calls or text-only chats, many at once, and repeat each scenario as often as you need for a pass rate you can trust.
• Judges. Score every conversation with code checks, LLM judges and audio checks for latency, dead air and interruptions.
• Production. Monitor live calls as they land. Co-Pilot reads every transcript, groups calls by the reasons behind them and ranks the recurring problems worth fixing, with the calls behind each one.
• Fixes and releases. Diagnose failing calls and generate prompt rewrites, then rerun your suite to prove the fix before you ship. Suites can run on a schedule or from your CI pipeline through the API.

It works with Vapi, Retell, LiveKit and Pipecat agents. It's the right fit if you want simulation, grading and production monitoring without building and maintaining your own harness.

2. LangSmith and Phoenix

LangSmith and Arize Phoenix are tracing and observability tools. They record every step your agent takes: each model call, each tool call and its arguments, and how long each one took. LangSmith is LangChain's hosted platform, and Phoenix is free to self-host.

They're strongest at the debugging step of this guide. When an eval fails, the trace shows you whether the fault was in the prompt, the tool or the model, which is exactly what you need before you touch the agent. Both let you run evaluators over traces too.

The catch is that a trace shows you what happened, not what went wrong for the business. Finding issues in production is mostly manual: someone reads through traces, spots a pattern, works out the cause, and turns it into a test case by hand. Neither tool simulates callers or tests audio on its own, so for a voice agent you'll pair them with something that does.

3. Pipecat Evals

Pipecat is an open-source framework for voice and multimodal agents, with about 17,000 GitHub stars and a BSD 2-Clause license. Pipecat Evals is its built-in testing system, and it runs scenarios against the same pipeline and services you deploy.

Scenarios are YAML files, and there are two kinds. A scripted scenario writes out the caller's turns and what the bot should say after each one. A simulated scenario has an LLM play a caller with a goal, and a judge reads the whole conversation to decide whether the bot did its job. You can run a scenario in text mode, which is fast while you work on prompts, or in audio mode, which speaks the caller's lines and transcribes the bot's replies. The judge is pluggable. The Pipecat team's own release suite uses Jev, and Cloudflare's Clef or a local model through Ollama also work.

Why should voice teams consider it? If your agent is built on Pipecat, it's the cheapest place to start. Because it drives your real bot through the real pipeline, it catches problems with turn-taking, interruptions and service settings that a text-only test would miss. The Pipecat team uses it to check more than a hundred example bots before every release.

The limit is that it's a harness, not a platform. It only works with Pipecat bots, and you build and maintain the scenarios, result history, comparisons between versions and production monitoring yourself.

Which one do you need?

They aren't mutually exclusive. A common setup is Pipecat Evals or RubricHQ to run your scenarios before every release, a tracing tool like LangSmith or Phoenix to debug individual failures, and RubricHQ's monitoring and Co-Pilot to find the production problems you didn't think to test for.

Whatever you pick, don't compare on cost per test alone. Count the engineering time to maintain it, how much of your workflow it covers, and how easily a production failure turns into a regression test.

Where to Start

If you take one thing from this guide, make it this. Check the outcome in your systems, because a conversation can sound perfect and still leave the wrong value in your database.

Start with your most important workflow and write five scenarios for it: a golden path, a missing-information case, a tool failure, a policy case and a hostile caller. Give each one a code check on the business outcome. Then run each scenario five times. Those 25 runs will tell you more about how consistent your agent is than any number of demo calls. They won't prove it's reliable, as the rule of three showed, but they'll catch the big inconsistencies, and that's enough to start.

For the payment agent, that first five could be a caller who pays in full, a caller who can't find their account number, a payment that times out, the "I'm his wife" call, and a caller who says the debt isn't theirs and gets angry when the agent keeps asking for payment. Once those pass reliably, add the next workflow.