Test your AI agent with simulated customers

Tests in Chatbase check how your AI agent handles important conversations, before you launch it and after every change you publish. A test is a conversation your AI agent has with a simulated customer: you describe who the customer is, what they want, and what has to happen for the conversation to count as a success. Chatbase then runs that conversation as many times as you ask, grades each one automatically, and shows you the transcripts. Use tests to check how a change affects your AI agent: new instructions, a new Data Source, a Procedure, or an Action. Tests run against your published AI agent, not your draft, so publish the change first, then run the same tests again and compare the pass rate. If the pass rate drops, restore the previous version from Version history. Tests live under Tests in your AI agent’s sidebar.

​

How a test run works

Each run starts one or more conversations. In each conversation:

1. A simulated customer, played by another AI model, opens the conversation based on the scenario you wrote. It only knows the scenario and the conversation so far. It never sees your agent’s instructions or the success criterion.
2. Your AI agent replies using its real setup: its instructions, Data Sources, Actions, and Procedures.
3. They keep talking until the customer is done or the conversation reaches its max turns.
4. A judge, another AI model, reads the finished conversation and decides whether it met your success criterion. The judge sees the messages and every Action the agent called, with what it sent and what came back.
5. If the test lists expected Actions, Chatbase also checks which Actions the agent called.

A conversation passes only when the judge passes it and the Action check passes. The simulated customer and the judge use the same model in every test. This keeps pass rates comparable between tests and between runs.

​

What to test first

You can have up to 50 tests per AI agent, so start with the conversations that matter most:

1. Your most common questions. Check Analytics for your top topics and write a test for each.
2. Anything that triggers an Action. Refunds, bookings, order lookups, and subscription changes are where a wrong answer costs the most. Use Expected Actions to check the right Action is called.
3. Each Procedure. One test per Procedure, with a customer who needs it from start to finish.
4. Handing off to a person. A frustrated customer who asks for a human should reach your escalation path.
5. Questions your AI agent shouldn’t answer. Off-topic or out-of-scope requests check that your instructions and Guardrails hold.
6. Failures. Turn on Return as error in an Action response to check the AI agent handles a failed Action gracefully.

​

Example tests by use case

Adjust the wording to your own products and Actions. The Expected Actions column uses the names of the Actions you’ve added to your AI agent.

​

Create a test

On the Tests page, click Create a test.

​

Expected Actions

This field decides whether Chatbase checks the Actions your agent calls. It behaves differently when it’s empty and when it has Actions. Empty. This is the default. Chatbase doesn’t check Actions. The agent can call any Action, or none, and only the judge grades the conversation. Use this when the outcome is all that matters, for example “the agent explains the refund policy correctly”. The test’s Configuration tab shows Not asserted. With Actions. The agent must call exactly the Actions on the list. No more, no fewer. The Action check fails if the agent:

• never called an Action on the list, or
• called an Action that isn’t on the list.

A failed Action check fails the conversation, even when the judge passed it. Adding one Action makes the check strict for every Action. Any Action the agent calls that isn’t on the list fails the conversation. So list every Action the agent should call in this scenario, not only the main one. A few rules for how the check counts calls:

• Order doesn’t matter.
• Calling the same Action several times counts as one call.
• A call that returns an error still counts. The agent chose to call it.
• A Procedure loading its own steps isn’t an Action, so it never counts.

The Action check only looks at which Actions the agent called, not what it sent to them. To check the inputs, for example that the agent booked the right date, write that into the success criterion. The judge sees each Action’s inputs. When the Action check fails, the conversation shows why under the judge’s verdict. Never called lists the missing Actions and Not expected lists the extra ones. Each Action can appear on the list once. Click Set response next to an expected Action to control what it returns. See Action responses.

​

Action responses

By default, every Action in a test returns a generic success without running, so a test never books a meeting, issues a refund, or creates a ticket. Add an Action response to control what a specific Action returns. Each Action response is either: Each Action shows whether it only reads data (Read-only) or changes something (Writes data). Live is locked for Actions that write data, and for Custom Actions, since Chatbase can’t tell what your API does. These Actions can run live:

• Stripe: get subscription info, get invoices, find payments
• Shopify: show products, show order, get cart, get metafields
• Cal.com and Calendly: get available slots
• Web search

​

Run a test

• From the editor: click Save and run to start 3 conversations, or open the menu next to it to choose 1, 3, 5, 10, or 20.
• From the Tests page: select one or more tests, click Run selected, and choose how many times to run each one.
• From a test’s row: open the ⋯ menu and click Run test.
• From a test’s page: click Run again, or open the menu next to it to choose how many times.

A single run can start up to 200 conversations, counted as tests times repeats. Each account runs up to 5 conversations at once. The rest wait as Queued. Most runs finish within a few minutes. While a test is running, you can’t edit it or run it again. Run selected skips any selected test that is already running.

​

Manage tests

On the Tests page, search tests by name or filter them by status: Completed, In-progress, or Never run. Open the ⋯ menu on a test’s row to:

• Edit test: change its setup. The next run uses the new setup.
• Run test: start a run.
• Duplicate test: open the editor with a copy of the test, so you can make a variation.
• Export results: download the latest run as a CSV file.
• Delete test: remove the test and its run history. This can’t be undone.

​

Read the results

Open a test to see its latest run.

• Pass rate: the share of graded conversations that passed. Conversations with No verdict aren’t counted, since they say nothing about your agent.
• Runs completed: how many of the run’s conversations have finished.
• Runs table: one row per conversation, with its status, the number of turns, the Actions the agent called, and how long it took.

Click a row to open the conversation. The judge’s reason for its verdict is at the top. If the Action check failed, Never called and Not expected sit under it. Under each agent reply, Completed N Actions lists the Actions the agent called. Click an Action to see what it returned, labelled Mocked response or Response. The Configuration tab shows the test’s setup, including its expected Actions and Action responses. To share or analyse results elsewhere, click Export results. The CSV has one row per conversation with its status, turns, the Actions called, missing, and unexpected, the time it took, the judge’s reason, and any error.

​

Test credits

Test credits are free credits added to your account every month, on top of your message credits. Tests use them first, so you can test your agent without spending message credits.

• Test credits reset on the 1st of each month, at midnight UTC. Unused credits don’t carry over.
• They belong to your account, so all AI agents in the account share them.
• Only your agent’s replies use credits, the same amount as a reply in a real conversation. The simulated customer and the judge are free.

On the Free plan, tests use message credits from the start. The Tests page shows how many test credits you’ve used this month and when they reset. When your test credits run out, tests use your message credits. Before a run that could use them, Chatbase shows how many test credits you have left, the most the run can use, and how much of that could come from message credits. Click Run anyway to start it. The real cost is usually lower, since it depends on how many turns each conversation takes. When both are used up, you can’t start new runs until your test credits reset or you add message credits.

​

Limits

​

Frequently asked questions