> ## Documentation Index > Fetch the complete documentation index at: https://chatbase.co/docs/llms.txt > Use this file to discover all available pages before exploring further. # Test your AI agent with simulated customers > Check how your AI agent handles a conversation before your customers do. A simulated customer talks to your agent, and each conversation is graded automatically. Tests in Chatbase check how your AI agent handles important conversations, before you launch it and after every change you publish. A **test** is a conversation your AI agent has with a simulated customer: you describe who the customer is, what they want, and what has to happen for the conversation to count as a success. Chatbase then runs that conversation as many times as you ask, grades each one automatically, and shows you the transcripts. Use tests to check how a change affects your AI agent: new instructions, a new Data Source, a Procedure, or an Action. Tests run against your published AI agent, not your draft, so publish the change first, then run the same tests again and compare the pass rate. If the pass rate drops, restore the previous version from [Version history](https://chatbase.co/docs/user-guides/chatbot/versions.md#restore-a-version). ![The Tests page, listing an agent's tests with their latest results](https://mintcdn.com/chatbase/T4eAdO1W8XNxl22V/user-guides/chatbot/images/tests/tests-list.png?fit=max\&auto=format\&n=T4eAdO1W8XNxl22V\&q=85\&s=74b52301fbdf9bb4cf20efffe3ec103d) Tests live under **Tests** in your AI agent's sidebar. ## How a test run works Each run starts one or more conversations. In each conversation: 1. A **simulated customer**, played by another AI model, opens the conversation based on the scenario you wrote. It only knows the scenario and the conversation so far. It never sees your agent's instructions or the success criterion. 2. Your **AI agent** replies using its real setup: its instructions, Data Sources, Actions, and Procedures. 3. They keep talking until the customer is done or the conversation reaches its **max turns**. 4. A **judge**, another AI model, reads the finished conversation and decides whether it met your **success criterion**. The judge sees the messages and every Action the agent called, with what it sent and what came back. 5. If the test lists **expected Actions**, Chatbase also checks which Actions the agent called. A conversation passes only when the judge passes it **and** the Action check passes. The simulated customer and the judge use the same model in every test. This keeps pass rates comparable between tests and between runs. > [!NOTE] > > Tests never reach your customers. The conversations aren't saved to your chat logs, and Actions don't run for real unless you set a read-only Action to **Live** (see [Action responses](#action-responses)). ## What to test first You can have up to 50 tests per AI agent, so start with the conversations that matter most: 1. **Your most common questions.** Check [Analytics](https://chatbase.co/docs/user-guides/chatbot/analytics.md) for your top topics and write a test for each. 2. **Anything that triggers an Action.** Refunds, bookings, order lookups, and subscription changes are where a wrong answer costs the most. Use [Expected Actions](#expected-actions) to check the right Action is called. 3. **Each Procedure.** One test per [Procedure](https://chatbase.co/docs/user-guides/chatbot/procedures/procedures-overview.md), with a customer who needs it from start to finish. 4. **Handing off to a person.** A frustrated customer who asks for a human should reach your [escalation](https://chatbase.co/docs/user-guides/chatbot/actions/escalate-to-human.md) path. 5. **Questions your AI agent shouldn't answer.** Off-topic or out-of-scope requests check that your instructions and [Guardrails](https://chatbase.co/docs/user-guides/chatbot/guardrails.md) hold. 6. **Failures.** Turn on **Return as error** in an [Action response](#action-responses) to check the AI agent handles a failed Action gracefully. ### Example tests by use case | Use case | Simulated user | Success criterion | Expected Actions | | - | - | - | - | | **Customer support** | "A customer who can't find where to change their billing email. Mildly annoyed, wants a quick answer." | "The agent gives the correct steps from the help docs without escalating." | Empty | | **Customer support** | "A customer whose issue wasn't solved after two tries and who now asks to speak to a person." | "The agent stops troubleshooting and hands the conversation to the team." | Your escalation Action | | **Ecommerce** | "A shopper who placed order #1042 five days ago and wants to know where it is." | "The agent looks up the order and states its current status." | Shopify show order | | **Ecommerce** | "A shopper looking for a waterproof jacket under \$100 who asks for a recommendation." | "The agent recommends only products that match both requirements." | Shopify show products | | **Sales** | "A marketing manager at a 50-person company who wants a demo next Tuesday afternoon." | "The agent books a demo on Tuesday afternoon and confirms the time." | Get slots, Book meeting | | **Sales** | "A visitor comparing plans who is interested but not ready to buy." | "The agent answers the pricing question and asks for the visitor's email." | Your lead collection Action, if you use one | Adjust the wording to your own products and Actions. The Expected Actions column uses the names of the Actions you've added to your AI agent. ## Create a test On the Tests page, click **Create a test**. | Field | What to write | | - | - | | **Test name** | A label for the test, shown in the list and in exported results. Only you see it. | | **Simulated user** | Who the customer is and what they want, for example: "A Premium customer who ordered 12 days ago and wants to return one item for a refund. Polite but firm." | | **Success criterion** | The one thing that has to be true for the conversation to pass, for example: "The agent confirms the order, explains the refund policy once, and issues the refund without escalating." | | **Expected Actions** | Optional. Leave it empty to skip the Action check, or add the Actions the agent must call. See [Expected Actions](#expected-actions). | | **Action responses** | Optional. What each Action returns during the test. See [Action responses](#action-responses). | | **Max conversation turns** | How many messages the customer can send, from 1 to 20. A conversation that hasn't met the criterion by then fails. | > [!TIP] > > Keep the success criterion to one clear outcome. The judge grades against it alone, so "books a demo for next week" gives more reliable results than a list of five things. ### Expected Actions This field decides whether Chatbase checks the Actions your agent calls. It behaves differently when it's empty and when it has Actions. **Empty.** This is the default. Chatbase doesn't check Actions. The agent can call any Action, or none, and only the judge grades the conversation. Use this when the outcome is all that matters, for example "the agent explains the refund policy correctly". The test's **Configuration** tab shows **Not asserted**. **With Actions.** The agent must call exactly the Actions on the list. No more, no fewer. The Action check fails if the agent: * never called an Action on the list, or * called an Action that isn't on the list. A failed Action check fails the conversation, even when the judge passed it. Adding one Action makes the check strict for every Action. Any Action the agent calls that isn't on the list fails the conversation. So list every Action the agent should call in this scenario, not only the main one. | Expected Actions | The agent calls | Result | | - | - | - | | Empty | Any Actions, or none | No Action check. Only the judge grades. | | Book meeting | Book meeting | Action check passes. | | Book meeting | Nothing | Fails. Book meeting was never called. | | Book meeting | Book meeting, Create ticket | Fails. Create ticket wasn't expected. | | Get slots, Book meeting | Book meeting, Get slots | Action check passes. Order doesn't matter. | A few rules for how the check counts calls: * Order doesn't matter. * Calling the same Action several times counts as one call. * A call that returns an error still counts. The agent chose to call it. * A Procedure loading its own steps isn't an Action, so it never counts. The Action check only looks at which Actions the agent called, not what it sent to them. To check the inputs, for example that the agent booked the right date, write that into the success criterion. The judge sees each Action's inputs. When the Action check fails, the conversation shows why under the judge's verdict. **Never called** lists the missing Actions and **Not expected** lists the extra ones. Each Action can appear on the list once. Click **Set response** next to an expected Action to control what it returns. See [Action responses](#action-responses). ### Action responses By default, every Action in a test returns a generic success without running, so a test never books a meeting, issues a refund, or creates a ticket. Add an **Action response** to control what a specific Action returns. Each Action response is either: | Mode | What happens | | - | - | | **Mock** | The Action returns the JSON you write, and nothing real runs. Built-in Actions start with a sample response shaped like the real one. Custom Actions start empty, so paste what your API returns. Turn on **Return as error** to test how the agent handles a failure. | | **Live** | The agent calls the real Action. Only Actions that just read data can run live, so nothing is created or changed. Results can differ between runs. | Each Action shows whether it only reads data (**Read-only**) or changes something (**Writes data**). **Live** is locked for Actions that write data, and for Custom Actions, since Chatbase can't tell what your API does. These Actions can run live: * Stripe: get subscription info, get invoices, find payments * Shopify: show products, show order, get cart, get metafields * Cal.com and Calendly: get available slots * Web search > [!NOTE] > > A live Action uses your connected account, for example your Stripe or Cal.com account, the same way it would in a real conversation. ## Run a test * **From the editor:** click **Save and run** to start 3 conversations, or open the menu next to it to choose 1, 3, 5, 10, or 20. * **From the Tests page:** select one or more tests, click **Run selected**, and choose how many times to run each one. * **From a test's row:** open the **⋯** menu and click **Run test**. * **From a test's page:** click **Run again**, or open the menu next to it to choose how many times. A single run can start up to 200 conversations, counted as tests times repeats. Each account runs up to 5 conversations at once. The rest wait as **Queued**. Most runs finish within a few minutes. While a test is running, you can't edit it or run it again. **Run selected** skips any selected test that is already running. ## Manage tests On the Tests page, search tests by name or filter them by status: **Completed**, **In-progress**, or **Never run**. Open the **⋯** menu on a test's row to: * **Edit test**: change its setup. The next run uses the new setup. * **Run test**: start a run. * **Duplicate test**: open the editor with a copy of the test, so you can make a variation. * **Export results**: download the latest run as a CSV file. * **Delete test**: remove the test and its run history. This can't be undone. ## Read the results Open a test to see its latest run. * **Pass rate**: the share of graded conversations that passed. Conversations with **No verdict** aren't counted, since they say nothing about your agent. * **Runs completed**: how many of the run's conversations have finished. * **Runs table**: one row per conversation, with its status, the number of turns, the Actions the agent called, and how long it took. | Status | Meaning | | - | - | | **Queued** | Waiting to start. | | **In-progress** | The conversation is running. | | **Passed** | The judge passed it, and the Action check passed if the test has expected Actions. | | **Failed** | The judge failed it, or the agent called the wrong Actions. | | **No verdict** | Something went wrong before the conversation could be graded, for example a timeout. | Click a row to open the conversation. The judge's reason for its verdict is at the top. If the Action check failed, **Never called** and **Not expected** sit under it. Under each agent reply, **Completed N Actions** lists the Actions the agent called. Click an Action to see what it returned, labelled **Mocked response** or **Response**. The **Configuration** tab shows the test's setup, including its expected Actions and Action responses. To share or analyse results elsewhere, click **Export results**. The CSV has one row per conversation with its status, turns, the Actions called, missing, and unexpected, the time it took, the judge's reason, and any error. ## Test credits Test credits are free credits added to your account every month, on top of your message credits. Tests use them first, so you can test your agent without spending message credits. * Test credits reset on the 1st of each month, at midnight UTC. Unused credits don't carry over. * They belong to your account, so all AI agents in the account share them. * Only your agent's replies use credits, the same amount as a reply in a real conversation. The simulated customer and the judge are free. | Plan | Test credits per month | | - | - | | Free | 0 | | Hobby | 200 | | Basic | 500 | | Business | 1,000 | | Enterprise | 1,000 | On the Free plan, tests use message credits from the start. The Tests page shows how many test credits you've used this month and when they reset. When your test credits run out, tests use your message credits. Before a run that could use them, Chatbase shows how many test credits you have left, the most the run can use, and how much of that could come from message credits. Click **Run anyway** to start it. The real cost is usually lower, since it depends on how many turns each conversation takes. When both are used up, you can't start new runs until your test credits reset or you add message credits. ## Limits | Limit | Value | | - | - | | Tests per AI agent | 50 | | Max conversation turns per test | 20 | | Conversations per run | 200 | | Conversations running at once, per account | 5 | | Size of a mock response | 50,000 characters | ## Frequently asked questions > [accordion] How many times should I run a test? > > **Save and run** starts 3 conversations, and you can choose 1, 3, 5, 10, or 20. Each conversation is graded separately, and the pass rate shows the share that passed. Running a test several times gives you a pass rate instead of a single result, which makes it easier to compare runs before and after a change. > [accordion] How does Chatbase decide if a test passed? > > A judge, another AI model, reads the finished conversation and decides whether it met your success criterion. It sees the messages and every Action the AI agent called, with what was sent and what came back. If the test has expected Actions, Chatbase also checks that the AI agent called exactly those Actions. The conversation passes only when both checks pass. The judge's reason appears at the top of each conversation. > [accordion] Do tests cost message credits? > > Tests use your free monthly test credits first: 200 on Hobby, 500 on Basic, and 1,000 on Business and Enterprise. Only your AI agent's replies use credits, the same amount as a real reply. The simulated customer and the judge are free. When test credits run out, tests use message credits, and Chatbase shows the most a run can use before you start it. See [Test credits](#test-credits). > [accordion] Can a test book a real meeting or issue a real refund? > > No. By default, every Action returns a generic success without running. You can set a read-only Action, such as getting available slots or showing an order, to **Live**, but Actions that write data can't run live. Test conversations also aren't saved to your chat logs. > [accordion] Why did a conversation get No verdict? > > Something went wrong before the conversation could be graded, for example a timeout. Conversations with **No verdict** aren't counted in the pass rate. Run the test again. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.