Smart Agent Test Cases
AI is non-deterministic — ask the same question twice and you may get slightly different wording, and a change to an agent's instructions, model, or data...
Merged manual — this file documents four related pages in order: the test-case list, the test-case card, the steps subpage embedded in the card, and the dialog used to save a conversation as a test case.
In plain terms
AI is non-deterministic — ask the same question twice and you may get slightly different wording, and a change to an agent's instructions, model, or data can quietly change its answers. A test case is how you guard against that: it's a saved conversation with an expected outcome that you can replay on demand.
Test cases are regression tests for your agent — the same idea as rerunning a suite of posting scenarios after an upgrade to confirm nothing broke. The easiest way to make one is to capture a real, good conversation: from the chat or Sessions list, choose Save as Test Case, and every prompt and the agent's good answer become the baseline ("steps").
You then run the test case whenever you want — after editing instructions, switching model tier, or before a release — and the app grades whether the agent still behaves. This page is where you manage those saved tests.
Worked example. You capture a "Create customer — happy path" conversation as a test case. Next month you tweak the agent's instructions; you click Run Now and a green Pass confirms the change didn't break customer creation.
Smart Agent Test Cases (list)
Page type: List Source table:
Smart Agent Test CaseObject:Page 72778349 "SA Agent Test Cases QUA"—Page.72778349.SAAgentTestCases.al
This page lists every AI test case stored in the system. You can filter the list to a specific agent, run individual tests from here, and review recent verdicts at a glance. It also opens when you drill into test cases from a Smart Agent card.
How to open it
- Tell Me (Alt+Q) → search "Smart Agent Test Cases".
- Navigate from the Role Center or from the Test Cases action on a Smart Agent card.
Fields
List columns
| Field | Type | Description |
|---|---|---|
| Test No. | Integer | The unique number identifying this test case. Read-only — assigned automatically. |
| Name | Text[100] | Friendly name of the test case. |
| Agent No. | Integer | The agent this test case belongs to. |
| Agent Name | Text[100] | Name of the agent. |
| Step Count | Integer | Number of prompts in this test case. |
| Enabled | Boolean | Disable to skip this test in batch runs. |
| Last Verdict | Enum | Verdict from the most recent execution. Values: Untested, Pass, Partial, Fail, Error. The value is colour-coded: Pass = green, Partial = amber, Fail / Error = red. |
| Last Score | Decimal | Score from the most recent execution. Blank when zero. |
| Last Run At | DateTime | When this test was last executed. |
| Run Count | Integer | Total number of executions. Blank when zero. |
| Pass Count | Integer | Total number of passing executions. Blank when zero. |
| Created At | DateTime | When this test case was created. |
Actions
| Action | What it does |
|---|---|
| Run Now | Queue this test case to run on a background session and open the test run card so you can watch progress live. |
| View Runs | Show every execution of this test case. |
| Re-promote from Source | Rebuild this test case from its original chat session, re-applying the latest rules for how a conversation is turned into test steps. Existing prompts and expected responses are refreshed from the source messages. |
Related pages
Smart Agent Test Case (card)
Page type: Card Source table:
Smart Agent Test CaseObject:Page 72778350 "SA Test Case Card QUA"—Page.72778350.SATestCaseCard.al
The card view of a single test case. It shows the test's identity and settings, a summary of its most recent result, and the full list of steps captured from the original chat session. You can edit the name, description, and enabled flag here, and run or view runs for this test.
How to open it
- Select a row in the Smart Agent Test Cases list and press Enter (or choose View).
- Opened from the Run Now action — the card for the newly created run opens automatically after submission.
Fields
General
| Field | Type | Description |
|---|---|---|
| Test No. | Integer | The unique number identifying this test case. Read-only. |
| Name | Text[100] | Friendly name of the test case. |
| Description | Text[250] | Optional description of what this test verifies. |
| Agent No. | Integer | The agent under test. Read-only. |
| Agent Name | Text[100] | Name of the agent. |
| Step Count | Integer | Number of prompts (steps) in this test case. Read-only. |
| Enabled | Boolean | Disable to skip this test in batch runs. |
| Source Session ID | Guid | The chat session this test was promoted from. Read-only. |
Last Result
| Field | Type | Description |
|---|---|---|
| Last Verdict | Enum | Verdict from the most recent execution. Values: Untested, Pass, Partial, Fail, Error. Read-only. |
| Last Score | Decimal | Score from the most recent execution. Read-only. Blank when zero. |
| Last Run At | DateTime | When this test was last executed. Read-only. |
| Last Run No. | Integer | Most recent test run number. Read-only. |
| Run Count | Integer | Total number of executions. Read-only. Blank when zero. |
| Pass Count | Integer | Total number of passing executions. Read-only. Blank when zero. |
Steps
The Steps subpage is embedded directly on the card. See Test Case Steps below.
Actions
| Action | What it does |
|---|---|
| Run Now | Queue this test case to run on a background session and open the test run card so you can watch progress live. |
| View Runs | Show every execution of this test case. |
Related pages
Test Case Steps (subpage)
Page type: ListPart (embedded in the Smart Agent Test Case card) Source table:
Smart Agent Test Case StepObject:Page 72778351 "SA Test Case Steps Sub QUA"—Page.72778351.SATestCaseStepsSub.al
Shows the ordered list of prompts and expected outcomes that make up a test case. Each row corresponds to one turn captured from the original chat session. You can view the full prompt or expected response text using the assist-edit button on each row or the actions in the subpage toolbar.
How to open it
This subpage is embedded in the Steps section of the Smart Agent Test Case card. It cannot be opened standalone.
Fields
| Field | Type | Description |
|---|---|---|
| Step No. | Integer | Order of this step within the test (1-based). |
| Prompt Preview | Text[250] | Short preview of the prompt. Select the assist-edit button (...) to view the full text. |
| Expected Response Preview | Text[250] | Short preview of the expected response. Select the assist-edit button (...) to view the full text. |
| Expected Tool Call Count | Integer | Number of tool calls observed when the source conversation produced this step. |
| Expected Latency (ms) | Integer | Soft baseline latency. Used as one input to the AI evaluator's latency sub-score. |
Actions
| Action | What it does |
|---|---|
| View Prompt | Show the full prompt text for the selected step. |
| View Expected Response | Show the full expected agent response for the selected step. |
Related pages
Save Conversation as Test Case
Page type: StandardDialog (no source table — dialog variables only) Object:
Page 72778355 "SA Test Case Input Dlg QUA"—Page.72778355.SATestCaseInputDlg.al
A short modal dialog that collects a name and an optional description before the current chat conversation is promoted into a saved test case. You confirm with OK; cancelling discards the action and leaves the conversation unchanged.
How to open it
- Opened from the Save as Test Case action on the chat page for the active conversation. It cannot be opened through Tell Me or the menu.
Fields
Test case
| Field | Type | Description |
|---|---|---|
| Name | Text[100] | Friendly name for the test case (e.g. "Create Customer happy path"). Required — the dialog will not confirm while this field is blank. |
| Description | Text[250] | Optional description of what this test verifies. Supports multiple lines. |
Actions
The dialog uses the standard OK / Cancel buttons. Choosing OK promotes the active chat conversation into a new test case using the name and description you entered.
Related pages
Notes
- The Last Verdict column on the list is colour-coded: Pass appears in green, Partial in amber, and Fail or Error in red. This colouring is applied automatically and cannot be changed.
- Disabling a test case (clearing Enabled) prevents it from running in batch operations but does not delete any existing run history.
- Re-promote from Source refreshes the prompts and expected responses from the original chat session. Use this after the rules for turning a conversation into test steps have changed, to update the baseline without manually editing each step.
- The full prompt and expected response texts are stored as binary blobs. The Prompt Preview and Expected Response Preview columns show the first 250 characters; use View Prompt or View Expected Response to read the complete text.