Running evaluations
In AEL, the Agent Engineering Language, prompts are treated as releasable business logic, with evaluations, recorded-model regression tests, drift checks, and promotion with one-step rollback. ael eval runs an evaluation suite against a prompt or model change, before you promote it.
Usage
ael eval --suite support-regression
ael eval --suite support-regression --json
| Option | Meaning |
|---|---|
--suite <suite> | The evaluation suite to run. |
--json | The report in machine-readable form. |
What a suite holds
A suite is a set of cases. Each case gives the input; the scripted or recorded model responses, or a live model binding; the expected typed outcome, or the scores that count as success; tool calls that must never happen; and limits for the case. Evaluations and prompt releases describes suites in full.
What it runs
| Cases | Model responses | Result |
|---|---|---|
| Contract cases | Scripted responses. | Must pass exactly: types, tool lists, denied permissions, retries, repairs and budgets. |
| Regression cases | Responses recorded from a real model earlier, played back with no network and no credentials. | Must pass exactly. A recording that no longer matches its request, types or tools is marked incompatible, never replayed against the wrong request. |
| Live cases | A real model, called live. | Scores, measured over a sample. Run only when you ask for them. |
Live cases declare their sample size, how many times each case repeats, and ceilings for time, tokens and cost. A live run stops at its ceilings.
The report
ael eval compares the candidate, your working version, with a baseline, and reports the differences in a short summary for people and, with --json, in machine-readable form. It keeps three kinds of difference apart:
- contract regressions: a case that passed exactly now fails;
- drift: live scores have moved beyond what chance explains;
- missing coverage: a behaviour has no case that tests it.
The report names the exact prompts, model bindings, tools and configuration it evaluated, so a result always belongs to one version of the agent. The command ends with a failure status when a contract or regression case fails.
What the results mean
Scores from a live model are observations with uncertainty, not proof that an agent is correct in every case, and model output is not reproducible run for run. Pin model versions where the provider allows it, and re-run live cases after every provider or model change to catch drift early.
Before promotion
A typical release of a prompt change:
ael eval --suite support-regression
ael prompt diff --baseline <release>
ael deploy plan
The deployment plan shows the evaluation results together with what the release changes. Comparing prompts and Deployment plans describe the next two steps.
Related
- Evaluations and prompt releases: versions, suites, promotion and rollback.
- Testing: prompt evaluations beside the tests that must pass exactly.
- Running tests: the project's tests.