Evaluations and prompt releases
AEL, the Agent Engineering Language, treats prompts as releasable business logic, with evaluations, recorded-model regression tests, drift checks, and promotion with one-step rollback, while active runs stay pinned to their version. Changing a system prompt is a change to what your agent does, and it gets the same care as a change to its code.
What identifies a version
An agent's behaviour depends on more than its prompt. Its version is identified by:
- the exact text of every prompt it resolves, and its input builders;
- its input and output types, tools and permissions;
- its model bindings and their parameters;
- its configuration and its rules for building context.
A prompt-only change gives the agent a new version, even when no code changes. Each run records the version it was admitted with.
Three layers of tests
| Layer | Model responses | What it shows |
|---|---|---|
| Contract tests | Scripted responses. | Exactly whether types, tool lists, denied permissions, retries, repairs and budgets behave as specified. They must pass exactly. |
| Regression tests | Responses recorded from a real model earlier. | Whether a change breaks behaviour that worked, with no network and no credentials. |
| Behavioural evaluations | A real model, called live. | How well the agent does its job on representative cases. |
- A recording is tied to the requests, types and tools it was made with. When those change, the recording is marked incompatible instead of being replayed against the wrong request.
- A live evaluation declares its sample size, how many times each case repeats, and limits on time, tokens and cost.
- Evaluation results describe how the agent performed on its cases. They are not proof of how a model behaves in every case, and model output is not reproducible run for run.
Evaluation suites
A suite is a set of cases. Each case gives:
- the input;
- the scripted or recorded model responses, or a live model binding;
- the expected typed outcome, or the scores that count as success;
- tool calls that must never happen;
- limits for the case.
Good suites cover normal work and the hard cases: missing information, contradictory input, invalid model output, tool errors, instructions hidden in retrieved documents, a budget running out and a delegated agent failing. Scores for live evaluations can include success on the case, evidence quality, unnecessary tool calls and time.
Running a suite
You run a suite from the command line:
ael eval --suite support-regression
ael eval compares the candidate, your working version, with a baseline, and reports the differences in a short summary for people and, with --json, in machine-readable form. The report keeps three kinds of difference apart:
- contract regressions: a case that passed exactly now fails;
- drift: live scores have moved beyond what chance explains;
- missing coverage: a behaviour has no case that tests it.
Drift checks
A hosted model can change without any change on your side. Re-running your live evaluations regularly, and after every provider or model change, shows drift early. Pin model versions where the provider allows it; each run records the model identity the provider returned.
Comparing prompts
ael prompt diff shows how a prompt changed against a released one:
ael prompt diff --baseline <release>
Promotion and rollback
A new version reaches your deployment through a reviewed plan:
ael deploy plan
ael deploy apply --plan <plan>
- The plan shows what changes: prompts, model bindings, tools and their permissions, and whether each change is compatible with what is deployed, together with the evaluation results.
- Applying the plan stages the new version, checks that it is ready, then switches new runs to it.
- Runs already under way stay pinned to the version they were admitted with, to the end.
- Rolling back is one step, to the previous compatible version.
- A change that widens a tool's permissions needs a new, authorized plan: a prompt update alone can never widen them.
Deployment and rollback describes staged deployments in full. Running evaluations, Comparing prompts and Deployment plans describe the commands.