Testing
Tests in AEL, the Agent Engineering Language, are AEL code, checked like the rest of your project. The command line runs them, and the standard library gives them what an agent needs to be tested without a network, a model account or a device on the desk.
Running tests
ael test runs a project's tests. Give it a folder or a file to run only the tests there.
ael test
ael test src/nodes
Like the other commands, it takes --json for machine-readable results.
Tests that give the same result every time
A test of an agent should not depend on a live model, the time of day or the order in which work happens to run. The library lets a test fix each of these:
| What varies | What a test uses instead |
|---|---|
| A model's answers | Mock responses you write, or responses recorded from a real model and played back. |
| The clock | A controlled clock that moves only when the test advances it. |
| Random choices | A fixed seed. |
| Devices and sensors | Simulated inputs, such as sensor readings or serial data, fed to the same application code. |
| Scheduling | A chosen order for concurrent work, so a failure can be reproduced. |
- Recorded responses play back without credentials or a network. A recording stops matching when the request, the output type or the tools change, and the test says so rather than accepting a response to a different question.
- A controlled clock lets a test cross a timeout, a schedule's time-zone change or a timer's wraparound without waiting for it.
- A test can make things fail on purpose: a full mailbox, a cancellation, a timeout, a fault that the supervisor handles.
The application code stays the same. What changes is what it is connected to.
What to check
- Results: an operation's
Result, including each error your code handles. - Agent behaviour: which tools a run called and which it did not, what it returned, and whether it stayed within its budget of attempts, repairs and tokens.
- Fallbacks and checks: that a missing system prompt, a malformed output or a failed output check leads where you expect.
Prompt evaluations
A prompt is business logic, and a change to it changes what an agent does. Prompts are treated as releasable business logic, with evaluations, recorded-model regression tests, drift checks, and promotion with one-step rollback, while active runs stay pinned to their version.
ael eval runs an evaluation suite against a prompt or model change before you promote it:
ael eval --suite <suite>
- A suite lists inputs, the typed outcome each should produce, tool calls that must not happen, limits and how answers are scored.
- Deterministic cases run against mock or recorded responses. Cases against a live model run only when you ask, with a sample size, a number of repeats, and time, token and cost ceilings that you set.
- A report compares the new prompt or model with the one it replaces, and keeps exact regressions apart from changes in a live model's behaviour.
- Scores from a live model are observations with uncertainty, not proof that an agent is correct.
Evaluations and prompt releases describes suites, promotion and rollback, and Running evaluations and Comparing prompts the commands.
Record and replay
You record control and orchestration runs and replay them to find the first point where behaviour diverged. Replay uses the recorded responses from outside the program, so it reproduces your program's decisions; it does not make a live model's output repeatable. See Reporting, logs and replay.