Running tests
In AEL, the Agent Engineering Language, ael test runs a project's tests. Tests are AEL code, checked like the rest of your project and held to the same limits as every other source file.
Usage
ael test
ael test src/nodes
ael test src/nodes/summarize.node.ael
ael test --json
| Argument or option | Meaning |
|---|---|
[path] | Run only the tests in one folder or one file. Without a path, every test of the project in the current folder. |
--json | Results in machine-readable form. |
What it does
- It checks the project first, through the same checks as
ael check. A project that fails them runs no tests. - It runs the tests it finds, and reports each one that fails with the place in your source.
- It ends with a failure status when any test fails, so an automated build stops there.
Tests without a network
A test of an agent should not depend on a live model, the time of day or the order in which work happens to run. The standard library lets a test replace each of these: mock model responses you write, or responses recorded from a real model and played back; a controlled clock; a fixed random seed; simulated devices and sensors; and a chosen order for concurrent work.
Recorded responses play back without credentials or a network. A recording stops matching when the request, the output type or the tools change, and the test says so. This is how prompts are treated as releasable business logic, with recorded-model regression tests you can run on every change.
Testing describes what a test can replace and check.
Tests and evaluations
ael test runs tests that must pass exactly. To measure how well an agent does its job against a live model, with a sample size and limits on time, tokens and cost, use ael eval instead.
Related
- Testing: writing tests.
- Running evaluations: evaluation suites against prompt and model changes.
- Evaluations and prompt releases: the three layers of agent tests.