Running tests
oel test runs a project's tests. Tests are ordinary code in the language, checked like the rest of your project and held to the same limits as every other source file.
Usage
oel test
oel test src/nodes
oel test src/nodes/summarize.node.oel
oel test --json
| Argument or option | Meaning |
|---|---|
[path] | Run only the tests in one folder or one file. Without a path, every test of the project in the current folder. |
--json | Results in machine-readable form. |
What it does
- It checks the project first, through the same checks as
oel check. A project that fails them runs no tests. - It runs the tests it finds, and reports each one that fails with the place in your source.
- It ends with a failure status when any test fails, so an automated build stops there.
Tests without a network
A test of an agent should not depend on a live model, the time of day or the order in which work happens to run. The standard library lets a test replace each of these: mock model responses you write, or responses recorded from a real model and played back; a controlled clock; a fixed random seed; simulated devices and sensors; and a chosen order for concurrent work.
Recorded responses play back without credentials or a network. A recording stops matching when the request, the output type or the tools change, and the test says so. This is how prompts are treated as releasable business logic, with recorded-model regression tests you can run on every change.
Testing describes what a test can replace and check.
Tests and evaluations
oel test runs tests that must pass exactly. To measure how well an agent does its job against a live model, with a sample size and limits on time, tokens and cost, use oel eval instead.
Related
- Testing: writing tests.
- Running evaluations: evaluation suites against prompt and model changes.
- Evaluations and prompt releases: the three layers of agent tests.