Browse the documentation

Early preview

Running tests

oel test runs a project's tests. Tests are ordinary code in the language, checked like the rest of your project and held to the same limits as every other source file.

Usage

oel test
oel test src/nodes
oel test src/nodes/summarize.node.oel
oel test --json
Argument or optionMeaning
[path]Run only the tests in one folder or one file. Without a path, every test of the project in the current folder.
--jsonResults in machine-readable form.

What it does

  • It checks the project first, through the same checks as oel check. A project that fails them runs no tests.
  • It runs the tests it finds, and reports each one that fails with the place in your source.
  • It ends with a failure status when any test fails, so an automated build stops there.

Tests without a network

A test of an agent should not depend on a live model, the time of day or the order in which work happens to run. The standard library lets a test replace each of these: mock model responses you write, or responses recorded from a real model and played back; a controlled clock; a fixed random seed; simulated devices and sensors; and a chosen order for concurrent work.

Recorded responses play back without credentials or a network. A recording stops matching when the request, the output type or the tools change, and the test says so. This is how prompts are treated as releasable business logic, with recorded-model regression tests you can run on every change.

Testing describes what a test can replace and check.

Tests and evaluations

oel test runs tests that must pass exactly. To measure how well an agent does its job against a live model, with a sample size and limits on time, tokens and cost, use oel eval instead.