Durable runs and approval
Runs in AEL, the Agent Engineering Language, pause, wait for human approval and reconcile side effects safely, and a run whose agent declares a storage point resumes and survives restarts. A run is durable only when its agent declares a storage point, a storage package you select that keeps the run's state in a directory, in object storage or in an SQL database you run; without one, the run runs to completion and keeps nothing. A durable run keeps its tenant, its budget and its record of what it has done through all of these.
The life of a run
| State | Meaning |
|---|---|
| Queued | Admitted, with its budget reserved, and waiting to start. |
| Running | Doing work. |
| Waiting | Waiting for a reply, a timer or more input. |
| Awaiting approval | Paused until a person approves, rejects or revises a proposed action. |
| Cancelling | Stopping its child work and cleaning up. |
Every run ends in exactly one final state:
| Final state | Meaning |
|---|---|
| Succeeded | It produced its typed output. |
| Partial | It produced part of its output, and recorded what is unfinished. Allowed only when the output type accepts partial results. |
| Failed | It ended with a typed error, including a rejected approval. |
| Timed out | Its deadline passed, including an approval that expired. |
| Exhausted | Its budget ran out. |
| Cancelled | It was cancelled. |
A final state never changes, and you cannot resume a finished run. To continue its work, you start a new run that refers to the original one and keeps its budget. A run's result is stored before its reply is delivered, so delivering a reply again never runs the work again.
Pausing and resuming
A run whose agent declares a storage point saves checkpoints there as it goes. A checkpoint holds:
- where the run is in its workflow, and the version of its state;
- effects it has started but not confirmed;
- its budget reservations and approvals;
- the prompt, configuration and model binding it was admitted with, and its bindings.
A checkpoint holds the decisions and actions the run took, never a model's private reasoning. A resumed run continues with the prompt and configuration it started with, or moves to a new version only through an explicit, tested migration.
Without a storage point, a run keeps its state only in the running program, and nothing of it is kept once the program ends: resuming the run later, or reading the status or events of an earlier run, is refused. Bindings and storage describes how an agent declares its storage point.
Human approval
A node can pause for a person's approval, or for more input, before it acts. Approval is your policy, where you want it; it is not required for every action.
- An approval is bound to the exact proposed action: its arguments, the policy version, who may approve and an expiry time.
- The approver can approve, reject or revise. A revised proposal needs a new approval.
- A stale, replayed or altered approval is refused.
- On approval, the run checks the caller's permissions again before acting, and draws on the same budget.
- A rejection ends the run as failed; an approval that expires ends it as timed out.
- A subagent, a short-lived agent spawned for one task, never pauses for approval: the check refuses an operation that waits for human approval in a subagent, in its helpers and in anything they call. See Subagents.
Surviving restarts
A run survives a restart only when its agent declares a storage point; without one, nothing of the run survives the end of the program. With a storage point:
- Runs, timers, deadlines and child work are recovered after a restart or a crash.
- Only one worker owns a run at a time. A worker that comes back after another has taken over the run cannot continue it.
- Work whose deadline passed while the program was down ends as timed out.
- When a client reconnects, it receives the run's events and result: work that was already paid for is not started again.
- A retried run keeps the original deadline and budget, and the record of side effects it already caused.
When the storage point fails at a checkpoint, the run fails closed: it never continues with weaker durability, and usage whose outcome is unknown stays counted.
Side effects
Some effects happen outside your program: a payment, a message, a change in another service. AEL cannot make such an effect happen exactly once, and says so. Instead, runs make effects safe to reconcile:
- Idempotency keys let a service recognize a repeated request for the same effect.
- Receipts record what a service confirmed.
- Unknown outcomes stay visible, when a reply was lost after an effect may have happened, until your code or a person resolves them.
- Compensation steps that you write can undo or offset an effect, where the other service allows it.
Cancelling or restarting a run never claims that an effect was undone.
Where durable state lives
AEL itself contains no database: the memory, knowledge and durable state of your runs live in the storage, database and vector-store packages you select. A durable run keeps its checkpoints in the storage point its agent declares. A deployment that only answers short requests can do without one: its runs start and finish within one request, and keep nothing. Bindings and storage describes the storage point, and Memory and state how state is scoped and kept.