Exit events
An exit event is any condition you configure that ends your application's execution cleanly. It can be based on time, on resource utilization, on an external source, on a message, on the application's own progress, or on an operator action. Every exit event is a rule under exit_events.rules, evaluated asynchronously inside the supervisor, outside your application; when it matches, the Executor runs the state-preserving stop.
The catalog
| Category | Rule type | Exits when |
|---|---|---|
| Time | duration | A monotonic runtime limit elapses (after: "1h"). |
cron | A scheduled cutoff window is reached. | |
timestamp | A fixed date and time is reached. | |
idle | The application has been idle (no CPU activity and no output) for a configured period. | |
| Utilization | utilization | CPU, memory, process count or disk usage stays above or below a threshold for a sustained period. |
| External source, pull | http_poll | A polled URL returns a matching status, header or body, with or without authorization. |
command | A probe command you supply reports a match: the universal adapter for any other system. | |
tcp_probe | A dependency endpoint becomes unreachable, or reachable. | |
process_watch | Another process, such as a parent or a sidecar, exits. | |
file_watch | A file is deleted, created or modified, or its content matches. | |
| External source, push | webhook | An inbound API request or status update matches. |
signal | The supervisor receives a configured operating-system signal. | |
| Message | kafka, mqtt, nats, redis | A message on a topic, subject or channel matches. |
| Application | checkpoint | The application reports a named checkpoint over local IPC. |
log_match | A sentinel pattern appears on stdout or stderr. | |
on_task_complete | The primary command finishes its work. | |
| Operator | (built in) | An operator runs oee stop. |
Every rule has an id and a type, may set its own stop_mode (GRACEFUL or IMMEDIATE), and maps to an exit code (see Exit codes by rule). All remote and message sources support the FAIL_OPEN and FAIL_CLOSED outage policies. Message-broker sources and the webhook listener need the extended supervisor variant, which oee build selects automatically.
Combining rules: ANY or ALL
exit_events:
combinator: "ANY"
rules:
- id: "one-hour-max-ttl"
type: "duration"
after: "1h"
- id: "remote-api-exit"
type: "http_poll"
url: "https://controlplane.example.org/api/v1/runtime/gate"
interval: "1m"
stop_when:
conditions:
- json_path: "$.ep_flag"
in: ["ABC", "XYZ"]
combinator: "ANY"(the default, logical OR): the first rule that matches stops the application.combinator: "ALL"(logical AND): the application stops when every rule has matched.
Inside a rule that carries a payload, stop_when.match combines its conditions the same way: ANY stops on the first matching condition, ALL only when all of them match.
If several triggers fire at the same moment, the stop is latched once; the stop pipeline is idempotent.
Time exit events
duration
Stops the application after a configured elapsed runtime, such as "1h", "30m" or "24h".
- id: "one-hour-max-ttl"
type: "duration"
after: "1h"
The duration is measured strictly on hardware monotonic clocks: CLOCK_MONOTONIC_RAW and the suspend-aware CLOCK_BOOTTIME on Linux, QueryPerformanceCounter on Windows, mach_continuous_time on macOS. Setting the system clock back (date -s "-2 hours") or an NTP jump cannot extend execution past the limit.
cron
Stops the application at a scheduled window, for example at 23:00 UTC every day:
- id: "end-of-window-cutoff"
type: "cron"
schedule: "0 23 * * *"
timezone: "UTC"
timestamp
Stops the application at a fixed ISO-8601 time, cross-checked against monotonic elapsed time:
- id: "year-end"
type: "timestamp"
at: "2026-12-31T23:59:59Z"
idle
Stops the application when it has done no work for a period. It is idle while the CPU usage of its whole process tree stays below cpu_below (default "1%" of one core) and it writes nothing to stdout or stderr. Any activity restarts the idle period, which is measured on the monotonic clock.
- id: "idle-for-30-minutes"
type: "idle"
after: "30m"
cpu_below: "1%"
Utilization
Stops the application when a metric of the guest process tree stays above (above) or below (below) a threshold for a sustained period (for).
metric | Threshold |
|---|---|
cpu | A percentage of the configured CPU cap, of the machine when no cap is set, or an absolute amount. |
memory | A percentage of the configured memory cap, of the machine when no cap is set, or an absolute amount such as "2GB". |
pids | A process count. |
disk_used | The total size of a declared path, such as ./reports exceeding "10GB". |
disk_free | The free space of the filesystem holding a path, falling below a threshold. |
- id: "sustained-memory-pressure"
type: "utilization"
metric: "memory"
above: "95%"
for: "5m"
- id: "reports-volume-full"
type: "utilization"
metric: "disk_used"
path: "./reports"
above: "10GB"
for: "1m"
A utilization exit is a planned, state-preserving stop with its own exit code, 76. It is distinct from a strict cap, which the kernel enforces as a hard ceiling; a utilization exit can stop the application before a cap is reached.
The HTTP and HTTPS poller
http_poll governs any wrapped application from an external REST API without changing the application's code.
- id: "remote-api-exit"
type: "http_poll"
url: "https://controlplane.example.org/api/v1/runtime/gate"
method: "GET"
interval: "1m"
jitter: "5s"
timeout: "8s"
check_before_start: true
auth:
scheme: "bearer"
from_env: "OEE_AUTH_TOKEN"
scrub_from_guest_env: true
tls:
min_version: "1.3"
pinned_spki_sha256: "sha256/VjLZe/p3W/PJnd6lL8JVNBCGQBZynFLdZSTIqcO0SJ8="
stop_when:
match: "ANY"
conditions:
- json_path: "$.ep_flag"
in: ["ABC", "XYZ", "HALT"]
- body_exact: "ABC"
- http_status: [410, 423]
on_network_error:
mode: "FAIL_OPEN"
Endpoint, cadence and telemetry
- Request.
GET,POSTorHEADagainst anyhttps://endpoint, or an explicitly permitted internalhttp://endpoint, with a requesttimeout(such as"5s"), connection pooling and exponential retry backoff within each polling window. - Interval and jitter. Any
interval, such as"15s","1m"or"5m", with optional randomizedjitter:jitter: "5s"adds plus or minus 5 seconds of uniform variance, so thousands of deployed binaries never create synchronized spikes on your control API. - Heartbeat payload. For
POST, a JSON body template can interpolate live supervisor metrics:{{instance_id}},{{app_name}},{{app_version}},{{hostname}},{{uptime_secs}},{{child_pid_count}},{{memory_rss_mb}},{{cpu_usage_percent}}and{{replica_count}}.
Authorization
Authorization is optional. An endpoint that needs no key is polled without credentials when the auth block is left out or set to scheme: "none".
| Scheme | Sent as |
|---|---|
none | No credentials. |
| Bearer token | Authorization: Bearer <key> |
| Custom header | A header such as X-API-Key: <key> |
| HTTP Basic | Basic authentication |
| HMAC-SHA256 | A timestamped request signature in X-OEE-Signature |
| Mutual TLS | A client certificate and key |
Credentials come from one of three sources:
| Source | Example | Notes |
|---|---|---|
| Environment variable | from_env: "OEE_EXIT_AUTH_KEY" | Stripped from the guest's environment by default (scrub_from_guest_env: true), so untrusted sandboxed code cannot read the exit-event key. |
| Mounted secret file | from_file: "/run/secrets/oee_api_token" | For Docker Swarm and Kubernetes secrets. |
| Static manifest token | Sealed inside the signed oee.yaml | Only when you choose it explicitly. |
TLS 1.3, pinning and signed responses
- SPKI pinning.
pinned_spki_sha256pins one or more SHA-256 Subject Public Key Info hashes inrustls, so a local man-in-the-middle proxy or a modified host CA store cannot spoof a "keep running" response. - Signed responses. Optionally, the Executor verifies an Ed25519 signature header,
X-OEE-Response-Sig, over the response body and the request nonce, which prevents replayed responses.
Conditions
stop_when.conditions lists the criteria that trigger a stop:
| Condition | Example | Matches |
|---|---|---|
| JSONPath | json_path: "$.ep_flag" with in: ["ABC", "XYZ"] or equals: "ABC" | A field of a JSON response: $.ep_flag is "ABC" or "XYZ", $.should_exit is true, or $.lease.status is not "VALID". |
| Exact body | body_exact: "ABC" | A raw text body that is exactly the value. |
| Substring or regex | A substring, or a regular expression such as ^STOP_INSTANCE | A raw text body that contains or matches it. |
| Status code | http_status: [410, 423] | Codes such as 410 Gone, 403 Forbidden or 423 Locked. |
| Header | A response header such as X-OEE-Action: STOP | A header value. |
Remote stop parameters
With stop_policy.allow_remote_stop_params: true, the response can choose how the Executor stops: {"ep_flag": "ABC", "stop_mode": "GRACEFUL", "grace_period_ms": 15000} for a planned maintenance drain, {"ep_flag": "XYZ", "stop_mode": "IMMEDIATE"} for an emergency halt. A response can never raise or disable a sealed CPU or memory cap.
Pre-launch gate
With check_before_start: true, the Executor polls once, synchronously, before it starts your application. If the stop tag is already active, startup is blocked and the binary exits immediately with code 78.
Outage policies
on_network_error.mode | Behavior |
|---|---|
FAIL_OPEN (default, for intermittent connectivity) | DNS failures, timeouts and 5xx errors are logged and the workload keeps running; evaluation resumes on the next tick. |
FAIL_CLOSED (strict lease mode) | After max_consecutive_failures polls without a valid keep-alive response (for example 5 missed one-minute polls), the Executor treats the lost lease as a stop and halts the application cleanly with exit code 75. |
External sources
webhook
The supervisor listens on a local address (default 127.0.0.1) and path. An external system stops the application by sending a request, such as POST /oee/status with a JSON status update. The request's body, headers and method are evaluated with the same stop_when conditions as polled responses, so {"status": "REVOKED"} stops the application and {"status": "ACTIVE"} does not.
- id: "status-update-webhook"
type: "webhook"
listen: "127.0.0.1:9911"
path: "/oee/status"
auth:
scheme: "none"
stop_when:
match: "ANY"
conditions:
- json_path: "$.status"
in: ["REVOKED", "SUSPENDED"]
Authorization is optional: bearer, header, hmac signed requests with timestamp replay protection, or none for an open endpoint. The listener runs in the supervisor's network context and is never reachable through the guest sandbox.
file_watch
Stops the application when a watched path is deleted, created or modified, or its content matches (content_match), for example when another system removes a license or lock file.
- id: "license-file-removed"
type: "file_watch"
path: "/var/run/payments/license.lock"
events: ["deleted"]
It uses native notification (inotify on Linux, kqueue on macOS, directory change notifications on Windows), falls back to polling on network filesystems, and detects atomic replacement and renames. The permissions of the watched path are the trust boundary: whoever can change the file can stop the application.
command
The universal adapter for any other system. The supervisor runs your command periodically, outside the guest sandbox, and stops the application when its exit code or output matches: a script that checks a Redis flag, a database row, a cloud queue or a hardware signal.
- id: "redis-flag-probe"
type: "command"
run: ["/usr/local/bin/check-halt-flag.sh"]
interval: "30s"
timeout: "5s"
stop_when:
match: "ANY"
conditions:
- exit_code: [10]
Besides exit_code, conditions can be stdout_contains or stdout_regex. Each run has a timeout and an output size cap, and the probe is never part of the guest process tree.
tcp_probe
Opens a TCP connection to an address periodically and stops the application after a number of consecutive failures (exit_when: "unreachable", such as a required database going away), or as soon as the address accepts connections (exit_when: "reachable").
- id: "database-gone"
type: "tcp_probe"
address: "db.internal:5432"
interval: "10s"
exit_when: "unreachable"
failures: 3
process_watch
Ties the application's lifetime to another process: the supervisor's own parent (watch: "parent"), or the process recorded in a PID file (watch: "pid_file" with pid_file: "/run/app/sidecar.pid"). Use it to stop with a launcher, a sidecar or a companion service.
- id: "launcher-exited"
type: "process_watch"
watch: "parent"
signal
Treats a signal delivered to the supervisor, such as SIGUSR1, as an exit event instead of forwarding it to the application. On Windows, the equivalent is a named event object.
- id: "operator-signal"
type: "signal"
signal: "SIGUSR1"
Message sources: Kafka, MQTT, NATS and Redis
The supervisor subscribes with a pure-Rust client and stops the application when a message matches stop_when, for example $.action equal to "STOP" addressed to this application or instance.
type | Subscribes to | Authentication |
|---|---|---|
kafka | A topic; can match the record key or value | None, or SASL plain, scram-sha-256 or scram-sha-512 |
mqtt | A topic filter, such as fleet/+/control | None, or username and password |
nats | A subject | None, a token, or username and password |
redis | A publish/subscribe channel | None, or username and password |
All four support plaintext and TLS connections, take credentials from from_env or from_file (for Kafka, username_from_env and password_from_env), consider only messages produced after the supervisor started, and follow the same FAIL_OPEN and FAIL_CLOSED outage policies as the HTTP poller.
- id: "ops-control-topic"
type: "kafka"
brokers: ["kafka-1.internal:9093", "kafka-2.internal:9093"]
topic: "ops.runtime.control"
tls:
enabled: true
auth:
mechanism: "scram-sha-512"
username_from_env: "OEE_KAFKA_USER"
password_from_env: "OEE_KAFKA_PASSWORD"
stop_when:
match: "ALL"
conditions:
- json_path: "$.app"
equals: "payment-reconciliation-api"
- json_path: "$.action"
in: ["STOP", "HALT"]
- id: "fleet-control-mqtt"
type: "mqtt"
broker: "mqtts://mqtt.internal:8883"
topic: "fleet/+/control"
stop_when:
match: "ANY"
conditions:
- json_path: "$.command"
equals: "shutdown"
Other messaging systems, such as AMQP and cloud queues, are reached through the webhook or command sources.
Application exit events
checkpoint
The Executor injects two variables into the guest environment: OEE_IPC_SOCKET, a Unix domain socket on Linux and macOS or a named pipe on Windows, and OEE_IPC_TOKEN, a random token for this run. Your application authenticates with the token and sends structured checkpoint messages, such as {"checkpoint": "BATCH_SYNC_DONE", "action": "stop"}, or increments milestone counters, stopping automatically once a checkpoint such as "ITERATION_COMPLETE" has been reached a given number of times.
- id: "reconciliation-done-checkpoint"
type: "checkpoint"
ipc_event: "RECONCILIATION_FINISHED"
stop_mode: "GRACEFUL"
A checkpoint stop exits with code 0. With auto-scaling, a checkpoint from any replica counts for the whole instance.
log_match
A zero-code matcher on the application's stdout and stderr: when a sentinel pattern you configure appears, such as a fatal upstream state string, the Executor stops the service.
on_task_complete
For batch work: once the primary command finishes, the Executor stops any lingering background helper processes in the sandbox or job cleanly and passes on the command's exact exit code.
Exit codes by rule
| Rules | Default exit code | stop_policy.exit_codes key |
|---|---|---|
duration, cron, timestamp, idle | 124 | on_ttl_expired |
utilization | 76 | on_utilization |
http_poll, webhook, file_watch, command, tcp_probe, process_watch, signal, kafka, mqtt, nats, redis, and oee stop | 143 | on_exit_event |
checkpoint | 0 | on_checkpoint |
Lease lost under FAIL_CLOSED | 75 | on_lease_lost |
on_task_complete | The guest's own exit code |
Stopping and exit codes gives the complete table, including the codes of startup refusals.
Extending the catalog
Every source implements one internal exit-source contract: start, emit matched events, report health, shut down. That is why any source combines freely with any other under ANY or ALL, and why every source uses the same stop pipeline.