Stopping and exit codes
When any exit event matches (time, utilization, an external source, a message, an application checkpoint or an operator stop), the Executor moves atomically into the STOPPING state and runs a deterministic five-stage shutdown. It guarantees zero orphaned worker processes and complete preservation of disk state: the binary, the runtime cache, the logs and every file your application created stay on disk.
The five stages
[Exit event matched: ep_flag == "ABC", TTL reached, utilization breach, ...]
|
v
Stage 1: LATCH STOP STATE AND RECORD AUDIT EVENT
|
v
Stage 2: DELIVER GRACEFUL STOP SIGNAL (skipped in IMMEDIATE mode)
|
v
Stage 3: BOUNDED DRAIN WAIT (grace_period_ms)
|
v
Stage 4: HARD PROCESS-TREE TERMINATION
|
v
Stage 5: FLUSH, RELEASE HANDLES, EXIT WITH A DETERMINISTIC CODE
| Stage | What happens |
|---|---|
| 1. Latch and audit | The state is set to STOPPING atomically, idempotent when several triggers fire at once. A structured stop event goes to the supervisor's stderr or your configured log sink. |
| 2. Graceful signal | On POSIX systems, the configured signal (SIGTERM or SIGINT) goes to the entire guest process group. On Windows, CTRL_BREAK_EVENT goes to the console process group, or an IPC shutdown notification. uvicorn, Go HTTP servers and JVM shutdown hooks drain active requests and flush their I/O. Skipped in IMMEDIATE mode. |
| 3. Bounded drain | The Executor watches the PID namespace or the Job Object completion port for up to grace_period_ms (such as 10,000 ms). If every process exits in time, it jumps to stage 5. |
| 4. Hard termination | Linux writes 1 to cgroup.kill (or sends SIGKILL across the PID namespace and subreaper children). Windows calls TerminateJobObject. macOS sends SIGKILL across the process group and every kqueue-tracked descendant. The Executor reaps every child status until no process remains. |
| 5. Flush and exit | Standard output and error are flushed, so final log lines are never truncated. Namespace descriptors, WFP port filters and the local control socket are released. The supervisor exits with the mapped exit code. |
With auto-scaling, the load balancer stops accepting new connections in stage 1, stages 2 to 4 run across every replica's process tree concurrently under one shared grace deadline, and the listener is released in stage 5.
Graceful and immediate
| Mode | Behavior |
|---|---|
GRACEFUL | Signal, bounded drain, then hard termination of whatever remains. A guest that ignores SIGTERM is still stopped when grace_period_ms expires, with the uncatchable SIGKILL or TerminateJobObject. |
IMMEDIATE | Skips the signal and the drain, and terminates the whole tree at once. The design target is under 5 ms from exit-event evaluation to complete termination across 50 concurrent child workers. |
Configure the default in stop_policy, override it per rule with stop_mode, and, with allow_remote_stop_params: true, let the exit-event response choose:
stop_policy:
default_mode: "GRACEFUL"
graceful_signal: "SIGTERM"
grace_period_ms: 10000
allow_remote_stop_params: true
exit_codes:
on_exit_event: 143
on_ttl_expired: 124
on_utilization: 76
on_checkpoint: 0
on_lease_lost: 75
Exit codes
The Executor uses explicit, configurable exit codes, so it works cleanly with Docker restart policies (--restart=on-failure), Kubernetes Jobs, systemd units and CI/CD pipelines.
| Termination cause | Default exit code | What it means for an orchestrator |
|---|---|---|
| Normal guest exit | The guest's code (0 to 255) | The wrapped application's own exit code, passed on unchanged. |
| Checkpoint stop | 0 | The application reached a planned completion checkpoint and stopped cleanly. |
| Pre-launch exit gate block | 78 | A remote exit event was already active before the application started (check_before_start: true). |
Exit event lease lost (FAIL_CLOSED) | 75 | The control-plane API was unreachable past max_consecutive_failures. |
| Required enforcement unavailable | 69 | A strict resource cap or a container_fallback: "strict" sandbox primitive cannot be enforced on this host, so the guest was never started. |
| Sandbox quota or policy violation | 70 | The guest exceeded a strict memory cap or the PID quota, or triggered a fatal syscall trap. |
| Time-based exit event | 124 | The configured duration, scheduled cutoff, fixed timestamp or idle timeout was reached. |
| Utilization exit event | 76 | A utilization rule matched: CPU, memory, process count or disk usage stayed beyond its threshold for the configured period. |
| External or message exit event | 143 | An external-source or message exit event matched: a remote API returned a stop flag (such as ep_flag in ["ABC", "XYZ"]), or a webhook, file event, message, probe, watched process or configured signal matched its rule. Also used for oee stop. |
The codes of the configurable causes are set in stop_policy.exit_codes; see the oee.yaml reference.
What a stop never touches
- The Executor binary itself.
- The runtime cache.
- Application logs and databases.
- Every file the workload wrote, such as the files in
read_write_paths.
A stop ends execution; it never deletes or alters state.