Browse the documentation

Early preview

Stopping and exit codes

When any exit event matches (time, utilization, an external source, a message, an application checkpoint or an operator stop), the Executor moves atomically into the STOPPING state and runs a deterministic five-stage shutdown. It guarantees zero orphaned worker processes and complete preservation of disk state: the binary, the runtime cache, the logs and every file your application created stay on disk.

The five stages

[Exit event matched: ep_flag == "ABC", TTL reached, utilization breach, ...]
       |
       v
Stage 1: LATCH STOP STATE AND RECORD AUDIT EVENT
       |
       v
Stage 2: DELIVER GRACEFUL STOP SIGNAL (skipped in IMMEDIATE mode)
       |
       v
Stage 3: BOUNDED DRAIN WAIT (grace_period_ms)
       |
       v
Stage 4: HARD PROCESS-TREE TERMINATION
       |
       v
Stage 5: FLUSH, RELEASE HANDLES, EXIT WITH A DETERMINISTIC CODE
StageWhat happens
1. Latch and auditThe state is set to STOPPING atomically, idempotent when several triggers fire at once. A structured stop event goes to the supervisor's stderr or your configured log sink.
2. Graceful signalOn POSIX systems, the configured signal (SIGTERM or SIGINT) goes to the entire guest process group. On Windows, CTRL_BREAK_EVENT goes to the console process group, or an IPC shutdown notification. uvicorn, Go HTTP servers and JVM shutdown hooks drain active requests and flush their I/O. Skipped in IMMEDIATE mode.
3. Bounded drainThe Executor watches the PID namespace or the Job Object completion port for up to grace_period_ms (such as 10,000 ms). If every process exits in time, it jumps to stage 5.
4. Hard terminationLinux writes 1 to cgroup.kill (or sends SIGKILL across the PID namespace and subreaper children). Windows calls TerminateJobObject. macOS sends SIGKILL across the process group and every kqueue-tracked descendant. The Executor reaps every child status until no process remains.
5. Flush and exitStandard output and error are flushed, so final log lines are never truncated. Namespace descriptors, WFP port filters and the local control socket are released. The supervisor exits with the mapped exit code.

With auto-scaling, the load balancer stops accepting new connections in stage 1, stages 2 to 4 run across every replica's process tree concurrently under one shared grace deadline, and the listener is released in stage 5.

Graceful and immediate

ModeBehavior
GRACEFULSignal, bounded drain, then hard termination of whatever remains. A guest that ignores SIGTERM is still stopped when grace_period_ms expires, with the uncatchable SIGKILL or TerminateJobObject.
IMMEDIATESkips the signal and the drain, and terminates the whole tree at once. The design target is under 5 ms from exit-event evaluation to complete termination across 50 concurrent child workers.

Configure the default in stop_policy, override it per rule with stop_mode, and, with allow_remote_stop_params: true, let the exit-event response choose:

stop_policy:
  default_mode: "GRACEFUL"
  graceful_signal: "SIGTERM"
  grace_period_ms: 10000
  allow_remote_stop_params: true
  exit_codes:
    on_exit_event: 143
    on_ttl_expired: 124
    on_utilization: 76
    on_checkpoint: 0
    on_lease_lost: 75

Exit codes

The Executor uses explicit, configurable exit codes, so it works cleanly with Docker restart policies (--restart=on-failure), Kubernetes Jobs, systemd units and CI/CD pipelines.

Termination causeDefault exit codeWhat it means for an orchestrator
Normal guest exitThe guest's code (0 to 255)The wrapped application's own exit code, passed on unchanged.
Checkpoint stop0The application reached a planned completion checkpoint and stopped cleanly.
Pre-launch exit gate block78A remote exit event was already active before the application started (check_before_start: true).
Exit event lease lost (FAIL_CLOSED)75The control-plane API was unreachable past max_consecutive_failures.
Required enforcement unavailable69A strict resource cap or a container_fallback: "strict" sandbox primitive cannot be enforced on this host, so the guest was never started.
Sandbox quota or policy violation70The guest exceeded a strict memory cap or the PID quota, or triggered a fatal syscall trap.
Time-based exit event124The configured duration, scheduled cutoff, fixed timestamp or idle timeout was reached.
Utilization exit event76A utilization rule matched: CPU, memory, process count or disk usage stayed beyond its threshold for the configured period.
External or message exit event143An external-source or message exit event matched: a remote API returned a stop flag (such as ep_flag in ["ABC", "XYZ"]), or a webhook, file event, message, probe, watched process or configured signal matched its rule. Also used for oee stop.

The codes of the configurable causes are set in stop_policy.exit_codes; see the oee.yaml reference.

What a stop never touches

  • The Executor binary itself.
  • The runtime cache.
  • Application logs and databases.
  • Every file the workload wrote, such as the files in read_write_paths.

A stop ends execution; it never deletes or alters state.