Auto-scaling and load balancer
Auto-scaling is a native, optional capability: autoscale.enabled: true or false. The supervisor runs one or more replicas of the same service on one host and balances traffic across them itself. A replica is a complete, independent guest process tree, started with the same entry point, the same sandbox policy and its own copy of the resource caps. No orchestrator, sidecar or external proxy is involved.
autoscale:
enabled: true
min_replicas: 1
max_replicas: 4
metrics:
memory:
upper_threshold: "80%"
lower_threshold: "30%"
cpu:
upper_threshold: "75%"
lower_threshold: "20%"
scale_out_after: "30s"
scale_in_after: "5m"
cooldown: "60s"
replica:
port_range: "auto"
port_env: "PORT"
readiness:
type: "http"
path: "/healthz"
timeout: "30s"
load_balancer:
listen: "0.0.0.0:8000"
mode: "tcp"
algorithm: "round_robin"
health_check:
interval: "5s"
unhealthy_threshold: 3
drain_timeout: "30s"
total_limits:
cpu: "50%"
memory: "8GB"
The scaling model
- Thresholds. You set a lower and an upper threshold per metric. Thresholds on
memoryandcpuare percentages of the per-replica cap, or absolute amounts, so auto-scaling needs a cap on every metric it uses. - Scale out. When the average usage across ready replicas stays above the
upper_thresholdof any configured metric for the wholescale_out_afterwindow, and fewer thanmax_replicasare running, the supervisor starts one more replica. - Scale in. When usage stays below the
lower_thresholdof every configured metric for the wholescale_in_afterwindow, and more thanmin_replicasare running, the supervisor removes the most recently started replica. - Stability. The gap between the thresholds gives hysteresis, a
cooldownfollows every scaling action, and at most one replica is added or removed per decision, so the replica count cannot oscillate. - Caps and scaling together. Each replica is held to the configured cap on its own, so no replica ever exceeds a
strictcap. The optionalautoscale.total_limitsis a hard ceiling over all replicas together: the supervisor never starts a replica that would take the total beyond it, andmax_replicasbounds the worst case.
The replica lifecycle
- Port allocation. Each replica gets a different available port, chosen automatically (
port_range: "auto") or from a range such as"20000-20100". The port reaches the replica through the{{port}}placeholder in the entry point and through the environment variable named byport_env. Replicas listen on loopback; only the load balancer listens on the public address. If the chosen port is taken between selection and bind, the start is retried on the next free port. - Launch. A replica starts through the same path as a single workload: its own process-tree containment, its own cap group, the same sandbox rules and the same scrubbed environment.
- Readiness gating. A new replica receives traffic only after its readiness check passes: a TCP connect, or an HTTP
GETreturning a success status. A replica that is not ready within the timeout is stopped and counted as a failed start. - Removal. A replica chosen for scale-in is taken out of rotation, its open connections drain for up to
drain_timeout, and it is stopped with the state-preserving stop. Its files stay on disk. - Crash handling. A replica that exits unexpectedly leaves rotation at once and is replaced after an exponential backoff while the count is below
min_replicasor load requires it. Repeated failures past the restart limit stop the whole instance and pass on the replica's exit code. - Exit events. Every exit event applies to the instance as a whole and stops the load balancer and every replica. A checkpoint from any replica counts for the instance.
The built-in load balancer
| Aspect | Behavior |
|---|---|
| Placement | Inside oee-stub, on the supervisor's event loop. Not a separate process; it needs no configuration beyond autoscale.load_balancer. |
| Modes | tcp (default) balances connections and works for any TCP protocol. http balances individual HTTP/1.1 requests across replicas, including requests on keep-alive connections, and adds X-Forwarded-For. |
| Algorithms | round_robin (default) and least_connections. |
| Health | Backends are checked actively every health_check.interval and passively on connection failure. An unhealthy backend gets no new connections. A connection attempt that fails before any byte is exchanged is retried on another healthy backend. |
| Sandbox | With network sandboxing, the load balancer accepts on the host side and forwards into each replica's isolated network context, so sandbox.network.inbound_allow_ports governs the public port only and replica ports are never exposed. |
| Single replica | With autoscale.enabled: false no load balancer runs, {{port}} resolves to the port of autoscale.load_balancer.listen when that block is present (8000 in the example), and the workload binds its port directly. |
Performance
These figures are design targets, not measured results.
| Measure | Design target |
|---|---|
| Added median latency per proxied connection on loopback | Less than 1 ms |
Supervisor memory per idle proxied connection in tcp mode | Less than 20 KB |
| Launch of a new replica after the scale-out window elapses | Within 1 s |
| Traffic to a new replica after it passes readiness | Within 100 ms |
Observability
oee status reports the replica count, each replica's port, readiness and usage against its cap, and each load-balancer backend's health and active connections. Every scaling decision emits a structured audit event with the metric values that caused it.
Auto-scaling and the load balancer are part of the extended supervisor variant, which oee build selects automatically.