Browse the documentation

Early preview

Auto-scaling and load balancer

Auto-scaling is a native, optional capability: autoscale.enabled: true or false. The supervisor runs one or more replicas of the same service on one host and balances traffic across them itself. A replica is a complete, independent guest process tree, started with the same entry point, the same sandbox policy and its own copy of the resource caps. No orchestrator, sidecar or external proxy is involved.

autoscale:
  enabled: true
  min_replicas: 1
  max_replicas: 4
  metrics:
    memory:
      upper_threshold: "80%"
      lower_threshold: "30%"
    cpu:
      upper_threshold: "75%"
      lower_threshold: "20%"
  scale_out_after: "30s"
  scale_in_after: "5m"
  cooldown: "60s"
  replica:
    port_range: "auto"
    port_env: "PORT"
    readiness:
      type: "http"
      path: "/healthz"
      timeout: "30s"
  load_balancer:
    listen: "0.0.0.0:8000"
    mode: "tcp"
    algorithm: "round_robin"
    health_check:
      interval: "5s"
      unhealthy_threshold: 3
    drain_timeout: "30s"
  total_limits:
    cpu: "50%"
    memory: "8GB"

The scaling model

  • Thresholds. You set a lower and an upper threshold per metric. Thresholds on memory and cpu are percentages of the per-replica cap, or absolute amounts, so auto-scaling needs a cap on every metric it uses.
  • Scale out. When the average usage across ready replicas stays above the upper_threshold of any configured metric for the whole scale_out_after window, and fewer than max_replicas are running, the supervisor starts one more replica.
  • Scale in. When usage stays below the lower_threshold of every configured metric for the whole scale_in_after window, and more than min_replicas are running, the supervisor removes the most recently started replica.
  • Stability. The gap between the thresholds gives hysteresis, a cooldown follows every scaling action, and at most one replica is added or removed per decision, so the replica count cannot oscillate.
  • Caps and scaling together. Each replica is held to the configured cap on its own, so no replica ever exceeds a strict cap. The optional autoscale.total_limits is a hard ceiling over all replicas together: the supervisor never starts a replica that would take the total beyond it, and max_replicas bounds the worst case.

The replica lifecycle

  1. Port allocation. Each replica gets a different available port, chosen automatically (port_range: "auto") or from a range such as "20000-20100". The port reaches the replica through the {{port}} placeholder in the entry point and through the environment variable named by port_env. Replicas listen on loopback; only the load balancer listens on the public address. If the chosen port is taken between selection and bind, the start is retried on the next free port.
  2. Launch. A replica starts through the same path as a single workload: its own process-tree containment, its own cap group, the same sandbox rules and the same scrubbed environment.
  3. Readiness gating. A new replica receives traffic only after its readiness check passes: a TCP connect, or an HTTP GET returning a success status. A replica that is not ready within the timeout is stopped and counted as a failed start.
  4. Removal. A replica chosen for scale-in is taken out of rotation, its open connections drain for up to drain_timeout, and it is stopped with the state-preserving stop. Its files stay on disk.
  5. Crash handling. A replica that exits unexpectedly leaves rotation at once and is replaced after an exponential backoff while the count is below min_replicas or load requires it. Repeated failures past the restart limit stop the whole instance and pass on the replica's exit code.
  6. Exit events. Every exit event applies to the instance as a whole and stops the load balancer and every replica. A checkpoint from any replica counts for the instance.

The built-in load balancer

AspectBehavior
PlacementInside oee-stub, on the supervisor's event loop. Not a separate process; it needs no configuration beyond autoscale.load_balancer.
Modestcp (default) balances connections and works for any TCP protocol. http balances individual HTTP/1.1 requests across replicas, including requests on keep-alive connections, and adds X-Forwarded-For.
Algorithmsround_robin (default) and least_connections.
HealthBackends are checked actively every health_check.interval and passively on connection failure. An unhealthy backend gets no new connections. A connection attempt that fails before any byte is exchanged is retried on another healthy backend.
SandboxWith network sandboxing, the load balancer accepts on the host side and forwards into each replica's isolated network context, so sandbox.network.inbound_allow_ports governs the public port only and replica ports are never exposed.
Single replicaWith autoscale.enabled: false no load balancer runs, {{port}} resolves to the port of autoscale.load_balancer.listen when that block is present (8000 in the example), and the workload binds its port directly.

Performance

These figures are design targets, not measured results.

MeasureDesign target
Added median latency per proxied connection on loopbackLess than 1 ms
Supervisor memory per idle proxied connection in tcp modeLess than 20 KB
Launch of a new replica after the scale-out window elapsesWithin 1 s
Traffic to a new replica after it passes readinessWithin 100 ms

Observability

oee status reports the replica count, each replica's port, readiness and usage against its cap, and each load-balancer backend's health and active connections. Every scaling decision emits a structured audit event with the metric values that caused it.

Auto-scaling and the load balancer are part of the extended supervisor variant, which oee build selects automatically.