Kubernetes

The Agent’s Kustomize base is published alongside this documentation, one per version: Namespace, ConfigMap, a StatefulSet with one identity volume per Agent, exec probes and a NetworkPolicy. It carries no environment values at all: the image, the credentials and the StorageClass come from your overlay. There is no Helm chart.

The requirements further down apply to the base and to anyone who prefers to write their own manifests.

Installation

Download the base for the same version as the image you run:

VERSION=X.Y.Z   # your image's tag, with or without the "v"
BASE="https://docs.rootpilot.sh/kubernetes/v${VERSION#v}/base"
mkdir -p base
FILES=$(curl -fsSL "$BASE/files.txt") &&
  for f in $FILES; do curl -fsSL -o "base/$f" "$BASE/$f"; done

The published folder carries the release tag’s v (/kubernetes/v1.25.0/base/); the image tag does not (rootpilot-agent:1.25.0). ${VERSION#v} accepts both forms. If the version has no published base, the first curl fails with a 404 and nothing is downloaded.

The base and the image come from the same release

A version’s base may depend on something that only exists in that version’s image. The first one already does: the probes run rootpilot-health, which is only in the image from v1.23.0 on. With the base from one version and the image from another, the probe can fail and restart the pod in a loop. When you upgrade the image, download the base for the same version again.

Next to it, create the overlay with the image you received, by registry and digest (see Image):

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - base
images:
  - name: rootpilot-agent
    newName: <address>/rootpilot-agent
    digest: sha256:<digest>

Create the Secret and apply:

kubectl apply -f base/namespace.yaml
kubectl -n rootpilot-agent create secret generic rootpilot-agent-env --from-env-file=agent.env
kubectl apply -k .
kubectl -n rootpilot-agent rollout status statefulset/rootpilot-agent --timeout=5m

agent.env carries RUNNER_BOOTSTRAP_TOKEN and, with RUNNER_SECRET_STORE_MODE=env (the base’s default), the connector credentials. After the first hello accepted, remove RUNNER_BOOTSTRAP_TOKEN from the Secret: it has been consumed, and restarts renew the identity from the volume.

If the identity is lost (the PVC was deleted, or it expired beyond the renewal grace, see When identity is lost), boot fails with RUNNER_ATTESTATION_MODE=bootstrap-token requires RUNNER_BOOTSTRAP_TOKEN and the pod goes into CrashLoopBackOff. The old token will not work again: mint a new one under Onboarding, put it back in the Secret and recreate the pod (kubectl -n rootpilot-agent delete pod rootpilot-agent-0), because the Secret is only read when the container starts.

Changing the configuration

The base’s non-secret configuration comes from a configMapGenerator. To change a key, merge it in your overlay; the ConfigMap’s name carries a hash of its content, and kubectl apply -k . recreates the pod with the new value:

# kustomization.yaml (the same overlay as above)
configMapGenerator:
  - name: rootpilot-agent-config
    behavior: merge
    literals:
      - RUNNER_SECRET_STORE_MODE=aws
      - RUNNER_SECRETS_MANAGER_REGION=us-east-1

More than one Agent

N Agents are N installations, not N replicas. The bootstrap token is single-use and there is one Secret per installation, so scaling the StatefulSet makes the second pod fail enrollment. Each Agent gets an overlay with its own namespace (Kustomize also renames the base’s Namespace), its own Secret and its own token:

agent-b/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: rootpilot-agent-b
resources:
  - ../base
images:
  - name: rootpilot-agent
    newName: <address>/rootpilot-agent
    digest: sha256:<digest>

kubectl create namespace rootpilot-agent-b
kubectl -n rootpilot-agent-b create secret generic rootpilot-agent-env --from-env-file=agent-b.env
kubectl apply -k agent-b
kubectl -n rootpilot-agent-b rollout status statefulset/rootpilot-agent --timeout=5m

agent-b.env carries a new bootstrap token, minted for this Agent: the first one’s is already consumed.

A namespace just for the Agent

The overlay’s namespace: must be a dedicated namespace. Kustomize renames the base’s Namespace to it, and apply stamps the restricted Pod Security profile on it: pointed at a namespace that already exists, the profile then applies to every pod in it.

The base’s RUNNER_INSTANCE_ID is <pod>.<namespace>.<pod uid>: unique across clusters too, so Agents in different clusters with the same namespace show up separately in Fleet. The UID changes when the pod is recreated: the old entry shows as disconnected and leaves Fleet on its own.

The container

The image is built by apps/runner/Dockerfile, with the repository root as context. The entrypoint is apps/runner/docker-entrypoint.sh and the command is:

node --import @rootpilotsh/otel/register apps/runner/dist/index.js

The image runs as node (uid 1000), not root, which is why Compose mounts ~/.aws at /home/node/.aws. The mount target is coupled to the image’s USER: the SDK’s default chain resolves the SSO cache from $HOME, and no env var repoints that cache. If you mount ~/.aws elsewhere, mount it at the $HOME of the user the pod runs as.

Requirement 1: health over exec, not HTTP

The runner is an outbound client. It listens on no port, exposes no /healthz, and has no readiness endpoint. Do not configure httpGet on any probe: every one of them will fail.

The health signal is a file. The runner writes to RUNNER_HEALTH_DIR (default /tmp/rootpilot-health) and the image ships rootpilot-health, which reads it from inside the container:

• rootpilot-health live: the process is alive and its event loop is running (heartbeat under 60 s);
• rootpilot-health ready: alive and enrolled, with the tunnel connected and the hello accepted.

          livenessProbe:
            exec:
              command: [rootpilot-health, live]
            periodSeconds: 20
            timeoutSeconds: 5
          readinessProbe:
            exec:
              command: [rootpilot-health, ready]
            periodSeconds: 10
            timeoutSeconds: 5

Never use ready as liveness. The tunnel drops when the network drops, and reconnecting is the runner’s backoff job: restarting the pod for it trades a runner that reconnects on its own for one that starts over.

With readOnlyRootFilesystem: true, mount an emptyDir at /tmp (or point RUNNER_HEALTH_DIR at a volume writable by uid 1000). With nowhere to write, the probe fails, which is the honest way of saying the signal does not exist. Per-connector health stays in the handshake and the control-plane’s Fleet.

Requirement 2: termination grace longer than the drain

The runner handles SIGTERM by draining: it stops accepting new work, finishes what’s in flight, and only then exits. The ceiling is RUNNER_DRAIN_TIMEOUT_MS, which defaults to 120,000 ms.

Kubernetes defaults terminationGracePeriodSeconds to 30 s. Leave it there and SIGKILL arrives mid-drain.

terminationGracePeriodSeconds: 150   # > RUNNER_DRAIN_TIMEOUT_MS (120s), with headroom

A hard reclaim with no drain (SIGKILL, OOM, spot) loses learned state since the last flush. RUNNER_LEARNED_FLUSH_INTERVAL_MS (default 60 s) exists to bound that loss, but it doesn’t eliminate it.

Requirement 3: identity on a volume, cache in memory

The mTLS identity persists in RUNNER_IDENTITY_DIR: a restart renews the certificate from it instead of spending another bootstrap token. On Kubernetes that is one PVC per Agent: the base’s StatefulSet creates one per ordinal, writable by uid 1000 (fsGroup: 1000). Never share the volume between Agents, because each boot would overwrite the other’s identity.

The learned-store cache is a different thing and stays :memory: by default: what persists is identity, not state, and the source of truth for what was learned is the control-plane. Do not point RUNNER_LEARNED_STORE_PATH at the identity volume.

With platform attestation (gke-oidc, aws-sts) the Agent re-attests on every boot and the volume is optional. Some providers have a minimum disk size far above what is needed (MGC and GCE require 10 GiB, even though the identity takes a few KiB), and the base already requests that size. Check your StorageClass’s reclaimPolicy: with Retain, deleting the PVC does not delete the disk.

Requirement 4: identity with no secret in the image

This is where Kubernetes changes the design, and for the better.

GKE: gke-oidc

The runner reads the projected OIDC token of the Kubernetes ServiceAccount and sends it as attestation. Nothing secret enters the image: the token is a short-lived, audience-bound JWT the kubelet projects.

The default path is /var/run/secrets/tokens/gke-oidc/token, configurable via RUNNER_GKE_TOKEN_PATH. So the pod needs a projected volume with a serviceAccountToken mounted exactly there, projected for the rootpilot-enroll audience. On top of the base this is a patch in your overlay; the base’s container is named agent, and Kustomize matches containers by name:

patch-gke-oidc.yaml

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: rootpilot-agent
spec:
  template:
    spec:
      volumes:
        - name: whystack-token
          projected:
            sources:
              - serviceAccountToken:
                  path: token
                  audience: rootpilot-enroll   # the control-plane requires this audience
                  expirationSeconds: 3600
      containers:
        - name: agent
          volumeMounts:
            - name: whystack-token
              mountPath: /var/run/secrets/tokens/gke-oidc
              readOnly: true

# kustomization.yaml (the overlay)
patches:
  - path: patch-gke-oidc.yaml
configMapGenerator:
  - name: rootpilot-agent-config
    behavior: merge
    literals:
      - RUNNER_ATTESTATION_MODE=gke-oidc

The audience isn’t decoration: it’s what stops a KSA token projected for some other service from being replayed into an enrollment. The control-plane asserts it and refuses anything that doesn’t match, including tokens projected with no audience at all.

If the volume isn’t mounted, the error says so explicitly:

RUNNER_ATTESTATION_MODE=gke-oidc: failed to read projected KSA token at
/var/run/secrets/tokens/gke-oidc/token (is the Workload Identity token volume mounted?)

If the audience is wrong, the volume mounts, the runner reads the token, and the enroll is refused. The error arrives as attestation_invalid, exactly what a bad credential would produce. Before investigating the KSA or Workload Identity, check the audience field above.

EKS: aws-sts, not eks-oidc

eks-oidc is not implemented

RUNNER_ATTESTATION_MODE=eks-oidc passes configuration validation but resolves to an unimplemented provider that throws at boot: attestation mode 'eks-oidc' not implemented.

On EKS, use RUNNER_ATTESTATION_MODE=aws-sts.

aws-sts signs a GetCallerIdentity request using the SDK’s default credential chain, which on EKS is IRSA. That means an annotated ServiceAccount with an IAM role is enough, with no bootstrap token and no human in a browser.

It requires AWS_REGION explicitly. Without it, boot fails with RUNNER_ATTESTATION_MODE=aws-sts requires AWS_REGION. The region is not guessed on purpose, because the control-plane only accepts regions from its own allowlist, and a guess would surface as an attestation_invalid that reads like a credential problem.

Requirement 5: secrets

Prefer the secret manager you already operate over Kubernetes Secrets: RUNNER_SECRET_STORE_MODE=aws (AWS Secrets Manager) or vault (HashiCorp Vault KV v2). aws resolves through the workload identity chain, with no static keys. vault authenticates with a token or AppRole: the pod’s ServiceAccount token is not yet a Vault login method. See Secret stores. The base uses env, with the rootpilot-agent-env Secret.

For AWS read credentials, RUNNER_AWS_CRED_MODE=chain makes the SDK resolve through IRSA directly, with no key passing through the cluster at all.

One runner per account, by design

chain mode is single-account. For multiple AWS accounts, run one runner per account, each with its own IRSA identity. The graph is merged on the control-plane side.

Requirement 6: egress

Outbound only, nothing inbound. If you use NetworkPolicy, the destination list is in Requirements. Remember that the allowlist declared in connector manifests is documentation, not enforcement: for native connectors, the cluster is what enforces egress.

Helm and sizing

There is no Helm chart. The base’s resources are a starting point, not a benchmark: measure CPU and memory with your real connectors before tightening the limits. The per-provider attestation choice also belongs to your overlay (see Changing the configuration): the base uses bootstrap-token, which works on any cluster.