Runner specification Draft 0.1 for review; pipelines live under .oxyhub/pipelines f160cbc57756

# Oxygen Runner — Specification

| | |

|---|---|

| **Status** | Draft 0.1, for review |

| **Scope** | Continuous integration and delivery for Oxygen repositories: pipeline definitions in the repository, a runner daemon (`oxyrun`) that executes them in fresh containers, the hub's queue and job API, and results as signed seals that gate landing. |

| **Depends on** | Oxygenhub ≥ 0.6 (checks, required checks, webhooks, API), `oxy-core` for checkouts and seals, an OCI runtime (Podman preferred, Docker supported) |

| **Date** | 2026-09-27 |

## 1. Why another CI system

GitHub Actions and GitLab CI work, and both carry mistakes that cannot be removed without breaking their users. This runner starts from Oxygen's model (content-addressed states, signed changes, seals, a trust policy) and keeps only what those systems got right: jobs in containers, a dependency graph of jobs, reusable steps, required checks, caches and artifacts.

What it refuses to repeat, and what it does instead:

| The mistake | Where | Here |

|---|---|---|

| YAML with implicit typing (`on: yes`, `NO` → false, `1.10` → 1.1), anchors, and string templating (`${{ … }}`) evaluated in several phases, so errors appear only at run time | Actions, GitLab | A typed KDL document with a published schema, validated by `oxyrun check` and by the hub on push, before anything runs. No template language: parameters are typed values; the only computation is matrix expansion (§4.5). |

| Actions and images referenced by mutable tags (`@v4`, `:latest`), so a compromised tag runs in every repository that trusts it | Actions (tj-actions, 2025), GitLab includes | Every reusable step and image is pinned to content: an Oxygen state hash or a signed tag verified against the action repository's policy, and images by digest. The hub refuses a pipeline that pins nothing (§6). |

| `pull_request_target` and similar triggers run a fork's proposal with the base repository's secrets | Actions | No such trigger. Secrets belong to named environments (§7); a proposal from someone without write access runs with no secrets until a maintainer approves that exact projected state. |

| "Works on CI, not locally" because the CI environment cannot be reproduced | both | `oxyrun run` executes the same document in the same image with the same inputs on a laptop, and must produce the same result; divergence is a runner bug (§9). |

| The unit of CI is a commit SHA, so two branches with the same tree build twice, and a merge commit is tested after merging | both | The unit is a **state hash**. A job's identity is (pipeline, job, state, inputs); the same state is built once whichever proposal or bookmark reaches it, and a proposal is tested on its **projected** state, which is exactly what landing produces (§5). |

| Green results are rows in a database, trusted because the server says so | both | A passed job is a **seal** signed by the runner's key over the state's changes (§8), verifiable by any `oxy` client offline and admitted by the repository's trust policy. |

| Cache keys are hand-written strings (`${{ hashFiles('**/Cargo.lock') }}-v2`), wrong in both directions | both | Keys are the BLAKE3 of the declared inputs, computed by the runner (§10). Restores are exact-key unless a fallback is declared. |

| Implicit shells, `set -e` off, and errors swallowed by pipes | both | A step runs a program with arguments. A `sh` step names its shell; `set -eu -o pipefail` is on unless the step turns it off (§4.4). |

| Logs are a text blob; secrets masked by string matching after the fact | both | Logs are structured records (time, job, step, stream, text) streamed to the hub and folded by step. Secrets are delivered as files or environment entries the runner knows, and redacted at the source (§11). |

| Long-lived runners that keep state between jobs, sometimes shared across repositories | both (self-hosted) | Every job gets a fresh container (or VM); nothing survives a job but its declared outputs, caches and artifacts. Runners are cattle (§12). |

| CI definitions that can do anything, including push to protected branches with a broad token | both | A job's token can read the repository and report its own checks; nothing else. Deploy jobs get more only through an environment (§7). |

## 2. Terms

- **Pipeline**: one document under `.oxyhub/pipelines/NAME.kdl` in the state being built. (Not under `.oxy/`, which is the repository's own metadata directory and is never tracked.)

- **Trigger**: what makes the hub run a pipeline: a push to a bookmark, a proposal (on its projected state), a tag, a schedule, or a manual request.

- **Run**: one execution of one pipeline on one state; identified by `(repository, pipeline, state, trigger inputs)`.

- **Job**: a node in the run's graph; runs in one container with one image; has steps.

- **Step**: a program with arguments, or a reusable action, inside the job's container.

- **Runner**: a machine running `oxyrun daemon`, registered with a hub, claiming jobs whose labels it has.

- **Environment**: a named set of secrets and protection rules (who may run jobs in it, for which references, whether proposals may use it).

## 3. Where the document lives and how it is chosen

The hub reads pipelines from the state being built, never from the target: a proposal runs the pipeline it proposes. Because a proposal from an outsider could change its own pipeline, outsider proposals run with no secrets and no environments (§7.3), which makes that safe.

## 4. The pipeline document

KDL (https://kdl.dev) was chosen over YAML for explicit types, comments, no anchors and a small grammar; over Starlark or other programs because a pipeline should be data a reviewer can read, with the only computation being matrix expansion. The schema is `pipeline.kdl-schema` published by `oxyrun schema`.

### 4.1 Example

```kdl

// .oxyhub/pipelines/ci.kdl

pipeline "ci" version=1

on {

    push bookmark="main"

    proposal target="main"

    tag pattern="v*"

}

image "rust" "docker.io/library/rust@sha256:4f8b…" // by digest, always

job "test" {

    image "rust"

    runs-on "linux" "x86_64"

    matrix {

        toolchain "stable" "1.85"

    }

    cache "cargo" {

        paths "~/.cargo/registry" "target"

        inputs "Cargo.lock" "rust-toolchain.toml"

    }

    step "fmt" run="cargo" args=["fmt", "--all", "--", "--check"]

    step "clippy" run="cargo" args=["clippy", "--workspace", "--all-targets", "--", "-D", "warnings"]

    step "test" run="cargo" args=["test", "--workspace"] {

        env "RUSTUP_TOOLCHAIN" matrix="toolchain"

    }

    timeout minutes=30

}

job "package" {

    needs "test"

    image "rust"

    runs-on "linux" "x86_64"

    step "build" run="cargo" args=["build", "--release"]

    artifact "oxy" path="target/release/oxy" retention-days=30

}

job "deploy" {

    needs "package"

    when tag pattern="v*"

    environment "production"

    image "rust"

    uses "hub:andymccall/deploy-action" state="3f2a…" {

        input "binary" artifact="oxy"

    }

}

```

### 4.2 Types

Strings, integers, booleans and lists as KDL defines them; durations as `minutes=`/`seconds=`; paths relative to the checkout unless they start with `~/`. A value is never reinterpreted: `"1.10"` stays a string.

### 4.3 Jobs

`image` (a named image, pinned by digest), `runs-on` (labels a runner must have), `needs` (jobs that must pass first), `when` (a trigger predicate), `matrix`, `env`, `cache`, `artifact`, `service` (sidecar containers, by digest), `environment`, `network` (`none`, `egress`, or `egress` with allowed hosts), `resources` (cpu, memory), `timeout`, `continue-on-error=false`, and steps.

### 4.4 Steps

`run=PROGRAM args=[…]` executes the program directly with no shell. `sh "script"` runs `/bin/sh -eu -o pipefail -c script` (bash with the same flags via `shell="bash"`); `strict=false` drops the flags explicitly. `uses=REF` runs a reusable action (§6). Each step has `env`, `workdir`, `timeout`, and `continue-on-error`. A step's outputs are files under `$OXYRUN_OUTPUT/NAME`, read by later steps as typed values, not parsed out of logs.

### 4.5 Matrix

A `matrix` block lists axes with values; the job expands into the cartesian product, each named `job (axis=value, …)`. `exclude` and `include` blocks adjust it. There is no other templating.

## 5. What a run builds

The runner checks out the exact state with `oxy` (a shallow checkout of the state's heads into the job's workspace), so every job of a run sees the same files. For a proposal the state is the **projected** state the hub computed (hub spec §7.2): the job tests what landing will produce, and its result carries over to the landed bookmark because the state is the same object.

Jobs are identified by `(pipeline, job, matrix values, state, image digest, input hashes)`. A passed job with the same identity is not re-run; the hub reuses its result and seal.

## 6. Reusable actions

An action is an Oxygen repository with `.oxyhub/action.kdl` declaring inputs, outputs and its steps (or a container image by digest). It is referenced as:

- `hub:owner/name` with `state="…"` (a state hash), or `tag="v1"` **only if** the tag is signed by a key the action repository's policy authorises for `tags/*`, which the runner verifies before running it;

- `git+https://host/path` with `commit="…40 hex…"`, for actions that live in Git.

A reference with neither a state, a signed tag, nor a commit is a validation error. Actions run in the job's container unless they declare their own image. They receive only their declared inputs; they cannot read secrets the job did not pass them explicitly.

## 7. Secrets and environments

### 7.1 Environments

An environment belongs to a repository (or an organisation) and holds secrets and rules: which bookmarks and tags may use it, whether a human must approve each run (and who), and a wait timer. Only jobs that name the environment receive its secrets.

### 7.2 Delivery

Secrets reach a step as files under `$OXYRUN_SECRETS/NAME` (default) or as environment variables when the step asks. They are never written into the checkout, the cache, or artifacts. The runner redacts every secret value from log records before they leave the container, and refuses to start a step whose command line contains a secret value.

### 7.3 Proposals

A proposal whose author lacks write access runs its pipeline with **no** environments and no secrets, and no `network egress` unless the repository allows it. A maintainer may approve one run of one projected state with its environments; the approval is bound to the state hash, so a later push to the proposal needs a new approval. There is no trigger that runs untrusted code with the base repository's secrets.

## 8. Results as seals

When a job passes, the runner signs a **seal** attestation (spec §14.4) over the changes of the state it built, with the runner's own Oxygen identity, and the hub stores it with the check (hub spec §13.1). A repository's policy can name runner keys as checkers; a required check then means "a seal from an authorised checker key over this state", which any `oxy` client can verify offline: `oxy verify` shows which checks sealed a state. A failed job produces a check without a seal.

## 9. Running locally

`oxyrun check` validates every pipeline in the working copy against the schema and the pinning rules. `oxyrun run [JOB] [--matrix axis=value]` runs the job on the working copy's current state (or `--state HASH`) in the same image, with the same inputs, caches in `~/.cache/oxyrun`, secrets from `--secret NAME=file` only. The same runner code executes both; "passes locally, fails on CI" with the same state and image is a bug in the runner.

## 10. Caches and artifacts

A cache's key is BLAKE3 over: the cache name, the image digest, the job's `runs-on` labels, and the content of every file matching `inputs`. Restore is by exact key; `fallback` lists older input sets explicitly. Caches and artifacts are content-addressed tarballs stored by the hub (the same store as release assets), with per-repository quotas and retention. Artifacts can be downloaded from the run page and promoted to a release asset.

## 11. Logs

Every line a step writes becomes a record `{time, job, step, stream, text}` streamed to the hub over the job API; the run page folds by step, shows timings, and highlights the failing step. Records are stored compressed per job with retention. Structured annotations (`::error file=src/x.rs line=12::message` is replaced by a JSON line on a dedicated file descriptor) attach to files in the proposal's Files tab.

## 12. Runners

`oxyrun daemon --hub URL --token REGISTRATION` registers once and receives a runner identity and credential; it then long-polls `POST /api/v1/runners/claim` with its labels and capacity. Each claimed job runs in a fresh container with a fresh workspace volume, `--network none` unless allowed, no access to the host's container socket, resource limits from the job, and a job token that can read the repository at that state and report that job's check, expiring when the job ends. After the job the container and workspace are destroyed. A runner may be limited to repositories or organisations; the hub never gives a runner a job its scope excludes.

## 13. Hub side

New tables: pipelines (parsed per state, cached), runs, jobs (queued, claimed, running, passed, failed, cancelled, skipped), job logs, caches, artifacts, runners, environments, secrets (encrypted at rest with a key in `<data>/keys/`), approvals. New pages: Runs tab per repository, run and job pages with live logs, runners in organisation and site settings, environments in repository settings. New API: runner registration and claim, job log and status streams, artifact and cache upload and download, manual dispatch, re-run, cancel. Triggers are evaluated on push, proposal projection, tag and schedule events inside the hub. Check names are `pipeline/job` (with matrix values), so required checks name them.

## 14. Security summary

- A pipeline is data validated before it runs; nothing mutable is ever fetched by name.

- Untrusted proposals run without secrets or environments until a maintainer approves that exact state.

- Secrets live in environments, reach only jobs that name them, and are redacted at the source.

- Jobs are isolated and ephemeral; tokens are per job and minimal.

- Results are seals by runner keys the repository's policy admits, verifiable without trusting the hub.

## 15. Milestones

| | Delivers |

|---|---|

| **R0.1** | KDL schema, `oxyrun check`, `oxyrun run` locally with Podman or Docker, steps, matrix, local caches |

| **R0.2** | Hub queue, triggers, runner registration and claim, `oxyrun daemon`, logs streamed and folded, checks per job, runs pages |

| **R0.3** | Caches and artifacts on the hub, re-run and cancel, manual dispatch, schedules |

| **R0.4** | Environments, secrets, approvals for outsider proposals and protected environments |

| **R0.5** | Seals from runner keys, policy checkers, required checks as seals, `oxy verify` shows them |

| **R1.0** | Reusable actions with verification, services, network policy with allowed hosts, VM isolation option, documentation |

The first real pipeline is Oxygen's own: fmt, clippy, tests and a release build on Earth, as a required check on `main`.

## 16. Open questions

1. KDL 2.0 or 1.0 syntax: the Rust `kdl` crate supports both; 2.0 is cleaner and is the proposal here.

2. Podman only, or Docker too: Earth runs Docker today; supporting both through their CLIs is cheap, the socket API is not needed.

3. VM isolation (Firecracker) for untrusted proposals: stronger than containers, more to operate. Proposed for R1.0 as an option.

4. Whether runner keys should be per runner or per runner group in the policy.

1. **Pipelines are typed, validated documents, not templated YAML.** The definition lives in the repository (`.oxyhub/pipelines/*.kdl`; not under `.oxy/`, the repository's own metadata directory), has a schema, and is validated by `oxyrun check` and by the hub on push before anything runs. Errors point at a line. There is no string templating stage; parameters are typed values.

5. **Pipeline language.** KDL is the current choice for `.oxyhub/pipelines`; Starlark would give loops and functions at the cost of a real evaluator. Settle in `docs/runner.md`.