OxygenFlow (flow) — Deterministic Execution & Provenance Pipeline Specification

The key words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are to be interpreted as described in RFC 2119.

1. Overview and Core Invariants

OxygenFlow (flow) is a content-addressed, hermetic execution engine, runner system, and pipeline language designed natively for the Oxygen (oxy) version control system.

Existing runner engines (GitHub Actions, GitLab CI, Forgejo Runner) suffer from structural flaws:

1. Workspace Bloat: Full clones, index unpacks, and checkout scans dominate I/O.
2. String Injection: Shell interpolation within untyped YAML files produces expression injection and command execution exploits.
3. Decoupled Supply Chain: Attestations (SLSA, in-toto, Cosign) exist outside the VCS, relying on external registries rather than native repository trust.
4. Coarse Caching: Opaque tarballs keyed by arbitrary string hashes fail to leverage commutative history or granular dependency graphs.

OxygenFlow solves these liabilities through direct integration with Oxygen:

• Zero-Copy Workspaces: Jobs mount workspaces directly from Oxygen's FastCDC- chunked, BLAKE3-addressed Content-Addressable Store (CAS) using Copy-on-Write (COW) overlays.
• Commutative History & Impact Trees: Pipelines evaluate over change closures rather than linear snapshot sequences. Commuting changes automatically reuse cached execution receipts without redundant re-execution.
• Micro-Engine Sandboxing: Jobs run inside isolated environments (WASM, gVisor, or MicroVMs) with strict, deny-by-default capabilities and no ambient network egress.
• Native Cryptographic Provenance: Provenance and execution receipts are generated outside the untrusted guest, encoded as deterministic CBOR, and signed using composite Ed25519 + ML-DSA-65 keys as native Oxygen Attest objects governed by in-repository Policy rules.

2. Threat Model and Security Architecture

2.1 Adversaries

• Untrusted Proposal Author: Submits code to an open repository containing malicious build configurations intended to escape the runner, exfiltrate secrets, or corrupt shared worker caches.
• Malicious Dependency / Upstream: Third-party dependencies invoked during build time attempting out-of-band network calls or host file modification.
• Compromised Runner / Malicious Worker: An execution node attempting to forge successful build receipts or inject backdoors into published artifacts.

2.2 Security Invariants

• I1 Deny-by-Default Execution: Sandboxes possess no network interfaces (excluding loopback when requested) and no host filesystem access unless explicitly granted in policy-signed configuration.
• I2 Injection-Free Invocation: Configuration parsing directly yields an argv string vector executed via execve(). Shell interpretation (/bin/sh -c) does not exist.
• I3 Host-Enforced Provenance: The running payload cannot author, inspect, or alter its own provenance statement. Provenance is emitted solely by the isolation hypervisor boundary.
• I4 Cryptographic Identity & Authorization: Runners execute using an Oxygen Identity whose KeyId is enrolled as a delegate in the repository's Policy. Unenrolled runners cannot publish recognized attestations or advance gates.

3. The OxygenFlow Configuration Language (.flow)

Pipelines are declared in KDL syntax within .flow/pipeline.kdl at the repository root. KDL provides strict typing, structured nesting, and rejects arbitrary text macro expansion.

Strings follow KDL 2.0, which has no \033, \e or \x escapes: ESC, for ANSI colours in a job's output, is \u{1b} ("\u{1b}[32mok\u{1b}[0m"), and flow check says so when it meets one of the others. Inside the guest, /dev/fd and /dev/stdin|stdout|stderr link to /proc/self/fd as on any Linux system, so bash process substitution works.

3.1 Pipeline Specification Schema

schema-version 1

// Define the hermetic execution boundary
sandbox "rust-builder" {
    runtime "microvm"       // Options: "wasm" | "gvisor" | "microvm"
    rootfs "cas://blob/9b5c2a1e8f3d4e7a6b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a"
    memory-limit "4GiB"
    cpu-limit 4
    network "isolated"      // Options: "isolated" | "loopback" | "egress" { allow "host"... } (§4.3.1)
}

// Job evaluated based on Oxygen change closure
job "compile-core" {
    sandbox "rust-builder"

    // Commutation boundary: runs only if change closure touches these paths
    scope "crates/oxy-core/**" "crates/oxy-server/**" "Cargo.*"

    // Zero-copy workspace mount backed by Oxygen CAS
    mount {
        source "oxy://workspace"
        target "/src"
        include "crates/oxy-core" "crates/oxy-server" "Cargo.toml" "Cargo.lock"
        readonly #true
    }

    // Ephemeral scratch space
    mount {
        source "ephemeral://tmpfs"
        target "/src/target"
        size "8GiB"
    }

    env {
        RUSTFLAGS "-Dwarnings"
        CARGO_INCREMENTAL "0"
    }

    // Direct argv execution vector (no subshell)
    exec "cargo" "build" "--release" "--package" "oxy-core"

    output {
        artifact "oxy-core-lib" {
            path "/src/target/release/liboxy_core.rlib"
            format "blob"
        }
    }
}

job "unit-tests" {
    sandbox "rust-builder"
    depends-on "compile-core"

    mount {
        source "oxy://workspace"
        target "/src"
        readonly #true
    }

    mount {
        source "artifact://compile-core/oxy-core-lib"
        target "/src/target/release/liboxy_core.rlib"
    }

    exec "cargo" "test" "--release" "--package" "oxy-core"
}

publish "release-provenance" {
    depends-on "unit-tests"

    attest {
        kind "provenance"
        bind-ref "bookmarks/main"
        include-artifacts "compile-core/oxy-core-lib"
    }
}


**What starts a job.** An `on` block, at the top of the pipeline or in a job (the job's wins), names the events that run it: `push` with bookmark globs (none: any bookmark), `tag` with tag globs (none: any tag), `manual` ("Run pipeline", `oh run start`) and `schedule` with five-field cron lines on the hub's clock (UTC), run on the default bookmark. Without `on`, a job runs on every push and by hand, as before. A run has only the jobs its event starts (and what they depend on); an event that starts no job starts no run. A rerun repeats its run's event.

```kdl
on {
    push "main" "feature/*"
    manual
}
job "dist" {
    on {
        tag "v*"
        manual
    }
    …
}

Time limits. A job may say how long it may run with timeout "90m" (hours, minutes and seconds, as "2h", "1h30m" or "45s", from a minute to a day); a sandbox's timeout is the default for its jobs. Without one, the hub's default applies, and the hub caps every job at the site's maximum (both set by an administrator on Admin → Runners). The runner stops a job at its limit, and the hub ends one its runner stopped reporting a few minutes after it. A timeout is part of the compiled job, so a job without one compiles as before.

Priority. A pipeline may lower its jobs' place in the hub's queue, never raise it (oxygenhub#244): priority "low" (or "normal") on a job, or at the top of the file for every job that does not say. A job's level is then the lower of its own and its repository's (the repository's, organisation's or site's, set on the hub), so nightly or bulk jobs can stand back for everyday ones; high and urgent are refused here, being the hub's to give, and a run a person expedites stays urgent whatever its jobs say. A priority is part of the compiled job; a job without one compiles as before.

Runner labels. A sandbox or a job may say runs-on "LABEL" ...: lower-case letters, digits, -, _ and ., at most 64 bytes each, given on one or more lines. A job needs a runner that has every label its sandbox and it name. A runner reports its labels and architecture when it registers and on each claim (flow runner register --label). The hub hands a job only to a runner with all of them, within the runner's scope (OxygenHub spec 13.2), and a job no runner can take waits, saying so on the queue. Labels are part of the compiled job, so a job that asks for none compiles as before.

3.2 Canonical Compilation

Upon commit, .flow/pipeline.kdl is validated and compiled into a deterministic CBOR job-graph blob. Runners evaluate the canonical CBOR to prevent syntax or parser ambiguities across varying client platforms.

4. Sandbox Execution Engine

OxygenFlow implements three distinct isolation tiers depending on the workload:

4.1 Level 0: WASM Component Sandbox

• Intended for static linters, formatters, doc generators, policy checks, and test suites built for wasm32-wasip1; the tier phones and browsers run (docs/handover.md).
• Built (oxygenflow#75): wasmtime 49 runs WASI preview 1 modules, each in a fresh instance, compiled by Cranelift (tens of milliseconds a module). Preview 2 components later.
• runtime "wasm"; no rootfs and no network (network absent or "isolated"); memory-limit holds the module's linear memory.
• exec names the module, a path among the job's files (its scope or an earlier job's artifact), optionally under /src/, or a glob with * in its file name: each match runs in turn, and the job's exit code is the first non-zero one after all have run. The rest of exec are the module's arguments. A module that traps exits 134.
• The job's files are the only host files it sees, preopened as /src (and as the current directory, for relative paths); each mount is preopened at its target. Secrets and identities are not offered yet.
• The receipt's environment names wasmtime and its version, the runner binary, the memory, and every module run with its BLAKE3.

4.2 Level 1: Intercepted Syscall Isolation (gVisor / Landlock + seccomp-bpf)

• Intended for standard compilation and hermetic test suites.
• Emulates or filters Linux system call interfaces in user space.
• Blocks access to raw network sockets, real-time hardware clocks, and device nodes.
• Memory and execution CPU quotas are strictly bounded via cgroups v2.

4.3 Level 2: MicroVM Isolation (Firecracker / Cloud-Hypervisor)

• Intended for kernel tests, multi-container integration, and untrusted code.
• Ephemeral Linux microVM booting in < 80 ms.
• The root filesystem is presented as a read-only virtio-block device mapped directly to FastCDC chunks stored in Oxygen's CAS.
• Guest VM memory is cleared upon termination.

4.3.1 Allow-listed egress (oxygenflow#55)

A sandbox reaches the network only by naming its hosts:

sandbox "deploy" {
    runtime "microvm"
    network "egress" { allow "*.googleapis.com" "registry.opentofu.org" }
}

The VM still has no network device. flow-init serves a proxy on the guest's 127.0.0.1:3128 (the job gets HTTPS_PROXY, HTTP_PROXY and their lower-case forms, unless it sets them) and relays each connection, unread, over Firecracker's vsock to the runner. The runner accepts only CONNECT host:443 for a host the list allows (a name exactly, or any name under a *. domain, never the domain itself; never an address), resolves it itself (the guest does not resolve), refuses any name that resolves to an address that is not public, and relays the bytes without opening the TLS. Every decision, allowed or refused, is a flow line in the job's log. Patterns are lower-case host names of two or more labels, at most 32 per sandbox, and part of the sandbox's hash.

Egress is off unless a runner turns it on (egress = true under [runner] in flow.toml); such a runner offers the egress label, and an egress sandbox needs it, so the hub hands its jobs only to runners that allow it, and a runner without it refuses one anyway. The syscall runtime has no egress yet.

Keyless receipts (oxygenhub#241). With keyless = true under [runner] (or in runner.toml), a runner signs each job's receipt with a fresh key instead of its own: just before signing, it asks the hub to certify the key (POST /api/v1/keyless/certify with the job's token and the key's proof of possession, scope seal), checks the certificate is for that key and signed by the key the hub publishes at /.well-known/oxyhub-key, signs, attaches the certificate and the hub's identity to the receipt (Oxygen spec §14.6), and drops the key. When the hub cannot certify, the runner's own key signs and the job's log says why. The runner's key then only authenticates it to the hub.

Metrics (oxygenflow#35). With metrics = "127.0.0.1:9464" under [runner] (off unless set), a runner serves Prometheus text at /metrics on that address, from a thread of its own: its version and uptime, jobs claimed and declined, finished jobs by result (passed, failed, error, cancelled), the seconds they ran together, and the jobs running now, each labelled with the runner's name. With metrics_token, a scrape must send it as a bearer token; without one, bind it to an address only the scraper reaches. The hub's own /metrics has the queue and run times of every runner's jobs.

5. Content-Addressable Storage (CAS) & Zero-Copy Mounting

OxygenFlow runners bypass traditional clone operations. Repositories are accessed directly via the local object store (.oxy/objects/) or streamed on demand using Oxygen's wire protocol.

5.1 FUSE Overlay Mounting (flow-casfs)

1. The runner receives a task targeting an Oxygen State ObjectRef (a sorted list of maximal heads).
2. The state's tree is read from a state index (Oxygen's flat, checksummed description of a history-only state: its changes, tree view and each file's deferred history with its graph key), built once from the nearest checkpoint plus the changes after it and kept beside the store. Listing a directory needs nothing else.
3. The tree is exposed to the sandbox read-only through flow-casfs (FUSE):
   • An opaque file's bytes are its stored blob, whose chunks (FastCDC, 32 KiB to 256 KiB, mean 64 KiB, Oxygen spec Section 13.2) are read as they are needed.
   • A text file's bytes are a rendering of its history, not a stored blob. Renderings are cached by graph key (equal keys mean equal renderings) as ordinary Blob and Chunk objects: a file is rendered on first read or stat, and every later state in which it has the same history finds it at once. A file added once and never edited is its own text blob and needs no rendering at all.
   • Missing objects are fetched lazily using the POST /objects protocol.
   • To prevent FUSE context-switch bottlenecks on many small reads, flow-casfs caches whole small files and mmaps chunk data, relying on the host page cache.
   • All guest mutations are diverted to an overlay on the job's ephemeral disk (or a tmpfs); base objects remain immutable.
4. The same mount backs web workspaces (OxygenHub spec 13.3): a workspace job keeps it for as long as its editor is open.
5. Targets, measured on the Linux tip: mount under 100 ms, a first read of a file in milliseconds, and a full read of the tree no slower than a checkout writing it.

5.2 Artifact Capture

1. At task completion, flow-casfs scans the paths declared in the output block.
2. Modified outputs are chunked with FastCDC, hashed via BLAKE3-256, and written as canonical Blob and Chunk objects.
3. Output ObjectRefs are stored directly in the Oxygen repository.
4. Outputs are read when the job passes. An artifact with when "always" (default when "passed") is also read when the job fails or is stopped at its time limit, so a test report survives the failure it describes. From a job that did not pass it is kept and shown, but its receipt never names it, and one that cannot be read is only noted in the log.

output {
    artifact "junit" {
        path "/src/target/junit.xml"
        when "always"
    }
}

5. A report output (OxygenHub docs/reports.md) names a directory holding report.toml at its top. The runner reads its regular files (links and anything else are left out, never followed; file names hold letters, digits, ., _, - and +; at most 2000 files and 512 MiB), uploads each to POST /api/v1/flow/jobs/{job}/reports/{name}/files with its path in X-Flow-Report-Path, then uploads a manifest, {"version":1,"files":[{"path","blob","size"}]}, as the output under the report's name, checking every blob the hub answers. The receipt names the manifest. Reports and artifacts share names; a report's kind is part of the job's hash ([name, path, "report"]).

output {
    report "frame-timing" {
        path "/src/out/report"
        when "always"
    }
}

5.3.1 Artifacts between jobs (oxygenflow#58)

A job takes an artifact of a job it depends on:

job "apply" {
    depends-on "plan"
    input {
        artifact "tfplan" from "plan"
        path "/src/tfplan"
    }
}

The producer must be named in depends-on and declare that name as an artifact (not a report); paths are clean absolute paths under /src, at most 8 inputs, and they are part of the job's hash ([artifact, from, path], field 10). Since the producer has passed when the job is handed out, the hub puts each input in the claim with the blob the producer's receipt names (a passing job's receipt names exactly the artifacts it uploaded). Before the job starts, the runner fetches each from GET /api/v1/flow/jobs/{job}/inputs/{index}, checks its Oxygen blob reference against that blob, refuses the job if they differ, and writes it at its path in the workspace (not part of the scope hash: it belongs to the run, not the repository). The job's receipt lists the blobs as its inputs, so an apply job's receipt proves which plan it applied.

5.3.2 Approval gates (oxygenflow#57)

A job can wait for a person:

job "apply" {
    depends-on "plan"
    approval {
        by "maintain"        // the least role: "write", "maintain" (default) or "admin"
        others-only #true    // not the person whose push started the run
        expires "24h"        // from when the jobs it depends on are done; up to 720h
        sealed #true         // signed with the approver's key; default: on with an environment
    }
}

The hub hands such a job to no runner until someone with that role on the repository approves it, on the run page or through the API; others-only leaves out whoever pushed. Approving or rejecting is recorded on the job (who, when) and in the audit log. A rejected or expired job ends as cancelled and the jobs after it are skipped; a newer push to the bookmark cancels the run as before, so a stale plan is never applied. The approval is part of the job's hash (field 11).

Sealed approvals (oxygenhub#235). With sealed (the default for a job that names an environment, §5.3.3), pressing Approve is not enough: the approver signs an Oxygen approval attestation with their own key (oxy seal --flow URL --job NAME; Oxygen spec §14.4) covering the run, the job's hash, the state, the input blobs (the reviewed plan) and the environment. The hub starts the job only once it verifies, the key is linked to an account with the role, and it is not a delegated key; it keeps the approval in the repository, and the job's identity tokens name the approver and the approval's hash. Rejecting needs no seal.

Agents (oxygenhub#229, #258). A gate is there so a person decides. A decision made through an agent's token (an agent-marked token, or a keyless agent's grant) is refused unless the repository lets agents decide its gates (its Settings › OxygenFlow, for maintainers; off by default, audited when changed). Where allowed, it says so on the job, the run and the audit log ("approved by carol via claude-code"). A sealed approval is a person's own key, whatever carried it, and is not affected.

5.3.3 Environments (oxygenflow#59)

environment "prod" on a job says where it deploys (1 to 32 of a-z, 0-9, -; part of the job's hash, field 12). The hub keeps what each environment of a repository runs, from the jobs that passed for it (state, run, when, who approved), on the repository's Deployments page. The job's identity tokens carry an environment claim only when the job was approved (§5.3.2): a job with no approval block, or one not approved, gets tokens without it. So a cloud's trust condition can require environment == "prod", and nothing unapproved deploys there, even from the deploy bookmark. The approval block is the environment's protection rule for now.

5.4 The Build Cache (oxygenflow#28)

An ephemeral://disk mount may name a cache, and the files that key it:

mount {
    source "ephemeral://disk"
    target "/src/target"
    size "24GiB"
    cache "target"
    key "Cargo.lock" "rust-toolchain.toml"
}

Such a disk is not always empty at the start. Each runner keeps a pool of the disks of passing jobs, by repository, job, cache name and bookmark, named by the key: the BLAKE3 of the sandbox image, the job's name, the cache name, and each key file's content in the job's state. A job's disk starts as, in order: its bookmark's disk with the same key (moved), its bookmark's newest disk (moved), another bookmark's with the same key and then another's newest (copied, so a proposal never takes the default bookmark's), and otherwise an empty disk. A microVM's disk must be the size the mount asks for.

• Only passing jobs keep a disk. When the job and its outputs pass, its cached disks go back into the pool under their key; otherwise they are discarded with the scratch directory, so nothing a failing job wrote reaches a later one.
• The receipt says so. The job's environment gains cache TARGET with the key and what the disk started from, so the receipt's environment hash differs from a cold build's.
• Bounded. The pool keeps under cache_gb in the runner's config (default 64 GiB) by dropping the disks used longest ago.
• Pruned before it is kept (oxygenflow#72). A kept disk would otherwise carry every build's files (each crate under every hash a lockfile or feature set gave it) until the job's disk is full. When a passing job's cache holds more than cache_prune_percent of its mount's size (runner config, default 60), the files the job did not use are removed, used longest ago first, until it is under that cap; one still over it is not kept, so the next run starts empty. A file was used if it was read, written or changed since the job started, or it shares a unit hash with one that was (a 16-digit hex word after a - in its path, as cargo names a unit's fingerprint, rlibs, binaries and build-script output: a crate cargo found fresh is only looked at, but its fingerprint is read), or is another link of one, or lies under one of the job's outputs. A microVM's guest (flow-init) prunes its disks before it powers off, having mounted them strictatime so every read is seen, and tells the runner what it kept (\x1eC frames); without word from it the runner only measures the image (dumpe2fs) and keeps it if it is under the cap. The syscall tier's runner prunes a directory itself, having set each file's atime to the epoch when the job took it, so even a relatime file system records the job's first read of each; on a noatime one nothing is pruned and only the size decides. The job's log says what happened: cache target: kept 12.3 GiB of 23.1 GiB (pruned 10.8 GiB in 4113 files this run did not use; the cap is 60% of its 24.0 GiB disk), or not kept: and why.
• Unchanged sources keep their times (oxygenflow#52). Beside each kept disk the pool keeps its sources: each workspace file's content hash and the mtime it had, in whole seconds (what the workspace image keeps). A job that takes the disk gives each file whose content is unchanged that mtime back before the workspace is imaged; a changed or new file keeps the time of its checkout, newer than anything on the disk. A build tool that judges freshness by time, such as cargo, then rebuilds exactly the crates whose sources changed, where a fresh checkout would make every path crate look changed. Directories are kept the same way, by their listing (the names in them): an unchanged one gets its old time back, one that gained or lost an entry keeps its new time, so a build script's rerun-if-changed on a directory sees exactly what changed. The log says how many files and directories kept their times. A build script that reads a per-state value (the Oxygen crates' FLOW_HEADS) reruns on every state; those crates read it only when OXY_BUILD is unset, so a pipeline that sets OXY_BUILD keeps them fresh.
• Jobs go where their cache is (oxygenflow#51). Each claim lists the caches the runner holds a disk of (caches, each flow_core::api::cache_slot(repo, job)), and the hub ranks a runner holding a job's cache above one holding its image (§11.4): a job another free runner holds the cache for is held for it for up to 15 seconds, and placed says "holds its build cache". When several hold it, the freshest copy wins (oxygenflow#54): claims also say when each slot last kept a disk (cache_times, Unix ms), and a copy kept at least a minute later ranks above an older one, before the image and speed are weighed; placed says how old the copy is.
• The syscall tier keeps directories (oxygenflow#70). Its ephemeral://disk mount is a host directory bound at the target, so its pool holds directories (<key>.tree) beside the microVM's images (<key>.ext4), taken and kept the same way: moved out of the pool for a job of the same bookmark (so two jobs of one slot running at once never take the same one), copied with their modes and times from another bookmark's, kept only when the job and its outputs pass and otherwise thrown away, and counted by the blocks of everything in them toward cache_gb. A directory has no size, so size does not choose between them. Each tier takes only its own form: a syscall job never starts from a microVM's image, nor a microVM from a directory. The job runs as the runner's user, so what it wrote is the runner's; before a directory goes into the pool nothing in it is left writable by group or others, or set-id, and the pool is readable by the runner alone. Claims report the slots of both forms in caches and cache_times, so jobs go where their cache is whichever tier they run in.
• A cache is a speed-up, never a source of truth: jobs must build correctly from an empty disk, which a key change or an eviction gives them at any time.

6. Commutative Execution & Change Caching

In Oxygen, changes that do not share scopes (referenced atoms, placements, or nodes) commute. OxygenFlow exploits this invariant to provide granular caching.

6.1 Execution Receipt Object

Every successful or failed job run emits an immutable Receipt:

Receipt {
  version        [0]: 1
  job_name       [1]: text
  pipeline_hash  [2]: ObjectRef
  scope_tree_hash [3]: ObjectRef     // BLAKE3 tree hash of the materialized scope
  inputs         [4]: [ObjectRef]   // Artifacts and input files consumed
  environment    [5]: ObjectRef     // Hash of rootfs, compiler flags, and argv
  outputs        [6]: [ObjectRef]   // Produced artifact blobs
  exit_code      [7]: u32
  duration_ms    [8]: u64
}

A Receipt is hashed as an Oxygen object with type name "receipt": ObjectRef = BLAKE3-256("oxy1 receipt " || decimal(len) || 0x00 || body)

6.2 Cache Re-use Invariant

Let Scope(J) be the file paths and tree nodes declared by job J. Because changes that do not share scopes commute, we do not need to track lists of ChangeHashes. Instead, we hash the materialized tree of Scope(J).

If an existing Receipt R exists such that: R.job_name == J.name && R.pipeline_hash == Hash(J) && R.scope_tree_hash == TreeHash(Scope(J)) && R.environment == CurrentEnvironmentHash

Then the job execution is bypassed. R.outputs are mapped into downstream dependents immediately as a cached-commuted hit.

7. Native Provenance Specification

OxygenFlow formalizes build provenance as a native, tamper-evident Oxygen Attest object (kind: "attest"). Provenance is generated exclusively by the runner hypervisor, not the guest payload.

7.1 Provenance Object Schema

Attest {
  version [0]: 1
  kind    [1]: "attest"
  repo    [2]: bytes(16)               // Oxygen Repository ID
  hlc     [3]: HLC                     // Hybrid Logical Clock
  body    [4]: {
    "type": "flow/provenance/v1",
    
    // Subject artifacts produced by the build
    "subjects": [
      {
        "name": text,
        "ref": ObjectRef,              // BLAKE3 hash of output blob
        "media_type": text
      }
    ],

    // Builder platform and execution identity
    "builder": {
      "runner_id": KeyId,              // ObjectRef of runner's Oxygen Identity
      "isolation_level": text,         // "wasm" | "gvisor" | "microvm"
      "hypervisor_image": ObjectRef    // Rootfs blob digest
    },

    // Exact input materials
    "materials": {
      "state_heads": [ChangeHash],     // Oxygen heads of the state built
      "scope_tree_hash": ObjectRef,        // Captures exact state without O(N) array bloat
      "inputs": [
        {
          "name": text,
          "ref": ObjectRef             // Content hash of locked dependencies
        }
      ]
    },

    // Structured execution arguments
    "invocation": {
      "job": text,
      "pipeline_spec": ObjectRef,      // ObjectRef of .flow/pipeline.kdl
      "argv": [text],                  // Unmodified execution vector
      "env": map<text, text>           // Explicitly whitelisted environment
    },

    // Environment observations
    "metadata": {
      "started_on": u64,               // Wallclock milliseconds
      "finished_on": u64,
      "hermetic": bool,                // Verified zero outbound network access
      "reproducible": bool
    }
  }
}

7.2 Composite Signatures

All provenance attestations are signed using Oxygen's composite signature construction (Ed25519 and ML-DSA-65):

Signature {
  key [0]: KeyId
  alg [1]: 1                           // 1 = Ed25519 + ML-DSA-65
  ed  [2]: bytes(64)                   // Classical signature
  ml  [3]: bytes(3309)                 // Post-quantum signature
}

Both signatures are computed over: "oxy1 sig attest\0" || KeyId || ObjectRef

8. Policy Integration & Bookmark Gates

Oxygen repositories enforce branch and bookmark protections via signed Policy objects (Oxygen Spec Section 14.5). OxygenFlow integrates provenance directly into policy evaluation during push validation.

8.1 Policy Gate Definition

A Policy rule governing a bookmark MAY demand valid OxygenFlow attestations. To mitigate a compromised authorized worker, high-security repositories can require Consensus Builds by setting the threshold > 1. This requires multiple independent runner identities to arrive at the exact same output hashes and both sign it.

PolicyRule {
  pattern: "bookmarks/production"
  keys: [ KeyId ]
  threshold: 1
  require: "signed"
  allow_conflicts: false
  require_attestations: [
    {
      type: "flow/provenance/v1",
      job: "release-provenance",
      require_hermetic: true,
      signer_roles: [ "official-worker" ]
    }
  ]
}

### 8.2 Push Verification Sequence (VCS Engine Enforcement)
When a push request arrives at an Oxygen server attempting to update a
protected bookmark:
1. Object Integrity: Validate that all changes, chunks, and blobs exist and match
   their BLAKE3 hashes.
2. Attestation Resolution: Verify that an Attest object of type
   "flow/provenance/v1" is present, referencing the exact heads of the new
   State.
3. Identity Verification: Validate that the attestation's composite signature
   resolves to a KeyId listed as an authorized delegate of the repository
   policy roots.
4. Invariant Checking: Confirm that the provenance statement records
   "hermetic: true" and exit code 0.
5. Atomic Update: If validation fails at any point, the entire push is refused
   with ok = false.

## 9. Zero-Environment Secret Brokering

OxygenFlow forbids the injection of raw secret strings into process environment
tables to eliminate accidental leaking via child process dumping, procfs
reads, and verbose compiler logging.

### 9.1 Ephemeral Secret Broker
1. If a job requires a credential (e.g., signing key, cloud deployment token),
   it declares a secret mount:
     secret "deploy-token" {
         vault-ref "infra/tokens/prod"
         mount-path "/run/secrets/token"
     }
2. The runner negotiates a short-lived token lease from the configured vault.
3. The secret is mounted onto an ephemeral, in-memory ramfs location inside
   the sandbox with access mode 0400, owned exclusively by the worker process.
4. Upon process termination, the memory pages are zeroized and the token is
   marked for immediate revocation at the vault.

## 10. Conformance and Verification Requirements

A conforming OxygenFlow implementation MUST pass the following tests:

1. Injection Immunity: Attempting to supply unescaped shell metacharacters
   (; && | ` $() \n) in any job field MUST pass those characters as literal
   bytes to argv; subshells MUST NOT be spawned.
2. FastCDC Determinism: Chunks produced by artifact extraction MUST match
   the gear table cuts and BLAKE3 hashes generated by Oxygen's core chunker.
3. Commutation Hit Invariance: For any two commuting changes A and B, running
   a pipeline on State(A, B) MUST reuse cached receipts generated on State(A)
   for any job whose scope does not intersect the touched paths of B.
4. Non-Downgrade Attestation: An attestation missing the ML-DSA-65 component
   MUST be rejected by policy verification.
5. Hermetic Network Isolation: Jobs configured with network "isolated" MUST
   have outgoing TCP SYN and UDP packets dropped at the hypervisor boundary.
## 11. Runner Capabilities and Placement (proposed 2026-10-04; §11.2 and `arch` of §11.3 done the same day; tiers of §11.1, §11.5 and the waiting reasons of §11.4 done 2026-10-05, oxygenflow#36)

A runner MUST never take a job it cannot run or cannot finish well: the
wrong architecture, no hardware virtualisation for the MicroVM tier, too
little memory, an image it does not hold, or a machine too slow for the
job's time limit. Today a runner reports its architecture, OS, labels,
job slots, memory and vCPUs when it registers and on each claim, and a
claim carries the tiers it can run, the memory and vCPUs free now, and
the images it holds; the hub matches tier, room, labels, scope and image.
What is missing: architecture is reported but never matched (an image's
architecture is implicit in its bytes), a pipeline cannot say which
architecture it needs or offer a rootfs per architecture, speed and load
play no part, and a runner cannot decline a job it was handed.

### 11.1 Capability Profile

At registration and on every claim a runner sends its profile:

| Field | Meaning |
|---|---|
| `arch`, `os`, `kernel` | `x86_64`, `aarch64`, `riscv64`, …; the kernel release |
| `tiers` | The sandbox tiers it can honour: `microvm` only with `/dev/kvm` usable (a board without KVM, such as an OrangePi RV2, offers `syscall`, §4.2, and `wasm`) |
| `memory_mib`, `cpus`, `max_jobs` | What its jobs share |
| `images` | The rootfs images it holds, each with the architecture its manifest records (§11.2) |
| `labels` | As §3.1 |
| `speed` | A score and a class from 1 (slow) to 5 (fast), measured by the runner at start, again every 15 minutes while it runs no job, and by `flow runner bench`: a fixed one-thread workload (BLAKE3 over 256 MiB, and a sort of 4 Mi words, where a C compiler cannot be assumed) whose time on the reference machine (Earth, about 225 ms) scores 100; class 3 is 70–139, each class a factor of two (done 2026-10-11, `facts` on the claim) |
| `load` | The 1-minute load average (×100), and the free disk where jobs' disks are made, sent with every claim (`facts`); memory free and jobs running as before |

### 11.2 Image Manifests Carry the Architecture

`flow rootfs import` reads the ELF header of the image's `/bin/sh` and
records its architecture in the CAS manifest beside the hash. A runner
refuses to import an image for another architecture (there is no
emulation tier).

An image whose manifest records no architecture (imported before this
section, or fetched from a hub that did not know it) is read, not
guessed: `jet::rootfs` follows `/bin/sh` (else busybox, env or bash)
through the image's ext4, symlinks included, to its ELF header, with no
mount and no privileges. The runner records the answer in the manifest,
so each image is read once, reports it with its claims, and declines a
job whose image is for another architecture instead of running it and
failing (§11.5). The hub reads the images it keeps the same way
(OxygenHub, oxygenhub#173), and an image nobody can read is offered only
to the architecture its jobs have passed on before.

### 11.3 Job Requirements

Hard requirements, all of which a runner MUST satisfy to be offered the
job: the sandbox's tier, its rootfs (hence the image's architecture),
`memory-limit`, `cpus`, every `runs-on` label of the job and its sandbox,
and the runner's scope (OxygenHub spec §13.2). Two additions to the
language:

- `arch "riscv64"` on a sandbox pins the architecture; without it the
  architecture is the rootfs image's.
- A rootfs per architecture: `rootfs { x86_64 "cas://blob/…"; riscv64 "cas://blob/…" }`.
  The job runs on whichever architecture takes it; the receipt and the
  provenance record the architecture and image used, and a check's
  description names them.

Soft requirements guide placement but never block it: a job's `timeout`
against the runner's speed class. The hub keeps, per job hash and speed
class, the median duration of past runs; a runner whose class predicts a
duration over the limit is offered the job only when no better runner
is free.

### 11.4 Placement

The hub filters by the hard requirements, then ranks the runners that
remain: idle before busy (`load`), a higher speed class first, a runner
already holding the image before one that must fetch it (§5), fewer jobs
waiting on it first. A job no runner can satisfy waits and the queue
says what it waits for: tier, architecture, image, memory, labels or
scope, so the gap is visible rather than a silent stall.

### 11.5 Declining

A runner double-checks a claim before it starts: the architecture, that
`/dev/kvm` opens for the MicroVM tier, that the image's blob is present
and verifies, that the memory is free. If any check fails it answers the
hub with `declined` and a reason instead of running and failing; the hub
re-queues the job, never offers it to that runner again until its
profile changes, records the decline on the runner's page, and after
three declines by different runners marks the job `unplaceable` with the
reasons. A decline is never a failed check on the state.

As built: the runner sends `POST /api/v1/flow/jobs/{id}/decline` with
`{"reason": …}` before it fetches anything. The profile is a hash of the
tiers, images, architecture, labels and capacity a runner claims with.
A decline also lapses after ten minutes, so a passing reason such as
memory in use does not bar a runner for good. An unplaceable job ends
with status `error` and an error beginning `unplaceable:`. The hub
works out why each ready job waits every 15 seconds, and the run page and
the API (`waiting`) show it.

### 11.5.1 Restarts (oxygenflow#49, #50)

Neither a hub restart nor a runner restart costs a job anything.

- **The hub away.** A job's uploads (artifacts, report files) and declines
  are tried again through refused connections and server errors (5xx,
  as a proxy answers while the hub restarts) for three minutes, waiting
  1, 2, 4, 8 and then 15 seconds; a refusal (4xx) is final. Log lines
  the hub could not take are kept and sent with the next batch, 200 lines
  at a time, up to 50,000 (beyond that the oldest go, and the log says how
  many). A job's result is kept on disk and offered for two hours, as
  before. Claims are not retried through a lost connection, so a job is
  never claimed twice; the runner simply asks again.
- **The runner restarted.** The first SIGTERM or SIGINT drains the runner:
  it claims nothing more and stops once every job it holds has ended and
  been reported. A second signal stops it at once. Its systemd unit says
  `KillMode=mixed`, so only the runner is signalled and its microVMs run
  on, and `TimeoutStopSec` longer than the longest job, after which
  systemd kills what is left. A new binary is installed by replacing it
  and restarting the unit (`systemctl restart --no-block`): the old runner
  finishes its job, and `Restart=always` starts the new one.
  `flow runner drain` asks the same by a `drain.request` file in the
  runner's home, which its loop reads: the way on Windows, which has no
  SIGTERM (oxygenflow#83).

### 11.5.2 Handing back (oxygenflow#76)

A runner that leaves while it holds a job, with no time to finish it,
hands the job back: `POST /api/v1/flow/jobs/{id}/release` with
`{"reason": …}` (a phone off its allowed network, docs/handover.md §6).
The hub queues the job again **ahead of new work**, whatever its
priority, adds a line to its log saying who handed it back and why, and
lets the next attempt's log follow on. It is not a decline: the runner is
not held from the job and it does not count towards unplaceable.

**Leases.** Holding a job is a lease the runner keeps by being heard from:
claims, a heartbeat every 30 seconds while it claims nothing, its jobs'
logs. When the hub has not heard from a runner for 3 minutes, its jobs
are handed back the same way, so a runner that vanishes (out of every
network, crashed) costs a restart of its job, not the job. A job handed
back more than 5 times ends as an error saying so.

**Notice.** A departure may come with notice (a phone taken off charge
has a minute; a spot reclaim 30 seconds to 2 minutes). A claim carries
`expected_ms`, how long the job is expected to take, when the hub has
learned it: the median of the job's last five passing runs in that
repository (reused passes left out). A leaving runner finishes and
reports a job whose expected rest fits within the notice, and hands back
the others at once, and any job still running when the notice is up. A
job with no expected time is handed back at once.

**Checkpoints (#77).** A job may name directories that hold its progress:
`checkpoint "/src/target"` (one or more, under /src, not /src itself).
When its runner hands it back, the runner packs them (files with their
modes and modification times, which cargo's fingerprints read; at most
512 MiB) and keeps them with `POST /api/v1/flow/jobs/{id}/checkpoint`
before the release; the claim of the next attempt names the blob
(`checkpoint`), and that runner fetches it (`GET` on the same path),
checks it, and restores it over the job's cache before running the same
exec, so a tool that skips finished work carries on. The receipt lists
the checkpoint among the job's inputs. As built: the WASM and syscall
tiers save and restore checkpoints, under a disk mount or in the tree;
the MicroVM tier, whose files are in an image, starts again for now.

As built (flow 0.3.2, OxygenHub 0.9.4): the runner as a library has
`runner::leave(reason)` (no notice) and `runner::leave_within(reason,
notice)` beside `runner::stop()`: each claims nothing more and stops the
jobs it does not finish (their logs sent first), releasing them; if the
release cannot be sent, the lease does it. A host's SIGTERM still drains
by finishing what it holds. Checkpoints (#77), so the next attempt
resumes rather than restarts, come next.

### 11.5.3 Updating a runner (oxygenflow#85)

`flow runner update --from FILE` installs a new `flow` build in place of
the one it runs as (`--exe` names another), the runner's side of
self-updating runners (docs/jetstream.md §8.2):

1. **Refused** when the home's `flow.toml` says `[runner.update] allow =
   false`, when an earlier update is not yet proven, or when the build is
   older than the installed one (unless `--allow-older`).
2. **Checked:** a 64-bit little-endian ELF for this machine; it names
   itself (`--version`, `flow X.Y.Z …`) and finds a tier it can run here
   (`tiers` against the runner's home), each within 30 seconds.
3. **Staged and swapped:** copied beside the installed binary as
   `flow.new` and checked there; the installed one is kept as
   `flow.prev` (a hard link, else a copy), then `flow.new` replaces it in
   one rename, so `flow` is never missing. `--dry-run` stops before the
   swap and changes nothing.
4. **The running runner leaves:** a runner writes its pid to
   `runner.pid` in its home, and the update sends it SIGUSR1 (after
   checking that pid is still a `flow … runner`). It then leaves with
   `--notice` (default 600 s, §11.5.2): jobs expected to finish in time
   finish, the rest are handed back with their checkpoints, and it exits
   0, so its unit's `Restart=always` starts the new build. A runner from
   before #85 is not told; restarting its unit drains it as before.
5. **Proven or rolled back:** the update waits in `update.json` until the
   new build's first claim gets an answer from the hub. Each start counts;
   on the fourth start without that answer, or after ten minutes running
   without it, `flow.prev` is put back (and on a start, run in its place).
   `updates.log` in the home records every install, proof and rollback.
   The runner tells the hub at its next answer (`POST /api/v1/flow/runner/update`):
   `runs flow X (was Y); proven`, or `rolled back from flow X to Y: why`,
   on which the hub holds X back from every other runner until an
   administrator lifts the hold (oxygenhub#326).
6. **On Windows** (oxygenflow#83) the same steps run with Windows' means:
   the build is checked as a PE executable for this machine; `flow.exe`
   is renamed to `flow.prev.exe` and `flow.new.exe` into its place (a
   running program can be renamed but not replaced, and cannot rename
   its own file, so a copy of it, `flow __rename`, does that); the runner
   is told by an `update.request` file in its home, which it reads in its
   loop, there being no SIGUSR1; and having left, it exits 75, so a Task
   Scheduler task set to restart on failure starts the new build. A
   rollback runs the restored build in its place and ends with it.

### 11.6 Conformance

1. Architecture Safety: a job whose image is `x86_64` MUST never be
   offered to a `riscv64` runner, with or without the image copied there.
2. Tier Safety: a runner without `/dev/kvm` MUST NOT be offered a
   MicroVM job, and MUST decline one if handed it.
3. Room: a job whose `memory-limit` exceeds the runner's free memory
   MUST wait, not be killed.
4. Visibility: a job nobody can take MUST say why on the queue within
   one scheduling pass.
5. Preference: with two idle runners that both satisfy a job, the hub
   MUST offer it to the higher speed class, and to the one holding the
   image when classes tie.