# OxygenFlow (flow) — Deterministic Execution & Provenance Pipeline Specification

| | |

|---|---|

| **Format version** | 1 |

| **Date** | 2026-09-28 |

| **Status** | Draft 0.1 |

| **Target VCS** | Oxygen (oxy) Draft 0.8i |

| **Audience** | Implementers, Security Architects, and Systems Engineers |

The key words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are to be

interpreted as described in RFC 2119.

## 1. Overview and Core Invariants

OxygenFlow (flow) is a content-addressed, hermetic execution engine, runner system,

and pipeline language designed natively for the Oxygen (oxy) version control

system.

Existing runner engines (GitHub Actions, GitLab CI, Forgejo Runner) suffer from

structural flaws:

1. Workspace Bloat: Full clones, index unpacks, and checkout scans dominate I/O.

2. String Injection: Shell interpolation within untyped YAML files produces

   expression injection and command execution exploits.

3. Decoupled Supply Chain: Attestations (SLSA, in-toto, Cosign) exist outside

   the VCS, relying on external registries rather than native repository trust.

4. Coarse Caching: Opaque tarballs keyed by arbitrary string hashes fail to

   leverage commutative history or granular dependency graphs.

OxygenFlow solves these liabilities through direct integration with Oxygen:

- Zero-Copy Workspaces: Jobs mount workspaces directly from Oxygen's FastCDC-

  chunked, BLAKE3-addressed Content-Addressable Store (CAS) using Copy-on-Write

  (COW) overlays.

- Commutative History & Impact Trees: Pipelines evaluate over change closures

  rather than linear snapshot sequences. Commuting changes automatically reuse

  cached execution receipts without redundant re-execution.

- Micro-Engine Sandboxing: Jobs run inside isolated environments (WASM, gVisor,

  or MicroVMs) with strict, deny-by-default capabilities and no ambient

  network egress.

- Native Cryptographic Provenance: Provenance and execution receipts are

  generated outside the untrusted guest, encoded as deterministic CBOR, and

  signed using composite Ed25519 + ML-DSA-65 keys as native Oxygen Attest

  objects governed by in-repository Policy rules.

## 2. Threat Model and Security Architecture

### 2.1 Adversaries

- Untrusted Proposal Author: Submits code to an open repository containing

  malicious build configurations intended to escape the runner, exfiltrate

  secrets, or corrupt shared worker caches.

- Malicious Dependency / Upstream: Third-party dependencies invoked during

  build time attempting out-of-band network calls or host file modification.

- Compromised Runner / Malicious Worker: An execution node attempting to

  forge successful build receipts or inject backdoors into published artifacts.

### 2.2 Security Invariants

- I1 Deny-by-Default Execution: Sandboxes possess no network interfaces

  (excluding loopback when requested) and no host filesystem access unless

  explicitly granted in policy-signed configuration.

- I2 Injection-Free Invocation: Configuration parsing directly yields an argv

  string vector executed via execve(). Shell interpretation (/bin/sh -c) does

  not exist.

- I3 Host-Enforced Provenance: The running payload cannot author, inspect, or

  alter its own provenance statement. Provenance is emitted solely by the

  isolation hypervisor boundary.

- I4 Cryptographic Identity & Authorization: Runners execute using an Oxygen

  Identity whose KeyId is enrolled as a delegate in the repository's Policy.

  Unenrolled runners cannot publish recognized attestations or advance gates.

## 3. The OxygenFlow Configuration Language (.flow)

Pipelines are declared in KDL syntax within .flow/pipeline.kdl at the

repository root. KDL provides strict typing, structured nesting, and rejects

arbitrary text macro expansion.

Strings follow KDL 2.0, which has no `\033`, `\e` or `\x` escapes: ESC,

for ANSI colours in a job's output, is `\u{1b}` (`"\u{1b}[32mok\u{1b}[0m"`),

and `flow check` says so when it meets one of the others. Inside the

guest, `/dev/fd` and `/dev/stdin|stdout|stderr` link to `/proc/self/fd`

as on any Linux system, so bash process substitution works.

### 3.1 Pipeline Specification Schema

```kdl

schema-version 1

// Define the hermetic execution boundary

sandbox "rust-builder" {

    runtime "microvm"       // Options: "wasm" | "gvisor" | "microvm"

    rootfs "cas://blob/9b5c2a1e8f3d4e7a6b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a"

    memory-limit "4GiB"

    cpu-limit 4

    network "isolated"      // Options: "isolated" | "loopback" | "egress"

}

// Job evaluated based on Oxygen change closure

job "compile-core" {

    sandbox "rust-builder"

    // Commutation boundary: runs only if change closure touches these paths

    scope "crates/oxy-core/**" "crates/oxy-server/**" "Cargo.*"

    // Zero-copy workspace mount backed by Oxygen CAS

    mount {

        source "oxy://workspace"

        target "/src"

        include "crates/oxy-core" "crates/oxy-server" "Cargo.toml" "Cargo.lock"

        readonly #true

    }

    // Ephemeral scratch space

    mount {

        source "ephemeral://tmpfs"

        target "/src/target"

        size "8GiB"

    }

    env {

        RUSTFLAGS "-Dwarnings"

        CARGO_INCREMENTAL "0"

    }

    // Direct argv execution vector (no subshell)

    exec "cargo" "build" "--release" "--package" "oxy-core"

    output {

        artifact "oxy-core-lib" {

            path "/src/target/release/liboxy_core.rlib"

            format "blob"

        }

    }

}

job "unit-tests" {

    sandbox "rust-builder"

    depends-on "compile-core"

    mount {

        source "oxy://workspace"

        target "/src"

        readonly #true

    }

    mount {

        source "artifact://compile-core/oxy-core-lib"

        target "/src/target/release/liboxy_core.rlib"

    }

    exec "cargo" "test" "--release" "--package" "oxy-core"

}

publish "release-provenance" {

    depends-on "unit-tests"

    attest {

        kind "provenance"

        bind-ref "bookmarks/main"

        include-artifacts "compile-core/oxy-core-lib"

    }

}

**What starts a job.** An `on` block, at the top of the pipeline or in a job (the job's wins), names the events that run it: `push` with bookmark globs (none: any bookmark), `tag` with tag globs (none: any tag), `manual` ("Run pipeline", `oh run start`) and `schedule` with five-field cron lines on the hub's clock (UTC), run on the default bookmark. Without `on`, a job runs on every push and by hand, as before. A run has only the jobs its event starts (and what they depend on); an event that starts no job starts no run. A rerun repeats its run's event.

```kdl

on {

    push "main" "feature/*"

    manual

}

job "dist" {

    on {

        tag "v*"

        manual

    }

    …

}

```

**Time limits.** A job may say how long it may run with `timeout "90m"` (hours, minutes and seconds, as `"2h"`, `"1h30m"` or `"45s"`, from a minute to a day); a sandbox's `timeout` is the default for its jobs. Without one, the hub's default applies, and the hub caps every job at the site's maximum (both set by an administrator on Admin → Runners). The runner stops a job at its limit, and the hub ends one its runner stopped reporting a few minutes after it. A timeout is part of the compiled job, so a job without one compiles as before.

**Runner labels.** A sandbox or a job may say `runs-on "LABEL" ...`: lower-case letters, digits, `-`, `_` and `.`, at most 64 bytes each, given on one or more lines. A job needs a runner that has every label its sandbox and it name. A runner reports its labels and architecture when it registers and on each claim (`flow runner register --label`). The hub hands a job only to a runner with all of them, within the runner's scope (OxygenHub spec 13.2), and a job no runner can take waits, saying so on the queue. Labels are part of the compiled job, so a job that asks for none compiles as before.

### 3.2 Canonical Compilation

Upon commit, .flow/pipeline.kdl is validated and compiled into a deterministic

CBOR job-graph blob. Runners evaluate the canonical CBOR to prevent syntax or

parser ambiguities across varying client platforms.

## 4. Sandbox Execution Engine

OxygenFlow implements three distinct isolation tiers depending on the workload:

### 4.1 Level 0: WASM Component Sandbox

- Intended for static linters, formatters, doc generators, and policy checks.

- Runs on a standard WebAssembly System Interface (WASI) runtime.

- Startup latency: < 5 ms.

- Zero host I/O; virtualized file system presented in-memory.

### 4.2 Level 1: Intercepted Syscall Isolation (gVisor / Landlock + seccomp-bpf)

- Intended for standard compilation and hermetic test suites.

- Emulates or filters Linux system call interfaces in user space.

- Blocks access to raw network sockets, real-time hardware clocks, and device

  nodes.

- Memory and execution CPU quotas are strictly bounded via cgroups v2.

### 4.3 Level 2: MicroVM Isolation (Firecracker / Cloud-Hypervisor)

- Intended for kernel tests, multi-container integration, and untrusted code.

- Ephemeral Linux microVM booting in < 80 ms.

- The root filesystem is presented as a read-only virtio-block device mapped

  directly to FastCDC chunks stored in Oxygen's CAS.

- Guest VM memory is cleared upon termination.

## 5. Content-Addressable Storage (CAS) & Zero-Copy Mounting

OxygenFlow runners bypass traditional clone operations. Repositories are accessed

directly via the local object store (.oxy/objects/) or streamed on demand

using Oxygen's wire protocol.

### 5.1 FUSE Overlay Mounting (flow-casfs)

1. The runner receives a task targeting an Oxygen State ObjectRef (a sorted

   list of maximal heads).

2. The state's tree is read from a state index (Oxygen's flat, checksummed

   description of a history-only state: its changes, tree view and each

   file's deferred history with its graph key), built once from the

   nearest checkpoint plus the changes after it and kept beside the store.

   Listing a directory needs nothing else.

3. The tree is exposed to the sandbox read-only through flow-casfs (FUSE):

   - An opaque file's bytes are its stored blob, whose chunks (FastCDC,

     32 KiB to 256 KiB, mean 64 KiB, Oxygen spec Section 13.2) are read as

     they are needed.

   - A text file's bytes are a rendering of its history, not a stored blob.

     Renderings are cached by **graph key** (equal keys mean equal

     renderings) as ordinary Blob and Chunk objects: a file is rendered on

     first read or stat, and every later state in which it has the same

     history finds it at once. A file added once and never edited is its

     own text blob and needs no rendering at all.

   - Missing objects are fetched lazily using the POST /objects protocol.

   - To prevent FUSE context-switch bottlenecks on many small reads,

     flow-casfs caches whole small files and mmaps chunk data, relying on

     the host page cache.

   - All guest mutations are diverted to an overlay on the job's ephemeral

     disk (or a tmpfs); base objects remain immutable.

4. The same mount backs web workspaces (OxygenHub spec 13.3): a

   `workspace` job keeps it for as long as its editor is open.

5. Targets, measured on the Linux tip: mount under 100 ms, a first read of

   a file in milliseconds, and a full read of the tree no slower than a

   checkout writing it.

### 5.2 Artifact Capture

1. At task completion, flow-casfs scans the paths declared in the output block.

2. Modified outputs are chunked with FastCDC, hashed via BLAKE3-256, and

   written as canonical Blob and Chunk objects.

3. Output ObjectRefs are stored directly in the Oxygen repository.

4. Outputs are read when the job passes. An artifact with `when "always"`

   (default `when "passed"`) is also read when the job fails or is stopped

   at its time limit, so a test report survives the failure it describes.

   From a job that did not pass it is kept and shown, but its receipt never

   names it, and one that cannot be read is only noted in the log.

```kdl

output {

    artifact "junit" {

        path "/src/target/junit.xml"

        when "always"

    }

}

```

5. A `report` output (OxygenHub `docs/reports.md`) names a directory

   holding `report.toml` at its top. The runner reads its regular files

   (links and anything else are left out, never followed; file names hold

   letters, digits, `.`, `_`, `-` and `+`; at most 2000 files and 512 MiB),

   uploads each to `POST /api/v1/flow/jobs/{job}/reports/{name}/files` with

   its path in `X-Flow-Report-Path`, then uploads a manifest,

   `{"version":1,"files":[{"path","blob","size"}]}`, as the output under

   the report's name, checking every blob the hub answers. The receipt

   names the manifest. Reports and artifacts share names; a report's kind

   is part of the job's hash (`[name, path, "report"]`).

```kdl

output {

    report "frame-timing" {

        path "/src/out/report"

        when "always"

    }

}

```

## 6. Commutative Execution & Change Caching

In Oxygen, changes that do not share scopes (referenced atoms, placements, or

nodes) commute. OxygenFlow exploits this invariant to provide granular caching.

### 6.1 Execution Receipt Object

Every successful or failed job run emits an immutable Receipt:

```cbor

Receipt {

  version        [0]: 1

  job_name       [1]: text

  pipeline_hash  [2]: ObjectRef

  scope_tree_hash [3]: ObjectRef     // BLAKE3 tree hash of the materialized scope

  inputs         [4]: [ObjectRef]   // Artifacts and input files consumed

  environment    [5]: ObjectRef     // Hash of rootfs, compiler flags, and argv

  outputs        [6]: [ObjectRef]   // Produced artifact blobs

  exit_code      [7]: u32

  duration_ms    [8]: u64

}

```

A Receipt is hashed as an Oxygen object with type name "receipt":

ObjectRef = BLAKE3-256("oxy1 receipt " || decimal(len) || 0x00 || body)

### 6.2 Cache Re-use Invariant

Let Scope(J) be the file paths and tree nodes declared by job J.

Because changes that do not share scopes commute, we do not need to track lists of ChangeHashes. Instead, we hash the materialized tree of Scope(J).

If an existing Receipt R exists such that:

  R.job_name == J.name &&

  R.pipeline_hash == Hash(J) &&

  R.scope_tree_hash == TreeHash(Scope(J)) &&

  R.environment == CurrentEnvironmentHash

Then the job execution is bypassed. R.outputs are mapped into downstream

dependents immediately as a cached-commuted hit.

## 7. Native Provenance Specification

OxygenFlow formalizes build provenance as a native, tamper-evident Oxygen Attest

object (kind: "attest"). Provenance is generated exclusively by the runner

hypervisor, not the guest payload.

### 7.1 Provenance Object Schema

```json

Attest {

  version [0]: 1

  kind    [1]: "attest"

  repo    [2]: bytes(16)               // Oxygen Repository ID

  hlc     [3]: HLC                     // Hybrid Logical Clock

  body    [4]: {

    "type": "flow/provenance/v1",

    // Subject artifacts produced by the build

    "subjects": [

      {

        "name": text,

        "ref": ObjectRef,              // BLAKE3 hash of output blob

        "media_type": text

      }

    ],

    // Builder platform and execution identity

    "builder": {

      "runner_id": KeyId,              // ObjectRef of runner's Oxygen Identity

      "isolation_level": text,         // "wasm" | "gvisor" | "microvm"

      "hypervisor_image": ObjectRef    // Rootfs blob digest

    },

    // Exact input materials

    "materials": {

      "state_heads": [ChangeHash],     // Oxygen heads of the state built

      "scope_tree_hash": ObjectRef,        // Captures exact state without O(N) array bloat

      "inputs": [

        {

          "name": text,

          "ref": ObjectRef             // Content hash of locked dependencies

        }

      ]

    },

    // Structured execution arguments

    "invocation": {

      "job": text,

      "pipeline_spec": ObjectRef,      // ObjectRef of .flow/pipeline.kdl

      "argv": [text],                  // Unmodified execution vector

      "env": map<text, text>           // Explicitly whitelisted environment

    },

    // Environment observations

    "metadata": {

      "started_on": u64,               // Wallclock milliseconds

      "finished_on": u64,

      "hermetic": bool,                // Verified zero outbound network access

      "reproducible": bool

    }

  }

}

```

### 7.2 Composite Signatures

All provenance attestations are signed using Oxygen's composite signature

construction (Ed25519 and ML-DSA-65):

```cbor

Signature {

  key [0]: KeyId

  alg [1]: 1                           // 1 = Ed25519 + ML-DSA-65

  ed  [2]: bytes(64)                   // Classical signature

  ml  [3]: bytes(3309)                 // Post-quantum signature

}

```

Both signatures are computed over:

  "oxy1 sig attest\0" || KeyId || ObjectRef

## 8. Policy Integration & Bookmark Gates

Oxygen repositories enforce branch and bookmark protections via signed Policy

objects (Oxygen Spec Section 14.5). OxygenFlow integrates provenance directly into

policy evaluation during push validation.

### 8.1 Policy Gate Definition

A Policy rule governing a bookmark MAY demand valid OxygenFlow attestations. To mitigate a compromised authorized worker, high-security repositories can require **Consensus Builds** by setting the threshold > 1. This requires multiple independent runner identities to arrive at the exact same output hashes and both sign it.

```json

PolicyRule {

  pattern: "bookmarks/production"

  keys: [ KeyId ]

  threshold: 1

  require: "signed"

  allow_conflicts: false

  require_attestations: [

    {

      type: "flow/provenance/v1",

      job: "release-provenance",

      require_hermetic: true,

      signer_roles: [ "official-worker" ]

    }

  ]

}

### 8.2 Push Verification Sequence (VCS Engine Enforcement)

When a push request arrives at an Oxygen server attempting to update a

protected bookmark:

1. Object Integrity: Validate that all changes, chunks, and blobs exist and match

   their BLAKE3 hashes.

2. Attestation Resolution: Verify that an Attest object of type

   "flow/provenance/v1" is present, referencing the exact heads of the new

   State.

3. Identity Verification: Validate that the attestation's composite signature

   resolves to a KeyId listed as an authorized delegate of the repository

   policy roots.

4. Invariant Checking: Confirm that the provenance statement records

   "hermetic: true" and exit code 0.

5. Atomic Update: If validation fails at any point, the entire push is refused

   with ok = false.

## 9. Zero-Environment Secret Brokering

OxygenFlow forbids the injection of raw secret strings into process environment

tables to eliminate accidental leaking via child process dumping, procfs

reads, and verbose compiler logging.

### 9.1 Ephemeral Secret Broker

1. If a job requires a credential (e.g., signing key, cloud deployment token),

   it declares a secret mount:

     secret "deploy-token" {

         vault-ref "infra/tokens/prod"

         mount-path "/run/secrets/token"

     }

2. The runner negotiates a short-lived token lease from the configured vault.

3. The secret is mounted onto an ephemeral, in-memory ramfs location inside

   the sandbox with access mode 0400, owned exclusively by the worker process.

4. Upon process termination, the memory pages are zeroized and the token is

   marked for immediate revocation at the vault.

## 10. Conformance and Verification Requirements

A conforming OxygenFlow implementation MUST pass the following tests:

1. Injection Immunity: Attempting to supply unescaped shell metacharacters

   (; && | ` $() \n) in any job field MUST pass those characters as literal

   bytes to argv; subshells MUST NOT be spawned.

2. FastCDC Determinism: Chunks produced by artifact extraction MUST match

   the gear table cuts and BLAKE3 hashes generated by Oxygen's core chunker.

3. Commutation Hit Invariance: For any two commuting changes A and B, running

   a pipeline on State(A, B) MUST reuse cached receipts generated on State(A)

   for any job whose scope does not intersect the touched paths of B.

4. Non-Downgrade Attestation: An attestation missing the ML-DSA-65 component

   MUST be rejected by policy verification.

5. Hermetic Network Isolation: Jobs configured with network "isolated" MUST

   have outgoing TCP SYN and UDP packets dropped at the hypervisor boundary.

## 11. Runner Capabilities and Placement (proposed 2026-10-04; §11.2 and `arch` of §11.3 done the same day; tiers of §11.1, §11.5 and the waiting reasons of §11.4 done 2026-10-05, oxygenflow#36)

A runner MUST never take a job it cannot run or cannot finish well: the

wrong architecture, no hardware virtualisation for the MicroVM tier, too

little memory, an image it does not hold, or a machine too slow for the

job's time limit. Today a runner reports its architecture, OS, labels,

job slots, memory and vCPUs when it registers and on each claim, and a

claim carries the tiers it can run, the memory and vCPUs free now, and

the images it holds; the hub matches tier, room, labels, scope and image.

What is missing: architecture is reported but never matched (an image's

architecture is implicit in its bytes), a pipeline cannot say which

architecture it needs or offer a rootfs per architecture, speed and load

play no part, and a runner cannot decline a job it was handed.

### 11.1 Capability Profile

At registration and on every claim a runner sends its profile:

| Field | Meaning |

|---|---|

| `arch`, `os`, `kernel` | `x86_64`, `aarch64`, `riscv64`, …; the kernel release |

| `tiers` | The sandbox tiers it can honour: `microvm` only with `/dev/kvm` usable (a board without KVM, such as an OrangePi RV2, offers `syscall`, §4.2, and `wasm`) |

| `memory_mib`, `cpus`, `max_jobs` | What its jobs share |

| `images` | The rootfs images it holds, each with the architecture its manifest records (§11.2) |

| `labels` | As §3.1 |

| `speed` | A score and a class from 1 (slow) to 5 (fast), measured by the runner at start, again every 15 minutes while it runs no job, and by `flow runner bench`: a fixed one-thread workload (BLAKE3 over 256 MiB, and a sort of 4 Mi words, where a C compiler cannot be assumed) whose time on the reference machine (Earth, about 225 ms) scores 100; class 3 is 70–139, each class a factor of two (done 2026-10-11, `facts` on the claim) |

| `load` | The 1-minute load average (×100), and the free disk where jobs' disks are made, sent with every claim (`facts`); memory free and jobs running as before |

### 11.2 Image Manifests Carry the Architecture

`flow rootfs import` reads the ELF header of the image's `/bin/sh` and

records its architecture in the CAS manifest beside the hash. A runner

refuses to import an image for another architecture (there is no

emulation tier). An image imported before this section has no recorded

architecture and is treated as the importing runner's.

### 11.3 Job Requirements

Hard requirements, all of which a runner MUST satisfy to be offered the

job: the sandbox's tier, its rootfs (hence the image's architecture),

`memory-limit`, `cpus`, every `runs-on` label of the job and its sandbox,

and the runner's scope (OxygenHub spec §13.2). Two additions to the

language:

- `arch "riscv64"` on a sandbox pins the architecture; without it the

  architecture is the rootfs image's.

- A rootfs per architecture: `rootfs { x86_64 "cas://blob/…"; riscv64 "cas://blob/…" }`.

  The job runs on whichever architecture takes it; the receipt and the

  provenance record the architecture and image used, and a check's

  description names them.

Soft requirements guide placement but never block it: a job's `timeout`

against the runner's speed class. The hub keeps, per job hash and speed

class, the median duration of past runs; a runner whose class predicts a

duration over the limit is offered the job only when no better runner

is free.

### 11.4 Placement

The hub filters by the hard requirements, then ranks the runners that

remain: idle before busy (`load`), a higher speed class first, a runner

already holding the image before one that must fetch it (§5), fewer jobs

waiting on it first. A job no runner can satisfy waits and the queue

says what it waits for: tier, architecture, image, memory, labels or

scope, so the gap is visible rather than a silent stall.

### 11.5 Declining

A runner double-checks a claim before it starts: the architecture, that

`/dev/kvm` opens for the MicroVM tier, that the image's blob is present

and verifies, that the memory is free. If any check fails it answers the

hub with `declined` and a reason instead of running and failing; the hub

re-queues the job, never offers it to that runner again until its

profile changes, records the decline on the runner's page, and after

three declines by different runners marks the job `unplaceable` with the

reasons. A decline is never a failed check on the state.

As built: the runner sends `POST /api/v1/flow/jobs/{id}/decline` with

`{"reason": …}` before it fetches anything. The profile is a hash of the

tiers, images, architecture, labels and capacity a runner claims with.

A decline also lapses after ten minutes, so a passing reason such as

memory in use does not bar a runner for good. An unplaceable job ends

with status `error` and an error beginning `unplaceable:`. The hub

works out why each ready job waits every 15 seconds, and the run page and

the API (`waiting`) show it.

### 11.6 Conformance

1. Architecture Safety: a job whose image is `x86_64` MUST never be

   offered to a `riscv64` runner, with or without the image copied there.

2. Tier Safety: a runner without `/dev/kvm` MUST NOT be offered a

   MicroVM job, and MUST decline one if handed it.

3. Room: a job whose `memory-limit` exceeds the runner's free memory

   MUST wait, not be killed.

4. Visibility: a job nobody can take MUST say why on the queue within

   one scheduling pass.

5. Preference: with two idle runners that both satisfy a job, the hub

   MUST offer it to the higher speed class, and to the one holding the

   image when classes tie.