OXYGENHUB
# ea81796bb757 oxygenflow / docs / spec.md
26.6 KiB · Markdown · last changed Allow-listed egress over vsock: network "egress" { allow "host"... } 20 days ago renderedsource blame history raw
1
# OxygenFlow (flow) — Deterministic Execution & Provenance Pipeline Specification
2
3
| | |
4
|---|---|
5
| **Format version** | 1 |
6
| **Date** | 2026-09-28 |
7
| **Status** | Draft 0.1 |
8
| **Target VCS** | Oxygen (oxy) Draft 0.8i |
9
| **Audience** | Implementers, Security Architects, and Systems Engineers |
10
11
12
The key words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are to be
13
interpreted as described in RFC 2119.
14
15
## 1. Overview and Core Invariants
16
17
OxygenFlow (flow) is a content-addressed, hermetic execution engine, runner system,
18
and pipeline language designed natively for the Oxygen (oxy) version control
19
system.
20
21
Existing runner engines (GitHub Actions, GitLab CI, Forgejo Runner) suffer from
22
structural flaws:
23
1. Workspace Bloat: Full clones, index unpacks, and checkout scans dominate I/O.
24
2. String Injection: Shell interpolation within untyped YAML files produces
25
   expression injection and command execution exploits.
26
3. Decoupled Supply Chain: Attestations (SLSA, in-toto, Cosign) exist outside
27
   the VCS, relying on external registries rather than native repository trust.
28
4. Coarse Caching: Opaque tarballs keyed by arbitrary string hashes fail to
29
   leverage commutative history or granular dependency graphs.
30
31
OxygenFlow solves these liabilities through direct integration with Oxygen:
32
- Zero-Copy Workspaces: Jobs mount workspaces directly from Oxygen's FastCDC-
33
  chunked, BLAKE3-addressed Content-Addressable Store (CAS) using Copy-on-Write
34
  (COW) overlays.
35
- Commutative History & Impact Trees: Pipelines evaluate over change closures
36
  rather than linear snapshot sequences. Commuting changes automatically reuse
37
  cached execution receipts without redundant re-execution.
38
- Micro-Engine Sandboxing: Jobs run inside isolated environments (WASM, gVisor,
39
  or MicroVMs) with strict, deny-by-default capabilities and no ambient
40
  network egress.
41
- Native Cryptographic Provenance: Provenance and execution receipts are
42
  generated outside the untrusted guest, encoded as deterministic CBOR, and
43
  signed using composite Ed25519 + ML-DSA-65 keys as native Oxygen Attest
44
  objects governed by in-repository Policy rules.
45
46
## 2. Threat Model and Security Architecture
47
48
### 2.1 Adversaries
49
- Untrusted Proposal Author: Submits code to an open repository containing
50
  malicious build configurations intended to escape the runner, exfiltrate
51
  secrets, or corrupt shared worker caches.
52
- Malicious Dependency / Upstream: Third-party dependencies invoked during
53
  build time attempting out-of-band network calls or host file modification.
54
- Compromised Runner / Malicious Worker: An execution node attempting to
55
  forge successful build receipts or inject backdoors into published artifacts.
56
57
### 2.2 Security Invariants
58
- I1 Deny-by-Default Execution: Sandboxes possess no network interfaces
59
  (excluding loopback when requested) and no host filesystem access unless
60
  explicitly granted in policy-signed configuration.
61
- I2 Injection-Free Invocation: Configuration parsing directly yields an argv
62
  string vector executed via execve(). Shell interpretation (/bin/sh -c) does
63
  not exist.
64
- I3 Host-Enforced Provenance: The running payload cannot author, inspect, or
65
  alter its own provenance statement. Provenance is emitted solely by the
66
  isolation hypervisor boundary.
67
- I4 Cryptographic Identity & Authorization: Runners execute using an Oxygen
68
  Identity whose KeyId is enrolled as a delegate in the repository's Policy.
69
  Unenrolled runners cannot publish recognized attestations or advance gates.
70
71
## 3. The OxygenFlow Configuration Language (.flow)
72
73
Pipelines are declared in KDL syntax within .flow/pipeline.kdl at the
74
repository root. KDL provides strict typing, structured nesting, and rejects
75
arbitrary text macro expansion.
76
77
Strings follow KDL 2.0, which has no `\033`, `\e` or `\x` escapes: ESC,
78
for ANSI colours in a job's output, is `\u{1b}` (`"\u{1b}[32mok\u{1b}[0m"`),
79
and `flow check` says so when it meets one of the others. Inside the
80
guest, `/dev/fd` and `/dev/stdin|stdout|stderr` link to `/proc/self/fd`
81
as on any Linux system, so bash process substitution works.
82
83
### 3.1 Pipeline Specification Schema
84
85
```kdl
86
schema-version 1
87
88
// Define the hermetic execution boundary
89
sandbox "rust-builder" {
90
    runtime "microvm"       // Options: "wasm" | "gvisor" | "microvm"
91
    rootfs "cas://blob/9b5c2a1e8f3d4e7a6b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a"
92
    memory-limit "4GiB"
93
    cpu-limit 4
94
    network "isolated"      // Options: "isolated" | "loopback" | "egress" { allow "host"... } (§4.3.1)
95
}
96
97
// Job evaluated based on Oxygen change closure
98
job "compile-core" {
99
    sandbox "rust-builder"
100
101
    // Commutation boundary: runs only if change closure touches these paths
102
    scope "crates/oxy-core/**" "crates/oxy-server/**" "Cargo.*"
103
104
    // Zero-copy workspace mount backed by Oxygen CAS
105
    mount {
106
        source "oxy://workspace"
107
        target "/src"
108
        include "crates/oxy-core" "crates/oxy-server" "Cargo.toml" "Cargo.lock"
109
        readonly #true
110
    }
111
112
    // Ephemeral scratch space
113
    mount {
114
        source "ephemeral://tmpfs"
115
        target "/src/target"
116
        size "8GiB"
117
    }
118
119
    env {
120
        RUSTFLAGS "-Dwarnings"
121
        CARGO_INCREMENTAL "0"
122
    }
123
124
    // Direct argv execution vector (no subshell)
125
    exec "cargo" "build" "--release" "--package" "oxy-core"
126
127
    output {
128
        artifact "oxy-core-lib" {
129
            path "/src/target/release/liboxy_core.rlib"
130
            format "blob"
131
        }
132
    }
133
}
134
135
job "unit-tests" {
136
    sandbox "rust-builder"
137
    depends-on "compile-core"
138
139
    mount {
140
        source "oxy://workspace"
141
        target "/src"
142
        readonly #true
143
    }
144
145
    mount {
146
        source "artifact://compile-core/oxy-core-lib"
147
        target "/src/target/release/liboxy_core.rlib"
148
    }
149
150
    exec "cargo" "test" "--release" "--package" "oxy-core"
151
}
152
153
publish "release-provenance" {
154
    depends-on "unit-tests"
155
156
    attest {
157
        kind "provenance"
158
        bind-ref "bookmarks/main"
159
        include-artifacts "compile-core/oxy-core-lib"
160
    }
161
}
162
163
164
**What starts a job.** An `on` block, at the top of the pipeline or in a job (the job's wins), names the events that run it: `push` with bookmark globs (none: any bookmark), `tag` with tag globs (none: any tag), `manual` ("Run pipeline", `oh run start`) and `schedule` with five-field cron lines on the hub's clock (UTC), run on the default bookmark. Without `on`, a job runs on every push and by hand, as before. A run has only the jobs its event starts (and what they depend on); an event that starts no job starts no run. A rerun repeats its run's event.
165
166
```kdl
167
on {
168
    push "main" "feature/*"
169
    manual
170
}
171
job "dist" {
172
    on {
173
        tag "v*"
174
        manual
175
    }
176
    …
177
}
178
```
179
180
**Time limits.** A job may say how long it may run with `timeout "90m"` (hours, minutes and seconds, as `"2h"`, `"1h30m"` or `"45s"`, from a minute to a day); a sandbox's `timeout` is the default for its jobs. Without one, the hub's default applies, and the hub caps every job at the site's maximum (both set by an administrator on Admin → Runners). The runner stops a job at its limit, and the hub ends one its runner stopped reporting a few minutes after it. A timeout is part of the compiled job, so a job without one compiles as before.
181
182
**Runner labels.** A sandbox or a job may say `runs-on "LABEL" ...`: lower-case letters, digits, `-`, `_` and `.`, at most 64 bytes each, given on one or more lines. A job needs a runner that has every label its sandbox and it name. A runner reports its labels and architecture when it registers and on each claim (`flow runner register --label`). The hub hands a job only to a runner with all of them, within the runner's scope (OxygenHub spec 13.2), and a job no runner can take waits, saying so on the queue. Labels are part of the compiled job, so a job that asks for none compiles as before.
183
184
### 3.2 Canonical Compilation
185
Upon commit, .flow/pipeline.kdl is validated and compiled into a deterministic
186
CBOR job-graph blob. Runners evaluate the canonical CBOR to prevent syntax or
187
parser ambiguities across varying client platforms.
188
189
## 4. Sandbox Execution Engine
190
191
OxygenFlow implements three distinct isolation tiers depending on the workload:
192
193
### 4.1 Level 0: WASM Component Sandbox
194
- Intended for static linters, formatters, doc generators, and policy checks.
195
- Runs on a standard WebAssembly System Interface (WASI) runtime.
196
- Startup latency: < 5 ms.
197
- Zero host I/O; virtualized file system presented in-memory.
198
199
### 4.2 Level 1: Intercepted Syscall Isolation (gVisor / Landlock + seccomp-bpf)
200
- Intended for standard compilation and hermetic test suites.
201
- Emulates or filters Linux system call interfaces in user space.
202
- Blocks access to raw network sockets, real-time hardware clocks, and device
203
  nodes.
204
- Memory and execution CPU quotas are strictly bounded via cgroups v2.
205
206
### 4.3 Level 2: MicroVM Isolation (Firecracker / Cloud-Hypervisor)
207
- Intended for kernel tests, multi-container integration, and untrusted code.
208
- Ephemeral Linux microVM booting in < 80 ms.
209
- The root filesystem is presented as a read-only virtio-block device mapped
210
  directly to FastCDC chunks stored in Oxygen's CAS.
211
- Guest VM memory is cleared upon termination.
212
213
#### 4.3.1 Allow-listed egress (oxygenflow#55)
214
215
A sandbox reaches the network only by naming its hosts:
216
217
```kdl
218
sandbox "deploy" {
219
    runtime "microvm"
220
    network "egress" { allow "*.googleapis.com" "registry.opentofu.org" }
221
}
222
```
223
224
The VM still has no network device. `flow-init` serves a proxy on the
225
guest's `127.0.0.1:3128` (the job gets `HTTPS_PROXY`, `HTTP_PROXY` and
226
their lower-case forms, unless it sets them) and relays each connection,
227
unread, over Firecracker's vsock to the runner. The runner accepts only
228
`CONNECT host:443` for a host the list allows (a name exactly, or any
229
name under a `*.` domain, never the domain itself; never an address),
230
resolves it itself (the guest does not resolve), refuses any name that
231
resolves to an address that is not public, and relays the bytes without
232
opening the TLS. Every decision, allowed or refused, is a `flow` line in
233
the job's log. Patterns are lower-case host names of two or more labels,
234
at most 32 per sandbox, and part of the sandbox's hash.
235
236
Egress is off unless a runner turns it on (`egress = true` under
237
`[runner]` in `flow.toml`); such a runner offers the `egress` label, and
238
an egress sandbox needs it, so the hub hands its jobs only to runners
239
that allow it, and a runner without it refuses one anyway. The syscall
240
runtime has no egress yet.
241
242
## 5. Content-Addressable Storage (CAS) & Zero-Copy Mounting
243
244
OxygenFlow runners bypass traditional clone operations. Repositories are accessed
245
directly via the local object store (.oxy/objects/) or streamed on demand
246
using Oxygen's wire protocol.
247
248
### 5.1 FUSE Overlay Mounting (flow-casfs)
249
1. The runner receives a task targeting an Oxygen State ObjectRef (a sorted
250
   list of maximal heads).
251
2. The nearest Checkpoint is resolved. Uncommitted changes between the
252
   checkpoint and heads are replayed into a virtual tree in-memory.
253
3. The virtual tree is exposed to the sandbox via flow-casfs:
254
   - File reads resolve chunks using the FastCDC gear table (32 KiB to 256 KiB,
255
     mean 64 KiB) pinned in Oxygen Spec Section 13.2.
256
   - Missing chunks are fetched lazily using the POST /objects protocol.
257
   - To prevent FUSE context-switch bottlenecks on many small reads, flow-casfs aggressively mmaps chunk data and utilizes host page caching.
258
   - All guest mutations are diverted to a scratch tmpfs layer; base objects
259
     remain immutable.
260
261
### 5.2 Artifact Capture
262
1. At task completion, flow-casfs scans the paths declared in the output block.
263
2. Modified outputs are chunked with FastCDC, hashed via BLAKE3-256, and
264
   written as canonical Blob and Chunk objects.
265
3. Output ObjectRefs are stored directly in the Oxygen repository.
266
4. Outputs are read when the job passes. An artifact with `when "always"`
267
   (default `when "passed"`) is also read when the job fails or is stopped
268
   at its time limit, so a test report survives the failure it describes.
269
   From a job that did not pass it is kept and shown, but its receipt never
270
   names it, and one that cannot be read is only noted in the log.
271
272
```kdl
273
output {
274
    artifact "junit" {
275
        path "/src/target/junit.xml"
276
        when "always"
277
    }
278
}
279
```
280
281
5. A `report` output (OxygenHub `docs/reports.md`) names a directory
282
   holding `report.toml` at its top. The runner reads its regular files
283
   (links and anything else are left out, never followed; file names hold
284
   letters, digits, `.`, `_`, `-` and `+`; at most 2000 files and 512 MiB),
285
   uploads each to `POST /api/v1/flow/jobs/{job}/reports/{name}/files` with
286
   its path in `X-Flow-Report-Path`, then uploads a manifest,
287
   `{"version":1,"files":[{"path","blob","size"}]}`, as the output under
288
   the report's name, checking every blob the hub answers. The receipt
289
   names the manifest. Reports and artifacts share names; a report's kind
290
   is part of the job's hash (`[name, path, "report"]`).
291
292
```kdl
293
output {
294
    report "frame-timing" {
295
        path "/src/out/report"
296
        when "always"
297
    }
298
}
299
```
300
301
### 5.4 The Build Cache (oxygenflow#28)
302
303
An `ephemeral://disk` mount may name a `cache`, and the files that key it:
304
305
```kdl
306
mount {
307
    source "ephemeral://disk"
308
    target "/src/target"
309
    size "24GiB"
310
    cache "target"
311
    key "Cargo.lock" "rust-toolchain.toml"
312
}
313
```
314
315
Such a disk is not always empty at the start. Each runner keeps a pool of
316
the disks of passing jobs, by repository, job, cache name and bookmark,
317
named by the **key**: the BLAKE3 of the sandbox image, the job's name, the
318
cache name, and each key file's content in the job's state. A job's disk
319
starts as, in order: its bookmark's disk with the same key (moved), its
320
bookmark's newest disk (moved), another bookmark's with the same key and
321
then another's newest (copied, so a proposal never takes the default
322
bookmark's), and otherwise an empty disk. A disk must be the size the
323
mount asks for.
324
325
- **Only passing jobs keep a disk.** When the job and its outputs pass,
326
  its cached disks go back into the pool under their key; otherwise they
327
  are discarded with the scratch directory, so nothing a failing job wrote
328
  reaches a later one.
329
- **The receipt says so.** The job's environment gains `cache TARGET` with
330
  the key and what the disk started from, so the receipt's environment
331
  hash differs from a cold build's.
332
- **Bounded.** The pool keeps under `cache_gb` in the runner's config
333
  (default 64 GiB) by dropping the disks used longest ago.
334
- The syscall tier keeps its disks empty for now.
335
- A cache is a speed-up, never a source of truth: jobs must build
336
  correctly from an empty disk, which a key change or an eviction gives
337
  them at any time.
338
339
## 6. Commutative Execution & Change Caching
340
341
In Oxygen, changes that do not share scopes (referenced atoms, placements, or
342
nodes) commute. OxygenFlow exploits this invariant to provide granular caching.
343
344
### 6.1 Execution Receipt Object
345
Every successful or failed job run emits an immutable Receipt:
346
347
```cbor
348
Receipt {
349
  version        [0]: 1
350
  job_name       [1]: text
351
  pipeline_hash  [2]: ObjectRef
352
  scope_tree_hash [3]: ObjectRef     // BLAKE3 tree hash of the materialized scope
353
  inputs         [4]: [ObjectRef]   // Artifacts and input files consumed
354
  environment    [5]: ObjectRef     // Hash of rootfs, compiler flags, and argv
355
  outputs        [6]: [ObjectRef]   // Produced artifact blobs
356
  exit_code      [7]: u32
357
  duration_ms    [8]: u64
358
}
359
```
360
361
A Receipt is hashed as an Oxygen object with type name "receipt":
362
ObjectRef = BLAKE3-256("oxy1 receipt " || decimal(len) || 0x00 || body)
363
364
### 6.2 Cache Re-use Invariant
365
Let Scope(J) be the file paths and tree nodes declared by job J.
366
Because changes that do not share scopes commute, we do not need to track lists of ChangeHashes. Instead, we hash the materialized tree of Scope(J).
367
368
If an existing Receipt R exists such that:
369
  R.job_name == J.name &&
370
  R.pipeline_hash == Hash(J) &&
371
  R.scope_tree_hash == TreeHash(Scope(J)) &&
372
  R.environment == CurrentEnvironmentHash
373
374
Then the job execution is bypassed. R.outputs are mapped into downstream
375
dependents immediately as a cached-commuted hit.
376
377
## 7. Native Provenance Specification
378
379
OxygenFlow formalizes build provenance as a native, tamper-evident Oxygen Attest
380
object (kind: "attest"). Provenance is generated exclusively by the runner
381
hypervisor, not the guest payload.
382
383
### 7.1 Provenance Object Schema
384
385
```json
386
Attest {
387
  version [0]: 1
388
  kind    [1]: "attest"
389
  repo    [2]: bytes(16)               // Oxygen Repository ID
390
  hlc     [3]: HLC                     // Hybrid Logical Clock
391
  body    [4]: {
392
    "type": "flow/provenance/v1",
393
    
394
    // Subject artifacts produced by the build
395
    "subjects": [
396
      {
397
        "name": text,
398
        "ref": ObjectRef,              // BLAKE3 hash of output blob
399
        "media_type": text
400
      }
401
    ],
402
403
    // Builder platform and execution identity
404
    "builder": {
405
      "runner_id": KeyId,              // ObjectRef of runner's Oxygen Identity
406
      "isolation_level": text,         // "wasm" | "gvisor" | "microvm"
407
      "hypervisor_image": ObjectRef    // Rootfs blob digest
408
    },
409
410
    // Exact input materials
411
    "materials": {
412
      "state_heads": [ChangeHash],     // Oxygen heads of the state built
413
      "scope_tree_hash": ObjectRef,        // Captures exact state without O(N) array bloat
414
      "inputs": [
415
        {
416
          "name": text,
417
          "ref": ObjectRef             // Content hash of locked dependencies
418
        }
419
      ]
420
    },
421
422
    // Structured execution arguments
423
    "invocation": {
424
      "job": text,
425
      "pipeline_spec": ObjectRef,      // ObjectRef of .flow/pipeline.kdl
426
      "argv": [text],                  // Unmodified execution vector
427
      "env": map<text, text>           // Explicitly whitelisted environment
428
    },
429
430
    // Environment observations
431
    "metadata": {
432
      "started_on": u64,               // Wallclock milliseconds
433
      "finished_on": u64,
434
      "hermetic": bool,                // Verified zero outbound network access
435
      "reproducible": bool
436
    }
437
  }
438
}
439
```
440
441
### 7.2 Composite Signatures
442
All provenance attestations are signed using Oxygen's composite signature
443
construction (Ed25519 and ML-DSA-65):
444
445
```cbor
446
Signature {
447
  key [0]: KeyId
448
  alg [1]: 1                           // 1 = Ed25519 + ML-DSA-65
449
  ed  [2]: bytes(64)                   // Classical signature
450
  ml  [3]: bytes(3309)                 // Post-quantum signature
451
}
452
```
453
454
Both signatures are computed over:
455
  "oxy1 sig attest\0" || KeyId || ObjectRef
456
457
## 8. Policy Integration & Bookmark Gates
458
459
Oxygen repositories enforce branch and bookmark protections via signed Policy
460
objects (Oxygen Spec Section 14.5). OxygenFlow integrates provenance directly into
461
policy evaluation during push validation.
462
463
### 8.1 Policy Gate Definition
464
A Policy rule governing a bookmark MAY demand valid OxygenFlow attestations. To mitigate a compromised authorized worker, high-security repositories can require **Consensus Builds** by setting the threshold > 1. This requires multiple independent runner identities to arrive at the exact same output hashes and both sign it.
465
466
```json
467
PolicyRule {
468
  pattern: "bookmarks/production"
469
  keys: [ KeyId ]
470
  threshold: 1
471
  require: "signed"
472
  allow_conflicts: false
473
  require_attestations: [
474
    {
475
      type: "flow/provenance/v1",
476
      job: "release-provenance",
477
      require_hermetic: true,
478
      signer_roles: [ "official-worker" ]
479
    }
480
  ]
481
}
482
483
### 8.2 Push Verification Sequence (VCS Engine Enforcement)
484
When a push request arrives at an Oxygen server attempting to update a
485
protected bookmark:
486
1. Object Integrity: Validate that all changes, chunks, and blobs exist and match
487
   their BLAKE3 hashes.
488
2. Attestation Resolution: Verify that an Attest object of type
489
   "flow/provenance/v1" is present, referencing the exact heads of the new
490
   State.
491
3. Identity Verification: Validate that the attestation's composite signature
492
   resolves to a KeyId listed as an authorized delegate of the repository
493
   policy roots.
494
4. Invariant Checking: Confirm that the provenance statement records
495
   "hermetic: true" and exit code 0.
496
5. Atomic Update: If validation fails at any point, the entire push is refused
497
   with ok = false.
498
499
## 9. Zero-Environment Secret Brokering
500
501
OxygenFlow forbids the injection of raw secret strings into process environment
502
tables to eliminate accidental leaking via child process dumping, procfs
503
reads, and verbose compiler logging.
504
505
### 9.1 Ephemeral Secret Broker
506
1. If a job requires a credential (e.g., signing key, cloud deployment token),
507
   it declares a secret mount:
508
     secret "deploy-token" {
509
         vault-ref "infra/tokens/prod"
510
         mount-path "/run/secrets/token"
511
     }
512
2. The runner negotiates a short-lived token lease from the configured vault.
513
3. The secret is mounted onto an ephemeral, in-memory ramfs location inside
514
   the sandbox with access mode 0400, owned exclusively by the worker process.
515
4. Upon process termination, the memory pages are zeroized and the token is
516
   marked for immediate revocation at the vault.
517
518
## 10. Conformance and Verification Requirements
519
520
A conforming OxygenFlow implementation MUST pass the following tests:
521
522
1. Injection Immunity: Attempting to supply unescaped shell metacharacters
523
   (; && | ` $() \n) in any job field MUST pass those characters as literal
524
   bytes to argv; subshells MUST NOT be spawned.
525
2. FastCDC Determinism: Chunks produced by artifact extraction MUST match
526
   the gear table cuts and BLAKE3 hashes generated by Oxygen's core chunker.
527
3. Commutation Hit Invariance: For any two commuting changes A and B, running
528
   a pipeline on State(A, B) MUST reuse cached receipts generated on State(A)
529
   for any job whose scope does not intersect the touched paths of B.
530
4. Non-Downgrade Attestation: An attestation missing the ML-DSA-65 component
531
   MUST be rejected by policy verification.
532
5. Hermetic Network Isolation: Jobs configured with network "isolated" MUST
533
   have outgoing TCP SYN and UDP packets dropped at the hypervisor boundary.
534
## 11. Runner Capabilities and Placement (proposed 2026-10-04; §11.2 and `arch` of §11.3 done the same day, oxygenflow#36)
535
536
A runner MUST never take a job it cannot run or cannot finish well: the
537
wrong architecture, no hardware virtualisation for the MicroVM tier, too
538
little memory, an image it does not hold, or a machine too slow for the
539
job's time limit. Today a runner reports its architecture, OS, labels,
540
job slots, memory and vCPUs when it registers and on each claim, and a
541
claim carries the tiers it can run, the memory and vCPUs free now, and
542
the images it holds; the hub matches tier, room, labels, scope and image.
543
What is missing: architecture is reported but never matched (an image's
544
architecture is implicit in its bytes), a pipeline cannot say which
545
architecture it needs or offer a rootfs per architecture, speed and load
546
play no part, and a runner cannot decline a job it was handed.
547
548
### 11.1 Capability Profile
549
550
At registration and on every claim a runner sends its profile:
551
552
| Field | Meaning |
553
|---|---|
554
| `arch`, `os`, `kernel` | `x86_64`, `aarch64`, `riscv64`, …; the kernel release |
555
| `tiers` | The sandbox tiers it can honour: `microvm` only with `/dev/kvm` usable (a board without KVM, such as an OrangePi RV2, offers `syscall`, §4.2, and `wasm`) |
556
| `memory_mib`, `cpus`, `max_jobs` | What its jobs share |
557
| `images` | The rootfs images it holds, each with the architecture its manifest records (§11.2) |
558
| `labels` | As §3.1 |
559
| `speed` | A score and a class from 1 (slow) to 5 (fast), measured by the runner at start, again every 15 minutes while it runs no job, and by `flow runner bench`: a fixed one-thread workload (BLAKE3 over 256 MiB, and a sort of 4 Mi words, where a C compiler cannot be assumed) whose time on the reference machine (Earth, about 225 ms) scores 100; class 3 is 70–139, each class a factor of two (done 2026-10-11, `facts` on the claim) |
560
| `load` | The 1-minute load average (×100), and the free disk where jobs' disks are made, sent with every claim (`facts`); memory free and jobs running as before |
561
562
### 11.2 Image Manifests Carry the Architecture
563
564
`flow rootfs import` reads the ELF header of the image's `/bin/sh` and
565
records its architecture in the CAS manifest beside the hash. A runner
566
refuses to import an image for another architecture (there is no
567
emulation tier). An image imported before this section has no recorded
568
architecture and is treated as the importing runner's.
569
570
### 11.3 Job Requirements
571
572
Hard requirements, all of which a runner MUST satisfy to be offered the
573
job: the sandbox's tier, its rootfs (hence the image's architecture),
574
`memory-limit`, `cpus`, every `runs-on` label of the job and its sandbox,
575
and the runner's scope (OxygenHub spec §13.2). Two additions to the
576
language:
577
578
- `arch "riscv64"` on a sandbox pins the architecture; without it the
579
  architecture is the rootfs image's.
580
- A rootfs per architecture: `rootfs { x86_64 "cas://blob/…"; riscv64 "cas://blob/…" }`.
581
  The job runs on whichever architecture takes it; the receipt and the
582
  provenance record the architecture and image used, and a check's
583
  description names them.
584
585
Soft requirements guide placement but never block it: a job's `timeout`
586
against the runner's speed class. The hub keeps, per job hash and speed
587
class, the median duration of past runs; a runner whose class predicts a
588
duration over the limit is offered the job only when no better runner
589
is free.
590
591
### 11.4 Placement
592
593
The hub filters by the hard requirements, then ranks the runners that
594
remain: idle before busy (`load`), a higher speed class first, a runner
595
already holding the image before one that must fetch it (§5), fewer jobs
596
waiting on it first. A job no runner can satisfy waits and the queue
597
says what it waits for: tier, architecture, image, memory, labels or
598
scope, so the gap is visible rather than a silent stall.
599
600
### 11.5 Declining
601
602
A runner double-checks a claim before it starts: the architecture, that
603
`/dev/kvm` opens for the MicroVM tier, that the image's blob is present
604
and verifies, that the memory is free. If any check fails it answers the
605
hub with `declined` and a reason instead of running and failing; the hub
606
re-queues the job, never offers it to that runner again until its
607
profile changes, records the decline on the runner's page, and after
608
three declines by different runners marks the job `unplaceable` with the
609
reasons. A decline is never a failed check on the state.
610
611
### 11.6 Conformance
612
613
1. Architecture Safety: a job whose image is `x86_64` MUST never be
614
   offered to a `riscv64` runner, with or without the image copied there.
615
2. Tier Safety: a runner without `/dev/kvm` MUST NOT be offered a
616
   MicroVM job, and MUST decline one if handed it.
617
3. Room: a job whose `memory-limit` exceeds the runner's free memory
618
   MUST wait, not be killed.
619
4. Visibility: a job nobody can take MUST say why on the queue within
620
   one scheduling pass.
621
5. Preference: with two idle runners that both satisfy a job, the hub
622
   MUST offer it to the higher speed class, and to the one holding the
623
   image when classes tie.