OXYGENHUB
# 50354f576ea6 +4 oxygenflow / README.md
4.2 KiB · Markdown · last changed oxygen as a pinned subrepo in deps/ (make pins moves it), and fmt, clippy and tests in the pipeline, before the summary 1 month ago renderedsource blame history raw
a083cc80 Andy McCall ↶ 1 
# OxygenFlow
2
c140d0d8 Andy McCall ↶ 3 
[![flow](.flow/stamp)](.flow/run)
4
a083cc80 Andy McCall ↶ 5 
Hermetic pipelines for Oxygen repositories:
6
7 
- a typed pipeline in `.flow/pipeline.kdl`
8 
- jobs run in isolation tiers, Firecracker microVMs by default
9 
- the host writes the receipts
10 
- results go to OxygenHub as checks and appear on its Flow tab
11
12 
The specification is [docs/spec.md](docs/spec.md). What works today is
13 
[docs/m0.md](docs/m0.md).
14
15 
## A pipeline
16
17 
```kdl
18 
schema-version 1
19
20 
sandbox "alpine" {
21 
    runtime "microvm"
22 
    rootfs "cas://blob/…"      // printed by `flow rootfs import`
23 
    memory-limit "256MiB"
24 
    network "isolated"
25 
}
26
27 
job "build" {
28 
    sandbox "alpine"
29 
    exec "make"
30 
}
31
32 
job "test" {
33 
    sandbox "alpine"
34 
    depends-on "build"
35 
    exec "sh" "-eu" "test.sh"   // argv, run as written: no shell unless you name one
36 
}
37 
```
38
39 
Each job boots its own microVM:
40
41 
- it has no network device
42 
- the rootfs is read-only
43 
- the workspace is the state being built, read-write at `/src`
44 
- the job runs as a user that is not root
45
46 
This repository's own pipeline is [.flow/pipeline.kdl](.flow/pipeline.kdl).
c140d0d8 Andy McCall ↶ 47 
The stamp at the top of this page is its result on OxygenHub. It shows the
48 
run of the state you are viewing, not whatever the bookmark holds now, and
49 
it names the state and each job's receipt.
a083cc80 Andy McCall ↶ 50
fc10d9e8 Andy McCall ↶ 51 
## Deploying the runner on Earth
52
53 
`make deploy-runner` builds `flow` and the static `flow-init`, installs them
54 
under `/home/claude/flow` (keeping the previous files as `.prev`), re-pins
55 
`flow-init` in `flow.toml` and restarts the `flow-runner` service.
56
a083cc80 Andy McCall ↶ 57 
## Commands
58
59 
    flow check                 # validate .flow/pipeline.kdl, print jobs and the pipeline hash
60 
    flow run [JOB…]            # run the pipeline on this directory, locally, in microVMs
61 
    flow rootfs import T.tgz --sha256 HEX   # make a rootfs image; prints the `rootfs` line to pin
62 
    flow rootfs list
63 
    flow hash FILE             # BLAKE3, for pinning in flow.toml
64 
    flow runner                # claim and run jobs for the hub in flow.toml
65
66 
## A Flow home
67
68 
`$FLOW_HOME` (default `~/.local/share/flow`) holds:
69
70 
- `flow.toml`
71 
- the pinned Firecracker binary, guest kernel and `flow-init`
72 
- rootfs images under `cas/`
73 
- scratch space
74
75 
Every pinned file is checked against its BLAKE3 before use.
76
77 
```toml
78 
[microvm]
79 
firecracker = { path = "bin/firecracker", blake3 = "…" }
80 
kernel = { path = "kernel/vmlinux", blake3 = "…" }
81 
init = { path = "bin/flow-init", blake3 = "…" }
82
83 
[runner]                     # only for `flow runner`
84 
hub = "https://hub.example:7343"
85 
token = "…"                  # listed in the hub's [flow] runner_tokens
86 
ca = "/path/to/hub-ca.pem"   # when the hub's certificate is self-signed
87 
name = "runner-1"
88 
```
89
90 
Setting one up needs Linux x86-64 with KVM and e2fsprogs 1.43 or later:
91
92 
1. Build `flow` with `cargo build --release -p flow`. Build the guest init
93 
   as a static binary:
94 
   `RUSTFLAGS="-C target-feature=+crt-static" cargo build --release -p flow-init --target x86_64-unknown-linux-gnu`.
95 
2. Download Firecracker v1.17.0 from its GitHub release, and check the
96 
   published SHA-256. Download the guest kernel
97 
   `firecracker-ci/v1.15/x86_64/vmlinux-6.1.155` from Firecracker's CI
98 
   bucket.
99 
3. Run `flow rootfs import alpine-minirootfs-3.24.2-x86_64.tar.gz --sha256 …`.
100 
4. Write `flow.toml`, pinning each file by its `flow hash`.
101
102 
On OxygenHub, `hub.toml` lists the runner token and the repositories whose
103 
pushes run pipelines:
104
105 
```toml
106 
[flow]
107 
runner_tokens = ["…"]
108 
repos = ["owner/*"]
109 
```
110
111 
## Tests
112
113 
`cargo test --workspace` runs every test that needs no VM. The microVM
114 
conformance test also runs when `FLOW_TEST_HOME` names a Flow home and
115 
`FLOW_TEST_ROOTFS` is the hash of an Alpine rootfs in it. The test checks
116 
literal argv, network isolation, the read-only rootfs, and that dependents
117 
are skipped after a failure.
fe128fa6 Andy McCall ↶ 118
119 
## oxygen as a pinned subrepo
120
121 
`flow` builds against `deps/oxygen`, a subrepo (`oxy submodule`) pinned to an exact state, so builds are reproducible and OxygenFlow can build this repository without network. `make pins` moves it to oxygen's `main` on Earth (push oxygen first); the next commit records the new pin. Format only this workspace, `cargo fmt -p flow -p flow-core -p flow-init`: `--all` also reaches `deps/`. After a fresh clone, `oxy submodule update --init` fetches it.