README.md at 060d459ff6c6 · andymccall/oxygenflow

# 060d459ff6c6 oxygenflow / README.md

5.7 KiB · Markdown · last changed flow runner drain: drain the running runner by a request file 10 days ago rendered source blame history raw

# OxygenFlow

Hermetic pipelines for Oxygen repositories:

- a typed pipeline in `.flow/pipeline.kdl`
- jobs run in isolation tiers, Firecracker microVMs by default
- the host writes the receipts
- results go to OxygenHub as checks and appear on its Flow tab

The specification is [docs/spec.md](docs/spec.md). What works today is
[docs/m0.md](docs/m0.md).

## A pipeline

```kdl
schema-version 1

sandbox "alpine" {
    runtime "microvm"
    rootfs "cas://blob/…"      // printed by `flow rootfs import`
    memory-limit "256MiB"
    network "isolated"
}

job "build" {
    sandbox "alpine"
    exec "make"
}

job "test" {
    sandbox "alpine"
    depends-on "build"
    exec "sh" "-eu" "test.sh"   // argv, run as written: no shell unless you name one
}
```

Each job boots its own microVM:

- it has no network device
- the rootfs is read-only
- the workspace is the state being built, read-write at `/src`
- the job runs as a user that is not root

This repository's own pipeline is [.flow/pipeline.kdl](.flow/pipeline.kdl).

## Deploying the runner on Earth

`make deploy-runner` builds `flow` and the static `flow-init`, installs them
under `/home/claude/flow` (keeping the previous files as `.prev`), re-pins
`flow-init` in `flow.toml` and restarts the `flow-runner` service.

## A Windows runner

A Windows machine runs WASM jobs (the other tiers are Linux's). Each
release carries `flow-windows-x86_64.exe` (from the `dist` job):

    flow.exe --home C:\flow runner register https://oxygenhub.io --token flowreg_… --name win-1 --label windows
    flow.exe --home C:\flow runner

The token comes from the hub (Admin › Runners, or `POST /api/v1/user/runner-token`
for a runner of your own repositories). A pipeline sends it work with a
`runtime "wasm"` sandbox and `runs-on "windows"`. To keep it running, a
Task Scheduler task at start-up with that command (restarting on failure)
does; set it to restart on failure, as a runner that leaves for a new build
exits 75 (`flow runner update`, spec §11.5.3). To stop it without losing
its job, `flow.exe --home C:\flow runner drain` (Windows has no SIGTERM;
ending the task kills it). What a Windows runner does not report yet: load
and kernel (oxygenflow#83).

## Commands

    flow check                 # validate .flow/pipeline.kdl, print jobs and the pipeline hash
    flow run [JOB…]            # run the pipeline on this directory, locally, in microVMs
    flow rootfs import T.tgz --sha256 HEX   # make a rootfs image; prints the `rootfs` line to pin
                  [--env NAME=value …]      #   with the image's own environment (a Dockerfile's ENV)
    flow rootfs list
    flow hash FILE             # BLAKE3, for pinning in flow.toml
    flow runner                # claim and run jobs for the hub in flow.toml
    flow runner update --from FILE [--dry-run]   # install a new flow build in place of this one (spec §11.5.3)
    flow runner drain          # the running runner claims nothing more and stops once its jobs are reported

## A Flow home

`$FLOW_HOME` (default `~/.local/share/flow`) holds:

- `flow.toml`
- the pinned Firecracker binary, guest kernel and `flow-init`
- rootfs images under `cas/`
- scratch space

Every pinned file is checked against its BLAKE3 before use.

```toml
[microvm]
firecracker = { path = "bin/firecracker", blake3 = "…" }
kernel = { path = "kernel/vmlinux", blake3 = "…" }
init = { path = "bin/flow-init", blake3 = "…" }

[runner]                     # only for `flow runner`
hub = "https://hub.example:7343"
token = "…"                  # listed in the hub's [flow] runner_tokens
ca = "/path/to/hub-ca.pem"   # when the hub's certificate is self-signed
name = "runner-1"
max_jobs = 3                 # jobs at once (default 1)
memory_mib = 24576           # what they may use together; a lone job may use it all
cpus = 8
```

Setting one up needs Linux x86-64 with KVM and e2fsprogs 1.43 or later:

1. Build `flow` with `cargo build --release -p flow`. Build the guest init
   as a static binary:
   `RUSTFLAGS="-C target-feature=+crt-static" cargo build --release -p flow-init --target x86_64-unknown-linux-gnu`.
2. Download Firecracker v1.17.0 from its GitHub release, and check the
   published SHA-256. Download the guest kernel
   `firecracker-ci/v1.15/x86_64/vmlinux-6.1.155` from Firecracker's CI
   bucket.
3. Run `flow rootfs import alpine-minirootfs-3.24.2-x86_64.tar.gz --sha256 …`.
4. Write `flow.toml`, pinning each file by its `flow hash`.

On OxygenHub, `hub.toml` lists the runner token and the repositories whose
pushes run pipelines:

```toml
[flow]
runner_tokens = ["…"]
repos = ["owner/*"]
```

## Tests

**Changelog:** a commit that changes code adds a line under `## Unreleased` in CHANGELOG.md (Added, Changed, Fixed or Security, with the issue number); versions change only at release. `scripts/changelog-check.sh [PATH...]` checks it for the paths a commit takes (the working copy's changes without any) and runs in the gate; `NO_CHANGELOG=1` for code changes users won't notice.


`cargo test --workspace` runs every test that needs no VM. The microVM
conformance test also runs when `FLOW_TEST_HOME` names a Flow home and
`FLOW_TEST_ROOTFS` is the hash of an Alpine rootfs in it. The test checks
literal argv, network isolation, the read-only rootfs, and that dependents
are skipped after a failure.

## oxygen as a pinned subrepo

`flow` builds against `deps/oxygen`, a subrepo (`oxy submodule`) pinned to an exact state, so builds are reproducible and OxygenFlow can build this repository without network. `make pins` moves it to oxygen's `main` on Earth (push oxygen first); the next commit records the new pin. Format only this workspace, `cargo fmt -p flow -p flow-core -p flow-init`: `--all` also reaches `deps/`. After a fresh clone, `oxy submodule update --init` fetches it.

`/`	Search
`g` `c`	Code
`g` `h`	History
`g` `b`	Bookmarks
`g` `t`	Trust
`y`	Permalink to this state (files and trees)
`?`	This sheet

1	# OxygenFlow
2
3	Hermetic pipelines for Oxygen repositories:
4
5	- a typed pipeline in `.flow/pipeline.kdl`
6	- jobs run in isolation tiers, Firecracker microVMs by default
7	- the host writes the receipts
8	- results go to OxygenHub as checks and appear on its Flow tab
9
10	The specification is [docs/spec.md](docs/spec.md). What works today is
11	[docs/m0.md](docs/m0.md).
12
13	## A pipeline
14
15	```kdl
16	schema-version 1
17
18	sandbox "alpine" {
19	runtime "microvm"
20	rootfs "cas://blob/…" // printed by `flow rootfs import`
21	memory-limit "256MiB"
22	network "isolated"
23	}
24
25	job "build" {
26	sandbox "alpine"
27	exec "make"
28	}
29
30	job "test" {
31	sandbox "alpine"
32	depends-on "build"
33	exec "sh" "-eu" "test.sh" // argv, run as written: no shell unless you name one
34	}
35	```
36
37	Each job boots its own microVM:
38
39	- it has no network device
40	- the rootfs is read-only
41	- the workspace is the state being built, read-write at `/src`
42	- the job runs as a user that is not root
43
44	This repository's own pipeline is [.flow/pipeline.kdl](.flow/pipeline.kdl).
45
46	## Deploying the runner on Earth
47
48	`make deploy-runner` builds `flow` and the static `flow-init`, installs them
49	under `/home/claude/flow` (keeping the previous files as `.prev`), re-pins
50	`flow-init` in `flow.toml` and restarts the `flow-runner` service.
51
52	## A Windows runner
53
54	A Windows machine runs WASM jobs (the other tiers are Linux's). Each
55	release carries `flow-windows-x86_64.exe` (from the `dist` job):
56
57	flow.exe --home C:\flow runner register https://oxygenhub.io --token flowreg_… --name win-1 --label windows
58	flow.exe --home C:\flow runner
59
60	The token comes from the hub (Admin › Runners, or `POST /api/v1/user/runner-token`
61	for a runner of your own repositories). A pipeline sends it work with a
62	`runtime "wasm"` sandbox and `runs-on "windows"`. To keep it running, a
63	Task Scheduler task at start-up with that command (restarting on failure)
64	does; set it to restart on failure, as a runner that leaves for a new build
65	exits 75 (`flow runner update`, spec §11.5.3). To stop it without losing
66	its job, `flow.exe --home C:\flow runner drain` (Windows has no SIGTERM;
67	ending the task kills it). What a Windows runner does not report yet: load
68	and kernel (oxygenflow#83).
69
70	## Commands
71
72	flow check # validate .flow/pipeline.kdl, print jobs and the pipeline hash
73	flow run [JOB…] # run the pipeline on this directory, locally, in microVMs
74	flow rootfs import T.tgz --sha256 HEX # make a rootfs image; prints the `rootfs` line to pin
75	[--env NAME=value …] # with the image's own environment (a Dockerfile's ENV)
76	flow rootfs list
77	flow hash FILE # BLAKE3, for pinning in flow.toml
78	flow runner # claim and run jobs for the hub in flow.toml
79	flow runner update --from FILE [--dry-run] # install a new flow build in place of this one (spec §11.5.3)
80	flow runner drain # the running runner claims nothing more and stops once its jobs are reported
81
82	## A Flow home
83
84	`$FLOW_HOME` (default `~/.local/share/flow`) holds:
85
86	- `flow.toml`
87	- the pinned Firecracker binary, guest kernel and `flow-init`
88	- rootfs images under `cas/`
89	- scratch space
90
91	Every pinned file is checked against its BLAKE3 before use.
92
93	```toml
94	[microvm]
95	firecracker = { path = "bin/firecracker", blake3 = "…" }
96	kernel = { path = "kernel/vmlinux", blake3 = "…" }
97	init = { path = "bin/flow-init", blake3 = "…" }
98
99	[runner] # only for `flow runner`
100	hub = "https://hub.example:7343"
101	token = "…" # listed in the hub's [flow] runner_tokens
102	ca = "/path/to/hub-ca.pem" # when the hub's certificate is self-signed
103	name = "runner-1"
104	max_jobs = 3 # jobs at once (default 1)
105	memory_mib = 24576 # what they may use together; a lone job may use it all
106	cpus = 8
107	```
108
109	Setting one up needs Linux x86-64 with KVM and e2fsprogs 1.43 or later:
110
111	1. Build `flow` with `cargo build --release -p flow`. Build the guest init
112	as a static binary:
113	`RUSTFLAGS="-C target-feature=+crt-static" cargo build --release -p flow-init --target x86_64-unknown-linux-gnu`.
114	2. Download Firecracker v1.17.0 from its GitHub release, and check the
115	published SHA-256. Download the guest kernel
116	`firecracker-ci/v1.15/x86_64/vmlinux-6.1.155` from Firecracker's CI
117	bucket.
118	3. Run `flow rootfs import alpine-minirootfs-3.24.2-x86_64.tar.gz --sha256 …`.
119	4. Write `flow.toml`, pinning each file by its `flow hash`.
120
121	On OxygenHub, `hub.toml` lists the runner token and the repositories whose
122	pushes run pipelines:
123
124	```toml
125	[flow]
126	runner_tokens = ["…"]
127	repos = ["owner/*"]
128	```
129
130	## Tests
131
132	Changelog: a commit that changes code adds a line under `## Unreleased` in CHANGELOG.md (Added, Changed, Fixed or Security, with the issue number); versions change only at release. `scripts/changelog-check.sh [PATH...]` checks it for the paths a commit takes (the working copy's changes without any) and runs in the gate; `NO_CHANGELOG=1` for code changes users won't notice.
133
134
135	`cargo test --workspace` runs every test that needs no VM. The microVM
136	conformance test also runs when `FLOW_TEST_HOME` names a Flow home and
137	`FLOW_TEST_ROOTFS` is the hash of an Alpine rootfs in it. The test checks
138	literal argv, network isolation, the read-only rootfs, and that dependents
139	are skipped after a failure.
140
141	## oxygen as a pinned subrepo
142
143	`flow` builds against `deps/oxygen`, a subrepo (`oxy submodule`) pinned to an exact state, so builds are reproducible and OxygenFlow can build this repository without network. `make pins` moves it to oxygen's `main` on Earth (push oxygen first); the next commit records the new pin. Format only this workspace, `cargo fmt -p flow -p flow-core -p flow-init`: `--all` also reaches `deps/`. After a fresh clone, `oxy submodule update --init` fetches it.

Keyboard