README.md at main · andymccall/oxygenflow

◆ main oxygenflow / README.md

6.0 KiB · Markdown · last changed flow runner drain: drain the running runner by a request file 9 days ago rendered source blame history raw

# OxygenFlow

[![OxygenFlow](https://oxygenhub.io/andymccall/oxygenflow/flow/badge.svg)](https://oxygenhub.io/andymccall/oxygenflow/flow)

Hermetic pipelines for Oxygen repositories:

- a typed pipeline in `.flow/pipeline.kdl`
- jobs run in isolation tiers, Firecracker microVMs by default
- the host writes the receipts
- results go to OxygenHub as checks and appear on its Flow tab

The specification is [docs/spec.md](docs/spec.md). What works today is
[docs/m0.md](docs/m0.md).

## A pipeline

```kdl
schema-version 1

sandbox "alpine" {
    runtime "microvm"
    rootfs "cas://blob/…"      // printed by `flow rootfs import`
    memory-limit "256MiB"
    network "isolated"
}

job "build" {
    sandbox "alpine"
    exec "make"
}

job "test" {
    sandbox "alpine"
    depends-on "build"
    exec "sh" "-eu" "test.sh"   // argv, run as written: no shell unless you name one
}
```

Each job boots its own microVM:

- it has no network device
- the rootfs is read-only
- the workspace is the state being built, read-write at `/src`
- the job runs as a user that is not root

This repository's own pipeline is [.flow/pipeline.kdl](.flow/pipeline.kdl).
The stamp at the top of this page is its result on OxygenHub. It shows the
run of the state you are viewing, not whatever the bookmark holds now, and
it names the state and each job's receipt.

## Deploying the runner on Earth

`make deploy-runner` builds `flow` and the static `flow-init`, installs them
under `/home/claude/flow` (keeping the previous files as `.prev`), re-pins
`flow-init` in `flow.toml` and restarts the `flow-runner` service.

## A Windows runner

A Windows machine runs WASM jobs (the other tiers are Linux's). Each
release carries `flow-windows-x86_64.exe` (from the `dist` job):

    flow.exe --home C:\flow runner register https://oxygenhub.io --token flowreg_… --name win-1 --label windows
    flow.exe --home C:\flow runner

The token comes from the hub (Admin › Runners, or `POST /api/v1/user/runner-token`
for a runner of your own repositories). A pipeline sends it work with a
`runtime "wasm"` sandbox and `runs-on "windows"`. To keep it running, a
Task Scheduler task at start-up with that command (restarting on failure)
does; set it to restart on failure, as a runner that leaves for a new build
exits 75 (`flow runner update`, spec §11.5.3). To stop it without losing
its job, `flow.exe --home C:\flow runner drain` (Windows has no SIGTERM;
ending the task kills it). What a Windows runner does not report yet: load
and kernel (oxygenflow#83).

## Commands

    flow check                 # validate .flow/pipeline.kdl, print jobs and the pipeline hash
    flow run [JOB…]            # run the pipeline on this directory, locally, in microVMs
    flow rootfs import T.tgz --sha256 HEX   # make a rootfs image; prints the `rootfs` line to pin
                  [--env NAME=value …]      #   with the image's own environment (a Dockerfile's ENV)
    flow rootfs list
    flow hash FILE             # BLAKE3, for pinning in flow.toml
    flow runner                # claim and run jobs for the hub in flow.toml
    flow runner update --from FILE [--dry-run]   # install a new flow build in place of this one (spec §11.5.3)
    flow runner drain          # the running runner claims nothing more and stops once its jobs are reported

## A Flow home

`$FLOW_HOME` (default `~/.local/share/flow`) holds:

- `flow.toml`
- the pinned Firecracker binary, guest kernel and `flow-init`
- rootfs images under `cas/`
- scratch space

Every pinned file is checked against its BLAKE3 before use.

```toml
[microvm]
firecracker = { path = "bin/firecracker", blake3 = "…" }
kernel = { path = "kernel/vmlinux", blake3 = "…" }
init = { path = "bin/flow-init", blake3 = "…" }

[runner]                     # only for `flow runner`
hub = "https://hub.example:7343"
token = "…"                  # listed in the hub's [flow] runner_tokens
ca = "/path/to/hub-ca.pem"   # when the hub's certificate is self-signed
name = "runner-1"
max_jobs = 3                 # jobs at once (default 1)
memory_mib = 24576           # what they may use together; a lone job may use it all
cpus = 8
```

Setting one up needs Linux x86-64 with KVM and e2fsprogs 1.43 or later:

1. Build `flow` with `cargo build --release -p flow`. Build the guest init
   as a static binary:
   `RUSTFLAGS="-C target-feature=+crt-static" cargo build --release -p flow-init --target x86_64-unknown-linux-gnu`.
2. Download Firecracker v1.17.0 from its GitHub release, and check the
   published SHA-256. Download the guest kernel
   `firecracker-ci/v1.15/x86_64/vmlinux-6.1.155` from Firecracker's CI
   bucket.
3. Run `flow rootfs import alpine-minirootfs-3.24.2-x86_64.tar.gz --sha256 …`.
4. Write `flow.toml`, pinning each file by its `flow hash`.

On OxygenHub, `hub.toml` lists the runner token and the repositories whose
pushes run pipelines:

```toml
[flow]
runner_tokens = ["…"]
repos = ["owner/*"]
```

## Tests

**Changelog:** a commit that changes code adds a line under `## Unreleased` in CHANGELOG.md (Added, Changed, Fixed or Security, with the issue number); versions change only at release. `scripts/changelog-check.sh [PATH...]` checks it for the paths a commit takes (the working copy's changes without any) and runs in the gate; `NO_CHANGELOG=1` for code changes users won't notice.


`cargo test --workspace` runs every test that needs no VM. The microVM
conformance test also runs when `FLOW_TEST_HOME` names a Flow home and
`FLOW_TEST_ROOTFS` is the hash of an Alpine rootfs in it. The test checks
literal argv, network isolation, the read-only rootfs, and that dependents
are skipped after a failure.

## oxygen as a pinned subrepo

`flow` builds against `deps/oxygen`, a subrepo (`oxy submodule`) pinned to an exact state, so builds are reproducible and OxygenFlow can build this repository without network. `make pins` moves it to oxygen's `main` on Earth (push oxygen first); the next commit records the new pin. Format only this workspace, `cargo fmt -p flow -p flow-core -p flow-init`: `--all` also reaches `deps/`. After a fresh clone, `oxy submodule update --init` fetches it.

`/`	Search
`g` `c`	Code
`g` `h`	History
`g` `b`	Bookmarks
`g` `t`	Trust
`y`	Permalink to this state (files and trees)
`?`	This sheet

1	# OxygenFlow
2
3	[![OxygenFlow](https://oxygenhub.io/andymccall/oxygenflow/flow/badge.svg)](https://oxygenhub.io/andymccall/oxygenflow/flow)
4
5	Hermetic pipelines for Oxygen repositories:
6
7	- a typed pipeline in `.flow/pipeline.kdl`
8	- jobs run in isolation tiers, Firecracker microVMs by default
9	- the host writes the receipts
10	- results go to OxygenHub as checks and appear on its Flow tab
11
12	The specification is [docs/spec.md](docs/spec.md). What works today is
13	[docs/m0.md](docs/m0.md).
14
15	## A pipeline
16
17	```kdl
18	schema-version 1
19
20	sandbox "alpine" {
21	runtime "microvm"
22	rootfs "cas://blob/…" // printed by `flow rootfs import`
23	memory-limit "256MiB"
24	network "isolated"
25	}
26
27	job "build" {
28	sandbox "alpine"
29	exec "make"
30	}
31
32	job "test" {
33	sandbox "alpine"
34	depends-on "build"
35	exec "sh" "-eu" "test.sh" // argv, run as written: no shell unless you name one
36	}
37	```
38
39	Each job boots its own microVM:
40
41	- it has no network device
42	- the rootfs is read-only
43	- the workspace is the state being built, read-write at `/src`
44	- the job runs as a user that is not root
45
46	This repository's own pipeline is [.flow/pipeline.kdl](.flow/pipeline.kdl).
47	The stamp at the top of this page is its result on OxygenHub. It shows the
48	run of the state you are viewing, not whatever the bookmark holds now, and
49	it names the state and each job's receipt.
50
51	## Deploying the runner on Earth
52
53	`make deploy-runner` builds `flow` and the static `flow-init`, installs them
54	under `/home/claude/flow` (keeping the previous files as `.prev`), re-pins
55	`flow-init` in `flow.toml` and restarts the `flow-runner` service.
56
57	## A Windows runner
58
59	A Windows machine runs WASM jobs (the other tiers are Linux's). Each
60	release carries `flow-windows-x86_64.exe` (from the `dist` job):
61
62	flow.exe --home C:\flow runner register https://oxygenhub.io --token flowreg_… --name win-1 --label windows
63	flow.exe --home C:\flow runner
64
65	The token comes from the hub (Admin › Runners, or `POST /api/v1/user/runner-token`
66	for a runner of your own repositories). A pipeline sends it work with a
67	`runtime "wasm"` sandbox and `runs-on "windows"`. To keep it running, a
68	Task Scheduler task at start-up with that command (restarting on failure)
69	does; set it to restart on failure, as a runner that leaves for a new build
70	exits 75 (`flow runner update`, spec §11.5.3). To stop it without losing
71	its job, `flow.exe --home C:\flow runner drain` (Windows has no SIGTERM;
72	ending the task kills it). What a Windows runner does not report yet: load
73	and kernel (oxygenflow#83).
74
75	## Commands
76
77	flow check # validate .flow/pipeline.kdl, print jobs and the pipeline hash
78	flow run [JOB…] # run the pipeline on this directory, locally, in microVMs
79	flow rootfs import T.tgz --sha256 HEX # make a rootfs image; prints the `rootfs` line to pin
80	[--env NAME=value …] # with the image's own environment (a Dockerfile's ENV)
81	flow rootfs list
82	flow hash FILE # BLAKE3, for pinning in flow.toml
83	flow runner # claim and run jobs for the hub in flow.toml
84	flow runner update --from FILE [--dry-run] # install a new flow build in place of this one (spec §11.5.3)
85	flow runner drain # the running runner claims nothing more and stops once its jobs are reported
86
87	## A Flow home
88
89	`$FLOW_HOME` (default `~/.local/share/flow`) holds:
90
91	- `flow.toml`
92	- the pinned Firecracker binary, guest kernel and `flow-init`
93	- rootfs images under `cas/`
94	- scratch space
95
96	Every pinned file is checked against its BLAKE3 before use.
97
98	```toml
99	[microvm]
100	firecracker = { path = "bin/firecracker", blake3 = "…" }
101	kernel = { path = "kernel/vmlinux", blake3 = "…" }
102	init = { path = "bin/flow-init", blake3 = "…" }
103
104	[runner] # only for `flow runner`
105	hub = "https://hub.example:7343"
106	token = "…" # listed in the hub's [flow] runner_tokens
107	ca = "/path/to/hub-ca.pem" # when the hub's certificate is self-signed
108	name = "runner-1"
109	max_jobs = 3 # jobs at once (default 1)
110	memory_mib = 24576 # what they may use together; a lone job may use it all
111	cpus = 8
112	```
113
114	Setting one up needs Linux x86-64 with KVM and e2fsprogs 1.43 or later:
115
116	1. Build `flow` with `cargo build --release -p flow`. Build the guest init
117	as a static binary:
118	`RUSTFLAGS="-C target-feature=+crt-static" cargo build --release -p flow-init --target x86_64-unknown-linux-gnu`.
119	2. Download Firecracker v1.17.0 from its GitHub release, and check the
120	published SHA-256. Download the guest kernel
121	`firecracker-ci/v1.15/x86_64/vmlinux-6.1.155` from Firecracker's CI
122	bucket.
123	3. Run `flow rootfs import alpine-minirootfs-3.24.2-x86_64.tar.gz --sha256 …`.
124	4. Write `flow.toml`, pinning each file by its `flow hash`.
125
126	On OxygenHub, `hub.toml` lists the runner token and the repositories whose
127	pushes run pipelines:
128
129	```toml
130	[flow]
131	runner_tokens = ["…"]
132	repos = ["owner/*"]
133	```
134
135	## Tests
136
137	Changelog: a commit that changes code adds a line under `## Unreleased` in CHANGELOG.md (Added, Changed, Fixed or Security, with the issue number); versions change only at release. `scripts/changelog-check.sh [PATH...]` checks it for the paths a commit takes (the working copy's changes without any) and runs in the gate; `NO_CHANGELOG=1` for code changes users won't notice.
138
139
140	`cargo test --workspace` runs every test that needs no VM. The microVM
141	conformance test also runs when `FLOW_TEST_HOME` names a Flow home and
142	`FLOW_TEST_ROOTFS` is the hash of an Alpine rootfs in it. The test checks
143	literal argv, network isolation, the read-only rootfs, and that dependents
144	are skipped after a failure.
145
146	## oxygen as a pinned subrepo
147
148	`flow` builds against `deps/oxygen`, a subrepo (`oxy submodule`) pinned to an exact state, so builds are reproducible and OxygenFlow can build this repository without network. `make pins` moves it to oxygen's `main` on Earth (push oxygen first); the next commit records the new pin. Format only this workspace, `cargo fmt -p flow -p flow-core -p flow-init`: `--all` also reaches `deps/`. After a fresh clone, `oxy submodule update --init` fetches it.

Keyboard