OXYGENHUB

Compare newer-pipeline … main

⇄

131 changes in main not in newer-pipeline. 77 files differ between the two states.

Release 0.5.3 mainv0.5.3 head
Andy McCall via agent:claude@andymccall agent ·
A changelog check on the WASM tier, a job in Flow's pipeline mainv0.5.3 head
Andy McCall via agent:claude@andymccall agent ·
Cargo.lock for the oxygen pin at 3ce2d633 (refs andymccall/oxygenhub#355) mainv0.5.3 head
Andy McCall via agent:claude@andymccall agent ·
OxygenFlow 0.5.2 v0.5.2
Andy McCall via agent:claude@andymccall agent ·
OxygenFlow 0.5.0 v0.5.0
Andy McCall via agent:claude@andymccall agent ·
Offering held images: a hub that answers 405 predates it too, and is left quiet (refs #94) mainv0.5.0v0.5.1v0.5.2v0.5.3 head
Andy McCall via agent:claude@andymccall agent ·
Design: images only runners hold, moved by the swarm (refs #94)
Andy McCall via agent:claude@andymccall agent ·
CHANGELOG.md merge=union: entries added side by side keep both (oxygen#145) mainv0.5.0v0.5.1v0.5.2v0.5.3 head
Andy McCall via agent:claude@andymccall agent ·
Android image README: the job needs network "loopback" for Gradle (refs #19) mainv0.5.0v0.5.1v0.5.2v0.5.3 head
Andy McCall via agent:claude@andymccall agent ·
Android image: warm the coverage task's dependencies too (refs #19) mainv0.5.0v0.5.1v0.5.2v0.5.3 head
Andy McCall via agent:claude@andymccall agent ·
Pin oxygen main: author.for (refs andymccall/oxygen#82)
Andy McCall via agent:claude@andymccall agent ·
flow runner bench says when load is not measured
Andy McCall via agent:claude@andymccall agent ·
OxygenFlow 0.4.0 v0.4.0
Andy McCall via agent:claude@andymccall agent ·
Docs: the rollback report the hub holds a build on; runner updates' status mainv0.4.0v0.5.0v0.5.1v0.5.2v0.5.3 head
Andy McCall via agent:claude@andymccall agent ·
A Windows runner reports its Windows version
Andy McCall via agent:claude@andymccall agent ·
Design: runners update themselves (refs #84)
Andy McCall via agent:claude@andymccall agent ·
Checkpoints: a handed-back job resumes (refs #77)
Andy McCall via agent:claude@andymccall agent ·
A leaving runner finishes what fits its notice (refs #76)
Andy McCall via agent:claude@andymccall agent ·
A runner that leaves hands its jobs back (refs #76)
Andy McCall via agent:claude@andymccall agent ·
The x86_64 seccomp block builds for Android too (refs oxygenhub-android#8) mainv0.4.0v0.5.0v0.5.1v0.5.2v0.5.3 head
Andy McCall via agent:claude@andymccall agent ·
Pin the Rust image with wasmtime's crates and cargo-nextest (refs #75 #81) v0.4.0v0.5.0
Andy McCall via agent:claude@andymccall agent ·
OxygenFlow 0.3.1: pins oxy 0.6.0 v0.3.1
Andy McCall via agent:claude@andymccall agent ·
Pin oxygen with keyless agents' grant signers carried (oxygen#134)
Andy McCall via agent:claude@andymccall agent ·
OxygenFlow 0.3.0 v0.3.0
Andy McCall via agent:claude@andymccall agent ·
A runner's own metrics (refs #35)
Andy McCall via agent:claude@andymccall agent ·
Pin oxygen 0.5.1
Andy McCall via agent:claude@andymccall agent ·
Pin oxygen: keyless agents' release tags (oxygen#129)
Andy McCall via agent:claude@andymccall agent ·
Prune build caches before they are kept
Andy McCall via agent:claude@andymccall agent ·
OxygenFlow 0.2.0, its first tagged release v0.2.0
Andy McCall via agent:claude@andymccall agent ·
Pin oxygen ec1dc9bd: Backend::vet_push for hosts (oxygenhub#268)
Andy McCall via agent:claude@andymccall agent ·
Changelog per change: a check in the gate, and Unreleased caught up
Andy McCall via agent:claude@andymccall agent ·
Build caches for the syscall tier
Andy McCall via agent:claude@andymccall agent ·
Pin oxygen ee497ab1: fetches carry attestation signers' identities
Andy McCall via agent:claude@andymccall agent ·
Spec: agents and approval gates (oxygenhub#229, #258) mainv0.2.0v0.3.0v0.3.1v0.4.0v0.5.0v0.5.1v0.5.2v0.5.3 head
Andy McCall via agent:claude@andymccall agent ·
Pin oxygen 89ffcd8d: readers never wait (oxygenhub#249)
Andy McCall via agent:claude@andymccall agent ·
Pin oxygen: no request per pack on open
Andy McCall via agent:claude@andymccall agent ·
Pin oxygen: listing counters
Andy McCall via agent:claude@andymccall agent ·
Pin oxygen: pack listings kept a moment
Andy McCall via agent:claude@andymccall agent ·
Pin oxygen with transparency-log witnesses
Andy McCall via agent:claude@andymccall agent ·
Pin oxygen: a new log key is a note
Andy McCall via agent:claude@andymccall agent ·
Pin oxygen with oxy tlog verify
Andy McCall via agent:claude@andymccall agent ·
Pin oxygen: fetch checks the host's transparency log
Andy McCall via agent:claude@andymccall agent ·
Pin oxygen with the transparency log library
Andy McCall via agent:claude@andymccall agent ·
Pin oxygen: grant tokens reach hubs
Andy McCall via agent:claude@andymccall agent ·
Pin oxygen: subject grants and certified pushes
Andy McCall via Claude Code agent ·
Pin oxygen: receipts may carry a certificate
Andy McCall via Claude Code agent ·
Pin oxygen: certificates for short-lived keys
Andy McCall via Claude Code agent ·
docs/workspaces.md: paused until the promised work is delivered mainv0.2.0v0.3.0v0.3.1v0.4.0v0.5.0v0.5.1v0.5.2v0.5.3 head
Andy McCall via Claude Code agent ·
pipeline.kdl: priority lowers a job's place in the queue, never raises it v0.2.0v0.3.0v0.3.1
Andy McCall via Claude Code agent ·
Pin oxygen 0.4.4
Andy McCall via Claude Code agent ·
Pin oxygen: strata counts its requests by kind
Andy McCall via Claude Code agent ·
Pin oxygen a23729f0: shared cloud store clients
Andy McCall via Claude Code agent ·
Runner: a heartbeat while it has no room to claim
Andy McCall via Claude Code agent ·
tofu image: curl, jq, xz, sfdisk and debugfs for baking VM images mainv0.2.0v0.3.0v0.3.1v0.4.0v0.5.0v0.5.1v0.5.2v0.5.3 head
Andy McCall via Claude Code agent ·
rust image: PostgreSQL and nss_wrapper for OxygenHub's Postgres tests mainv0.2.0v0.3.0v0.3.1v0.4.0v0.5.0v0.5.1v0.5.2v0.5.3 head
Andy McCall via Claude Code agent ·
Pin oxygen at the approval index (oxygenhub#235)
Andy McCall via Claude Code agent ·
Environments: environment "prod" on a job
Andy McCall via Claude Code agent ·
OpenTofu image: TFLint and its Google ruleset mainv0.2.0v0.3.0v0.3.1v0.4.0v0.5.0v0.5.1v0.5.2v0.5.3 head
Andy McCall via Claude Code agent ·
The OpenTofu image for deploy jobs
Andy McCall via Claude Code agent ·
Runners read an unrecorded image's architecture and decline another mainv0.2.0v0.3.0v0.3.1v0.4.0v0.5.0v0.5.1v0.5.2v0.5.3 head
Andy McCall ·
ci: keep clippy's and test's target from their last passing run (refs #28) mainv0.2.0v0.3.0v0.3.1v0.4.0v0.5.0v0.5.1v0.5.2v0.5.3 head
Andy McCall ·
feat(cache): directories keep their times too; FLOW_HEADS only without OXY_BUILD (refs #52) mainv0.2.0v0.3.0v0.3.1v0.4.0v0.5.0v0.5.1v0.5.2v0.5.3 head
Andy McCall ·
flow rootfs push: ask about missing chunks 20,000 at a time mainv0.2.0v0.3.0v0.3.1v0.4.0v0.5.0v0.5.1v0.5.2v0.5.3 head
Andy McCall ·
flow rootfs push and pull use the login oh already saved mainpush-uses-loginv0.2.0v0.3.0v0.3.1v0.4.0v0.5.0v0.5.1v0.5.2v0.5.3 head
Andy McCall ·
hide whitespace changes
added .config/nextest.toml +8 −0
1 +
# The CI profile (oxygenflow#82): every test runs, and a JUnit report is
2 +
# written to target/nextest/ci/junit.xml for the hub's test report
3 +
# (the job's `junit` artifact).
4 +
[profile.ci]
5 +
fail-fast = false
6 +
7 +
[profile.ci.junit]
8 +
path = "junit.xml"
added .flow/checks.wasm +0 −0

No line changes.

removed .flow/jobs/layout.sh +0 −5
1 -
# The workspace has the crates and documents it should.
2 -
for f in Cargo.toml docs/m0.md crates/flow-core/src/pipeline.rs crates/flow-init/src/main.rs crates/flow/src/microvm.rs; do
3 -
    test -f "$f" || { echo "missing: $f" >&2; exit 1; }
4 -
    echo "found $f"
5 -
done
removed .flow/jobs/size.sh +0 −5
1 -
# Lines of Rust per crate.
2 -
for c in crates/*/; do
3 -
    n=$(find "$c" -name '*.rs' -exec cat {} + | wc -l)
4 -
    printf '%-24s %6d lines\n' "$c" "$n"
5 -
done
removed .flow/jobs/summary.sh +0 −1
1 -
echo "$PROJECT: every check passed on this state."
removed .flow/jobs/todos.sh +0 −4
1 -
# Notes left in the code, for the log (they do not fail the job).
2 -
n=$(grep -rn -E 'TODO|FIXME|XXX' crates | wc -l)
3 -
echo "$n TODO/FIXME notes in crates/"
4 -
grep -rn -E 'TODO|FIXME|XXX' crates | head -20 || true
modified .flow/pipeline.kdl +100 −15
…
10 unchanged lines
11 11
    network "isolated"
12 12
}
13 13
14 +
// The cheap checks as one WASI module (#96): crates/flow-checks, built by
15 +
// scripts/checks-wasm.sh and committed as .flow/checks.wasm (the fmt job
16 +
// fails if it is not what the source builds). No image and no KVM, so any
17 +
// runner with the WASM tier takes them: the phone, plex, windows-1.
18 +
sandbox "wasm" {
19 +
    runtime "wasm"
20 +
    memory-limit "64MiB"
21 +
}
22 +
14 23
job "layout" {
15 -
    sandbox "alpine"
16 -
    exec "sh" "-eu" ".flow/jobs/layout.sh"
24 +
    sandbox "wasm"
25 +
    exec ".flow/checks.wasm" "layout" "Cargo.toml" "docs/m0.md" "crates/flow-core/src/pipeline.rs" "crates/flow-init/src/main.rs" "crates/flow/src/microvm.rs"
17 26
}
18 27
19 28
job "todos" {
20 -
    sandbox "alpine"
21 -
    scope "crates/**"
22 -
    exec "sh" "-eu" ".flow/jobs/todos.sh"
29 +
    sandbox "wasm"
30 +
    scope "crates/**" ".flow/checks.wasm"
31 +
    exec ".flow/checks.wasm" "todos" "crates"
32 +
}
33 +
34 +
// The changelog's shape, and a section for the version Cargo.toml says, so
35 +
// a release never finds its notes missing.
36 +
job "changelog" {
37 +
    sandbox "wasm"
38 +
    scope "CHANGELOG.md" "Cargo.toml" ".flow/checks.wasm"
39 +
    exec ".flow/checks.wasm" "changelog" "CHANGELOG.md" "Cargo.toml"
23 40
}
24 41
25 42
job "size" {
26 -
    sandbox "alpine"
27 -
    scope "crates/**"
28 -
    exec "sh" "-eu" ".flow/jobs/size.sh"
43 +
    sandbox "wasm"
44 +
    scope "crates/**" ".flow/checks.wasm"
45 +
    exec ".flow/checks.wasm" "size" "crates"
29 46
}
30 47
31 48
job "isolation" {
…
5 unchanged lines
37 54
// vendored crates. oxygen comes from the pinned subrepo in deps/.
38 55
sandbox "rust" {
39 56
    runtime "microvm"
40 -
    rootfs "cas://blob/7705d699a4fc06862e4297b7069396cf3b5b35b1d3f15191c46dc2031af17133"
57 +
    rootfs "cas://blob/e52c41a173956b6bacc270eb81208a1b118b652412b84d15fa24708e6f8ec392"
41 58
    memory-limit "8GiB"
42 59
    cpu-limit 4
43 60
    network "loopback"
…
9 unchanged lines
53 70
54 71
sandbox "rust-small" {
55 72
    runtime "microvm"
56 -
    rootfs "cas://blob/7705d699a4fc06862e4297b7069396cf3b5b35b1d3f15191c46dc2031af17133"
73 +
    rootfs "cas://blob/e52c41a173956b6bacc270eb81208a1b118b652412b84d15fa24708e6f8ec392"
57 74
    memory-limit "1GiB"
58 75
    cpu-limit 1
59 76
    network "isolated"
…
1 unchanged lines
61 78
62 79
job "fmt" {
63 80
    sandbox "rust-small"
64 -
    // This workspace only: --all would reach the pinned oxygen in deps/.
65 -
    exec "cargo" "fmt" "--check" "-p" "flow" "-p" "flow-core" "-p" "flow-init" "-p" "flow-casfs"
81 +
    // This workspace only (--all would reach the pinned oxygen in deps/),
82 +
    // and the checks' committed WASI module is what its source builds (#96).
83 +
    exec "sh" "-euc" "cargo fmt --check -p flow -p flow-core -p flow-init -p flow-casfs -p flow-checks && sh scripts/checks-wasm.sh --check"
66 84
}
67 85
68 86
job "clippy" {
…
3 unchanged lines
72 90
        source "ephemeral://disk"
73 91
        target "/src/target"
74 92
        size "16GiB"
93 +
        // Kept from this job's last passing run (oxygenflow#28): cargo
94 +
        // builds only what changed since.
95 +
        cache "target"
96 +
        key "Cargo.lock"
97 +
    }
98 +
    // One build identity for every CI state, so the cached target stays
99 +
    // fresh from one push to the next (oxygenflow#52).
100 +
    env {
101 +
        OXY_BUILD "ci"
75 102
    }
76 103
    exec "sh" ".flow/warm-target.sh" "cargo" "clippy" "--workspace" "--all-targets" "--locked" "--" "-D" "warnings"
77 104
}
…
5 unchanged lines
83 110
        source "ephemeral://disk"
84 111
        target "/src/target"
85 112
        size "16GiB"
113 +
        // Kept from this job's last passing run (oxygenflow#28): cargo
114 +
        // builds only what changed since.
115 +
        cache "target"
116 +
        key "Cargo.lock"
86 117
    }
87 -
    exec "sh" ".flow/warm-target.sh" "cargo" "test" "--workspace" "--locked"
118 +
    // One build identity for every CI state, so the cached target stays
119 +
    // fresh from one push to the next (oxygenflow#52).
120 +
    env {
121 +
        OXY_BUILD "ci"
122 +
    }
123 +
    // The tests under cargo-nextest and cargo-llvm-cov (#82): a test report
124 +
    // (JUnit, kept whether or not they pass) and line coverage (LCOV) from
125 +
    // one run, for the hub's Tests and Coverage tabs.
126 +
    exec "sh" ".flow/warm-target.sh" "cargo" "llvm-cov" "nextest" "--workspace" "--locked" "--profile" "ci" "--lcov" "--output-path" "/src/target/lcov.info"
127 +
    output {
128 +
        artifact "junit" {
129 +
            path "/src/target/nextest/ci/junit.xml"
130 +
            when "always"
131 +
        }
132 +
        artifact "coverage" {
133 +
            path "/src/target/lcov.info"
134 +
        }
135 +
    }
88 136
}
89 137
138 +
// Runner builds for releases (oxygenhub#322): flow-linux-ARCH for x86_64,
139 +
// aarch64 and riscv64, which runners that update themselves are offered.
140 +
job "dist" {
141 +
    sandbox "rust"
142 +
    depends-on "fmt"
143 +
    scope "Cargo.toml" "scripts/dist.sh"
144 +
    mount {
145 +
        source "ephemeral://disk"
146 +
        target "/src/target"
147 +
        size "24GiB"
148 +
        cache "target"
149 +
        key "Cargo.lock"
150 +
    }
151 +
    env {
152 +
        DIST_FLAT "/src/target/dist-flat"
153 +
        ZIG_GLOBAL_CACHE_DIR "/src/target/zig-cache"
154 +
    }
155 +
    exec "bash" "scripts/dist.sh"
156 +
    output {
157 +
        artifact "flow-linux-x86_64" {
158 +
            path "/src/target/dist-flat/flow-linux-x86_64"
159 +
        }
160 +
        artifact "flow-linux-aarch64" {
161 +
            path "/src/target/dist-flat/flow-linux-aarch64"
162 +
        }
163 +
        artifact "flow-linux-riscv64" {
164 +
            path "/src/target/dist-flat/flow-linux-riscv64"
165 +
        }
166 +
        artifact "flow-windows-x86_64.exe" {
167 +
            path "/src/target/dist-flat/flow-windows-x86_64.exe"
168 +
        }
169 +
        artifact "SHA256SUMS" {
170 +
            path "/src/target/dist-flat/SHA256SUMS"
171 +
        }
172 +
    }
173 +
}
174 +
90 175
job "summary" {
91 -
    sandbox "alpine"
176 +
    sandbox "wasm"
92 177
    depends-on "layout" "todos" "size" "isolation" "clippy" "test"
93 178
    env {
94 179
        PROJECT "OxygenFlow"
95 180
    }
96 -
    exec "sh" "-eu" ".flow/jobs/summary.sh"
181 +
    exec ".flow/checks.wasm" "summary"
97 182
}
added .oxyattributes +3 −0
1 +
# oxygen#145: two changes that each add a changelog entry in the same
2 +
# place keep both, without a conflict.
3 +
CHANGELOG.md merge=union
modified .oxyignore +4 −0
…
2 unchanged lines
3 3
images/rust/vendor/
4 4
images/rust/*.tar
5 5
images/rust/npm-cache/
6 +
images/android/dl/
7 +
images/android/app/
8 +
images/android/vendor/
9 +
images/android/*.tar
modified CHANGELOG.md +114 −2
1 1
# Changelog
2 2
3 -
OxygenFlow has no release yet: the runner on Earth runs `main`. This file starts on 2026-10-01; earlier history is in `oxy log`.
3 +
0.2.0 is OxygenFlow's first tagged release; until then the runners ran `main`. This file starts on 2026-10-01; earlier history is in `oxy log`.
4 +
5 +
## 0.5.3 (2026-11-04)
6 +
7 +
- **Added: a `changelog` check on the WASM tier.** `.flow/checks.wasm changelog CHANGELOG.md [MANIFEST]` checks the changelog's shape (`## Unreleased` first if at all, then `## X.Y.Z (YYYY-MM-DD)` sections newest first, none empty) and that the version a `Cargo.toml`, `.properties` or `package.json` says has its section, so a release never finds its notes missing. It is a job of OxygenFlow's pipeline, and of OxygenHub's, Oxygen's and the Android app's, any runner with the WASM tier taking it.
8 +
- **Added: `flow deps check` (#101).** Jobs have no network, so every crate a `Cargo.lock` needs must be vendored in the build image at that version. `flow deps check [Cargo.lock]` says whether it is, before the push: each missing crate with the versions the image does have and the `cargo update --precise` that pins one, exit 1. It reads the image's crate list (`--list`), a vendor directory (`--vendor`), or the lock files the image is built from (`--image-lock`, repeatable); inside a job, the image's own. `flow deps list` prints the set.
9 +
- **Changed: the build image publishes its crate list.** `images/rust/build.sh` writes `crates.txt` (`NAME VERSION` a line) beside the tarball and at `/opt/crates.txt` in the image.
10 +
- **What a runner is running, for its host (oxygenhub-android#8):** `flow::runner::running()` lists the jobs this runner is running now (`owner/name #run job`), each from just before it starts until it ends, so the Android app can show them on its card and notification.
11 +
- **`flow` builds and runs on FreeBSD (#99):** FreeBSD 15.1 with pkg's Rust 1.98 builds `flow` with the WASM tier (wasmtime runs there); the `noatime` check is Linux's only, as on Haiku. A FreeBSD machine can be a WASM runner (`flow tiers`: wasm available; the microvm and syscall tiers say they are Linux's).
12 +
13 +
## 0.5.2 (2026-10-29)
14 +
15 +
- **Runner builds through the swarm (#84):** a runner in a swarm fetches an offered build as the image its BLAKE3 names, from its site's peers and the hub together, as it fetches images, and checks its size and BLAKE3 before installing it; over HTTP as before when it is in no swarm, the hub has not shared the build yet, or the swarm cannot get it. The hub shares a build the first time it offers it (OxygenHub 0.9.15), so a site's runners fetch it from each other and the hub serves it about once per site. The build stays in the runner's images for its peers.
16 +
- **`flow` builds on Haiku (#91):** with `--no-default-features`, `flow` now builds natively on Haiku R1/beta6 (HaikuPorts' Rust 1.94): the syscall tier, the guest's job code and the load average are Linux's (and Android's) only, and the `noatime` check takes reads as recorded where statvfs has no such flag. It checks pipelines and runs `flow tiers`, which says no tier runs there yet: the host tier (#90) is next.
17 +
- **The WASM tier can be left out of a build (#91):** the `wasm` cargo feature (on by default) carries wasmtime; `--no-default-features` builds `flow` for systems wasmtime does not run on, such as Haiku, whose runners then offer no WASM tier and say why.
18 +
19 +
## 0.5.1 (2026-10-28)
20 +
21 +
- **Sealed alerts (`flow_core::alert`, oxygenhub-android#16):** what a phone is told about a job is encrypted to an X25519 key the phone made (a fresh key per alert, HKDF-SHA256, ChaCha20-Poly1305), so the push service carrying it sees only ciphertext. `seal` for the hub, `open` and `new_secret`/`public_key` for the phone, from crates already in the tree (curve25519-dalek, ring).
22 +
23 +
- **`flow runner register --ca` keeps the CA in the Flow home (#98):** it wrote the path as given, so `--ca hub-ca.pem` left `ca = "hub-ca.pem"` in runner.toml and a service started elsewhere could not find it. Register now copies the file to the home's `hub-ca.pem` and records that copy's absolute path, and a relative `ca` already in runner.toml is read against the home.
24 +
25 +
- **The cheap checks run as WASM (#96):** `layout`, `todos`, `size` and `summary` are one WASI module, `crates/flow-checks`, built by `scripts/checks-wasm.sh` and committed as `.flow/checks.wasm`, so they need no image or microVM and any runner with the WASM tier takes them (the phone, plex, windows-1); each runs in under 100 ms. The fmt job fails unless the committed module is what the source builds, and the build image has the wasm32-wasip1 target for that.
26 +
27 +
## 0.5.0 (2026-10-27)
28 +
29 +
- **A job's report says where its time went (#35).** The finish report carries `phases` in milliseconds: `image` (fetching its image, when the runner lacked it), `prepare` (its workspace and disks), `run` (the sandbox from start to stop) and `finish` (outputs, caches and receipt), so a slow job shows whether it waited on an image, a fetch or itself.
30 +
- **The swarm reads images in order (#97).** A fetcher took equally rare chunks at random, so a peer read its whole image in random order, a seek per chunk on a spinning disk: plex-1 got 2.3 GiB from earth-1 at about 2 MB/s. Neighbouring chunks now come in runs, shuffled so runners starting together still spread, and the picker no longer rescans what it has: picking a 142,244-chunk image went from 19 s to 0.35 s.
31 +
- **The swarm moves an image nobody pushed (#94).** A runner that serves its peers tells the hub, at start and every ten minutes, which images it holds, and sends the manifest of each the hub keeps none of: the manifest alone, not the image. With a hub that keeps such manifests (oxygenhub#342), runners in the same site fetch the image from it. A fetch told the hub holds no chunks asks peers alone, and fails plainly when none can serve it. Design: docs/jetstream.md §8.1.1.
32 +
- **A WASM job stopped while its module compiles stops (#93).** The time limit or a cancel moved the engine's epoch once; a store made after that set its deadline past it, so the job ran on forever (seen as a test spinning 35 minutes under load). Once stopped, the epoch now moves every tick until the job ends.
33 +
- **A runner claims only what its host can start (#92).** An idle runner told the hub it had no memory limit, so it was offered sandboxes bigger than its machine, fetched their images, and declined them one by one (mars-1, 7 GB, and the 8 GiB jobs). A claim now asks for the budget's room capped at the host's free memory.
34 +
- **`flow rootfs push` waits for the hub to check a big image (oxygenhub#341).** The hub reads the whole image before it accepts the manifest; the manifest request now waits up to an hour, not the 120 s every other request gets.
35 +
- **The Android image (#19).** `images/android`: OpenJDK 17, the Android SDK (command-line tools 23.0, platform 37.0, build-tools 36.0.0, NDK 30.0), Gradle 9.6.0, Rust with the Android targets and cargo-ndk, and the OxygenHub app's crates and Gradle dependencies, so the app builds and tests offline. Each download is checked against its published hash.
36 +
- **`flow runner bench` says when load is not measured (#83):** a Windows runner printed `load 0.00`; it now says `load not measured here`. Its kernel line reads `Windows 10.0.26300` (checked with 0.4.0 in the Windows VM).
37 +
## 0.4.0 (2026-10-26)
38 +
39 +
- **A Windows runner reports its Windows version (#83):** as its kernel fact (`Windows 10.0.26100`, from `RtlGetVersion`), shown on the hub's runner page and in `flow runner bench`. Built and linted for Windows; not yet run there.
40 +
- **`flow runner drain` (#83):** asks the home's running runner to drain, as SIGTERM does on Linux, through a `drain.request` file its loop reads: the way to stop a Windows runner without losing its job. A starting runner clears a stale request.
41 +
- **A Windows runner updates itself (#83, #85):** `flow runner update` on Windows checks a PE build, renames `flow.exe` to `flow.prev.exe` and `flow.new.exe` into its place with `MoveFileExW` (retrying while a virus scanner holds a new file; a running program cannot rename its own file, so a copy of it, `flow __rename`, does), tells the running runner through an `update.request` file in its home (no SIGUSR1 there), and the runner leaves and exits 75 so a Task Scheduler task set to restart on failure starts the new build, which proves itself by reaching the hub. Tried in the Windows VM: install, leave, exit 75, restart, proven. A rollback runs the restored build and ends with it. Spec §11.5.3.
42 +
- **`flow` builds for Windows (#83, step 2 begun):** `cargo build --target x86_64-pc-windows-gnu` gives a `flow.exe` that offers the WASM tier: the syscall tier, the guest helpers and the egress proxy are Linux's only (a Windows build says the syscall tier "runs only on Linux"), cache pruning by reads stays on Linux, and file modes, links and sizes go through a small `portable` module (modes are not kept on Windows). Its claims report free disk (`GetDiskFreeSpaceExW`); load and kernel are not measured there yet, and a Windows runner does not update itself. `dist` builds `flow-windows-x86_64.exe` too. Nothing changes on Linux or Android. Tried in the Windows VM on Earth: `flow.exe tiers` offers wasm, a local `flow run` of a 120-step WASM job passes, and registered as runner `windows-1` (label `windows`) it ran andymccall/windows-runner-demo's job from the hub (run #1). Helpers start from the running binary's path on Windows (`/proc/self/exe` on Linux and Android).
43 +
- **The syscall tier's loopback reaches TCP (#73):** with `network "loopback"`, a job may open IPv4 and IPv6 sockets, which in its own network namespace reach only its `lo`, so a server it starts on 127.0.0.1 answers it (a conformance suite over an emulator's HTTP control port, which skipped every test on narvi-1). With `network "isolated"` sockets other than `AF_UNIX` are still refused.
44 +
- **Runner builds in releases (oxygenhub#322):** a `dist` job (`scripts/dist.sh`, on a change to `Cargo.toml` or the script) builds `flow` stripped for Linux x86_64, aarch64 and riscv64 (glibc 2.28 and later) as `flow-linux-ARCH`, named by the architecture a runner reports, with `SHA256SUMS`: the files a release carries for runners that update themselves.
45 +
- **Runners ask the hub for their update (oxygenhub#322):** once the hub has answered a claim, and every 15 minutes, a runner asks `GET /api/v1/flow/runner/update`; when the hub offers a build, it downloads it (kept in the home's `updates/`), refuses it unless its size and BLAKE3 are those the hub attests, and installs it as `flow runner update` does, leaving within 600 s, or for a dry run checks it and says what would happen. What it did goes back to the hub once per outcome, and a proof or a rollback is told at the next start. `[runner.update] allow = false` refuses and says so; a hub without runner updates (404) is simply asked again later.
46 +
- **The runner update protocol's types (oxygenhub#322):** `UpdateAnswer`, `UpdatePolicy`, `UpdateOffer` and `UpdateReport` in flow-core, for `GET`/`POST /api/v1/flow/runner/update`: the policy that applies to a runner once every level is applied, and the release asset it should run.
47 +
- **`flow runner update --from FILE` (#85):** the runner's side of self-updating runners (docs/jetstream.md §8.2). It checks a new build (this machine's target, `--version`, a tier it can run here), keeps the installed one as `flow.prev`, swaps the new one in with a single rename, and tells the running runner (SIGUSR1, its pid in `runner.pid`) to leave within `--notice` seconds (600 by default): jobs that fit finish, the rest are handed back with their checkpoints, and the unit starts the new build. The new build proves itself by getting an answer from the hub; four starts or ten minutes without one put `flow.prev` back. `--dry-run` changes nothing; `[runner.update] allow = false` in `flow.toml` refuses every update; `updates.log` in the home records each install, proof and rollback. Spec §11.5.3.
48 +
- **A local `flow run` of a WASM or syscall job copies the workspace whole:** it copied into a directory it never made, so a file at the top of the workspace (a README) failed with `No such file or directory`.
49 +
- **Checkpoints: a handed-back job resumes (#77):** `checkpoint "/src/…"` on a job names directories that hold its progress; a runner handing the job back packs and keeps them (`POST …/checkpoint`), and the next attempt's runner restores them before running the same exec, so it carries on rather than starting again. The receipt lists the checkpoint among its inputs. WASM and syscall tiers for now. Spec §11.5.2.
50 +
- **A leaving runner finishes what fits its notice (#76):** a claim carries `expected_ms` (the median of the job's recent passing runs, from OxygenHub 0.9.4); `runner::leave_within(reason, notice)` finishes and reports a job whose expected rest fits the notice and hands back the others, and any still running when it is up. `runner::leave` is the same with no notice. Spec §11.5.2.
51 +
- **OxygenFlow's own CI writes a test report and coverage (#82):** its test job runs under cargo-nextest and cargo-llvm-cov (both in the Rust image, #81), keeping a JUnit report (whether or not the tests pass) and LCOV coverage for the hub's Tests and Coverage tabs; `.config/nextest.toml` has the `ci` profile.
52 +
- **A runner that leaves hands its jobs back (#76):** `runner::leave(reason)` (the runner as a library, as the Android app runs it) stops claiming, stops the jobs it holds, sends what they logged and hands them back with `POST /api/v1/flow/jobs/{id}/release`; the hub (OxygenHub #312) queues them again ahead of new work for another runner, and hands back the jobs of a runner silent for 3 minutes the same way. Spec §11.5.2.
53 +
- **A runner reads its load and kernel without /proc:** `sysinfo` and `uname`, the same numbers; an Android app may not read `/proc/loadavg` or `/proc/sys/kernel/osrelease`, so the phone runner was refused, and logged, at every claim (oxygenhub-android#8).
54 +
- **Built on oxygen's main with #140 and #141:** the runner's store puts objects in place where hard links are refused (Android, so the phone runner can clone), and the `deps/oxygen` pin conflict two replacements in one change left on main is settled with `oxy resolve deps/oxygen`.
55 +
- **The WASM tier (#75):** `runtime "wasm"` runs WASI preview 1 modules with wasmtime 49 on any runner (`flow tiers` offers `wasm` everywhere): no rootfs, no network, the job's files at `/src`, its mounts at their targets, the memory limit enforced by wasmtime. `exec` names a module or a glob of them (`target/wasm32-wasip1/debug/deps/*.wasm` runs each test binary in turn); the receipt names wasmtime's version and every module's hash. The tier phones and browsers will run (docs/handover.md). Secrets and identities are not offered in it yet.
56 +
- **The runner as a library, for the phone (oxygenhub-android#8):** `flow` is now a library and the command built on it, and the whole runner builds for Android (`aarch64-linux-android`); `runner::serve_embedded` and `runner::stop` run it inside an app and drain it. There the WASM tier runs modules in a thread rather than a child process (also with `FLOW_WASM_IN_PROCESS`), streaming their output from pipes, stopping them at the time limit or when the run is cancelled through wasmtime's epoch, and reporting the most memory they asked for. Two portability fixes on the way (the seccomp filter's BPF opcodes, for arm64 and x86_64, and an ioctl request type).
57 +
- **cargo-nextest in the Rust image (#81):** for JUnit test reports beside `cargo llvm-cov`'s coverage.
58 +
59 +
## 0.3.1 (2026-10-25)
60 +
61 +
- **Runners read repositories with oxy 0.6.0:** a tag withdrawn on the hub (oxygenhub#301) is dropped when a runner fetches, a keyless agent's changes verify in a repository that pins its owner's newest key (oxygen#135) or has never seen it (oxygen#134), and shallow clones of big repositories work (oxygen#137). The runner itself is unchanged.
62 +
63 +
## 0.3.0 (2026-10-21)
64 +
65 +
- **A runner's own metrics (#35):** `metrics = "127.0.0.1:9464"` under `[runner]` serves Prometheus text at `/metrics`: claims, declines, finished jobs by result, the seconds they ran, the jobs running now, version and uptime, optionally behind `metrics_token`. Off unless set.
66 +
- **Built with oxy 0.5.1.** A runner reads repositories whose release tags a keyless agent made (oxygen#129), and declares `certified-tags` to the hub (oxygen#131).
67 +
- **Build caches are pruned before they are kept (#72, spec §5.4).** A kept cache no longer grows until the job's disk is full. When a passing job's cache holds more than `cache_prune_percent` of its mount's size (runner config, default 60), the files the job did not read, write or share a cargo unit hash with are removed, oldest first, until it is under; one still over is not kept, and the next run starts empty. A microVM's guest init prunes its disk before it powers off; the runner prunes the syscall tier's directories. The job's log says how much was kept and pruned, or why the cache was not kept.
68 +
69 +
## 0.2.0 (2026-10-20)
70 +
71 +
- **Approval gates and environments (#57, #59, oxygenhub#235, spec §5.3.2, §5.3.3).** `approval { by "maintain"; others-only; expires "24h" }` on a job makes the hub hold it until a person with that role (write, maintain by default, or admin) approves it, and with `others-only` not the person who pushed. `approval { sealed #true }` asks for an approval signed with the approver's own key (`oxy seal --flow`), the default for a job that names an `environment "prod"`; the hub names the environment in the job's identity tokens only once it is approved. Both are part of the job's hash.
72 +
73 +
- **Artifacts between jobs (#58, spec §5.3.1).** `input { artifact "x" from "job"; path "/src/x" }` takes an artifact from a job named in `depends-on`, up to 8 per job. The runner fetches each from the hub, refuses the job when the blob is not the one the producer's receipt names, writes it at its path and lists it in the receipt's inputs.
74 +
75 +
- **Allow-listed egress (#55, spec §4.3.1).** `network "egress" { allow "host" "*.example.com" }` lets a microVM job reach those names on port 443 and nothing else. The VM still has no network device: a proxy on the runner takes only `CONNECT host:443` for an allowed name, refuses non-public addresses and logs every decision in the job's log; the job gets `HTTPS_PROXY`. Off unless a runner sets `egress = true` under `[runner]`; such jobs go only to runners that do.
76 +
77 +
- **Receipts signed with a key that dies with the job (oxygenhub#241).** With `keyless = true` under `[runner]`, the runner makes a fresh key just before signing a job's receipt, has the hub certify it with the job's token, signs, attaches the certificate and drops the key. If the hub cannot certify, the runner's own key signs and the job's log says why.
78 +
79 +
- **A pipeline may lower its jobs' priority (oxygenhub#244).** `priority "low"` (or `"normal"`) on a job, or at the top of the file for every job, puts it behind other work in the hub's queue. `high` and `urgent` are refused with the reason: they are the hub's to give.
80 +
81 +
- **Triggers (#24).** `on { push; tag; manual; schedule }` for the pipeline and each job: `push` and `tag` take globs, `manual` is Run pipeline, and `schedule` takes five-field cron lines. A job's own `on` wins over the pipeline's; without one a job runs on every push and by hand, as before.
82 +
83 +
- **Runner labels (oxygenhub#41).** A sandbox or a job may say `runs-on "LABEL"`; a job goes only to a runner that has every label it and its sandbox name (`flow runner register --label`).
84 +
85 +
- **Images by hash (#39, #44).** `flow rootfs push HASH` sends an image to a hub, only the chunks it lacks, and `flow rootfs pull HASH` fetches one. A runner whose job's image is missing fetches it before starting, saying so in the job's log, and keeps the chunks, so a rebuilt image costs only its changed chunks. Both commands use the login `oh auth login` saved when `OXY_TOKEN` is not set.
86 +
87 +
- **Runners say what they have and what jobs used (#40, #41).** Each claim carries the runner's free disk, load, kernel and a speed class from a short benchmark, taken at start and every 15 minutes while idle (`flow runner bench` prints them). A job's finish carries its peak disk and memory, for the hub to learn what its next run needs.
88 +
89 +
- **Reports (oxygenhub#127).** `report "NAME" { path "…"; when "…" }` in a job's output block uploads a directory holding `report.toml`, file by file with a manifest, within 2,000 files and 512 MiB; the receipt names the manifest.
90 +
91 +
- **Artifacts from a failing job (#37).** An artifact with `when "always"` is uploaded when the job fails or times out too, so a test report survives the failure it describes. It never goes in the receipt of a job that did not pass.
92 +
93 +
- **A cancelled run stops its jobs (#32).** A job whose run is cancelled is stopped: its microVM ends, or never boots if the job was still being prepared, and nothing more is sent.
94 +
95 +
- **The OpenTofu image (#56).** `images/tofu`: OpenTofu 1.13.1 with a provider mirror (hashicorp/google, cloudflare/cloudflare), TFLint with its Google ruleset, and curl, jq, xz, sfdisk and debugfs, for deploy jobs. Providers come only from the mirror, so `tofu init` and linting need no network.
4 96
5 -
## Unreleased
97 +
- **Coverage in the Rust build image (oxygenhub#113).** `cargo-llvm-cov` and the toolchain's `llvm-tools` are on the guest's `PATH`, so a job can write an LCOV report for the hub's Coverage tab.
98 +
99 +
- **The syscall tier keeps its build caches too (#70, spec §5.4).** A cached `ephemeral://disk` mount in a `runtime "syscall"` job now starts from the directory a passing job of the same slot kept, and goes back into the runner's pool when the job passes, with the same order, bookmark rules, `cache_gb` eviction and claim reporting as the microVM tier's disks, so jobs go where their cache is on boards without KVM as well. Nothing kept is writable by others or set-id, and the pool is the runner's alone.
100 +
101 +
- **The build cache pays off (#51, #52, spec §5.4).** A cached disk keeps the content hashes and times of the sources it was built from, and the next job gives every unchanged file its old time back, so cargo rebuilds only the crates that changed instead of every path crate. Claims list the caches a runner holds, and the hub sends a job to the runner holding its cache, as it does for images.
102 +
103 +
- **Restarts cost no job (#49, #50, spec §11.5.1).** A job's uploads and declines wait out a hub that is away for up to three minutes (a deploy restarting it), through refused connections as well as server errors, and log lines it could not send go with the next batch instead of being dropped. A runner told to stop drains: it claims nothing more, finishes and reports the jobs it holds, then exits; a second signal stops it at once. With `KillMode=mixed` and a `TimeoutStopSec` longer than the longest job in its unit, deploying a runner no longer has to wait for it to be idle.
104 +
105 +
- **A build cache (#28, spec §5.4).** A disk mount may name a `cache` and the `key` files it depends on, such as `Cargo.lock`. The runner keeps passing jobs' disks and starts the next job of the same repository, job and bookmark from one, so builds carry their compiled dependencies forward; another bookmark copies it rather than taking it. Failing jobs' disks are never kept, the receipt names what a disk started from, and the pool stays under `cache_gb` (default 64 GiB).
106 +
107 +
- **Runners share images (#47, docs/jetstream.md §8.1).** A runner with a `[runner.swarm]` section serves the image chunks it holds to other runners in its site, and fetches an image it lacks from them and the hub together, rarest chunks first. The hub is asked only for chunks no peer has or is fetching, so runners starting together split its work. Peers talk over TLS: each runner keeps its own certificate, the hub hands its fingerprint on with the peer, and a fetcher trusts no other. Peers admit each other with a ticket the hub signs, and every chunk is checked against its hash. Uploads are capped (`upload_mib_s`, default 50) at 16 connections. Without the section a runner fetches from the hub alone, as before.
6 108
7 109
- **A runner declines a job it cannot run (#36, spec §11.5).** Before fetching anything it checks the tier, the architecture, that the image is there and verifies, and that the sandbox's memory is free. If any check fails it hands the job back to the hub with the reason instead of running and failing it.
8 110
…
13 unchanged lines
22 124
23 125
- **A job says how long it may run (#23).** `timeout "90m"` on a job, or on a sandbox for its jobs (hours, minutes, seconds; a minute to a day). The hub caps it at the site's maximum.
24 126
127 +
- **A busy runner still says it is there (oxygenhub#243).** A runner whose slots are full, or that is draining, claims nothing, so the hub showed it offline through a quiet job. It now sends a heartbeat every 30 s while it cannot claim.
128 +
129 +
- **One slow fetch no longer holds every job on a runner (#63).** Workspaces were prepared under one runner-wide lock, so a job fetching a large repository from a bucket held all the others (one waited 819 s, and another's token expired before its fetch began). Fetches and exports now take only their repository's lock, and the job's log says how long it waited and how long fetching took.
130 +
131 +
- **An image with no recorded architecture is read, not trusted (#53).** Such an image (imported before architectures were recorded, or fetched from a hub that did not know it) matched every runner, so a riscv64 runner could be handed an x86_64 image and fail. The runner now reads the architecture from the image's shell once, records it, reports it and declines a job for another architecture.
132 +
133 +
- **`flow check` says when a pipeline needs a newer flow (#43).** A pipeline that names a node, output or event this flow does not know was called "not valid". `flow check` and `flow run` now say it uses something this version of flow does not know, and that a pipeline written for a newer OxygenFlow needs a newer flow.
134 +
135 +
- **Pushing a large image works against any hub (oxygenhub#147).** `flow rootfs push` asked about 50,000 chunks at once, over a hub's 2 MB body limit, so an 11 GB image failed with 413. It now asks 20,000 at a time.
136 +
25 137
- **A pinned file is re-read from the disk before its hash is called wrong (#22).**
26 138
27 139
- **A runner says which jobs it holds (oxygenhub#71).** Each claim lists the jobs the runner is running or still has a result for, so the hub ends at once any job it lost by restarting, instead of an hour later at the time limit.
…
6 unchanged lines
modified Cargo.lock +1513 −28
…
2 unchanged lines
3 3
version = 4
4 4
5 5
[[package]]
6 +
name = "addr2line"
7 +
version = "0.26.1"
8 +
source = "registry+https://github.com/rust-lang/crates.io-index"
9 +
checksum = "59317f77929f0e679d39364702289274de2f0f0b22cbf50b2b8cff2169a0b27a"
10 +
dependencies = [
11 +
 "gimli",
12 +
]
13 +
14 +
[[package]]
6 15
name = "adler2"
7 16
version = "2.0.1"
8 17
source = "registry+https://github.com/rust-lang/crates.io-index"
…
9 unchanged lines
18 27
]
19 28
20 29
[[package]]
30 +
name = "allocator-api2"
31 +
version = "0.2.21"
32 +
source = "registry+https://github.com/rust-lang/crates.io-index"
33 +
checksum = "683d7910e743518b0e34f1186f92494becacb047c7b6bf616c96772180fef923"
34 +
35 +
[[package]]
36 +
name = "ambient-authority"
37 +
version = "0.0.2"
38 +
source = "registry+https://github.com/rust-lang/crates.io-index"
39 +
checksum = "e9d4ee0d472d1cd2e28c97dfa124b3d8d992e10eb0a035f33f5d12e3a177ba3b"
40 +
41 +
[[package]]
21 42
name = "anstream"
22 43
version = "1.0.0"
23 44
source = "registry+https://github.com/rust-lang/crates.io-index"
…
50 unchanged lines
74 95
checksum = "330a5ed07fa54e4702c9d6c4174f74427fc0ef6e214bbd677ae50a5099946470"
75 96
76 97
[[package]]
98 +
name = "arbitrary"
99 +
version = "1.5.0"
100 +
source = "registry+https://github.com/rust-lang/crates.io-index"
101 +
checksum = "3bc62ac97cc33321f50863d514c3bc38a453947a8f9e781137e47c7401020aed"
102 +
103 +
[[package]]
77 104
name = "arrayvec"
78 105
version = "0.7.8"
79 106
source = "registry+https://github.com/rust-lang/crates.io-index"
80 107
checksum = "d3fb67a6e08acf24fdeccbac2cb6ac4305825bd1f117462e0e6f2f193345ad56"
81 108
82 109
[[package]]
110 +
name = "asn1-rs"
111 +
version = "0.7.2"
112 +
source = "registry+https://github.com/rust-lang/crates.io-index"
113 +
checksum = "b7f43a50ac4fdca5df8e885c21b835997f0a1cdee65494a6847694a98652d9d8"
114 +
dependencies = [
115 +
 "asn1-rs-derive",
116 +
 "asn1-rs-impl",
117 +
 "displaydoc",
118 +
 "nom",
119 +
 "num-traits",
120 +
 "rusticata-macros",
121 +
 "thiserror 2.0.21",
122 +
 "time",
123 +
]
124 +
125 +
[[package]]
126 +
name = "asn1-rs-derive"
127 +
version = "0.6.0"
128 +
source = "registry+https://github.com/rust-lang/crates.io-index"
129 +
checksum = "3109e49b1e4909e9db6515a30c633684d68cdeaa252f215214cb4fa1a5bfee2c"
130 +
dependencies = [
131 +
 "proc-macro2",
132 +
 "quote",
133 +
 "syn 2.0.119",
134 +
 "synstructure 0.13.2",
135 +
]
136 +
137 +
[[package]]
138 +
name = "asn1-rs-impl"
139 +
version = "0.2.0"
140 +
source = "registry+https://github.com/rust-lang/crates.io-index"
141 +
checksum = "7b18050c2cd6fe86c3a76584ef5e0baf286d038cda203eb6223df2cc413565f7"
142 +
dependencies = [
143 +
 "proc-macro2",
144 +
 "quote",
145 +
 "syn 2.0.119",
146 +
]
147 +
148 +
[[package]]
149 +
name = "async-trait"
150 +
version = "0.1.92"
151 +
source = "registry+https://github.com/rust-lang/crates.io-index"
152 +
checksum = "82f6aeea286b8eb4dd3431a1be1b59d290ace00f5bfd8e2a159bc2a05e2c1667"
153 +
dependencies = [
154 +
 "proc-macro2",
155 +
 "quote",
156 +
 "syn 3.0.6",
157 +
]
158 +
159 +
[[package]]
83 160
name = "autocfg"
84 161
version = "1.5.1"
85 162
source = "registry+https://github.com/rust-lang/crates.io-index"
…
6 unchanged lines
92 169
checksum = "72b3254f16251a8381aa12e40e3c4d2f0199f8c6508fbecb9d91f575e0fbb8c6"
93 170
94 171
[[package]]
172 +
name = "base64"
173 +
version = "0.23.1"
174 +
source = "registry+https://github.com/rust-lang/crates.io-index"
175 +
checksum = "ac07cdecf99051d9a5238b80f35af32cdeba5b336e55d957b318b50137e18da5"
176 +
177 +
[[package]]
95 178
name = "base64ct"
96 179
version = "1.8.3"
97 180
source = "registry+https://github.com/rust-lang/crates.io-index"
98 181
checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06"
99 182
100 183
[[package]]
184 +
name = "bit-vec"
185 +
version = "0.9.1"
186 +
source = "registry+https://github.com/rust-lang/crates.io-index"
187 +
checksum = "b71798fca2c1fe1086445a7258a4bc81e6e49dcd24c8d0dd9a1e57395b603f51"
188 +
dependencies = [
189 +
 "serde",
190 +
]
191 +
192 +
[[package]]
101 193
name = "bitflags"
102 194
version = "2.13.2"
103 195
source = "registry+https://github.com/rust-lang/crates.io-index"
…
9 unchanged lines
113 205
 "cc",
114 206
 "cfg-if",
115 207
 "constant_time_eq",
116 -
 "cpufeatures",
208 +
 "cpufeatures 0.3.1",
117 209
 "rayon-core",
118 210
]
119 211
120 212
[[package]]
121 213
name = "block-buffer"
214 +
version = "0.10.4"
215 +
source = "registry+https://github.com/rust-lang/crates.io-index"
216 +
checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71"
217 +
dependencies = [
218 +
 "generic-array",
219 +
]
220 +
221 +
[[package]]
222 +
name = "block-buffer"
122 223
version = "0.12.1"
123 224
source = "registry+https://github.com/rust-lang/crates.io-index"
124 225
checksum = "d2f6c7dbe95a6ed67ad9f18e57daf93a2f034c524b99fd2b76d18fdfeb6660aa"
…
12 unchanged lines
137 238
]
138 239
139 240
[[package]]
241 +
name = "bumpalo"
242 +
version = "3.20.3"
243 +
source = "registry+https://github.com/rust-lang/crates.io-index"
244 +
checksum = "72f5acc6cb2ba439de613abc23857ec3d78374d8ed5ac84e9d11336e87da8649"
245 +
dependencies = [
246 +
 "allocator-api2",
247 +
]
248 +
249 +
[[package]]
250 +
name = "bytes"
251 +
version = "1.12.1"
252 +
source = "registry+https://github.com/rust-lang/crates.io-index"
253 +
checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04"
254 +
255 +
[[package]]
256 +
name = "cap-primitives"
257 +
version = "4.0.3"
258 +
source = "registry+https://github.com/rust-lang/crates.io-index"
259 +
checksum = "8b5f74729fd2f44701d1a8eb47e906cdb3ccd9ec0f02baad85a744b791940b18"
260 +
dependencies = [
261 +
 "ambient-authority",
262 +
 "fs-set-times",
263 +
 "io-extras",
264 +
 "io-lifetimes 3.0.1",
265 +
 "ipnet",
266 +
 "maybe-owned",
267 +
 "rustix",
268 +
 "rustix-linux-procfs",
269 +
 "windows-sys 0.61.2",
270 +
 "winx",
271 +
]
272 +
273 +
[[package]]
140 274
name = "cc"
141 275
version = "1.5.1"
142 276
source = "registry+https://github.com/rust-lang/crates.io-index"
…
18 unchanged lines
161 295
checksum = "f079e83a288787bcd14a6aea84cee5c87a67c5a3e660c30f557a3d24761b3527"
162 296
163 297
[[package]]
298 +
name = "chacha20"
299 +
version = "0.10.2"
300 +
source = "registry+https://github.com/rust-lang/crates.io-index"
301 +
checksum = "65c35e4b699c7e15ccbe7ee35c005e4fc0a278d22238a2857e6ce2dadeda1b06"
302 +
dependencies = [
303 +
 "cfg-if",
304 +
 "cpufeatures 0.3.1",
305 +
 "rand_core 0.10.1",
306 +
]
307 +
308 +
[[package]]
164 309
name = "clap"
165 310
version = "4.6.7"
166 311
source = "registry+https://github.com/rust-lang/crates.io-index"
…
40 unchanged lines
207 352
checksum = "0c9ea0ac24bc397ab3c98583a3c9ba74fa56b09a4449bbe172b9b1ddb016027a"
208 353
209 354
[[package]]
355 +
name = "cobs"
356 +
version = "0.3.0"
357 +
source = "registry+https://github.com/rust-lang/crates.io-index"
358 +
checksum = "0fa961b519f0b462e3a3b4a34b64d119eeaca1d59af726fe450bbba07a9fc0a1"
359 +
dependencies = [
360 +
 "thiserror 2.0.21",
361 +
]
362 +
363 +
[[package]]
210 364
name = "colorchoice"
211 365
version = "1.0.5"
212 366
source = "registry+https://github.com/rust-lang/crates.io-index"
…
12 unchanged lines
225 379
checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b"
226 380
227 381
[[package]]
382 +
name = "core_detect"
383 +
version = "1.0.0"
384 +
source = "registry+https://github.com/rust-lang/crates.io-index"
385 +
checksum = "7f8f80099a98041a3d1622845c271458a2d73e688351bf3cb999266764b81d48"
386 +
387 +
[[package]]
388 +
name = "cpp_demangle"
389 +
version = "0.5.1"
390 +
source = "registry+https://github.com/rust-lang/crates.io-index"
391 +
checksum = "0667304c32ea56cb4cd6d2d7c0cfe9a2f8041229db8c033af7f8d69492429def"
392 +
dependencies = [
393 +
 "cfg-if",
394 +
]
395 +
396 +
[[package]]
397 +
name = "cpufeatures"
398 +
version = "0.2.17"
399 +
source = "registry+https://github.com/rust-lang/crates.io-index"
400 +
checksum = "59ed5838eebb26a2bb2e58f6d5b5316989ae9d08bab10e0e6d103e656d1b0280"
401 +
dependencies = [
402 +
 "libc",
403 +
]
404 +
405 +
[[package]]
228 406
name = "cpufeatures"
229 407
version = "0.3.1"
230 408
source = "registry+https://github.com/rust-lang/crates.io-index"
…
3 unchanged lines
234 412
]
235 413
236 414
[[package]]
415 +
name = "cranelift-assembler-x64"
416 +
version = "0.136.4"
417 +
source = "registry+https://github.com/rust-lang/crates.io-index"
418 +
checksum = "c9c587da2696fbaa58e7a2fa2e5bfdd365d0928219f36992304ac1f15e5bc254"
419 +
dependencies = [
420 +
 "cranelift-assembler-x64-meta",
421 +
]
422 +
423 +
[[package]]
424 +
name = "cranelift-assembler-x64-meta"
425 +
version = "0.136.4"
426 +
source = "registry+https://github.com/rust-lang/crates.io-index"
427 +
checksum = "f28a60027b7556b680b051651291053ec381edf4ffbc01784997ae51a64269bb"
428 +
dependencies = [
429 +
 "cranelift-srcgen",
430 +
]
431 +
432 +
[[package]]
433 +
name = "cranelift-bforest"
434 +
version = "0.136.4"
435 +
source = "registry+https://github.com/rust-lang/crates.io-index"
436 +
checksum = "62bfdb846acebfd591d601994ea283c98d525ac0bc9a84ca3f2c36bcf5b69ff1"
437 +
dependencies = [
438 +
 "cranelift-entity",
439 +
 "wasmtime-internal-core",
440 +
]
441 +
442 +
[[package]]
443 +
name = "cranelift-bitset"
444 +
version = "0.136.4"
445 +
source = "registry+https://github.com/rust-lang/crates.io-index"
446 +
checksum = "aa9f8cc1080739d5f72b4ee7e00aff7d589f1284beaf0a19472c07878be7f299"
447 +
dependencies = [
448 +
 "serde",
449 +
 "serde_derive",
450 +
 "wasmtime-internal-core",
451 +
]
452 +
453 +
[[package]]
454 +
name = "cranelift-codegen"
455 +
version = "0.136.4"
456 +
source = "registry+https://github.com/rust-lang/crates.io-index"
457 +
checksum = "d85417d493ecec833b5134546b4a252c78cc0bd5be11729a35d25dcff9e4e70b"
458 +
dependencies = [
459 +
 "bumpalo",
460 +
 "cranelift-assembler-x64",
461 +
 "cranelift-bforest",
462 +
 "cranelift-bitset",
463 +
 "cranelift-codegen-meta",
464 +
 "cranelift-codegen-shared",
465 +
 "cranelift-control",
466 +
 "cranelift-entity",
467 +
 "cranelift-isle",
468 +
 "gimli",
469 +
 "hashbrown 0.17.1",
470 +
 "libm",
471 +
 "log",
472 +
 "postcard",
473 +
 "pulley-interpreter",
474 +
 "regalloc2",
475 +
 "rustc-hash",
476 +
 "serde",
477 +
 "serde_derive",
478 +
 "sha2 0.10.9",
479 +
 "smallvec",
480 +
 "target-lexicon",
481 +
 "wasmtime-internal-core",
482 +
]
483 +
484 +
[[package]]
485 +
name = "cranelift-codegen-meta"
486 +
version = "0.136.4"
487 +
source = "registry+https://github.com/rust-lang/crates.io-index"
488 +
checksum = "d01c8ed0098360bae69ad6b82952d4247516799ce8050d45ad168d6689f4eae6"
489 +
dependencies = [
490 +
 "cranelift-assembler-x64-meta",
491 +
 "cranelift-codegen-shared",
492 +
 "cranelift-srcgen",
493 +
 "heck",
494 +
 "pulley-interpreter",
495 +
]
496 +
497 +
[[package]]
498 +
name = "cranelift-codegen-shared"
499 +
version = "0.136.4"
500 +
source = "registry+https://github.com/rust-lang/crates.io-index"
501 +
checksum = "04f9b86d809cec9c4e88b201f1b3ed3b0347674c9ccfbb8c355228669e34be81"
502 +
503 +
[[package]]
504 +
name = "cranelift-control"
505 +
version = "0.136.4"
506 +
source = "registry+https://github.com/rust-lang/crates.io-index"
507 +
checksum = "ce182038fb10b719841e65fdf5deeee258c684f63439f5be680c2be059274b58"
508 +
dependencies = [
509 +
 "arbitrary",
510 +
]
511 +
512 +
[[package]]
513 +
name = "cranelift-entity"
514 +
version = "0.136.4"
515 +
source = "registry+https://github.com/rust-lang/crates.io-index"
516 +
checksum = "7bca812b03a7b942796391f25bc02aa70a1a007fee3ce2756490c788429e9c39"
517 +
dependencies = [
518 +
 "cranelift-bitset",
519 +
 "serde",
520 +
 "serde_derive",
521 +
 "wasmtime-internal-core",
522 +
]
523 +
524 +
[[package]]
525 +
name = "cranelift-frontend"
526 +
version = "0.136.4"
527 +
source = "registry+https://github.com/rust-lang/crates.io-index"
528 +
checksum = "a0c30563d2aef95e8d1a00a920199c99117b350339599f0e9487a7a1cf5d59de"
529 +
dependencies = [
530 +
 "cranelift-codegen",
531 +
 "hashbrown 0.17.1",
532 +
 "log",
533 +
 "smallvec",
534 +
 "target-lexicon",
535 +
]
536 +
537 +
[[package]]
538 +
name = "cranelift-isle"
539 +
version = "0.136.4"
540 +
source = "registry+https://github.com/rust-lang/crates.io-index"
541 +
checksum = "9c54184333a275698edb1d3bdff226131a25d47133d792aa716e47e32b8b7f05"
542 +
543 +
[[package]]
544 +
name = "cranelift-native"
545 +
version = "0.136.4"
546 +
source = "registry+https://github.com/rust-lang/crates.io-index"
547 +
checksum = "eb67a4c8a3180b5f1d77dffcd6317c05386ced42e5a31c7004523a9d99dc85c1"
548 +
dependencies = [
549 +
 "cranelift-codegen",
550 +
 "libc",
551 +
 "target-lexicon",
552 +
]
553 +
554 +
[[package]]
555 +
name = "cranelift-srcgen"
556 +
version = "0.136.4"
557 +
source = "registry+https://github.com/rust-lang/crates.io-index"
558 +
checksum = "b533db1679b4b728568bd490da6533cf50974c48a4d7534e78f1e0e02bc39769"
559 +
560 +
[[package]]
237 561
name = "crc32fast"
238 562
version = "1.5.2"
239 563
source = "registry+https://github.com/rust-lang/crates.io-index"
…
29 unchanged lines
269 593
270 594
[[package]]
271 595
name = "crypto-common"
596 +
version = "0.1.7"
597 +
source = "registry+https://github.com/rust-lang/crates.io-index"
598 +
checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a"
599 +
dependencies = [
600 +
 "generic-array",
601 +
 "typenum",
602 +
]
603 +
604 +
[[package]]
605 +
name = "crypto-common"
272 606
version = "0.2.2"
273 607
source = "registry+https://github.com/rust-lang/crates.io-index"
274 608
checksum = "ce6e4c961d6cd6c9a86db418387425e8bdeaf05b3c8bc1411e6dca4c252f1453"
…
19 unchanged lines
294 628
checksum = "b5eed333089e2e1c1ac8c6c0398e5e2497b4c9926ca6d0365ed1e099afa5bc23"
295 629
dependencies = [
296 630
 "cfg-if",
297 -
 "cpufeatures",
631 +
 "cpufeatures 0.3.1",
298 632
 "curve25519-dalek-derive",
299 -
 "digest",
633 +
 "digest 0.11.3",
300 634
 "fiat-crypto",
301 635
 "rustc_version",
302 636
 "subtle",
…
12 unchanged lines
315 649
]
316 650
317 651
[[package]]
652 +
name = "data-encoding"
653 +
version = "2.11.1"
654 +
source = "registry+https://github.com/rust-lang/crates.io-index"
655 +
checksum = "4583a4551df46e2792f82ceeac45e850d2e2d5debba0b91f102385cda5b11f06"
656 +
657 +
[[package]]
318 658
name = "der"
319 659
version = "0.8.2"
320 660
source = "registry+https://github.com/rust-lang/crates.io-index"
…
4 unchanged lines
325 665
]
326 666
327 667
[[package]]
668 +
name = "der-parser"
669 +
version = "10.0.0"
670 +
source = "registry+https://github.com/rust-lang/crates.io-index"
671 +
checksum = "07da5016415d5a3c4dd39b11ed26f915f52fc4e0dc197d87908bc916e51bc1a6"
672 +
dependencies = [
673 +
 "asn1-rs",
674 +
 "displaydoc",
675 +
 "nom",
676 +
 "num-bigint",
677 +
 "num-traits",
678 +
 "rusticata-macros",
679 +
]
680 +
681 +
[[package]]
682 +
name = "deranged"
683 +
version = "0.5.8"
684 +
source = "registry+https://github.com/rust-lang/crates.io-index"
685 +
checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c"
686 +
687 +
[[package]]
688 +
name = "digest"
689 +
version = "0.10.7"
690 +
source = "registry+https://github.com/rust-lang/crates.io-index"
691 +
checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
692 +
dependencies = [
693 +
 "block-buffer 0.10.4",
694 +
 "crypto-common 0.1.7",
695 +
]
696 +
697 +
[[package]]
328 698
name = "digest"
329 699
version = "0.11.3"
330 700
source = "registry+https://github.com/rust-lang/crates.io-index"
331 701
checksum = "f1dd6dbb5841937940781866fa1281a1ff7bd3bf827091440879f9994983d5c2"
332 702
dependencies = [
333 -
 "block-buffer",
334 -
 "crypto-common",
703 +
 "block-buffer 0.12.1",
704 +
 "const-oid",
705 +
 "crypto-common 0.2.2",
335 706
]
336 707
337 708
[[package]]
…
24 unchanged lines
362 733
dependencies = [
363 734
 "curve25519-dalek",
364 735
 "ed25519",
365 -
 "sha2",
736 +
 "sha2 0.11.0",
366 737
 "subtle",
367 738
 "zeroize",
368 739
]
…
5 unchanged lines
374 745
checksum = "252afb9ae5eaa683babdc6a068b3f5726eb19e05070c731f9b2a23a7c3e8ed34"
375 746
376 747
[[package]]
748 +
name = "embedded-io"
749 +
version = "0.4.0"
750 +
source = "registry+https://github.com/rust-lang/crates.io-index"
751 +
checksum = "ef1a6892d9eef45c8fa6b9e0086428a2cca8491aca8f787c534a3d6d0bcb3ced"
752 +
753 +
[[package]]
754 +
name = "embedded-io"
755 +
version = "0.6.1"
756 +
source = "registry+https://github.com/rust-lang/crates.io-index"
757 +
checksum = "edd0f118536f44f5ccd48bcb8b111bdc3de888b58c74639dfb034a357d0f206d"
758 +
759 +
[[package]]
760 +
name = "encoding_rs"
761 +
version = "0.8.42"
762 +
source = "registry+https://github.com/rust-lang/crates.io-index"
763 +
checksum = "8e985e0451871ad22fb8d2b6b076e2028a502a0d3950998c2c5c0a4f9b5d9679"
764 +
dependencies = [
765 +
 "cfg-if",
766 +
 "core_detect",
767 +
 "multiversion_no_op",
768 +
 "rustversion",
769 +
 "scopeguard",
770 +
 "simdutf8",
771 +
]
772 +
773 +
[[package]]
377 774
name = "equivalent"
378 775
version = "1.0.2"
379 776
source = "registry+https://github.com/rust-lang/crates.io-index"
…
40 unchanged lines
420 817
421 818
[[package]]
422 819
name = "flow"
423 -
version = "0.1.0"
820 +
version = "0.5.3"
424 821
dependencies = [
425 822
 "anyhow",
426 823
 "blake3",
427 824
 "clap",
428 825
 "flow-core",
826 +
 "flow-init",
429 827
 "hex",
430 828
 "ignore",
431 829
 "jet",
432 830
 "libc",
433 831
 "oxy-core",
832 +
 "rcgen",
833 +
 "rustls",
434 834
 "serde",
435 835
 "serde_json",
436 836
 "tempfile",
437 837
 "toml",
438 838
 "ureq",
839 +
 "wasmtime",
840 +
 "wasmtime-wasi",
841 +
 "wat",
842 +
 "windows-sys 0.61.2",
439 843
]
440 844
441 845
[[package]]
442 846
name = "flow-casfs"
443 -
version = "0.1.0"
847 +
version = "0.5.3"
444 848
dependencies = [
445 849
 "anyhow",
446 850
 "fuser",
…
3 unchanged lines
450 854
]
451 855
452 856
[[package]]
857 +
name = "flow-checks"
858 +
version = "0.5.3"
859 +
dependencies = [
860 +
 "tempfile",
861 +
]
862 +
863 +
[[package]]
453 864
name = "flow-core"
454 -
version = "0.1.0"
865 +
version = "0.5.3"
455 866
dependencies = [
456 867
 "blake3",
868 +
 "curve25519-dalek",
457 869
 "globset",
458 870
 "hex",
459 871
 "kdl",
460 872
 "miette",
461 873
 "oxy-core",
874 +
 "ring",
462 875
 "serde",
463 876
 "serde_json",
464 -
 "thiserror",
877 +
 "thiserror 1.0.69",
465 878
]
466 879
467 880
[[package]]
468 881
name = "flow-init"
469 -
version = "0.1.0"
882 +
version = "0.5.3"
470 883
dependencies = [
471 884
 "libc",
472 885
 "serde",
473 886
 "serde_json",
887 +
 "tempfile",
474 888
]
475 889
476 890
[[package]]
891 +
name = "fnv"
892 +
version = "1.0.7"
893 +
source = "registry+https://github.com/rust-lang/crates.io-index"
894 +
checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1"
895 +
896 +
[[package]]
897 +
name = "foldhash"
898 +
version = "0.2.0"
899 +
source = "registry+https://github.com/rust-lang/crates.io-index"
900 +
checksum = "77ce24cb58228fbb8aa041425bb1050850ac19177686ea6e0f41a70416f56fdb"
901 +
902 +
[[package]]
477 903
name = "form_urlencoded"
478 904
version = "1.2.2"
479 905
source = "registry+https://github.com/rust-lang/crates.io-index"
…
3 unchanged lines
483 909
]
484 910
485 911
[[package]]
912 +
name = "fs-set-times"
913 +
version = "0.20.3"
914 +
source = "registry+https://github.com/rust-lang/crates.io-index"
915 +
checksum = "94e7099f6313ecacbe1256e8ff9d617b75d1bcb16a6fddef94866d225a01a14a"
916 +
dependencies = [
917 +
 "io-lifetimes 2.0.4",
918 +
 "rustix",
919 +
 "windows-sys 0.52.0",
920 +
]
921 +
922 +
[[package]]
486 923
name = "fuser"
487 924
version = "0.16.0"
488 925
source = "registry+https://github.com/rust-lang/crates.io-index"
…
9 unchanged lines
498 935
]
499 936
500 937
[[package]]
938 +
name = "futures"
939 +
version = "0.3.34"
940 +
source = "registry+https://github.com/rust-lang/crates.io-index"
941 +
checksum = "9a31d2a3fbaaeb2af2368bbdd904aa8e812d3c04a1ee10d3171f52d556e5d0a3"
942 +
dependencies = [
943 +
 "futures-channel",
944 +
 "futures-core",
945 +
 "futures-io",
946 +
 "futures-sink",
947 +
 "futures-task",
948 +
 "futures-util",
949 +
]
950 +
951 +
[[package]]
952 +
name = "futures-channel"
953 +
version = "0.3.34"
954 +
source = "registry+https://github.com/rust-lang/crates.io-index"
955 +
checksum = "b1f9e3d69d39e4862ffed03ed071a76f9a13ba1d9109d355b0f0aa6b15e393c4"
956 +
dependencies = [
957 +
 "futures-core",
958 +
 "futures-sink",
959 +
]
960 +
961 +
[[package]]
962 +
name = "futures-core"
963 +
version = "0.3.34"
964 +
source = "registry+https://github.com/rust-lang/crates.io-index"
965 +
checksum = "92d699e522242e69e3003b94ecc1f960f3a5e015aa7c5d7486e65ad01dd94f5e"
966 +
967 +
[[package]]
968 +
name = "futures-io"
969 +
version = "0.3.34"
970 +
source = "registry+https://github.com/rust-lang/crates.io-index"
971 +
checksum = "53c0fa8157de1303bfffdaa1cc2a673bfffb60102f76b0ef4441659124373fed"
972 +
973 +
[[package]]
974 +
name = "futures-sink"
975 +
version = "0.3.34"
976 +
source = "registry+https://github.com/rust-lang/crates.io-index"
977 +
checksum = "1944426bf7d03f1d14f708785e4b33efd750b36d48a157b836b3efc15ede8e1d"
978 +
979 +
[[package]]
980 +
name = "futures-task"
981 +
version = "0.3.34"
982 +
source = "registry+https://github.com/rust-lang/crates.io-index"
983 +
checksum = "cd417de3d1d015fc3bfd2b1ea46dfc7bab72ef86f1cc7cc9c78e728b34a6d1fd"
984 +
985 +
[[package]]
986 +
name = "futures-util"
987 +
version = "0.3.34"
988 +
source = "registry+https://github.com/rust-lang/crates.io-index"
989 +
checksum = "0d50a92467f8ba5dd6e3ee5d4bd04d73ab2e4e1c44474a0674821dfce14b79bc"
990 +
dependencies = [
991 +
 "futures-channel",
992 +
 "futures-core",
993 +
 "futures-io",
994 +
 "futures-sink",
995 +
 "futures-task",
996 +
 "memchr",
997 +
 "pin-project-lite",
998 +
 "slab",
999 +
]
1000 +
1001 +
[[package]]
1002 +
name = "generic-array"
1003 +
version = "0.14.7"
1004 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1005 +
checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a"
1006 +
dependencies = [
1007 +
 "typenum",
1008 +
 "version_check",
1009 +
]
1010 +
1011 +
[[package]]
501 1012
name = "getrandom"
502 1013
version = "0.2.17"
503 1014
source = "registry+https://github.com/rust-lang/crates.io-index"
…
17 unchanged lines
521 1032
]
522 1033
523 1034
[[package]]
1035 +
name = "gimli"
1036 +
version = "0.33.0"
1037 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1038 +
checksum = "0bf7f043f89559805f8c7cacc432749b2fa0d0a0a9ee46ce47164ed5ba7f126c"
1039 +
dependencies = [
1040 +
 "fnv",
1041 +
 "hashbrown 0.16.1",
1042 +
 "indexmap",
1043 +
 "stable_deref_trait",
1044 +
]
1045 +
1046 +
[[package]]
524 1047
name = "globset"
525 1048
version = "0.4.20"
526 1049
source = "registry+https://github.com/rust-lang/crates.io-index"
…
8 unchanged lines
535 1058
536 1059
[[package]]
537 1060
name = "hashbrown"
1061 +
version = "0.16.1"
1062 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1063 +
checksum = "841d1cc9bed7f9236f321df977030373f4a4163ae1a7dbfe1a51a2c1a51d9100"
1064 +
1065 +
[[package]]
1066 +
name = "hashbrown"
538 1067
version = "0.17.1"
539 1068
source = "registry+https://github.com/rust-lang/crates.io-index"
540 1069
checksum = "ed5909b6e89a2db4456e54cd5f673791d7eca6732202bbf2a9cc504fe2f9b84a"
1070 +
dependencies = [
1071 +
 "foldhash",
1072 +
 "serde",
1073 +
 "serde_core",
1074 +
]
541 1075
542 1076
[[package]]
543 1077
name = "heck"
…
101 unchanged lines
645 1179
]
646 1180
647 1181
[[package]]
1182 +
name = "id-arena"
1183 +
version = "2.3.0"
1184 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1185 +
checksum = "3d3067d79b975e8844ca9eb072e16b31c3c1c36928edf9c6789548c524d0d954"
1186 +
1187 +
[[package]]
648 1188
name = "idna"
649 1189
version = "1.1.0"
650 1190
source = "registry+https://github.com/rust-lang/crates.io-index"
…
37 unchanged lines
688 1228
checksum = "cc4e190f5d26ca7051642629da2c52fc03bde85a03197c99408dcd291734c855"
689 1229
dependencies = [
690 1230
 "equivalent",
691 -
 "hashbrown",
1231 +
 "hashbrown 0.17.1",
1232 +
 "serde",
1233 +
 "serde_core",
1234 +
]
1235 +
1236 +
[[package]]
1237 +
name = "io-extras"
1238 +
version = "0.19.0"
1239 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1240 +
checksum = "20fd6de4ccfcc187e38bc21cfa543cb5a302cb86a8b114eb7f0bf0dc9f8ac00f"
1241 +
dependencies = [
1242 +
 "io-lifetimes 3.0.1",
1243 +
 "windows-sys 0.52.0",
692 1244
]
693 1245
694 1246
[[package]]
1247 +
name = "io-lifetimes"
1248 +
version = "2.0.4"
1249 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1250 +
checksum = "06432fb54d3be7964ecd3649233cddf80db2832f47fec34c01f65b3d9d774983"
1251 +
1252 +
[[package]]
1253 +
name = "io-lifetimes"
1254 +
version = "3.0.1"
1255 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1256 +
checksum = "2f0fb0570afe1fed943c5c3d4102d5358592d8625fda6a0007fdbe65a92fba96"
1257 +
1258 +
[[package]]
1259 +
name = "ipnet"
1260 +
version = "2.12.2"
1261 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1262 +
checksum = "791930b43c0d5973160d90a8f3894509f2b273430f5c5c73b668636d0287c5c0"
1263 +
1264 +
[[package]]
695 1265
name = "is_terminal_polyfill"
696 1266
version = "1.70.2"
697 1267
source = "registry+https://github.com/rust-lang/crates.io-index"
698 1268
checksum = "a6cb138bb79a146c1bd460005623e142ef0181e3d0219cb493e02f7d08a35695"
699 1269
700 1270
[[package]]
1271 +
name = "itertools"
1272 +
version = "0.14.0"
1273 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1274 +
checksum = "2b192c782037fadd9cfa75548310488aabdbf3d2da73885b31bd0abd03351285"
1275 +
dependencies = [
1276 +
 "either",
1277 +
]
1278 +
1279 +
[[package]]
701 1280
name = "itoa"
702 1281
version = "1.0.18"
703 1282
source = "registry+https://github.com/rust-lang/crates.io-index"
…
1 unchanged lines
705 1284
706 1285
[[package]]
707 1286
name = "jet"
708 -
version = "0.1.0"
1287 +
version = "0.5.3"
709 1288
dependencies = [
710 1289
 "blake3",
711 1290
 "hex",
…
1 unchanged lines
713 1292
 "serde",
714 1293
 "serde_json",
715 1294
 "tempfile",
716 -
 "thiserror",
1295 +
 "thiserror 1.0.69",
717 1296
]
718 1297
719 1298
[[package]]
…
25 unchanged lines
745 1324
checksum = "d8f198d1db720e4940b5a493201d199d9f24f568f8f746bd13706243a2f71598"
746 1325
dependencies = [
747 1326
 "cfg-if",
748 -
 "cpufeatures",
1327 +
 "cpufeatures 0.3.1",
749 1328
]
750 1329
751 1330
[[package]]
1331 +
name = "lazy_static"
1332 +
version = "1.5.1"
1333 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1334 +
checksum = "20870f649af7073d53e38067b2a84312175d56ea15217e1b15bc83506ec50afb"
1335 +
1336 +
[[package]]
1337 +
name = "leb128"
1338 +
version = "0.2.7"
1339 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1340 +
checksum = "c83bff1d572d6b9aeef67ddfc8448e4a3737909cb28e81f97c791b9018703e52"
1341 +
1342 +
[[package]]
1343 +
name = "leb128fmt"
1344 +
version = "0.1.0"
1345 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1346 +
checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2"
1347 +
1348 +
[[package]]
752 1349
name = "libc"
753 1350
version = "0.2.189"
754 1351
source = "registry+https://github.com/rust-lang/crates.io-index"
755 1352
checksum = "3eaf3ede3fee6db1a4c2ee091bf8a8b4dccdc6d17f656fb07896ee72867612f2"
756 1353
757 1354
[[package]]
1355 +
name = "libm"
1356 +
version = "0.2.16"
1357 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1358 +
checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981"
1359 +
1360 +
[[package]]
758 1361
name = "linux-raw-sys"
759 1362
version = "0.12.1"
760 1363
source = "registry+https://github.com/rust-lang/crates.io-index"
…
12 unchanged lines
773 1376
checksum = "f9f8bd3e56ce4dfc153cf470fffbfa98c7620958b312ca5c3a4b8d5181fd13c6"
774 1377
775 1378
[[package]]
1379 +
name = "mach2"
1380 +
version = "0.6.0"
1381 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1382 +
checksum = "dae608c151f68243f2b000364e1f7b186d9c29845f7d2d85bd31b9ad77ad552b"
1383 +
1384 +
[[package]]
1385 +
name = "maybe-owned"
1386 +
version = "0.3.4"
1387 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1388 +
checksum = "4facc753ae494aeb6e3c22f839b158aebd4f9270f55cd3c79906c45476c47ab4"
1389 +
1390 +
[[package]]
776 1391
name = "memchr"
777 1392
version = "2.8.3"
778 1393
source = "registry+https://github.com/rust-lang/crates.io-index"
779 1394
checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
780 1395
781 1396
[[package]]
1397 +
name = "memfd"
1398 +
version = "0.6.6"
1399 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1400 +
checksum = "57804b2c9b69967f1536a56f86297e367a33b19e98852ed624b84551cdbc0d90"
1401 +
dependencies = [
1402 +
 "rustix",
1403 +
]
1404 +
1405 +
[[package]]
782 1406
name = "miette"
783 1407
version = "7.6.0"
784 1408
source = "registry+https://github.com/rust-lang/crates.io-index"
…
1 unchanged lines
786 1410
dependencies = [
787 1411
 "cfg-if",
788 1412
 "miette-derive",
789 -
 "unicode-width",
1413 +
 "unicode-width 0.1.14",
790 1414
]
791 1415
792 1416
[[package]]
…
8 unchanged lines
801 1425
]
802 1426
803 1427
[[package]]
1428 +
name = "minimal-lexical"
1429 +
version = "0.2.1"
1430 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1431 +
checksum = "68354c5c6bd36d73ff3feceb05efa59b6acb7626617f4962be322a825e61f79a"
1432 +
1433 +
[[package]]
804 1434
name = "miniz_oxide"
805 1435
version = "0.9.1"
806 1436
source = "registry+https://github.com/rust-lang/crates.io-index"
…
4 unchanged lines
811 1441
]
812 1442
813 1443
[[package]]
1444 +
name = "mio"
1445 +
version = "1.2.4"
1446 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1447 +
checksum = "1788edb87fdc09c7e26304471e2f5be8cdefb1b6930d6e3985fc02ff53bf86ee"
1448 +
dependencies = [
1449 +
 "libc",
1450 +
 "wasi",
1451 +
 "windows-sys 0.61.2",
1452 +
]
1453 +
1454 +
[[package]]
814 1455
name = "ml-dsa"
815 1456
version = "0.1.1"
816 1457
source = "registry+https://github.com/rust-lang/crates.io-index"
817 1458
checksum = "add6b9d92e496f16f4526d68ff29da1483aba4b119baeab8bed3b9e3544a6f3d"
818 1459
dependencies = [
819 1460
 "const-oid",
820 -
 "crypto-common",
1461 +
 "crypto-common 0.2.2",
821 1462
 "ctutils",
822 1463
 "hybrid-array",
823 1464
 "module-lattice",
…
14 unchanged lines
838 1479
]
839 1480
840 1481
[[package]]
1482 +
name = "multiversion_no_op"
1483 +
version = "1.0.0"
1484 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1485 +
checksum = "743fb55ba31b18fb1ecef6bdc9aa2743314978ac084044301a7eee33fb99a20d"
1486 +
1487 +
[[package]]
841 1488
name = "nix"
842 1489
version = "0.29.0"
843 1490
source = "registry+https://github.com/rust-lang/crates.io-index"
…
6 unchanged lines
850 1497
]
851 1498
852 1499
[[package]]
1500 +
name = "nom"
1501 +
version = "7.1.3"
1502 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1503 +
checksum = "d273983c5a657a70a3e8f2a01329822f3b8c8172b73826411a55751e404a0a4a"
1504 +
dependencies = [
1505 +
 "memchr",
1506 +
 "minimal-lexical",
1507 +
]
1508 +
1509 +
[[package]]
1510 +
name = "num-bigint"
1511 +
version = "0.4.8"
1512 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1513 +
checksum = "c89e69e7e0f03bea5ef08013795c25018e101932225a656383bd384495ecc367"
1514 +
dependencies = [
1515 +
 "num-integer",
1516 +
 "num-traits",
1517 +
]
1518 +
1519 +
[[package]]
1520 +
name = "num-conv"
1521 +
version = "0.2.2"
1522 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1523 +
checksum = "521739c6d2bac4aa25192232afe6841231376b2b26d4d9fae5ecf8ca5772e441"
1524 +
1525 +
[[package]]
1526 +
name = "num-integer"
1527 +
version = "0.1.47"
1528 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1529 +
checksum = "7ce2d95d4b3734dc35aa2f45e1aa22cd416814592a4f9d9205e11affd5b8e10b"
1530 +
dependencies = [
1531 +
 "num-traits",
1532 +
]
1533 +
1534 +
[[package]]
853 1535
name = "num-traits"
854 1536
version = "0.2.19"
855 1537
source = "registry+https://github.com/rust-lang/crates.io-index"
…
3 unchanged lines
859 1541
]
860 1542
861 1543
[[package]]
1544 +
name = "object"
1545 +
version = "0.40.0"
1546 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1547 +
checksum = "dd229a0361b9d0d4396176e02d65897f487eebeab7caa6d443855ee152ca0b9c"
1548 +
dependencies = [
1549 +
 "crc32fast",
1550 +
 "hashbrown 0.17.1",
1551 +
 "indexmap",
1552 +
 "memchr",
1553 +
]
1554 +
1555 +
[[package]]
1556 +
name = "oid-registry"
1557 +
version = "0.8.1"
1558 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1559 +
checksum = "12f40cff3dde1b6087cc5d5f5d4d65712f34016a03ed60e9c08dcc392736b5b7"
1560 +
dependencies = [
1561 +
 "asn1-rs",
1562 +
]
1563 +
1564 +
[[package]]
862 1565
name = "once_cell"
863 1566
version = "1.21.4"
864 1567
source = "registry+https://github.com/rust-lang/crates.io-index"
…
7 unchanged lines
872 1575
873 1576
[[package]]
874 1577
name = "oxy-core"
875 -
version = "0.4.3"
1578 +
version = "0.9.0"
876 1579
dependencies = [
877 1580
 "blake3",
878 1581
 "ed25519-dalek",
…
3 unchanged lines
882 1585
 "libc",
883 1586
 "memchr",
884 1587
 "ml-dsa",
885 -
 "rand",
1588 +
 "rand 0.8.8",
886 1589
 "rayon",
887 1590
 "rustls",
888 1591
 "rustls-pemfile",
1592 +
 "serde_json",
1593 +
 "sha2 0.11.0",
889 1594
 "similar",
890 -
 "thiserror",
1595 +
 "thiserror 1.0.69",
891 1596
 "ureq",
892 1597
 "webpki-roots 1.0.9",
893 1598
 "zstd",
…
10 unchanged lines
904 1609
]
905 1610
906 1611
[[package]]
1612 +
name = "pem"
1613 +
version = "4.0.0"
1614 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1615 +
checksum = "d354a98a3d1251555de99e8fdd8afda05573c31b82f59063a7b0a29b5527f120"
1616 +
dependencies = [
1617 +
 "base64 0.23.1",
1618 +
 "serde_core",
1619 +
]
1620 +
1621 +
[[package]]
907 1622
name = "percent-encoding"
908 1623
version = "2.3.2"
909 1624
source = "registry+https://github.com/rust-lang/crates.io-index"
910 1625
checksum = "9b4f627cb1b25917193a259e49bdad08f671f8d9708acfd5fe0a8c1455d87220"
911 1626
912 1627
[[package]]
1628 +
name = "pin-project-lite"
1629 +
version = "0.2.17"
1630 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1631 +
checksum = "a89322df9ebe1c1578d689c92318e070967d1042b512afbe49518723f4e6d5cd"
1632 +
1633 +
[[package]]
913 1634
name = "pkcs8"
914 1635
version = "0.11.0"
915 1636
source = "registry+https://github.com/rust-lang/crates.io-index"
…
10 unchanged lines
926 1647
checksum = "f6b464fbc74e149a392436b17d523f769e057cb6877f6a5c4618bc6f11800548"
927 1648
928 1649
[[package]]
1650 +
name = "postcard"
1651 +
version = "1.1.3"
1652 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1653 +
checksum = "6764c3b5dd454e283a30e6dfe78e9b31096d9e32036b5d1eaac7a6119ccb9a24"
1654 +
dependencies = [
1655 +
 "cobs",
1656 +
 "embedded-io 0.4.0",
1657 +
 "embedded-io 0.6.1",
1658 +
 "serde",
1659 +
]
1660 +
1661 +
[[package]]
929 1662
name = "potential_utf"
930 1663
version = "0.1.6"
931 1664
source = "registry+https://github.com/rust-lang/crates.io-index"
…
3 unchanged lines
935 1668
]
936 1669
937 1670
[[package]]
1671 +
name = "powerfmt"
1672 +
version = "0.2.1"
1673 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1674 +
checksum = "4a6394b9e965e73d0a289ee54f589087e2c676aedf60885baf52c76b771e4958"
1675 +
1676 +
[[package]]
938 1677
name = "ppv-lite86"
939 1678
version = "0.2.21"
940 1679
source = "registry+https://github.com/rust-lang/crates.io-index"
…
12 unchanged lines
953 1692
]
954 1693
955 1694
[[package]]
1695 +
name = "pulley-interpreter"
1696 +
version = "49.0.4"
1697 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1698 +
checksum = "5cd3de43f4f4cff536de8902097edbdf749321b97dda548f36a37918eb85bba3"
1699 +
dependencies = [
1700 +
 "cranelift-bitset",
1701 +
 "log",
1702 +
 "pulley-macros",
1703 +
 "wasmtime-internal-core",
1704 +
]
1705 +
1706 +
[[package]]
1707 +
name = "pulley-macros"
1708 +
version = "49.0.4"
1709 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1710 +
checksum = "8319a8fa5269b4895957f8795861c42822bd26218e495f70060a963323d94847"
1711 +
dependencies = [
1712 +
 "proc-macro2",
1713 +
 "quote",
1714 +
 "syn 2.0.119",
1715 +
]
1716 +
1717 +
[[package]]
956 1718
name = "quote"
957 1719
version = "1.0.47"
958 1720
source = "registry+https://github.com/rust-lang/crates.io-index"
…
20 unchanged lines
979 1741
]
980 1742
981 1743
[[package]]
1744 +
name = "rand"
1745 +
version = "0.10.3"
1746 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1747 +
checksum = "65c9fb96cbc91e3478eaae79a69fcd3f1ae4ad052e471fe6732fff548984b4af"
1748 +
dependencies = [
1749 +
 "chacha20",
1750 +
 "getrandom 0.4.3",
1751 +
 "rand_core 0.10.1",
1752 +
]
1753 +
1754 +
[[package]]
982 1755
name = "rand_chacha"
983 1756
version = "0.3.1"
984 1757
source = "registry+https://github.com/rust-lang/crates.io-index"
…
39 unchanged lines
1024 1797
]
1025 1798
1026 1799
[[package]]
1800 +
name = "rcgen"
1801 +
version = "0.14.10"
1802 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1803 +
checksum = "8774e05a7d0de114588e6a28fe7e71694b82614ed569d86d8b389dfbc98b8ad8"
1804 +
dependencies = [
1805 +
 "pem",
1806 +
 "ring",
1807 +
 "rustls-pki-types",
1808 +
 "time",
1809 +
 "x509-parser",
1810 +
 "yasna",
1811 +
]
1812 +
1813 +
[[package]]
1814 +
name = "regalloc2"
1815 +
version = "0.15.2"
1816 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1817 +
checksum = "757712e8e61590d6d4f5d563483755538b5aa13467837a3b41cd9832509a7f85"
1818 +
dependencies = [
1819 +
 "allocator-api2",
1820 +
 "bumpalo",
1821 +
 "hashbrown 0.17.1",
1822 +
 "log",
1823 +
 "rustc-hash",
1824 +
 "serde",
1825 +
 "smallvec",
1826 +
]
1827 +
1828 +
[[package]]
1027 1829
name = "regex-automata"
1028 1830
version = "0.4.18"
1029 1831
source = "registry+https://github.com/rust-lang/crates.io-index"
…
25 unchanged lines
1055 1857
]
1056 1858
1057 1859
[[package]]
1860 +
name = "rustc-demangle"
1861 +
version = "0.1.28"
1862 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1863 +
checksum = "b74b56ffa8bb2830709a538c2cbcae9aa062db0d2a42563bfb09bdaae44020eb"
1864 +
1865 +
[[package]]
1866 +
name = "rustc-hash"
1867 +
version = "2.1.3"
1868 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1869 +
checksum = "6b1e7f9a428571be2dc5bc0505c13fb6bf936822b894ec87abf8a08a4e51742d"
1870 +
1871 +
[[package]]
1058 1872
name = "rustc_version"
1059 1873
version = "0.4.1"
1060 1874
source = "registry+https://github.com/rust-lang/crates.io-index"
…
3 unchanged lines
1064 1878
]
1065 1879
1066 1880
[[package]]
1881 +
name = "rusticata-macros"
1882 +
version = "4.1.0"
1883 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1884 +
checksum = "faf0c4a6ece9950b9abdb62b1cfcf2a68b3b67a10ba445b3bb85be2a293d0632"
1885 +
dependencies = [
1886 +
 "nom",
1887 +
]
1888 +
1889 +
[[package]]
1067 1890
name = "rustix"
1068 1891
version = "1.1.5"
1069 1892
source = "registry+https://github.com/rust-lang/crates.io-index"
…
7 unchanged lines
1077 1900
]
1078 1901
1079 1902
[[package]]
1903 +
name = "rustix-linux-procfs"
1904 +
version = "0.1.1"
1905 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1906 +
checksum = "2fc84bf7e9aa16c4f2c758f27412dc9841341e16aa682d9c7ac308fe3ee12056"
1907 +
dependencies = [
1908 +
 "once_cell",
1909 +
 "rustix",
1910 +
]
1911 +
1912 +
[[package]]
1080 1913
name = "rustls"
1081 1914
version = "0.23.45"
1082 1915
source = "registry+https://github.com/rust-lang/crates.io-index"
…
38 unchanged lines
1121 1954
]
1122 1955
1123 1956
[[package]]
1957 +
name = "rustversion"
1958 +
version = "1.0.23"
1959 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1960 +
checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f"
1961 +
1962 +
[[package]]
1124 1963
name = "same-file"
1125 1964
version = "1.0.6"
1126 1965
source = "registry+https://github.com/rust-lang/crates.io-index"
…
3 unchanged lines
1130 1969
]
1131 1970
1132 1971
[[package]]
1972 +
name = "scopeguard"
1973 +
version = "1.2.0"
1974 +
source = "registry+https://github.com/rust-lang/crates.io-index"
1975 +
checksum = "94143f37725109f92c262ed2cf5e59bce7498c01bcc1502d7b9afe439a4e9f49"
1976 +
1977 +
[[package]]
1133 1978
name = "semver"
1134 1979
version = "1.0.28"
1135 1980
source = "registry+https://github.com/rust-lang/crates.io-index"
1136 1981
checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd"
1982 +
dependencies = [
1983 +
 "serde",
1984 +
 "serde_core",
1985 +
]
1137 1986
1138 1987
[[package]]
1139 1988
name = "serde"
…
49 unchanged lines
1189 2038
1190 2039
[[package]]
1191 2040
name = "sha2"
2041 +
version = "0.10.9"
2042 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2043 +
checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283"
2044 +
dependencies = [
2045 +
 "cfg-if",
2046 +
 "cpufeatures 0.2.17",
2047 +
 "digest 0.10.7",
2048 +
]
2049 +
2050 +
[[package]]
2051 +
name = "sha2"
1192 2052
version = "0.11.0"
1193 2053
source = "registry+https://github.com/rust-lang/crates.io-index"
1194 2054
checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4"
1195 2055
dependencies = [
1196 2056
 "cfg-if",
1197 -
 "cpufeatures",
1198 -
 "digest",
2057 +
 "cpufeatures 0.3.1",
2058 +
 "digest 0.11.3",
1199 2059
]
1200 2060
1201 2061
[[package]]
…
2 unchanged lines
1204 2064
source = "registry+https://github.com/rust-lang/crates.io-index"
1205 2065
checksum = "09057cb2149ad4cbd2da1e26b351f9a4c354219421229c69c3063e6f61947c4a"
1206 2066
dependencies = [
1207 -
 "digest",
2067 +
 "digest 0.11.3",
1208 2068
 "keccak",
1209 2069
 "sponge-cursor",
1210 2070
]
…
10 unchanged lines
1221 2081
source = "registry+https://github.com/rust-lang/crates.io-index"
1222 2082
checksum = "28d567dcbaf0049cb8ac2608a76cd95ff9e4412e1899d389ee400918ca7537f5"
1223 2083
dependencies = [
1224 -
 "digest",
2084 +
 "digest 0.11.3",
1225 2085
 "rand_core 0.10.1",
1226 2086
]
1227 2087
…
4 unchanged lines
1232 2092
checksum = "3a219298ac11a56ea9a6d2120044824d6f01aeb034955e7af7bc16858527deea"
1233 2093
1234 2094
[[package]]
2095 +
name = "simdutf8"
2096 +
version = "0.1.5"
2097 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2098 +
checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e"
2099 +
2100 +
[[package]]
1235 2101
name = "similar"
1236 2102
version = "2.7.0"
1237 2103
source = "registry+https://github.com/rust-lang/crates.io-index"
1238 2104
checksum = "bbbb5d9659141646ae647b42fe094daf6c6192d1620870b449d9557f748b2daa"
1239 2105
1240 2106
[[package]]
2107 +
name = "slab"
2108 +
version = "0.4.12"
2109 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2110 +
checksum = "0c790de23124f9ab44544d7ac05d60440adc586479ce501c1d6d7da3cd8c9cf5"
2111 +
2112 +
[[package]]
1241 2113
name = "smallvec"
1242 2114
version = "1.16.2"
1243 2115
source = "registry+https://github.com/rust-lang/crates.io-index"
1244 2116
checksum = "f9395f0f0eee849a9b707b2f06bb92a6a422090e2123bb2ef8e87a0e61892a8e"
2117 +
dependencies = [
2118 +
 "serde",
2119 +
]
2120 +
2121 +
[[package]]
2122 +
name = "socket2"
2123 +
version = "0.6.5"
2124 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2125 +
checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4"
2126 +
dependencies = [
2127 +
 "libc",
2128 +
 "windows-sys 0.61.2",
2129 +
]
1245 2130
1246 2131
[[package]]
1247 2132
name = "spki"
…
53 unchanged lines
1301 2186
1302 2187
[[package]]
1303 2188
name = "synstructure"
2189 +
version = "0.13.2"
2190 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2191 +
checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2"
2192 +
dependencies = [
2193 +
 "proc-macro2",
2194 +
 "quote",
2195 +
 "syn 2.0.119",
2196 +
]
2197 +
2198 +
[[package]]
2199 +
name = "synstructure"
1304 2200
version = "0.14.0"
1305 2201
source = "registry+https://github.com/rust-lang/crates.io-index"
1306 2202
checksum = "901704edd0dfe137f1987838ee4f259e4e063c31371bdb423f7ae38ec6f77f02"
…
4 unchanged lines
1311 2207
]
1312 2208
1313 2209
[[package]]
2210 +
name = "target-lexicon"
2211 +
version = "0.13.5"
2212 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2213 +
checksum = "adb6935a6f5c20170eeceb1a3835a49e12e19d792f6dd344ccc76a985ca5a6ca"
2214 +
2215 +
[[package]]
1314 2216
name = "tempfile"
1315 2217
version = "3.27.0"
1316 2218
source = "registry+https://github.com/rust-lang/crates.io-index"
…
7 unchanged lines
1324 2226
]
1325 2227
1326 2228
[[package]]
2229 +
name = "termcolor"
2230 +
version = "1.4.1"
2231 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2232 +
checksum = "06794f8f6c5c898b3275aebefa6b8a1cb24cd2c6c79397ab15774837a0bc5755"
2233 +
dependencies = [
2234 +
 "winapi-util",
2235 +
]
2236 +
2237 +
[[package]]
1327 2238
name = "thiserror"
1328 2239
version = "1.0.69"
1329 2240
source = "registry+https://github.com/rust-lang/crates.io-index"
1330 2241
checksum = "b6aaf5339b578ea85b50e080feb250a3e8ae8cfcdff9a461c9ec2904bc923f52"
1331 2242
dependencies = [
1332 -
 "thiserror-impl",
2243 +
 "thiserror-impl 1.0.69",
1333 2244
]
1334 2245
1335 2246
[[package]]
2247 +
name = "thiserror"
2248 +
version = "2.0.21"
2249 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2250 +
checksum = "09e52cb86a36cede5cb101bf8908837b3e4c6e5e59fe7fd85c23fb56200d189e"
2251 +
dependencies = [
2252 +
 "thiserror-impl 2.0.21",
2253 +
]
2254 +
2255 +
[[package]]
1336 2256
name = "thiserror-impl"
1337 2257
version = "1.0.69"
1338 2258
source = "registry+https://github.com/rust-lang/crates.io-index"
…
5 unchanged lines
1344 2264
]
1345 2265
1346 2266
[[package]]
2267 +
name = "thiserror-impl"
2268 +
version = "2.0.21"
2269 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2270 +
checksum = "fe5197923287db20a58125f0bc85c062f7f2c892de97b18c356f9efb14b28524"
2271 +
dependencies = [
2272 +
 "proc-macro2",
2273 +
 "quote",
2274 +
 "syn 3.0.6",
2275 +
]
2276 +
2277 +
[[package]]
2278 +
name = "time"
2279 +
version = "0.3.55"
2280 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2281 +
checksum = "cdb87b95ec50ddfa440816d227a17b2ccbdda963a316a727fda0fc4334f7d134"
2282 +
dependencies = [
2283 +
 "deranged",
2284 +
 "num-conv",
2285 +
 "powerfmt",
2286 +
 "serde_core",
2287 +
 "time-core",
2288 +
 "time-macros",
2289 +
]
2290 +
2291 +
[[package]]
2292 +
name = "time-core"
2293 +
version = "0.1.9"
2294 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2295 +
checksum = "9e1c906769ad99c88eaa54e728060edef082f8e358ff32030cb7c7d315e81109"
2296 +
2297 +
[[package]]
2298 +
name = "time-macros"
2299 +
version = "0.2.32"
2300 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2301 +
checksum = "7e689342a48d2ea927c87ea50cabf8594854bf940e9310208848d680d668ed85"
2302 +
dependencies = [
2303 +
 "num-conv",
2304 +
 "time-core",
2305 +
]
2306 +
2307 +
[[package]]
1347 2308
name = "tinystr"
1348 2309
version = "0.8.4"
1349 2310
source = "registry+https://github.com/rust-lang/crates.io-index"
…
4 unchanged lines
1354 2315
]
1355 2316
1356 2317
[[package]]
2318 +
name = "tokio"
2319 +
version = "1.54.1"
2320 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2321 +
checksum = "b52fd3255258d26fac7264ab5a76b53c13367136dc011d3d6ad4f8ed070396a3"
2322 +
dependencies = [
2323 +
 "bytes",
2324 +
 "libc",
2325 +
 "mio",
2326 +
 "pin-project-lite",
2327 +
 "socket2",
2328 +
 "windows-sys 0.61.2",
2329 +
]
2330 +
2331 +
[[package]]
1357 2332
name = "toml"
1358 2333
version = "0.8.23"
1359 2334
source = "registry+https://github.com/rust-lang/crates.io-index"
…
35 unchanged lines
1395 2370
checksum = "5d99f8c9a7727884afe522e9bd5edbfc91a3312b36a77b5fb8926e4c31a41801"
1396 2371
1397 2372
[[package]]
2373 +
name = "tracing"
2374 +
version = "0.1.44"
2375 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2376 +
checksum = "63e71662fa4b2a2c3a26f570f037eb95bb1f85397f3cd8076caed2f026a6d100"
2377 +
dependencies = [
2378 +
 "pin-project-lite",
2379 +
 "tracing-attributes",
2380 +
 "tracing-core",
2381 +
]
2382 +
2383 +
[[package]]
2384 +
name = "tracing-attributes"
2385 +
version = "0.1.31"
2386 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2387 +
checksum = "7490cfa5ec963746568740651ac6781f701c9c5ea257c58e057f3ba8cf69e8da"
2388 +
dependencies = [
2389 +
 "proc-macro2",
2390 +
 "quote",
2391 +
 "syn 2.0.119",
2392 +
]
2393 +
2394 +
[[package]]
2395 +
name = "tracing-core"
2396 +
version = "0.1.36"
2397 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2398 +
checksum = "db97caf9d906fbde555dd62fa95ddba9eecfd14cb388e4f491a66d74cd5fb79a"
2399 +
dependencies = [
2400 +
 "once_cell",
2401 +
]
2402 +
2403 +
[[package]]
1398 2404
name = "typenum"
1399 2405
version = "1.20.1"
1400 2406
source = "registry+https://github.com/rust-lang/crates.io-index"
…
12 unchanged lines
1413 2419
checksum = "7dd6e30e90baa6f72411720665d41d89b9a3d039dc45b8faea1ddd07f617f6af"
1414 2420
1415 2421
[[package]]
2422 +
name = "unicode-width"
2423 +
version = "0.2.2"
2424 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2425 +
checksum = "b4ac048d71ede7ee76d585517add45da530660ef4390e49b098733c6e897f254"
2426 +
2427 +
[[package]]
1416 2428
name = "untrusted"
1417 2429
version = "0.9.0"
1418 2430
source = "registry+https://github.com/rust-lang/crates.io-index"
…
5 unchanged lines
1424 2436
source = "registry+https://github.com/rust-lang/crates.io-index"
1425 2437
checksum = "02d1a66277ed75f640d608235660df48c8e3c19f3b4edb6a263315626cc3c01d"
1426 2438
dependencies = [
1427 -
 "base64",
2439 +
 "base64 0.22.1",
1428 2440
 "flate2",
1429 2441
 "log",
1430 2442
 "once_cell",
…
30 unchanged lines
1461 2473
checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821"
1462 2474
1463 2475
[[package]]
2476 +
name = "version_check"
2477 +
version = "0.9.5"
2478 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2479 +
checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
2480 +
2481 +
[[package]]
1464 2482
name = "walkdir"
1465 2483
version = "2.5.0"
1466 2484
source = "registry+https://github.com/rust-lang/crates.io-index"
…
10 unchanged lines
1477 2495
checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b"
1478 2496
1479 2497
[[package]]
2498 +
name = "wasm-encoder"
2499 +
version = "0.258.3"
2500 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2501 +
checksum = "f64765e43c000fae008a48db8b288e175e237fc9ce20a26586836878c1bee89f"
2502 +
dependencies = [
2503 +
 "leb128fmt",
2504 +
 "wasmparser 0.258.3",
2505 +
]
2506 +
2507 +
[[package]]
2508 +
name = "wasm-encoder"
2509 +
version = "0.262.0"
2510 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2511 +
checksum = "4495a6f85dd94c411c749303efcbc5defade0b0cbdbc0541495241ab917a9b5a"
2512 +
dependencies = [
2513 +
 "leb128fmt",
2514 +
 "wasmparser 0.262.0",
2515 +
]
2516 +
2517 +
[[package]]
2518 +
name = "wasm-metadata"
2519 +
version = "0.258.3"
2520 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2521 +
checksum = "47e10808eb27efde692f1b1706089fe19296728bf872291f5b4e0d50fba88f2f"
2522 +
dependencies = [
2523 +
 "anyhow",
2524 +
 "indexmap",
2525 +
 "wasm-encoder 0.258.3",
2526 +
 "wasmparser 0.258.3",
2527 +
]
2528 +
2529 +
[[package]]
2530 +
name = "wasmparser"
2531 +
version = "0.258.3"
2532 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2533 +
checksum = "fe9269dd817e0e7021bf534ccd9761bc2d64778b6708e41228b8439d75eaa0b1"
2534 +
dependencies = [
2535 +
 "bitflags",
2536 +
 "hashbrown 0.17.1",
2537 +
 "indexmap",
2538 +
 "semver",
2539 +
 "serde",
2540 +
]
2541 +
2542 +
[[package]]
2543 +
name = "wasmparser"
2544 +
version = "0.262.0"
2545 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2546 +
checksum = "974d643b125b272607e2fff8012ea14a8328acd7621e4bafa7afb32500052b1b"
2547 +
dependencies = [
2548 +
 "bitflags",
2549 +
 "indexmap",
2550 +
 "semver",
2551 +
]
2552 +
2553 +
[[package]]
2554 +
name = "wasmprinter"
2555 +
version = "0.258.3"
2556 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2557 +
checksum = "14a457ba8670a3db3d357c1aeac3ed4fe378a76054a221789f0df0ea507e292b"
2558 +
dependencies = [
2559 +
 "anyhow",
2560 +
 "termcolor",
2561 +
 "wasmparser 0.258.3",
2562 +
]
2563 +
2564 +
[[package]]
2565 +
name = "wasmtime"
2566 +
version = "49.0.4"
2567 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2568 +
checksum = "36abb25de5518707e5ba96b586366b09db168d2d4976cb9dac57265a317497ea"
2569 +
dependencies = [
2570 +
 "addr2line",
2571 +
 "async-trait",
2572 +
 "bitflags",
2573 +
 "bumpalo",
2574 +
 "cc",
2575 +
 "encoding_rs",
2576 +
 "futures",
2577 +
 "libc",
2578 +
 "log",
2579 +
 "mach2",
2580 +
 "memfd",
2581 +
 "object",
2582 +
 "once_cell",
2583 +
 "postcard",
2584 +
 "pulley-interpreter",
2585 +
 "rayon",
2586 +
 "rustix",
2587 +
 "semver",
2588 +
 "serde",
2589 +
 "serde_derive",
2590 +
 "smallvec",
2591 +
 "target-lexicon",
2592 +
 "wasmparser 0.258.3",
2593 +
 "wasmtime-environ",
2594 +
 "wasmtime-internal-component-macro",
2595 +
 "wasmtime-internal-component-util",
2596 +
 "wasmtime-internal-core",
2597 +
 "wasmtime-internal-cranelift",
2598 +
 "wasmtime-internal-fiber",
2599 +
 "wasmtime-internal-jit-debug",
2600 +
 "wasmtime-internal-jit-icache-coherence",
2601 +
 "wasmtime-internal-unwinder",
2602 +
 "wasmtime-internal-versioned-export-macros",
2603 +
 "windows-sys 0.61.2",
2604 +
]
2605 +
2606 +
[[package]]
2607 +
name = "wasmtime-environ"
2608 +
version = "49.0.4"
2609 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2610 +
checksum = "92f16a4d09fb1fbf2dcfeddb9818e1517a258e13b505b3bfdadcf47a19bf31ca"
2611 +
dependencies = [
2612 +
 "anyhow",
2613 +
 "cpp_demangle",
2614 +
 "cranelift-bforest",
2615 +
 "cranelift-bitset",
2616 +
 "cranelift-entity",
2617 +
 "gimli",
2618 +
 "hashbrown 0.17.1",
2619 +
 "indexmap",
2620 +
 "log",
2621 +
 "object",
2622 +
 "postcard",
2623 +
 "rustc-demangle",
2624 +
 "semver",
2625 +
 "serde",
2626 +
 "serde_derive",
2627 +
 "sha2 0.10.9",
2628 +
 "smallvec",
2629 +
 "target-lexicon",
2630 +
 "wasm-encoder 0.258.3",
2631 +
 "wasmparser 0.258.3",
2632 +
 "wasmprinter",
2633 +
 "wasmtime-internal-component-util",
2634 +
 "wasmtime-internal-core",
2635 +
]
2636 +
2637 +
[[package]]
2638 +
name = "wasmtime-internal-component-macro"
2639 +
version = "49.0.4"
2640 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2641 +
checksum = "d4fca03bd4425856574c0807d1dee16fad43f947feaf4e9ae1d2129596ee4de4"
2642 +
dependencies = [
2643 +
 "anyhow",
2644 +
 "proc-macro2",
2645 +
 "quote",
2646 +
 "syn 2.0.119",
2647 +
 "wasmtime-internal-component-util",
2648 +
 "wasmtime-internal-wit-bindgen",
2649 +
 "wit-parser",
2650 +
]
2651 +
2652 +
[[package]]
2653 +
name = "wasmtime-internal-component-util"
2654 +
version = "49.0.4"
2655 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2656 +
checksum = "4006009fc3598af4e895ab18c6c33d2c6b7c657b1bd0a755ca7ee1abef6f0c16"
2657 +
2658 +
[[package]]
2659 +
name = "wasmtime-internal-core"
2660 +
version = "49.0.4"
2661 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2662 +
checksum = "947a732ad659f42c10ec639706fc8cb1f068dd9a55eebde8a1f24dfd06dfa525"
2663 +
dependencies = [
2664 +
 "hashbrown 0.17.1",
2665 +
 "libm",
2666 +
 "serde",
2667 +
]
2668 +
2669 +
[[package]]
2670 +
name = "wasmtime-internal-cranelift"
2671 +
version = "49.0.4"
2672 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2673 +
checksum = "fbfb8b455d524939e03df42e3e749f8e26e26ba2acd8b08fa00b3bda55686750"
2674 +
dependencies = [
2675 +
 "cranelift-codegen",
2676 +
 "cranelift-control",
2677 +
 "cranelift-entity",
2678 +
 "cranelift-frontend",
2679 +
 "cranelift-native",
2680 +
 "gimli",
2681 +
 "itertools",
2682 +
 "log",
2683 +
 "object",
2684 +
 "pulley-interpreter",
2685 +
 "smallvec",
2686 +
 "target-lexicon",
2687 +
 "thiserror 2.0.21",
2688 +
 "wasmparser 0.258.3",
2689 +
 "wasmtime-environ",
2690 +
 "wasmtime-internal-core",
2691 +
 "wasmtime-internal-unwinder",
2692 +
 "wasmtime-internal-versioned-export-macros",
2693 +
]
2694 +
2695 +
[[package]]
2696 +
name = "wasmtime-internal-fiber"
2697 +
version = "49.0.4"
2698 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2699 +
checksum = "bd8aa8711f68b2cdd94c2b85ba5c8317edaa1ac4ee0a0534b957123680239307"
2700 +
dependencies = [
2701 +
 "cc",
2702 +
 "libc",
2703 +
 "rustix",
2704 +
 "wasmtime-environ",
2705 +
 "wasmtime-internal-versioned-export-macros",
2706 +
 "windows-sys 0.61.2",
2707 +
]
2708 +
2709 +
[[package]]
2710 +
name = "wasmtime-internal-jit-debug"
2711 +
version = "49.0.4"
2712 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2713 +
checksum = "e873e4504faf50aa8ce255d6ea7210d87b39d5f379c2486782e2c5f179c70ca8"
2714 +
dependencies = [
2715 +
 "cc",
2716 +
 "wasmtime-internal-versioned-export-macros",
2717 +
]
2718 +
2719 +
[[package]]
2720 +
name = "wasmtime-internal-jit-icache-coherence"
2721 +
version = "49.0.4"
2722 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2723 +
checksum = "197ab1fda5da958a5c0b8a97eb241beafc48ac568b82ef4f12f38d6866c74173"
2724 +
dependencies = [
2725 +
 "libc",
2726 +
 "wasmtime-internal-core",
2727 +
 "windows-sys 0.61.2",
2728 +
]
2729 +
2730 +
[[package]]
2731 +
name = "wasmtime-internal-unwinder"
2732 +
version = "49.0.4"
2733 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2734 +
checksum = "5577f3a5a979a7c3b1b23c1e9f13e70a8583acc8828471ec56f282117fc4952b"
2735 +
dependencies = [
2736 +
 "cranelift-codegen",
2737 +
 "log",
2738 +
 "object",
2739 +
 "wasmtime-environ",
2740 +
]
2741 +
2742 +
[[package]]
2743 +
name = "wasmtime-internal-versioned-export-macros"
2744 +
version = "49.0.4"
2745 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2746 +
checksum = "f6399c9c4050ed431cf66e21c05fadd6f9731db44476f452f8c7142b4597b883"
2747 +
dependencies = [
2748 +
 "proc-macro2",
2749 +
 "quote",
2750 +
 "syn 2.0.119",
2751 +
]
2752 +
2753 +
[[package]]
2754 +
name = "wasmtime-internal-wit-bindgen"
2755 +
version = "49.0.4"
2756 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2757 +
checksum = "b4eaf45d4da7681d234bb6efd846b4a528b08a75384a7fc23f125d0145364b06"
2758 +
dependencies = [
2759 +
 "anyhow",
2760 +
 "bitflags",
2761 +
 "heck",
2762 +
 "indexmap",
2763 +
 "wit-component",
2764 +
 "wit-parser",
2765 +
]
2766 +
2767 +
[[package]]
2768 +
name = "wasmtime-wasi"
2769 +
version = "49.0.4"
2770 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2771 +
checksum = "821e45765c35555d3b0bdf9277cb99478f7d1f29a17b3315ba76e01220f02a38"
2772 +
dependencies = [
2773 +
 "async-trait",
2774 +
 "bitflags",
2775 +
 "bytes",
2776 +
 "cap-primitives",
2777 +
 "futures",
2778 +
 "rand 0.10.3",
2779 +
 "rustix",
2780 +
 "rustix-linux-procfs",
2781 +
 "thiserror 2.0.21",
2782 +
 "tokio",
2783 +
 "tracing",
2784 +
 "url",
2785 +
 "wasmtime",
2786 +
 "wasmtime-wasi-io",
2787 +
 "wiggle",
2788 +
 "windows-sys 0.61.2",
2789 +
 "winx",
2790 +
]
2791 +
2792 +
[[package]]
2793 +
name = "wasmtime-wasi-io"
2794 +
version = "49.0.4"
2795 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2796 +
checksum = "8cdff38d1c730bdaf4fcfbe0eedf3265fb9c7a4deb119deecbfc1c1778950716"
2797 +
dependencies = [
2798 +
 "async-trait",
2799 +
 "bytes",
2800 +
 "futures",
2801 +
 "tracing",
2802 +
 "wasmtime",
2803 +
]
2804 +
2805 +
[[package]]
2806 +
name = "wast"
2807 +
version = "35.0.2"
2808 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2809 +
checksum = "2ef140f1b49946586078353a453a1d28ba90adfc54dde75710bc1931de204d68"
2810 +
dependencies = [
2811 +
 "leb128",
2812 +
]
2813 +
2814 +
[[package]]
2815 +
name = "wast"
2816 +
version = "262.0.0"
2817 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2818 +
checksum = "30612a0c29249b5f185888a107e8a519d920226b7d07f9d351fd094e9f75eb23"
2819 +
dependencies = [
2820 +
 "bumpalo",
2821 +
 "leb128fmt",
2822 +
 "memchr",
2823 +
 "unicode-width 0.2.2",
2824 +
 "wasm-encoder 0.262.0",
2825 +
]
2826 +
2827 +
[[package]]
2828 +
name = "wat"
2829 +
version = "1.262.0"
2830 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2831 +
checksum = "e228027f2b07bbaf946980ebbe2437fca73be6f54c40c2a8d4f42f277075f8d1"
2832 +
dependencies = [
2833 +
 "wast 262.0.0",
2834 +
]
2835 +
2836 +
[[package]]
1480 2837
name = "webpki-roots"
1481 2838
version = "0.26.11"
1482 2839
source = "registry+https://github.com/rust-lang/crates.io-index"
…
12 unchanged lines
1495 2852
]
1496 2853
1497 2854
[[package]]
2855 +
name = "wiggle"
2856 +
version = "49.0.4"
2857 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2858 +
checksum = "af3872869836d404f722e2361f869c09630e43a73fcc7060f7892a929b74b6a3"
2859 +
dependencies = [
2860 +
 "bitflags",
2861 +
 "thiserror 2.0.21",
2862 +
 "tracing",
2863 +
 "wasmtime",
2864 +
 "wasmtime-environ",
2865 +
 "wiggle-macro",
2866 +
]
2867 +
2868 +
[[package]]
2869 +
name = "wiggle-generate"
2870 +
version = "49.0.4"
2871 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2872 +
checksum = "cf7e31dec6de37e7e848aecbad54777dd246cd440a1a547ab1a1e31c27f55e49"
2873 +
dependencies = [
2874 +
 "heck",
2875 +
 "proc-macro2",
2876 +
 "quote",
2877 +
 "syn 2.0.119",
2878 +
 "wasmtime-environ",
2879 +
 "witx",
2880 +
]
2881 +
2882 +
[[package]]
2883 +
name = "wiggle-macro"
2884 +
version = "49.0.4"
2885 +
source = "registry+https://github.com/rust-lang/crates.io-index"
2886 +
checksum = "a2af323f33b28ea84b24e2bccef19e51c7a75aa1d2929280ad1a4865f2877dcc"
2887 +
dependencies = [
2888 +
 "proc-macro2",
2889 +
 "quote",
2890 +
 "syn 2.0.119",
2891 +
 "wiggle-generate",
2892 +
]
2893 +
2894 +
[[package]]
1498 2895
name = "winapi"
1499 2896
version = "0.3.9"
1500 2897
source = "registry+https://github.com/rust-lang/crates.io-index"
…
122 unchanged lines
1623 3020
]
1624 3021
1625 3022
[[package]]
3023 +
name = "winx"
3024 +
version = "0.36.4"
3025 +
source = "registry+https://github.com/rust-lang/crates.io-index"
3026 +
checksum = "3f3fd376f71958b862e7afb20cfe5a22830e1963462f3a17f49d82a6c1d1f42d"
3027 +
dependencies = [
3028 +
 "bitflags",
3029 +
 "windows-sys 0.52.0",
3030 +
]
3031 +
3032 +
[[package]]
3033 +
name = "wit-component"
3034 +
version = "0.258.3"
3035 +
source = "registry+https://github.com/rust-lang/crates.io-index"
3036 +
checksum = "f0fca88950af0c1f42bc77e17315e48798b8d68340171b6d204f62bc19a878a6"
3037 +
dependencies = [
3038 +
 "anyhow",
3039 +
 "bitflags",
3040 +
 "indexmap",
3041 +
 "log",
3042 +
 "serde",
3043 +
 "serde_derive",
3044 +
 "serde_json",
3045 +
 "wasm-encoder 0.258.3",
3046 +
 "wasm-metadata",
3047 +
 "wasmparser 0.258.3",
3048 +
 "wit-parser",
3049 +
]
3050 +
3051 +
[[package]]
3052 +
name = "wit-parser"
3053 +
version = "0.258.3"
3054 +
source = "registry+https://github.com/rust-lang/crates.io-index"
3055 +
checksum = "72a2127663dcf3ee16614fb0e3c33d5c6c66fec9dad317a8b0fdd3869487a533"
3056 +
dependencies = [
3057 +
 "anyhow",
3058 +
 "hashbrown 0.17.1",
3059 +
 "id-arena",
3060 +
 "indexmap",
3061 +
 "log",
3062 +
 "semver",
3063 +
 "serde",
3064 +
 "serde_derive",
3065 +
 "serde_json",
3066 +
 "unicode-ident",
3067 +
 "wasmparser 0.258.3",
3068 +
]
3069 +
3070 +
[[package]]
3071 +
name = "witx"
3072 +
version = "0.9.1"
3073 +
source = "registry+https://github.com/rust-lang/crates.io-index"
3074 +
checksum = "e366f27a5cabcddb2706a78296a40b8fcc451e1a6aba2fc1d94b4a01bdaaef4b"
3075 +
dependencies = [
3076 +
 "anyhow",
3077 +
 "log",
3078 +
 "thiserror 1.0.69",
3079 +
 "wast 35.0.2",
3080 +
]
3081 +
3082 +
[[package]]
1626 3083
name = "writeable"
1627 3084
version = "0.6.4"
1628 3085
source = "registry+https://github.com/rust-lang/crates.io-index"
1629 3086
checksum = "3ad82d2a33cdc9674dc7465672f271e096168fcdbe0f799d9e6db8c5892679dc"
1630 3087
1631 3088
[[package]]
3089 +
name = "x509-parser"
3090 +
version = "0.18.1"
3091 +
source = "registry+https://github.com/rust-lang/crates.io-index"
3092 +
checksum = "d43b0f71ce057da06bc0851b23ee24f3f86190b07203dd8f567d0b706a185202"
3093 +
dependencies = [
3094 +
 "asn1-rs",
3095 +
 "data-encoding",
3096 +
 "der-parser",
3097 +
 "lazy_static",
3098 +
 "nom",
3099 +
 "oid-registry",
3100 +
 "ring",
3101 +
 "rusticata-macros",
3102 +
 "thiserror 2.0.21",
3103 +
 "time",
3104 +
]
3105 +
3106 +
[[package]]
3107 +
name = "yasna"
3108 +
version = "0.6.0"
3109 +
source = "registry+https://github.com/rust-lang/crates.io-index"
3110 +
checksum = "b5f6765e852b9b4dc8e2a76843e4d64d1cea8e79bcde0b6901aea8e7c7f08282"
3111 +
dependencies = [
3112 +
 "bit-vec",
3113 +
 "time",
3114 +
]
3115 +
3116 +
[[package]]
1632 3117
name = "yoke"
1633 3118
version = "0.8.3"
1634 3119
source = "registry+https://github.com/rust-lang/crates.io-index"
…
13 unchanged lines
1648 3133
 "proc-macro2",
1649 3134
 "quote",
1650 3135
 "syn 3.0.6",
1651 -
 "synstructure",
3136 +
 "synstructure 0.14.0",
1652 3137
]
1653 3138
1654 3139
[[package]]
…
34 unchanged lines
1689 3174
 "proc-macro2",
1690 3175
 "quote",
1691 3176
 "syn 3.0.6",
1692 -
 "synstructure",
3177 +
 "synstructure 0.14.0",
1693 3178
]
1694 3179
1695 3180
[[package]]
…
77 unchanged lines
modified Cargo.toml +3 −2
1 1
[workspace]
2 -
members = ["crates/flow-core", "crates/flow-init", "crates/flow", "crates/flow-casfs", "crates/jet"]
2 +
members = ["crates/flow-core", "crates/flow-init", "crates/flow", "crates/flow-casfs", "crates/jet", "crates/flow-checks"]
3 3
# oxygen as a pinned subrepo (oxy submodule): the state flow builds against.
4 4
exclude = ["deps"]
5 5
resolver = "2"
6 6
7 7
[workspace.package]
8 -
version = "0.1.0"
8 +
version = "0.5.3"
9 9
edition = "2021"
10 10
license = "MIT OR Apache-2.0"
11 11
…
1 unchanged lines
13 13
oxy-core = { path = "deps/oxygen/crates/oxy-core" }
14 14
flow-core = { path = "crates/flow-core" }
15 15
jet = { path = "crates/jet" }
16 +
flow-init = { path = "crates/flow-init" }
16 17
serde = { version = "1", features = ["derive"] }
17 18
serde_json = "1"
18 19
blake3 = "1"
…
7 unchanged lines
modified Makefile +1 −1
…
21 unchanged lines
22 22
		mv bin/flow.new bin/flow; mv bin/flow-init.new bin/flow-init; \
23 23
		h=$$(bin/flow hash bin/flow-init | cut -d" " -f1); \
24 24
		sed -i "s|^init = .*|init = { path = \"bin/flow-init\", blake3 = \"$$h\" }|" flow.toml; \
25 -
		sudo systemctl restart flow-runner; systemctl is-active flow-runner'
25 +
		sudo systemctl restart --no-block flow-runner; echo "flow-runner: restarting (it drains: the job it holds finishes first)"'
26 26
27 27
# Move the pinned oxygen (deps/oxygen) to its main on Earth; the next commit
28 28
# records the new pin. Push oxygen first.
…
3 unchanged lines
modified README.md +23 −0
…
53 unchanged lines
54 54
under `/home/claude/flow` (keeping the previous files as `.prev`), re-pins
55 55
`flow-init` in `flow.toml` and restarts the `flow-runner` service.
56 56
57 +
## A Windows runner
58 +
59 +
A Windows machine runs WASM jobs (the other tiers are Linux's). Each
60 +
release carries `flow-windows-x86_64.exe` (from the `dist` job):
61 +
62 +
    flow.exe --home C:\flow runner register https://oxygenhub.io --token flowreg_… --name win-1 --label windows
63 +
    flow.exe --home C:\flow runner
64 +
65 +
The token comes from the hub (Admin › Runners, or `POST /api/v1/user/runner-token`
66 +
for a runner of your own repositories). A pipeline sends it work with a
67 +
`runtime "wasm"` sandbox and `runs-on "windows"`. To keep it running, a
68 +
Task Scheduler task at start-up with that command (restarting on failure)
69 +
does; set it to restart on failure, as a runner that leaves for a new build
70 +
exits 75 (`flow runner update`, spec §11.5.3). To stop it without losing
71 +
its job, `flow.exe --home C:\flow runner drain` (Windows has no SIGTERM;
72 +
ending the task kills it). What a Windows runner does not report yet: load
73 +
and kernel (oxygenflow#83).
74 +
57 75
## Commands
58 76
59 77
    flow check                 # validate .flow/pipeline.kdl, print jobs and the pipeline hash
…
3 unchanged lines
63 81
    flow rootfs list
64 82
    flow hash FILE             # BLAKE3, for pinning in flow.toml
65 83
    flow runner                # claim and run jobs for the hub in flow.toml
84 +
    flow runner update --from FILE [--dry-run]   # install a new flow build in place of this one (spec §11.5.3)
85 +
    flow runner drain          # the running runner claims nothing more and stops once its jobs are reported
66 86
67 87
## A Flow home
68 88
…
45 unchanged lines
114 134
115 135
## Tests
116 136
137 +
**Changelog:** a commit that changes code adds a line under `## Unreleased` in CHANGELOG.md (Added, Changed, Fixed or Security, with the issue number); versions change only at release. `scripts/changelog-check.sh [PATH...]` checks it for the paths a commit takes (the working copy's changes without any) and runs in the gate; `NO_CHANGELOG=1` for code changes users won't notice.
138 +
139 +
117 140
`cargo test --workspace` runs every test that needs no VM. The microVM
118 141
conformance test also runs when `FLOW_TEST_HOME` names a Flow home and
119 142
`FLOW_TEST_ROOTFS` is the hash of an Alpine rootfs in it. The test checks
…
6 unchanged lines
added crates/flow-checks/Cargo.toml +16 −0
1 +
[package]
2 +
name = "flow-checks"
3 +
version.workspace = true
4 +
edition.workspace = true
5 +
license.workspace = true
6 +
description = "OxygenFlow's cheap checks as one WASI module, so any runner with the WASM tier can take them (#96)"
7 +
8 +
# Built for wasm32-wasip1 by scripts/checks-wasm.sh and committed as
9 +
# .flow/checks.wasm; std only, so the build is the same on any machine with
10 +
# the same toolchain.
11 +
[[bin]]
12 +
name = "flow-checks"
13 +
path = "src/main.rs"
14 +
15 +
[dev-dependencies]
16 +
tempfile.workspace = true
added crates/flow-checks/src/main.rs +362 −0
1 +
//! OxygenFlow's cheap checks as one WASI module (#96): what
2 +
//! `.flow/jobs/{layout,todos,size,summary}.sh` did in an Alpine microVM, so
3 +
//! that any runner offering the WASM tier (a phone, plex, windows-1) can take
4 +
//! them. The job's files are preopened as `/src`, which is also the current
5 +
//! directory, so every path here is relative to the workspace.
6 +
//!
7 +
//! - `layout PATH...`: each path is a file; fails naming the first missing.
8 +
//! - `todos DIR...`: the lines holding a note marker, counted, the first 20
9 +
//!   listed; never fails.
10 +
//! - `size DIR`: lines of Rust in each crate directory under DIR.
11 +
//! - `summary`: one line naming `$PROJECT`.
12 +
//! - `changelog FILE [MANIFEST]`: the changelog's shape: `## Unreleased`
13 +
//!   first if at all, then `## X.Y.Z (YYYY-MM-DD)` sections, newest first,
14 +
//!   none empty; with a manifest (`Cargo.toml`, `*.properties`,
15 +
//!   `package.json`), its version has a section, as a release needs.
16 +
17 +
use std::fs;
18 +
use std::io::{self, Write};
19 +
use std::path::{Path, PathBuf};
20 +
use std::process::ExitCode;
21 +
22 +
/// The note markers, spelt in pieces so this file does not count itself.
23 +
const MARKERS: [&str; 3] = [concat!("TO", "DO"), concat!("FIX", "ME"), concat!("X", "XX")];
24 +
25 +
/// How many noted lines `todos` lists.
26 +
const LISTED: usize = 20;
27 +
28 +
const USAGE: &str = "usage: flow-checks layout PATH... | todos DIR... | size DIR | summary | changelog FILE [MANIFEST]";
29 +
30 +
fn main() -> ExitCode {
31 +
    let args: Vec<String> = std::env::args().skip(1).collect();
32 +
    let root = Path::new(".");
33 +
    let mut out = io::stdout().lock();
34 +
    let done = match args.split_first() {
35 +
        Some((cmd, rest)) => match cmd.as_str() {
36 +
            "layout" => layout(root, rest, &mut out),
37 +
            "todos" => todos(root, rest, &mut out),
38 +
            "size" => match rest {
39 +
                [dir] => size(root, dir, &mut out),
40 +
                _ => Err(USAGE.to_string()),
41 +
            },
42 +
            "summary" => summary(std::env::var("PROJECT").ok().as_deref(), &mut out),
43 +
            "changelog" => match rest {
44 +
                [file] => changelog(root, file, None, &mut out),
45 +
                [file, manifest] => changelog(root, file, Some(manifest), &mut out),
46 +
                _ => Err(USAGE.to_string()),
47 +
            },
48 +
            _ => Err(USAGE.to_string()),
49 +
        },
50 +
        None => Err(USAGE.to_string()),
51 +
    };
52 +
    match done {
53 +
        Ok(()) => ExitCode::SUCCESS,
54 +
        Err(e) => {
55 +
            eprintln!("{e}");
56 +
            ExitCode::FAILURE
57 +
        }
58 +
    }
59 +
}
60 +
61 +
type Done = Result<(), String>;
62 +
63 +
fn emit(out: &mut impl Write, line: &str) -> Done {
64 +
    writeln!(out, "{line}").map_err(|e| format!("write: {e}"))
65 +
}
66 +
67 +
/// The workspace has the files it should.
68 +
fn layout(root: &Path, paths: &[String], out: &mut impl Write) -> Done {
69 +
    if paths.is_empty() {
70 +
        return Err(USAGE.to_string());
71 +
    }
72 +
    for p in paths {
73 +
        if !root.join(p).is_file() {
74 +
            return Err(format!("missing: {p}"));
75 +
        }
76 +
        emit(out, &format!("found {p}"))?;
77 +
    }
78 +
    Ok(())
79 +
}
80 +
81 +
/// Notes left in the code, for the log.
82 +
fn todos(root: &Path, dirs: &[String], out: &mut impl Write) -> Done {
83 +
    if dirs.is_empty() {
84 +
        return Err(USAGE.to_string());
85 +
    }
86 +
    let mut noted = Vec::new();
87 +
    for d in dirs {
88 +
        for file in files(&root.join(d))? {
89 +
            // Text only, as grep reports a binary file rather than its lines.
90 +
            let Ok(text) = fs::read_to_string(&file) else { continue };
91 +
            let name = shown(root, &file);
92 +
            for (i, line) in text.lines().enumerate() {
93 +
                if MARKERS.iter().any(|m| line.contains(m)) {
94 +
                    noted.push(format!("{name}:{}:{line}", i + 1));
95 +
                }
96 +
            }
97 +
        }
98 +
    }
99 +
    emit(out, &format!("{} {} notes in {}", noted.len(), MARKERS.join("/"), shown_dirs(dirs)))?;
100 +
    for line in noted.iter().take(LISTED) {
101 +
        emit(out, line)?;
102 +
    }
103 +
    Ok(())
104 +
}
105 +
106 +
/// Lines of Rust per crate: each directory directly under `dir`.
107 +
fn size(root: &Path, dir: &str, out: &mut impl Write) -> Done {
108 +
    for krate in entries(&root.join(dir))?.into_iter().filter(|p| p.is_dir()) {
109 +
        let mut lines = 0;
110 +
        for file in files(&krate)?.into_iter().filter(|f| f.extension().is_some_and(|e| e == "rs")) {
111 +
            let bytes = fs::read(&file).map_err(|e| format!("{}: {e}", file.display()))?;
112 +
            lines += bytes.iter().filter(|&&b| b == b'\n').count();
113 +
        }
114 +
        emit(out, &format!("{:<24} {lines:>6} lines", format!("{}/", shown(root, &krate))))?;
115 +
    }
116 +
    Ok(())
117 +
}
118 +
119 +
fn summary(project: Option<&str>, out: &mut impl Write) -> Done {
120 +
    emit(out, &format!("{}: every check passed on this state.", project.unwrap_or("This project")))
121 +
}
122 +
123 +
/// A directory's entries, sorted by name so every runner prints the same.
124 +
/// A version as three numbers, or `None`.
125 +
fn version(s: &str) -> Option<(u64, u64, u64)> {
126 +
    let mut parts = s.split('.').map(|p| p.parse::<u64>().ok());
127 +
    let v = (parts.next()??, parts.next()??, parts.next()??);
128 +
    parts.next().is_none().then_some(v)
129 +
}
130 +
131 +
/// A `YYYY-MM-DD` date with a real month and day.
132 +
fn is_date(s: &str) -> bool {
133 +
    let b = s.as_bytes();
134 +
    b.len() == 10
135 +
        && b[4] == b'-'
136 +
        && b[7] == b'-'
137 +
        && s[..4].parse::<u32>().is_ok()
138 +
        && s[5..7].parse::<u32>().is_ok_and(|m| (1..=12).contains(&m))
139 +
        && s[8..].parse::<u32>().is_ok_and(|d| (1..=31).contains(&d))
140 +
}
141 +
142 +
/// The version a manifest declares: `version = "…"` in `Cargo.toml`'s
143 +
/// `[package]` or `[workspace.package]`, a `….version=` key of a
144 +
/// `.properties` file, or `"version"` in a `package.json`.
145 +
fn manifest_version(path: &str, text: &str) -> Option<String> {
146 +
    let unquote = |v: &str| v.trim().trim_matches('"').trim_end_matches(',').trim_matches('"').to_string();
147 +
    if path.ends_with(".properties") {
148 +
        return text.lines().find_map(|l| {
149 +
            let (k, v) = l.split_once('=')?;
150 +
            (k.trim() == "version" || k.trim().ends_with(".version")).then(|| v.trim().to_string())
151 +
        });
152 +
    }
153 +
    if path.ends_with(".json") {
154 +
        return text.lines().find_map(|l| l.trim().strip_prefix("\"version\":").map(unquote));
155 +
    }
156 +
    let mut section = "";
157 +
    for l in text.lines() {
158 +
        let l = l.trim();
159 +
        if l.starts_with('[') {
160 +
            section = l;
161 +
        } else if matches!(section, "[package]" | "[workspace.package]") {
162 +
            if let Some(v) = l.strip_prefix("version").and_then(|r| r.trim_start().strip_prefix('=')) {
163 +
                return Some(unquote(v));
164 +
            }
165 +
        }
166 +
    }
167 +
    None
168 +
}
169 +
170 +
/// The changelog's shape, and the manifest's version in it.
171 +
fn changelog(root: &Path, file: &str, manifest: Option<&String>, out: &mut impl Write) -> Done {
172 +
    let text = fs::read_to_string(root.join(file)).map_err(|e| format!("{file}: {e}"))?;
173 +
    let mut sections: Vec<(usize, String, usize)> = Vec::new(); // (line, heading, entries)
174 +
    for (i, line) in text.lines().enumerate() {
175 +
        if let Some(h) = line.strip_prefix("## ") {
176 +
            sections.push((i + 1, h.trim().to_string(), 0));
177 +
        } else if let Some(last) = sections.last_mut() {
178 +
            if line.trim_start().starts_with("- ") || line.trim_start().starts_with("* ") {
179 +
                last.2 += 1;
180 +
            }
181 +
        }
182 +
    }
183 +
    let mut problems = Vec::new();
184 +
    let mut last: Option<(u64, u64, u64)> = None;
185 +
    let mut released = Vec::new();
186 +
    for (k, (line, h, n)) in sections.iter().enumerate() {
187 +
        if h == "Unreleased" {
188 +
            if k != 0 {
189 +
                problems.push(format!("{file}:{line}: ## Unreleased comes first"));
190 +
            }
191 +
            continue;
192 +
        }
193 +
        let (v, rest) = h.split_once(' ').unwrap_or((h.as_str(), ""));
194 +
        let date = rest.trim().strip_prefix('(').and_then(|r| r.strip_suffix(')'));
195 +
        match (version(v), date) {
196 +
            (Some(ver), Some(d)) if is_date(d) => {
197 +
                if last.is_some_and(|l| ver >= l) {
198 +
                    problems.push(format!("{file}:{line}: {v} is not older than the section above it"));
199 +
                }
200 +
                last = Some(ver);
201 +
                released.push(v.to_string());
202 +
                if *n == 0 {
203 +
                    problems.push(format!("{file}:{line}: ## {h} has no entries"));
204 +
                }
205 +
            }
206 +
            _ => problems.push(format!("{file}:{line}: \"## {h}\" is not \"## X.Y.Z (YYYY-MM-DD)\" or \"## Unreleased\"")),
207 +
        }
208 +
    }
209 +
    let mut said = format!("changelog: {} released section{}", released.len(), if released.len() == 1 { "" } else { "s" });
210 +
    if let Some(m) = manifest {
211 +
        let mtext = fs::read_to_string(root.join(m)).map_err(|e| format!("{m}: {e}"))?;
212 +
        match manifest_version(m, &mtext) {
213 +
            Some(v) if released.contains(&v) => said.push_str(&format!(", {m} at {v} has its section")),
214 +
            Some(v) => problems.push(format!(
215 +
                "{m} says version {v}, but {file} has no \"## {v} (YYYY-MM-DD)\" section: a release of it would have no notes"
216 +
            )),
217 +
            None => problems.push(format!("{m}: no version found")),
218 +
        }
219 +
    }
220 +
    if problems.is_empty() {
221 +
        return emit(out, &said);
222 +
    }
223 +
    Err(problems.join("\n"))
224 +
}
225 +
226 +
fn entries(dir: &Path) -> Result<Vec<PathBuf>, String> {
227 +
    let read = fs::read_dir(dir).map_err(|e| format!("{}: {e}", dir.display()))?;
228 +
    let mut all = read.map(|e| e.map(|e| e.path())).collect::<Result<Vec<_>, _>>().map_err(|e| format!("{}: {e}", dir.display()))?;
229 +
    all.sort();
230 +
    Ok(all)
231 +
}
232 +
233 +
/// Every file under `dir`, depth first, in name order; links are not followed.
234 +
fn files(dir: &Path) -> Result<Vec<PathBuf>, String> {
235 +
    let mut found = Vec::new();
236 +
    for p in entries(dir)? {
237 +
        let kind = fs::symlink_metadata(&p).map_err(|e| format!("{}: {e}", p.display()))?.file_type();
238 +
        if kind.is_dir() {
239 +
            found.extend(files(&p)?);
240 +
        } else if kind.is_file() {
241 +
            found.push(p);
242 +
        }
243 +
    }
244 +
    Ok(found)
245 +
}
246 +
247 +
/// A path as the workspace names it, with `/` between its parts.
248 +
fn shown(root: &Path, p: &Path) -> String {
249 +
    let rel = p.strip_prefix(root).unwrap_or(p);
250 +
    rel.components().map(|c| c.as_os_str().to_string_lossy()).collect::<Vec<_>>().join("/")
251 +
}
252 +
253 +
fn shown_dirs(dirs: &[String]) -> String {
254 +
    dirs.iter().map(|d| format!("{}/", d.trim_end_matches('/'))).collect::<Vec<_>>().join(" ")
255 +
}
256 +
257 +
#[cfg(test)]
258 +
mod tests {
259 +
    use super::*;
260 +
261 +
    fn tree(files: &[(&str, &str)]) -> tempfile::TempDir {
262 +
        let dir = tempfile::tempdir().unwrap();
263 +
        for (path, text) in files {
264 +
            let p = dir.path().join(path);
265 +
            fs::create_dir_all(p.parent().unwrap()).unwrap();
266 +
            fs::write(p, text).unwrap();
267 +
        }
268 +
        dir
269 +
    }
270 +
271 +
    fn run(f: impl FnOnce(&mut Vec<u8>) -> Done) -> (Done, String) {
272 +
        let mut out = Vec::new();
273 +
        let done = f(&mut out);
274 +
        (done, String::from_utf8(out).unwrap())
275 +
    }
276 +
277 +
    fn args(a: &[&str]) -> Vec<String> {
278 +
        a.iter().map(|s| s.to_string()).collect()
279 +
    }
280 +
281 +
    #[test]
282 +
    fn changelog_shape_and_the_manifest_version() {
283 +
        let good = "# Changelog\n\nIntro.\n\n## Unreleased\n\n## 0.2.0 (2026-11-03)\n\n- **New.**\n\n## 0.1.9 (2026-10-01)\n\n- Old.\n";
284 +
        let dir = tree(&[
285 +
            ("CHANGELOG.md", good),
286 +
            ("Cargo.toml", "[workspace]\nmembers = []\n\n[workspace.package]\nversion = \"0.2.0\"\n"),
287 +
            ("gradle.properties", "android.useAndroidX=true\noxygenhub.version=0.1.9\n"),
288 +
            ("package.json", "{\n  \"name\": \"x\",\n  \"version\": \"0.3.0\",\n}\n"),
289 +
        ]);
290 +
        let r = |m: Option<&str>| run(|o| changelog(dir.path(), "CHANGELOG.md", m.map(String::from).as_ref(), o));
291 +
        let (done, out) = r(None);
292 +
        assert_eq!((done, out.trim()), (Ok(()), "changelog: 2 released sections"));
293 +
        assert!(r(Some("Cargo.toml")).1.contains("Cargo.toml at 0.2.0 has its section"));
294 +
        assert!(r(Some("gradle.properties")).0.is_ok(), "an older release's section counts");
295 +
        let e = r(Some("package.json")).0.unwrap_err();
296 +
        assert!(e.contains("version 0.3.0") && e.contains("no \"## 0.3.0 (YYYY-MM-DD)\""), "{e}");
297 +
        let bad = "## 0.1.0 (2026-10-01)\n\n- a\n\n## Unreleased\n\n## 0.2.0 (2026-13-01)\n\n- b\n\n## 0.0.9 (2026-09-01)\n\n## v1\n";
298 +
        let dir = tree(&[("CHANGELOG.md", bad)]);
299 +
        let e = run(|o| changelog(dir.path(), "CHANGELOG.md", None, o)).0.unwrap_err();
300 +
        assert!(e.contains(":5: ## Unreleased comes first"), "{e}");
301 +
        assert!(e.contains(":7: \"## 0.2.0 (2026-13-01)\" is not"), "a month 13: {e}");
302 +
        assert!(e.contains(":11: ## 0.0.9 (2026-09-01) has no entries"), "{e}");
303 +
        assert!(e.contains(":13: \"## v1\""), "{e}");
304 +
        let order = "## 0.1.0 (2026-10-01)\n\n- a\n\n## 0.2.0 (2026-11-01)\n\n- b\n";
305 +
        let dir = tree(&[("CHANGELOG.md", order)]);
306 +
        let e = run(|o| changelog(dir.path(), "CHANGELOG.md", None, o)).0.unwrap_err();
307 +
        assert!(e.contains("0.2.0 is not older than the section above it"), "{e}");
308 +
    }
309 +
310 +
    #[test]
311 +
    fn layout_lists_what_it_found_and_fails_on_the_first_missing() {
312 +
        let t = tree(&[("Cargo.toml", ""), ("docs/m0.md", "")]);
313 +
        let (done, out) = run(|o| layout(t.path(), &args(&["Cargo.toml", "docs/m0.md"]), o));
314 +
        assert_eq!(done, Ok(()));
315 +
        assert_eq!(out, "found Cargo.toml\nfound docs/m0.md\n");
316 +
        let (done, out) = run(|o| layout(t.path(), &args(&["Cargo.toml", "nope.rs", "docs/m0.md"]), o));
317 +
        assert_eq!(done, Err("missing: nope.rs".to_string()));
318 +
        assert_eq!(out, "found Cargo.toml\n");
319 +
        // A directory is not the file asked for.
320 +
        assert!(layout(t.path(), &args(&["docs"]), &mut Vec::new()).is_err());
321 +
    }
322 +
323 +
    #[test]
324 +
    fn todos_counts_every_marked_line_and_lists_the_first_twenty() {
325 +
        let marker = MARKERS[0];
326 +
        let many: String = (0..25).map(|i| format!("// {marker} {i}\n")).collect();
327 +
        let t = tree(&[
328 +
            ("crates/a/src/lib.rs", &format!("fn a() {{}}\n// {} later\n", MARKERS[1])),
329 +
            ("crates/b/src/many.rs", &many),
330 +
            ("crates/b/blob.bin", "\u{0}"),
331 +
        ]);
332 +
        // Invalid UTF-8 is skipped, as grep skips a binary file's lines.
333 +
        fs::write(t.path().join("crates/b/blob.bin"), [0xff, 0xfe, b'T']).unwrap();
334 +
        let (done, out) = run(|o| todos(t.path(), &args(&["crates"]), o));
335 +
        assert_eq!(done, Ok(()));
336 +
        let lines: Vec<&str> = out.lines().collect();
337 +
        assert_eq!(lines[0], format!("26 {} notes in crates/", MARKERS.join("/")));
338 +
        assert_eq!(lines[1], format!("crates/a/src/lib.rs:2:// {} later", MARKERS[1]));
339 +
        assert_eq!(lines[2], format!("crates/b/src/many.rs:1:// {marker} 0"));
340 +
        assert_eq!(lines.len(), 1 + LISTED);
341 +
    }
342 +
343 +
    #[test]
344 +
    fn size_counts_rust_lines_per_crate_in_name_order() {
345 +
        let t = tree(&[
346 +
            ("crates/zeta/src/main.rs", "a\nb\nc\n"),
347 +
            ("crates/alpha/src/lib.rs", "a\n"),
348 +
            ("crates/alpha/src/deep/mod.rs", "a\nb\n"),
349 +
            ("crates/alpha/README.md", "not rust\n"),
350 +
            ("crates/loose.txt", "a file, not a crate\n"),
351 +
        ]);
352 +
        let (done, out) = run(|o| size(t.path(), "crates", o));
353 +
        assert_eq!(done, Ok(()));
354 +
        assert_eq!(out, format!("{:<24} {:>6} lines\n{:<24} {:>6} lines\n", "crates/alpha/", 3, "crates/zeta/", 3));
355 +
    }
356 +
357 +
    #[test]
358 +
    fn summary_names_the_project() {
359 +
        let (_, out) = run(|o| summary(Some("OxygenFlow"), o));
360 +
        assert_eq!(out, "OxygenFlow: every check passed on this state.\n");
361 +
    }
362 +
}
modified crates/flow-core/Cargo.toml +3 −0
…
14 unchanged lines
15 15
kdl = "6"
16 16
miette = "7"
17 17
globset = "0.4"
18 +
# Sealed alerts (alert.rs): both already in the tree through oxy-core.
19 +
curve25519-dalek = "5"
20 +
ring = "0.17"
added crates/flow-core/src/alert.rs +94 −0
1 +
//! Sealed alerts (oxygenhub-android#16): what a phone is told about a job,
2 +
//! encrypted to a key the phone made, so the push service that carries it
3 +
//! (Firebase Cloud Messaging) sees only ciphertext.
4 +
//!
5 +
//! The phone keeps an X25519 secret and gives the hub its public key. For
6 +
//! each alert the hub makes a fresh X25519 key, agrees a secret with the
7 +
//! phone's key, derives a ChaCha20-Poly1305 key with HKDF-SHA256 (salted
8 +
//! with both public keys) and encrypts once under it with a zero nonce:
9 +
//! the key is never used twice. A sealed alert is the fresh public key (32
10 +
//! bytes) followed by the ciphertext and its tag.
11 +
12 +
use curve25519_dalek::montgomery::MontgomeryPoint;
13 +
use ring::aead::{Aad, LessSafeKey, Nonce, UnboundKey, CHACHA20_POLY1305};
14 +
use ring::hkdf;
15 +
use ring::rand::{SecureRandom, SystemRandom};
16 +
17 +
/// What HKDF binds the key to, so a key for anything else never fits.
18 +
const INFO: &[u8] = b"oxygenhub alert v1";
19 +
20 +
/// A new X25519 secret, for a phone to keep.
21 +
pub fn new_secret() -> [u8; 32] {
22 +
    let mut s = [0u8; 32];
23 +
    SystemRandom::new().fill(&mut s).expect("the system's random source");
24 +
    s
25 +
}
26 +
27 +
/// The public key of `secret`, for the hub.
28 +
pub fn public_key(secret: &[u8; 32]) -> [u8; 32] {
29 +
    MontgomeryPoint::mul_base_clamped(*secret).to_bytes()
30 +
}
31 +
32 +
fn key(shared: &MontgomeryPoint, ephemeral: &[u8; 32], recipient: &[u8; 32]) -> Option<LessSafeKey> {
33 +
    // An all-zero agreement means a low-order point was given: refuse it.
34 +
    if shared.to_bytes() == [0u8; 32] {
35 +
        return None;
36 +
    }
37 +
    let mut salt = [0u8; 64];
38 +
    salt[..32].copy_from_slice(ephemeral);
39 +
    salt[32..].copy_from_slice(recipient);
40 +
    let prk = hkdf::Salt::new(hkdf::HKDF_SHA256, &salt).extract(shared.as_bytes());
41 +
    let okm = prk.expand(&[INFO], &CHACHA20_POLY1305).ok()?;
42 +
    let mut bytes = [0u8; 32];
43 +
    okm.fill(&mut bytes).ok()?;
44 +
    Some(LessSafeKey::new(UnboundKey::new(&CHACHA20_POLY1305, &bytes).ok()?))
45 +
}
46 +
47 +
/// Seal `message` to the phone whose public key is `recipient`; `None`
48 +
/// when that key is unusable.
49 +
pub fn seal(recipient: &[u8; 32], message: &[u8]) -> Option<Vec<u8>> {
50 +
    let e = new_secret();
51 +
    let ephemeral = public_key(&e);
52 +
    let shared = MontgomeryPoint(*recipient).mul_clamped(e);
53 +
    let key = key(&shared, &ephemeral, recipient)?;
54 +
    let mut sealed = message.to_vec();
55 +
    key.seal_in_place_append_tag(Nonce::assume_unique_for_key([0u8; 12]), Aad::empty(), &mut sealed).ok()?;
56 +
    let mut out = ephemeral.to_vec();
57 +
    out.extend(sealed);
58 +
    Some(out)
59 +
}
60 +
61 +
/// Open an alert sealed to the phone whose secret is `secret`; `None` when
62 +
/// it was not sealed to it or was changed on the way.
63 +
pub fn open(secret: &[u8; 32], sealed: &[u8]) -> Option<Vec<u8>> {
64 +
    if sealed.len() < 32 + CHACHA20_POLY1305.tag_len() {
65 +
        return None;
66 +
    }
67 +
    let ephemeral: [u8; 32] = sealed[..32].try_into().ok()?;
68 +
    let shared = MontgomeryPoint(ephemeral).mul_clamped(*secret);
69 +
    let key = key(&shared, &ephemeral, &public_key(secret))?;
70 +
    let mut body = sealed[32..].to_vec();
71 +
    let plain = key.open_in_place(Nonce::assume_unique_for_key([0u8; 12]), Aad::empty(), &mut body).ok()?;
72 +
    Some(plain.to_vec())
73 +
}
74 +
75 +
#[cfg(test)]
76 +
mod tests {
77 +
    use super::*;
78 +
79 +
    #[test]
80 +
    fn a_sealed_alert_opens_only_with_its_phones_key() {
81 +
        let phone = new_secret();
82 +
        let sealed = seal(&public_key(&phone), b"run #5: test passed").unwrap();
83 +
        assert_eq!(open(&phone, &sealed).as_deref(), Some(&b"run #5: test passed"[..]));
84 +
        // Each alert has its own key: the same message seals differently.
85 +
        assert_ne!(seal(&public_key(&phone), b"run #5: test passed").unwrap(), sealed);
86 +
        assert!(open(&new_secret(), &sealed).is_none(), "another phone");
87 +
        let mut changed = sealed.clone();
88 +
        *changed.last_mut().unwrap() ^= 1;
89 +
        assert!(open(&phone, &changed).is_none(), "changed on the way");
90 +
        assert!(open(&phone, &sealed[..40]).is_none(), "cut short");
91 +
        // A low-order public key (all zeros) is refused, not sealed to.
92 +
        assert!(seal(&[0u8; 32], b"x").is_none());
93 +
    }
94 +
}
modified crates/flow-core/src/api.rs +184 −0
…
47 unchanged lines
48 48
    /// docs/jetstream.md §6). Absent: an older runner.
49 49
    #[serde(default, skip_serializing_if = "Option::is_none")]
50 50
    pub facts: Option<Facts>,
51 +
    /// Where this runner serves image chunks to its peers (docs/jetstream.md
52 +
    /// §8.1). Absent: it is not in a swarm.
53 +
    #[serde(default, skip_serializing_if = "Option::is_none")]
54 +
    pub swarm: Option<SwarmAddr>,
55 +
    /// The build caches this runner holds a disk of (oxygenflow#51), each
56 +
    /// [`cache_slot`] of a repository and job, so the hub can send a job
57 +
    /// where its cache is. Absent: an older runner.
58 +
    #[serde(default, skip_serializing_if = "Option::is_none")]
59 +
    pub caches: Option<Vec<String>>,
60 +
    /// When each of [`ClaimRequest::caches`] last kept a disk, Unix ms, so
61 +
    /// the hub can send a job to the runner whose copy is freshest when
62 +
    /// several hold one (oxygenflow#54). Absent: an older runner.
63 +
    #[serde(default, skip_serializing_if = "Option::is_none")]
64 +
    pub cache_times: Option<std::collections::BTreeMap<String, u64>>,
65 +
}
66 +
67 +
/// A name safe as one path segment: kept readable, with a short hash so
68 +
/// two names that clean up alike stay apart. The build cache's pool is laid
69 +
/// out by these (oxygenflow#28).
70 +
pub fn segment(s: &str) -> String {
71 +
    let clean: String = s.chars().map(|c| if c.is_ascii_alphanumeric() || "._-".contains(c) { c } else { '_' }).take(48).collect();
72 +
    format!("{clean}-{}", &blake3::hash(s.as_bytes()).to_hex()[..8])
73 +
}
74 +
75 +
/// Where a repository's job keeps its build caches, as runners report
76 +
/// them in [`ClaimRequest::caches`] (oxygenflow#51): `repo/job`, each part
77 +
/// a [`segment`].
78 +
pub fn cache_slot(repo: &str, job: &str) -> String {
79 +
    format!("{}/{}", segment(repo), segment(job))
80 +
}
81 +
82 +
/// A runner's place in the image swarm: the address peers reach it on,
83 +
/// the site it is in (peers are only ever in the same site), and the
84 +
/// BLAKE3 of its TLS certificate, which peers check it by.
85 +
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
86 +
pub struct SwarmAddr {
87 +
    pub addr: String,
88 +
    #[serde(default)]
89 +
    pub site: String,
90 +
    #[serde(default)]
91 +
    pub cert: String,
92 +
}
93 +
94 +
/// `POST /flow/swarm/announce`: a runner starting, continuing or ending a
95 +
/// fetch of `image`.
96 +
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
97 +
pub struct Announce {
98 +
    pub image: String,
99 +
    /// `fetching`, `done` or `stopped`.
100 +
    pub state: String,
101 +
}
102 +
103 +
/// The hub's answer to an announce: the peers to ask, and the ticket that
104 +
/// admits this runner to them.
105 +
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
106 +
pub struct Announced {
107 +
    pub ticket: String,
108 +
    pub peers: Vec<SwarmPeer>,
109 +
    /// Announce again after this many seconds while still fetching.
110 +
    pub interval_s: u64,
111 +
    /// Whether the hub holds the image's chunks and so may be asked for
112 +
    /// them (jetstream.md §8.1.1): `Some(false)` for an image only runners
113 +
    /// hold; absent from a hub that predates it, which always seeds.
114 +
    #[serde(default, skip_serializing_if = "Option::is_none")]
115 +
    pub seed: Option<bool>,
116 +
}
117 +
118 +
/// `POST /flow/images/held`: the images a runner holds, by hash
119 +
/// (oxygenflow#94).
120 +
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
121 +
pub struct HeldImages {
122 +
    pub images: Vec<String>,
123 +
}
124 +
125 +
/// The hub's answer to [`HeldImages`]: those it keeps no manifest for, which
126 +
/// the runner sends with `PUT /flow/images/{hash}/held`.
127 +
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
128 +
pub struct WantedManifests {
129 +
    pub wanted: Vec<String>,
130 +
}
131 +
132 +
/// A peer in the swarm for one image.
133 +
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
134 +
pub struct SwarmPeer {
135 +
    pub runner: i64,
136 +
    pub name: String,
137 +
    pub addr: String,
138 +
    /// The BLAKE3 of its TLS certificate: a peer that shows another is not
139 +
    /// this runner.
140 +
    pub cert: String,
141 +
    /// It holds the whole image (rather than fetching it now).
142 +
    pub complete: bool,
51 143
}
52 144
53 145
/// A runner's own measurements, sent with every claim (spec §11.1): cheap
…
153 unchanged lines
207 299
    /// them for the repository (oxygenhub#107).
208 300
    #[serde(default, skip_serializing_if = "Vec::is_empty")]
209 301
    pub secrets: Vec<SecretInput>,
302 +
    /// The job's inputs (oxygenflow#58), in the job's order, each with the
303 +
    /// blob the producing job's receipt names: fetched from
304 +
    /// `/api/v1/flow/jobs/{job_id}/inputs/{index}` and checked against it.
305 +
    #[serde(default, skip_serializing_if = "Vec::is_empty")]
306 +
    pub inputs: Vec<ArtifactInput>,
307 +
    /// How long the job is expected to take, from its recent passing runs,
308 +
    /// when the hub has learned it (#76): a runner leaving with some notice
309 +
    /// finishes a job whose rest fits it and hands back the others.
310 +
    #[serde(default, skip_serializing_if = "Option::is_none")]
311 +
    pub expected_ms: Option<u64>,
312 +
    /// The checkpoint an earlier attempt saved before its runner handed the
313 +
    /// job back (oxygenflow#77): the blob to fetch from
314 +
    /// `/api/v1/flow/jobs/{job_id}/checkpoint` and check, then restore
315 +
    /// before the job runs.
316 +
    #[serde(default, skip_serializing_if = "Option::is_none")]
317 +
    pub checkpoint: Option<String>,
318 +
}
319 +
320 +
/// An artifact a job takes (oxygenflow#58), as the hub hands it over.
321 +
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
322 +
pub struct ArtifactInput {
323 +
    pub artifact: String,
324 +
    /// The producing job.
325 +
    pub from: String,
326 +
    /// Where it goes, under `/src`.
327 +
    pub path: String,
328 +
    /// The Oxygen blob reference its receipt names, hex.
329 +
    pub blob: String,
210 330
}
211 331
212 332
impl Claim {
…
43 unchanged lines
256 376
    pub reason: String,
257 377
}
258 378
379 +
/// `GET /api/v1/flow/runner/update` (docs/jetstream.md §8.2, oxygenhub#322):
380 +
/// the update policy that applies to the asking runner, and the build it
381 +
/// should run when that is not the one it runs.
382 +
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
383 +
pub struct UpdateAnswer {
384 +
    pub policy: UpdatePolicy,
385 +
    /// The build to install now; none when updates are off, the runner is
386 +
    /// up to date, it is outside its window, or no build suits it.
387 +
    #[serde(default)]
388 +
    pub offer: Option<UpdateOffer>,
389 +
    /// Why there is no offer, or what the offer is for, in words.
390 +
    #[serde(default)]
391 +
    pub note: String,
392 +
}
393 +
394 +
/// A runner's update policy once every level is applied (nearest wins).
395 +
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
396 +
pub struct UpdatePolicy {
397 +
    /// Off unless some level turned it on.
398 +
    pub updates: bool,
399 +
    /// `stable`, `canary`, or a version such as `0.3.4`.
400 +
    pub channel: String,
401 +
    /// Hours (UTC) in which it may update, from and to, when limited.
402 +
    #[serde(default)]
403 +
    pub window: Option<(u8, u8)>,
404 +
    /// Fetch and check, say what would happen, do not swap.
405 +
    #[serde(default)]
406 +
    pub dry_run: bool,
407 +
    /// Where each setting came from: `site`, `owner acme`, `group canary`,
408 +
    /// `runner`, or `default`, in the order updates, channel, window, dry run.
409 +
    #[serde(default)]
410 +
    pub from: Vec<String>,
411 +
}
412 +
413 +
/// A build to install: a release asset of the site's runner releases.
414 +
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
415 +
pub struct UpdateOffer {
416 +
    pub version: String,
417 +
    pub tag: String,
418 +
    /// The asset's name, `flow-linux-x86_64`.
419 +
    pub asset: String,
420 +
    /// Where to download it, relative to the hub.
421 +
    pub url: String,
422 +
    /// Its BLAKE3, as the hub stores and attests it: the download must match.
423 +
    pub blake3: String,
424 +
    pub size: u64,
425 +
    /// Older than the runner's build: a pinned channel going back.
426 +
    #[serde(default)]
427 +
    pub older: bool,
428 +
}
429 +
430 +
/// `POST /api/v1/flow/runner/update`: what the runner did about its update,
431 +
/// shown on its page.
432 +
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
433 +
pub struct UpdateReport {
434 +
    pub note: String,
435 +
}
436 +
259 437
/// `POST /api/v1/flow/jobs/{id}/log`.
260 438
#[derive(Clone, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
261 439
pub struct LogBatch {
…
20 unchanged lines
282 460
    pub peak_disk_mib: Option<u64>,
283 461
    #[serde(default, skip_serializing_if = "Option::is_none")]
284 462
    pub peak_memory_mib: Option<u64>,
463 +
    /// Where the job's time went, in milliseconds (oxygenflow#35): `image`
464 +
    /// (fetching its image, when the runner lacked it), `prepare` (its
465 +
    /// workspace and disks), `run` (the sandbox, from start to stop) and
466 +
    /// `finish` (its outputs, caches and receipt).
467 +
    #[serde(default, skip_serializing_if = "std::collections::BTreeMap::is_empty")]
468 +
    pub phases: std::collections::BTreeMap<String, u64>,
285 469
}
286 470
287 471
/// Longest text of one record the hub keeps; longer lines are cut.
…
3 unchanged lines
modified crates/flow-core/src/canonical.rs +29 −1
…
46 unchanged lines
47 47
    if let Some(a) = &s.arch {
48 48
        fields.push((7, text(a)));
49 49
    }
50 +
    // The hosts egress may reach, sorted; absent without egress.
51 +
    if !s.egress_allow.is_empty() {
52 +
        fields.push((8, texts(&s.egress_allow)));
53 +
    }
50 54
    Value::map(fields)
51 55
}
52 56
…
15 unchanged lines
68 72
            Value::Array(
69 73
                j.mounts
70 74
                    .iter()
71 -
                    .map(|m| Value::Array(vec![text(m.source.as_str()), text(&m.target), Value::Uint(m.size_mib as u64)]))
75 +
                    .map(|m| {
76 +
                        // A cache mount adds its name and key files; a plain
77 +
                        // one encodes as before caches existed.
78 +
                        let mut v = vec![text(m.source.as_str()), text(&m.target), Value::Uint(m.size_mib as u64)];
79 +
                        if let Some(c) = &m.cache {
80 +
                            v.push(text(c));
81 +
                            v.push(Value::Array(m.key.iter().map(|k| text(k)).collect()));
82 +
                        }
83 +
                        Value::Array(v)
84 +
                    })
72 85
                    .collect(),
73 86
            ),
74 87
        ));
…
18 unchanged lines
93 106
    if !j.identities.is_empty() {
94 107
        fields.push((9, Value::Array(j.identities.iter().map(|i| Value::Array(vec![text(&i.audience), text(&i.mount_path)])).collect())));
95 108
    }
109 +
    // Inputs as [artifact, from, path] sorted by path; absent when none.
110 +
    // Approval as [by, others-only, expires seconds]; absent without one.
111 +
    if let Some(a) = &j.approval {
112 +
        fields.push((11, Value::Array(vec![text(&a.by), Value::Bool(a.others_only), Value::Uint(a.expires_s), Value::Bool(a.sealed)])));
113 +
    }
114 +
    // The environment it deploys to; absent without one.
115 +
    if let Some(e) = &j.environment {
116 +
        fields.push((12, text(e)));
117 +
    }
118 +
    if !j.inputs.is_empty() {
119 +
        fields.push((
120 +
            10,
121 +
            Value::Array(j.inputs.iter().map(|i| Value::Array(vec![text(&i.artifact), text(&i.from), text(&i.path)])).collect()),
122 +
        ));
123 +
    }
96 124
    Value::map(fields)
97 125
}
98 126
…
41 unchanged lines
modified crates/flow-core/src/lib.rs +3 −2
…
3 unchanged lines
4 4
/// This library's version, for hosts that report what they are built on.
5 5
pub const VERSION: &str = env!("CARGO_PKG_VERSION");
6 6
7 +
pub mod alert;
7 8
pub mod api;
8 9
pub mod canonical;
9 10
pub mod cron;
…
2 unchanged lines
12 13
13 14
pub use canonical::{object_hash, scope_tree_hash, Hash};
14 15
pub use pipeline::{
15 -
    arch_ok, Identity, Job, Mount, MountSource, Network, Output, OutputKind, Pipeline, Problem, Problems, Runtime, Sandbox, Secret,
16 -
    PIPELINE_PATH,
16 +
    arch_ok, egress_allows, egress_pattern_ok, Approval, Identity, Input, Job, Mount, MountSource, Network, Output, OutputKind, Pipeline,
17 +
    Problem, Problems, Runtime, Sandbox, Secret, EGRESS_LABEL, PIPELINE_PATH,
17 18
};
18 19
pub use receipt::{Environment, Receipt};
19 20
…
2 unchanged lines
modified crates/flow-core/src/pipeline.rs +406 −13
…
152 unchanged lines
153 153
    /// Labels a runner must have to run jobs here (`runs-on`), sorted.
154 154
    #[serde(default, skip_serializing_if = "Vec::is_empty")]
155 155
    pub runs_on: Vec<String>,
156 +
    /// The hosts a `network "egress"` sandbox may reach (oxygenflow#55):
157 +
    /// names, or `*.` and a domain for any name under it; sorted. Its jobs
158 +
    /// reach them through the runner's proxy, and nothing else.
159 +
    #[serde(default, skip_serializing_if = "Vec::is_empty")]
160 +
    pub egress_allow: Vec<String>,
156 161
    /// How long each job here may run, in seconds (`timeout "2h"`), when
157 162
    /// the job does not say; the hub caps it at the site's maximum.
158 163
    #[serde(default, skip_serializing_if = "Option::is_none")]
…
26 unchanged lines
185 190
    /// Files the job produces, sorted by name (spec §5.2 `output`).
186 191
    #[serde(default, skip_serializing_if = "Vec::is_empty")]
187 192
    pub outputs: Vec<Output>,
193 +
    /// Artifacts of jobs this one depends on, put in its workspace before it
194 +
    /// starts (oxygenflow#58), sorted by path. The runner checks each against
195 +
    /// the producing job's receipt; this job's receipt lists them as inputs.
196 +
    #[serde(default, skip_serializing_if = "Vec::is_empty")]
197 +
    pub inputs: Vec<Input>,
198 +
    /// Wait for a person before running (oxygenflow#57): `approval { by
199 +
    /// "maintain"; others-only #true; expires "24h" }`.
200 +
    #[serde(default, skip_serializing_if = "Option::is_none")]
201 +
    pub approval: Option<Approval>,
202 +
    /// The environment the job deploys to (oxygenflow#59), e.g. `prod`: the
203 +
    /// hub records what each environment runs, and the job's identity token
204 +
    /// names it once the job has been approved.
205 +
    #[serde(default, skip_serializing_if = "Option::is_none")]
206 +
    pub environment: Option<String>,
188 207
    /// Secrets the job mounts, sorted by name (spec §9 `secret`): the hub
189 208
    /// supplies each by name, the runner mounts it as a file.
190 209
    #[serde(default, skip_serializing_if = "Vec::is_empty")]
…
13 unchanged lines
204 223
    /// When it runs, over the pipeline's `on` (oxygenflow#24).
205 224
    #[serde(default, skip_serializing_if = "Option::is_none")]
206 225
    pub on: Option<On>,
226 +
    /// `low` or `normal` (`priority "low"`, or the pipeline's): a pipeline
227 +
    /// may lower its jobs below the repository's level, never raise them
228 +
    /// (oxygenhub#244); higher levels are the hub's to give.
229 +
    #[serde(default, skip_serializing_if = "Option::is_none")]
230 +
    pub priority: Option<String>,
231 +
    /// Directories under /src that hold the job's progress (`checkpoint
232 +
    /// "/src/target"`, oxygenflow#77), sorted: saved when its runner hands
233 +
    /// it back, restored before the next attempt runs the same exec.
234 +
    #[serde(default, skip_serializing_if = "Vec::is_empty")]
235 +
    pub checkpoints: Vec<String>,
207 236
}
208 237
209 238
impl Job {
…
23 unchanged lines
233 262
}
234 263
235 264
/// An ephemeral scratch mount: empty at the start of every job and gone
236 -
/// after it, so it cannot carry anything between jobs (spec §2.2 I4).
265 +
/// after it, so it cannot carry anything between jobs (spec §2.2 I4) --
266 +
/// unless it names a `cache` (spec §5.4, #28): then a disk kept from an
267 +
/// earlier passing job may start it, and the receipt says which.
237 268
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
238 269
pub struct Mount {
239 270
    pub source: MountSource,
240 271
    /// An absolute path under `/src` or `/tmp`.
241 272
    pub target: String,
242 273
    pub size_mib: u32,
274 +
    /// `cache "name"`: keep this disk after a passing job and start later
275 +
    /// jobs from it (disk mounts only).
276 +
    #[serde(default, skip_serializing_if = "Option::is_none")]
277 +
    pub cache: Option<String>,
278 +
    /// `key "Cargo.lock" ...`: files whose content picks the cache entry.
279 +
    #[serde(default, skip_serializing_if = "Vec::is_empty")]
280 +
    pub key: Vec<String>,
243 281
}
244 282
245 283
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]
…
32 unchanged lines
278 316
    pub kind: OutputKind,
279 317
}
280 318
319 +
/// An artifact a job takes from a job it depends on (oxygenflow#58):
320 +
/// `input { artifact "tfplan" from "plan"; path "/src/tfplan" }`.
321 +
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
322 +
pub struct Input {
323 +
    /// The producing job's artifact.
324 +
    pub artifact: String,
325 +
    /// The producing job: one this job depends on.
326 +
    pub from: String,
327 +
    /// Where it is put: a clean absolute path under `/src`.
328 +
    pub path: String,
329 +
}
330 +
331 +
/// Who must approve a job before it runs, and for how long it waits
332 +
/// (oxygenflow#57).
333 +
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
334 +
pub struct Approval {
335 +
    /// The least role that approves: `write`, `maintain` (the default) or `admin`.
336 +
    pub by: String,
337 +
    /// Not the person whose push started the run.
338 +
    #[serde(default, skip_serializing_if = "std::ops::Not::not")]
339 +
    pub others_only: bool,
340 +
    /// How long it waits once the jobs it depends on are done, in seconds.
341 +
    pub expires_s: u64,
342 +
    /// The approval must be signed with the approver's own key
343 +
    /// (`oxy seal --flow`; oxygenhub#235). Defaults to on for a job that
344 +
    /// names an environment.
345 +
    #[serde(default, skip_serializing_if = "std::ops::Not::not")]
346 +
    pub sealed: bool,
347 +
}
348 +
349 +
/// The longest a job waits for approval: 30 days.
350 +
const MAX_APPROVAL_S: u64 = 30 * 24 * 3600;
351 +
352 +
/// The most inputs a job takes.
353 +
const MAX_INPUTS: usize = 8;
354 +
281 355
/// What an [`Output`] is.
282 356
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
283 357
#[serde(rename_all = "lowercase")]
…
245 unchanged lines
529 603
        let mut sandboxes = BTreeMap::new();
530 604
        let mut jobs: BTreeMap<String, (Job, SourceSpan)> = BTreeMap::new();
531 605
        let mut on = None;
606 +
        let mut priority = None;
532 607
        for node in doc.nodes() {
533 608
            match node.name().value() {
609 +
                "priority" => {
610 +
                    if priority.is_some() {
611 +
                        ctx.err(node.span(), "`priority` is given twice");
612 +
                    }
613 +
                    priority = parse_priority(&mut ctx, node);
614 +
                }
534 615
                "on" => {
535 616
                    if on.is_some() {
536 617
                        ctx.err(node.span(), "`on` is given twice");
…
27 unchanged lines
564 645
                    }
565 646
                }
566 647
                n if LATER_TOP.contains(&n) => ctx.err(node.span(), format!("`{n}` is not in M0 yet")),
567 -
                n => ctx.err(node.span(), format!("unknown node `{n}`; expected schema-version, on, sandbox or job")),
648 +
                n => ctx.err(node.span(), format!("unknown node `{n}`; expected schema-version, on, priority, sandbox or job")),
568 649
            }
569 650
        }
570 651
        if schema.is_none() && !ctx.problems.iter().any(|p| p.message.contains("schema-version")) {
…
14 unchanged lines
585 666
                    ctx.err(*span, format!("job {:?} depends on itself", job.name));
586 667
                }
587 668
            }
669 +
            // Inputs come from a job this one depends on, which declares
670 +
            // that artifact (oxygenflow#58).
671 +
            for i in &job.inputs {
672 +
                let Some((producer, _)) = jobs.get(&i.from) else {
673 +
                    ctx.err(*span, format!("job {:?} takes an input from {:?}, which is not declared", job.name, i.from));
674 +
                    continue;
675 +
                };
676 +
                if !job.depends_on.contains(&i.from) {
677 +
                    ctx.err(*span, format!("job {:?} takes {:?} from {:?}: add depends-on \"{}\"", job.name, i.artifact, i.from, i.from));
678 +
                }
679 +
                match producer.outputs.iter().find(|o| o.name == i.artifact) {
680 +
                    Some(o) if o.kind == OutputKind::Artifact => {}
681 +
                    Some(_) => ctx.err(*span, format!("{:?} of job {:?} is a report; inputs are artifacts", i.artifact, i.from)),
682 +
                    None => ctx.err(*span, format!("job {:?} declares no artifact {:?}", i.from, i.artifact)),
683 +
                }
684 +
            }
588 685
        }
589 686
        let graph: BTreeMap<&str, Vec<&str>> =
590 687
            jobs.values().map(|(j, _)| (j.name.as_str(), j.depends_on.iter().map(String::as_str).collect())).collect();
…
5 unchanged lines
596 693
            ctx.problems.sort_by_key(|p| p.line);
597 694
            return Err(Problems(ctx.problems));
598 695
        }
599 -
        Ok(Pipeline { schema_version: 1, sandboxes, jobs: jobs.into_iter().map(|(k, (j, _))| (k, j)).collect(), on })
696 +
        // The pipeline's priority is each job's that does not say.
697 +
        let jobs = jobs
698 +
            .into_iter()
699 +
            .map(|(k, (mut j, _))| {
700 +
                if j.priority.is_none() {
701 +
                    j.priority.clone_from(&priority);
702 +
                }
703 +
                (k, j)
704 +
            })
705 +
            .collect();
706 +
        Ok(Pipeline { schema_version: 1, sandboxes, jobs, on })
600 707
    }
601 708
602 709
    /// Jobs in an order where each comes after what it depends on (ties by name).
…
93 unchanged lines
696 803
    let mut runs_on = Vec::new();
697 804
    let mut timeout_s = None;
698 805
    let mut arch: Option<String> = None;
806 +
    let mut egress_allow = Vec::new();
699 807
    for c in block(ctx, node) {
700 808
        let key = c.name().value();
701 809
        if key != "runs-on" && !seen.insert(key.to_string()) {
…
8 unchanged lines
710 818
            parse_labels(ctx, c, &mut runs_on);
711 819
            continue;
712 820
        }
713 -
        ctx.no_children(c);
821 +
        if key != "network" {
822 +
            ctx.no_children(c);
823 +
        }
714 824
        match key {
715 825
            "runtime" => {
716 826
                runtime = match ctx.one_string(c).as_deref() {
717 827
                    Some("microvm") => Some(Runtime::Microvm),
718 828
                    Some("syscall") | Some("gvisor") => Some(Runtime::Syscall),
719 -
                    Some("wasm") => {
720 -
                        ctx.err(c.span(), "runtime \"wasm\" is not in M0 yet; use \"microvm\" or \"syscall\"");
721 -
                        None
722 -
                    }
829 +
                    // A WASI module run by wasmtime (spec §4.1, oxygenflow#75).
830 +
                    Some("wasm") => Some(Runtime::Wasm),
723 831
                    Some(other) => {
724 832
                        ctx.err(c.span(), format!("runtime {other:?}: expected \"microvm\", \"syscall\" or \"wasm\""));
725 833
                        None
…
52 unchanged lines
778 886
                }
779 887
            }
780 888
            "network" => {
781 -
                network = match ctx.one_string(c).as_deref() {
889 +
                let kind = ctx.one_string(c);
890 +
                if kind.as_deref() != Some("egress") {
891 +
                    ctx.no_children(c);
892 +
                }
893 +
                network = match kind.as_deref() {
782 894
                    Some("isolated") => Some(Network::Isolated),
783 895
                    // Loopback only: the guest's `lo` comes up, and it still has
784 896
                    // no device that reaches anything outside it.
785 897
                    Some("loopback") => Some(Network::Loopback),
898 +
                    // Allow-listed egress (oxygenflow#55): still no network
899 +
                    // device; a proxy in the guest relays to the runner, which
900 +
                    // lets through only the hosts named here.
786 901
                    Some("egress") => {
787 -
                        ctx.err(c.span(), "network \"egress\" is not in M0; M0 has \"isolated\" and \"loopback\"");
788 -
                        None
902 +
                        egress_allow = parse_egress(ctx, c);
903 +
                        Some(Network::Egress)
789 904
                    }
790 905
                    Some(other) => {
791 906
                        ctx.err(c.span(), format!("network {other:?}: expected \"isolated\", \"loopback\" or \"egress\""));
…
6 unchanged lines
798 913
            other => ctx.err(c.span(), format!("unknown sandbox setting `{other}`")),
799 914
        }
800 915
    }
801 -
    if rootfs.is_none() && !seen.contains("rootfs") {
916 +
    // A WASM sandbox has no image: its module is a file of the job (its
917 +
    // scope or an earlier job's artifact), and it reaches no network.
918 +
    let wasm = runtime == Some(Runtime::Wasm);
919 +
    if wasm {
920 +
        if seen.contains("rootfs") {
921 +
            ctx.err(node.span(), format!("sandbox {name:?}: runtime \"wasm\" takes no rootfs; its module is a file the job has"));
922 +
        }
923 +
        if network.is_some_and(|n| n != Network::Isolated) {
924 +
            ctx.err(node.span(), format!("sandbox {name:?}: runtime \"wasm\" has no network; leave out `network` or say \"isolated\""));
925 +
        }
926 +
        rootfs = Some(String::new());
927 +
        network = Some(Network::Isolated);
928 +
    } else if rootfs.is_none() && !seen.contains("rootfs") {
802 929
        ctx.err(node.span(), format!("sandbox {name:?} needs a rootfs"));
803 930
    }
931 +
    if network == Some(Network::Egress) {
932 +
        if runtime == Some(Runtime::Syscall) {
933 +
            ctx.err(node.span(), format!("sandbox {name:?}: network \"egress\" needs runtime \"microvm\""));
934 +
        }
935 +
        // Only a runner that allows egress offers this label.
936 +
        if !runs_on.iter().any(|l| l == EGRESS_LABEL) {
937 +
            runs_on.push(EGRESS_LABEL.to_string());
938 +
            runs_on.sort();
939 +
        }
940 +
    }
804 941
    if !named {
805 942
        return None;
806 943
    }
…
6 unchanged lines
813 950
        network: network?,
814 951
        env,
815 952
        runs_on,
953 +
        egress_allow,
816 954
        timeout_s,
817 955
        arch,
818 956
    })
819 957
}
820 958
959 +
/// The label a runner offers when it allows egress, and an egress sandbox
960 +
/// needs (oxygenflow#55).
961 +
pub const EGRESS_LABEL: &str = "egress";
962 +
963 +
/// The most hosts one sandbox may name.
964 +
const MAX_EGRESS_HOSTS: usize = 32;
965 +
966 +
/// Whether `pattern` is a host name, or `*.` and a domain: lower-case
967 +
/// letters, digits, `-` and dots, at least two labels.
968 +
pub fn egress_pattern_ok(pattern: &str) -> bool {
969 +
    let name = pattern.strip_prefix("*.").unwrap_or(pattern);
970 +
    let labels: Vec<&str> = name.split('.').collect();
971 +
    name.len() <= 253
972 +
        && labels.len() >= 2
973 +
        && labels.iter().all(|l| {
974 +
            !l.is_empty()
975 +
                && l.len() <= 63
976 +
                && !l.starts_with('-')
977 +
                && !l.ends_with('-')
978 +
                && l.bytes().all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'-')
979 +
        })
980 +
}
981 +
982 +
/// Whether `host` is one `patterns` allow: a name exactly, or a name under
983 +
/// a `*.` domain (not the domain itself). Case and a final dot are ignored.
984 +
pub fn egress_allows(patterns: &[String], host: &str) -> bool {
985 +
    let host = host.trim_end_matches('.').to_ascii_lowercase();
986 +
    patterns.iter().any(|p| match p.strip_prefix("*.") {
987 +
        Some(domain) => host.len() > domain.len() + 1 && host.ends_with(domain) && host[..host.len() - domain.len()].ends_with('.'),
988 +
        None => host == *p,
989 +
    })
990 +
}
991 +
992 +
/// `network "egress" { allow "*.googleapis.com" "registry.opentofu.org" }`.
993 +
fn parse_egress(ctx: &mut Ctx, node: &KdlNode) -> Vec<String> {
994 +
    let mut out = Vec::new();
995 +
    for c in block(ctx, node) {
996 +
        ctx.no_children(c);
997 +
        if c.name().value() != "allow" {
998 +
            ctx.err(c.span(), format!("unknown egress setting `{}`; egress takes `allow \"host\"...`", c.name().value()));
999 +
            continue;
1000 +
        }
1001 +
        for h in ctx.strings(c) {
1002 +
            if egress_pattern_ok(&h) {
1003 +
                out.push(h);
1004 +
            } else {
1005 +
                ctx.err(
1006 +
                    c.span(),
1007 +
                    format!("allow {h:?}: expected a lower-case host name such as \"registry.opentofu.org\", or \"*.\" and a domain"),
1008 +
                );
1009 +
            }
1010 +
        }
1011 +
    }
1012 +
    out.sort();
1013 +
    out.dedup();
1014 +
    if out.is_empty() {
1015 +
        ctx.err(node.span(), "network \"egress\" needs at least one `allow \"host\"`");
1016 +
    } else if out.len() > MAX_EGRESS_HOSTS {
1017 +
        ctx.err(node.span(), format!("network \"egress\" may allow at most {MAX_EGRESS_HOSTS} hosts"));
1018 +
    }
1019 +
    out
1020 +
}
1021 +
821 1022
/// An `on` block: `push` and `tag` with globs (none: any), `manual`,
822 1023
/// `schedule` with five-field cron lines.
823 1024
fn parse_on(ctx: &mut Ctx, node: &KdlNode) -> On {
…
67 unchanged lines
891 1092
}
892 1093
893 1094
/// A `timeout` line: one duration, from a minute to a day.
1095 +
/// `priority "low"`: a pipeline may lower its jobs, never raise them
1096 +
/// (oxygenhub#244).
1097 +
fn parse_priority(ctx: &mut Ctx, c: &KdlNode) -> Option<String> {
1098 +
    ctx.no_children(c);
1099 +
    let p = ctx.one_string(c)?;
1100 +
    match p.as_str() {
1101 +
        "low" | "normal" => Some(p),
1102 +
        "high" | "urgent" => {
1103 +
            ctx.err(
1104 +
                c.span(),
1105 +
                format!("priority {p:?}: a pipeline may lower its jobs (`low`, `normal`), not raise them; higher levels are set on the hub, and a person may expedite a run"),
1106 +
            );
1107 +
            None
1108 +
        }
1109 +
        _ => {
1110 +
            ctx.err(c.span(), format!("priority {p:?}: expected `low` or `normal`"));
1111 +
            None
1112 +
        }
1113 +
    }
1114 +
}
1115 +
894 1116
fn parse_timeout(ctx: &mut Ctx, c: &KdlNode) -> Option<u64> {
895 1117
    let t = ctx.one_string(c)?;
896 1118
    match parse_duration(&t) {
…
55 unchanged lines
952 1174
/// `mount { source "ephemeral://disk"; target "/src/target"; size "8GiB" }`.
953 1175
fn parse_mount(ctx: &mut Ctx, node: &KdlNode) -> Option<Mount> {
954 1176
    let (mut source, mut target, mut size) = (None, None, None);
1177 +
    let (mut cache, mut keys) = (None, Vec::new());
955 1178
    let mut seen = BTreeSet::new();
956 1179
    for c in block(ctx, node) {
957 1180
        let key = c.name().value();
…
46 unchanged lines
1004 1227
                    None => None,
1005 1228
                }
1006 1229
            }
1230 +
            "cache" => {
1231 +
                cache = match ctx.one_string(c) {
1232 +
                    Some(n) if !n.is_empty() && n.len() <= 64 && n.bytes().all(|b| b.is_ascii_alphanumeric() || b"._-".contains(&b)) => {
1233 +
                        Some(n)
1234 +
                    }
1235 +
                    Some(n) => {
1236 +
                        ctx.err(c.span(), format!("cache name {n:?}: letters, digits, '.', '_' and '-', at most 64"));
1237 +
                        None
1238 +
                    }
1239 +
                    None => None,
1240 +
                }
1241 +
            }
1242 +
            "key" => {
1243 +
                keys = ctx.strings(c);
1244 +
                if keys.is_empty() {
1245 +
                    ctx.err(c.span(), "`key` names one or more files, as \"Cargo.lock\"");
1246 +
                }
1247 +
                for k in &keys {
1248 +
                    if k.starts_with('/') || k.split('/').any(|p| p.is_empty() || p == "." || p == "..") {
1249 +
                        ctx.err(c.span(), format!("key file {k:?} must be a path in the repository, without . or .."));
1250 +
                    }
1251 +
                }
1252 +
            }
1007 1253
            "readonly" | "include" => ctx.err(c.span(), format!("mount `{key}` is not in M0 yet")),
1008 1254
            other => ctx.err(c.span(), format!("unknown mount setting `{other}`")),
1009 1255
        }
…
3 unchanged lines
1013 1259
            ctx.err(node.span(), format!("a mount needs a {name}"));
1014 1260
        }
1015 1261
    }
1016 -
    Some(Mount { source: source?, target: target?, size_mib: size? })
1262 +
    if source == Some(MountSource::Tmpfs) && cache.is_some() {
1263 +
        ctx.err(node.span(), "only an ephemeral://disk mount can be a cache");
1264 +
    }
1265 +
    if !keys.is_empty() && cache.is_none() {
1266 +
        ctx.err(node.span(), "`key` goes with `cache`");
1267 +
    }
1268 +
    Some(Mount { source: source?, target: target?, size_mib: size?, cache, key: keys })
1017 1269
}
1018 1270
1019 1271
/// `output { artifact "name" { path "/src/target/release/tool" } }`.
…
140 unchanged lines
1160 1412
    }
1161 1413
}
1162 1414
1415 +
/// `approval { by "maintain"; others-only #true; expires "24h" }`, each
1416 +
/// optional: `maintain`, anyone with the role, 24 hours.
1417 +
/// The approval, and whether it said `sealed` itself (else the job's
1418 +
/// environment decides).
1419 +
fn parse_approval(ctx: &mut Ctx, node: &KdlNode) -> Option<(Approval, bool)> {
1420 +
    if !node.entries().is_empty() {
1421 +
        ctx.err(node.span(), "`approval` takes a block: approval { by \"maintain\"; others-only #true; expires \"24h\" }");
1422 +
    }
1423 +
    let mut a = Approval { by: "maintain".into(), others_only: false, expires_s: 24 * 3600, sealed: false };
1424 +
    let mut sealed_said = false;
1425 +
    let Some(children) = node.children() else { return Some((a, false)) };
1426 +
    for c in children.nodes() {
1427 +
        ctx.no_children(c);
1428 +
        match c.name().value() {
1429 +
            "by" => match ctx.one_string(c).as_deref() {
1430 +
                Some(r @ ("write" | "maintain" | "admin")) => a.by = r.to_string(),
1431 +
                Some(r) => ctx.err(c.span(), format!("approval by {r:?}: expected \"write\", \"maintain\" or \"admin\"")),
1432 +
                None => {}
1433 +
            },
1434 +
            "others-only" => match c.entries() {
1435 +
                [e] if e.name().is_none() => match e.value() {
1436 +
                    KdlValue::Bool(b) => a.others_only = *b,
1437 +
                    _ => ctx.err(c.span(), "`others-only` takes #true or #false"),
1438 +
                },
1439 +
                _ => ctx.err(c.span(), "`others-only` takes #true or #false"),
1440 +
            },
1441 +
            "sealed" => match c.entries() {
1442 +
                [e] if e.name().is_none() => match e.value() {
1443 +
                    KdlValue::Bool(b) => {
1444 +
                        a.sealed = *b;
1445 +
                        sealed_said = true;
1446 +
                    }
1447 +
                    _ => ctx.err(c.span(), "`sealed` takes #true or #false"),
1448 +
                },
1449 +
                _ => ctx.err(c.span(), "`sealed` takes #true or #false"),
1450 +
            },
1451 +
            "expires" => {
1452 +
                let Some(t) = ctx.one_string(c) else { continue };
1453 +
                match parse_duration(&t) {
1454 +
                    Some(secs) if (60..=MAX_APPROVAL_S).contains(&secs) => a.expires_s = secs,
1455 +
                    Some(_) => ctx.err(c.span(), format!("expires {t:?} must be from 1m to 720h")),
1456 +
                    None => ctx.err(c.span(), format!("expires {t:?}: expected hours, minutes and seconds, as \"24h\"")),
1457 +
                }
1458 +
            }
1459 +
            other => {
1460 +
                ctx.err(c.span(), format!("unknown approval setting `{other}`; approval takes `by`, `others-only`, `expires` and `sealed`"))
1461 +
            }
1462 +
        }
1463 +
    }
1464 +
    Some((a, sealed_said))
1465 +
}
1466 +
1467 +
/// `input { artifact "NAME" from "JOB"; path "/src/FILE" }`.
1468 +
fn parse_input(ctx: &mut Ctx, node: &KdlNode, job: &mut Job) {
1469 +
    if !node.entries().is_empty() {
1470 +
        ctx.err(node.span(), "`input` takes a block: input { artifact \"tfplan\" from \"plan\"; path \"/src/tfplan\" }");
1471 +
    }
1472 +
    let (mut artifact, mut from, mut path) = (None, None, None);
1473 +
    for c in block(ctx, node) {
1474 +
        ctx.no_children(c);
1475 +
        match c.name().value() {
1476 +
            "artifact" => match ctx.strings(c).as_slice() {
1477 +
                [name, word, producer] if word == "from" => {
1478 +
                    if ctx.name_ok(c.span(), "artifact", name) && ctx.name_ok(c.span(), "job", producer) {
1479 +
                        artifact = Some(name.clone());
1480 +
                        from = Some(producer.clone());
1481 +
                    }
1482 +
                }
1483 +
                _ => ctx.err(c.span(), "`artifact` names the artifact and its job: artifact \"tfplan\" from \"plan\""),
1484 +
            },
1485 +
            "path" => match ctx.one_string(c) {
1486 +
                Some(p) if output_path_ok(&p) => path = Some(p),
1487 +
                Some(p) => {
1488 +
                    ctx.err(c.span(), format!("input path {p:?} must be an absolute path under /src, without . or .., quotes or spaces"))
1489 +
                }
1490 +
                None => {}
1491 +
            },
1492 +
            other => ctx.err(c.span(), format!("unknown input setting `{other}`; an input has `artifact` and `path`")),
1493 +
        }
1494 +
    }
1495 +
    match (artifact, from, path) {
1496 +
        (Some(artifact), Some(from), Some(path)) => {
1497 +
            if job.inputs.iter().any(|i| i.path == path) {
1498 +
                ctx.err(node.span(), format!("two inputs are put at {path:?}"));
1499 +
            } else if job.inputs.len() == MAX_INPUTS {
1500 +
                ctx.err(node.span(), format!("a job takes at most {MAX_INPUTS} inputs"));
1501 +
            } else {
1502 +
                job.inputs.push(Input { artifact, from, path });
1503 +
            }
1504 +
        }
1505 +
        (a, _, p) => {
1506 +
            if a.is_some() && p.is_none() {
1507 +
                ctx.err(node.span(), "an input needs a `path`");
1508 +
            } else if a.is_none() && p.is_some() {
1509 +
                ctx.err(node.span(), "an input needs `artifact \"NAME\" from \"JOB\"`");
1510 +
            }
1511 +
        }
1512 +
    }
1513 +
}
1514 +
1163 1515
fn parse_job(ctx: &mut Ctx, node: &KdlNode) -> Option<Job> {
1164 1516
    let name = ctx.one_string(node)?;
1165 1517
    let named = ctx.name_ok(node.span(), "job", &name);
1518 +
    let mut approval: Option<(Approval, bool)> = None;
1166 1519
    let mut job = Job {
1167 1520
        name: name.clone(),
1168 1521
        sandbox: String::new(),
…
5 unchanged lines
1174 1527
        secrets: vec![],
1175 1528
        identities: vec![],
1176 1529
        outputs: vec![],
1530 +
        inputs: vec![],
1531 +
        approval: None,
1532 +
        environment: None,
1177 1533
        runs_on: vec![],
1178 1534
        timeout_s: None,
1179 1535
        on: None,
1536 +
        priority: None,
1537 +
        checkpoints: Vec::new(),
1180 1538
    };
1181 1539
    let mut seen = BTreeSet::new();
1182 1540
    let mut outputs_at = None;
…
5 unchanged lines
1188 1546
            && key != "secret"
1189 1547
            && key != "identity"
1190 1548
            && key != "runs-on"
1549 +
            && key != "checkpoint"
1191 1550
            && !seen.insert(key.to_string())
1192 1551
        {
1193 1552
            ctx.err(c.span(), format!("`{key}` is given twice"));
…
3 unchanged lines
1197 1556
                ctx.no_children(c);
1198 1557
                job.sandbox = ctx.one_string(c).unwrap_or_default();
1199 1558
            }
1559 +
            "checkpoint" => {
1560 +
                ctx.no_children(c);
1561 +
                for p in ctx.strings(c) {
1562 +
                    if output_path_ok(&p) && p.trim_end_matches('/') != "/src" {
1563 +
                        job.checkpoints.push(p);
1564 +
                    } else {
1565 +
                        ctx.err(c.span(), format!("checkpoint {p:?} must be a directory under /src, without . or .., quotes or spaces"));
1566 +
                    }
1567 +
                }
1568 +
            }
1200 1569
            "scope" => {
1201 1570
                ctx.no_children(c);
1202 1571
                for g in ctx.strings(c) {
…
15 unchanged lines
1218 1587
                ctx.no_children(c);
1219 1588
                job.timeout_s = parse_timeout(ctx, c);
1220 1589
            }
1590 +
            "priority" => job.priority = parse_priority(ctx, c),
1221 1591
            "on" => job.on = Some(parse_on(ctx, c)),
1222 1592
            "env" => parse_env(ctx, c, &mut job.env),
1223 1593
            "exec" => {
…
27 unchanged lines
1251 1621
                outputs_at = Some(c.span());
1252 1622
                parse_output(ctx, c, &mut job);
1253 1623
            }
1624 +
            "input" => parse_input(ctx, c, &mut job),
1625 +
            "approval" => approval = parse_approval(ctx, c),
1626 +
            "environment" => {
1627 +
                ctx.no_children(c);
1628 +
                if let Some(e) = ctx.one_string(c) {
1629 +
                    let ok = !e.is_empty() && e.len() <= 32 && e.bytes().all(|b| b.is_ascii_lowercase() || b.is_ascii_digit() || b == b'-');
1630 +
                    if ok {
1631 +
                        job.environment = Some(e);
1632 +
                    } else {
1633 +
                        ctx.err(c.span(), format!("environment {e:?}: 1 to 32 of a-z 0-9 -, as \"prod\""));
1634 +
                    }
1635 +
                }
1636 +
            }
1254 1637
            "secret" => parse_secret(ctx, c, &mut job),
1255 1638
            "identity" => parse_identity(ctx, c, &mut job),
1256 1639
            n if LATER_JOB.contains(&n) => ctx.err(c.span(), format!("`{n}` is not in M0 yet")),
1257 1640
            other => ctx.err(c.span(), format!("unknown job setting `{other}`")),
1258 1641
        }
1259 1642
    }
1643 +
    // A job for an environment takes a sealed approval unless it says not.
1644 +
    job.approval = approval.map(|(mut a, said)| {
1645 +
        if !said {
1646 +
            a.sealed = job.environment.is_some();
1647 +
        }
1648 +
        a
1649 +
    });
1260 1650
    job.mounts.sort_by(|a, b| a.target.cmp(&b.target));
1261 1651
    job.outputs.sort_by(|a, b| a.name.cmp(&b.name));
1652 +
    job.inputs.sort_by(|a, b| a.path.cmp(&b.path));
1262 1653
    job.secrets.sort_by(|a, b| a.name.cmp(&b.name));
1263 1654
    job.identities.sort_by(|a, b| a.audience.cmp(&b.audience));
1264 1655
    // A tmpfs is gone when the job ends; nothing on it can be kept.
…
16 unchanged lines
1281 1672
    }
1282 1673
    job.depends_on.sort();
1283 1674
    job.depends_on.dedup();
1675 +
    job.checkpoints.sort();
1676 +
    job.checkpoints.dedup();
1284 1677
    // Kept even when incomplete, so its references are checked too; the
1285 1678
    // problems above already make the pipeline invalid.
1286 1679
    named.then_some(job)
…
1 unchanged lines
modified crates/flow-core/src/tests.rs +200 −9
…
94 unchanged lines
95 95
96 96
#[test]
97 97
fn m0_refuses_what_it_cannot_honour() {
98 -
    let src = format!("schema-version 1\nsandbox \"s\" {{\n  runtime \"wasm\"\n  rootfs \"cas://blob/{ROOTFS}\"\n  network \"egress\"\n}}\njob \"a\" {{ sandbox \"s\"; exec \"true\"; }}\n");
98 +
    let src = format!("schema-version 1\nsandbox \"s\" {{\n  runtime \"wasm\"\n  rootfs \"cas://blob/{ROOTFS}\"\n  network \"wifi\"\n}}\njob \"a\" {{ sandbox \"s\"; exec \"true\"; }}\n");
99 99
    let ps = problems(&src);
100 -
    assert!(ps.iter().any(|(l, m)| *l == 3 && m.contains("wasm")), "{ps:?}");
100 +
    // A WASM sandbox has no image (oxygenflow#75).
101 +
    assert!(ps.iter().any(|(l, m)| *l == 2 && m.contains("takes no rootfs")), "{ps:?}");
101 102
    assert!(ps.iter().any(|(l, m)| *l == 5 && m.contains("isolated")), "{ps:?}");
102 103
    assert!(problems("schema-version 2\n").iter().any(|(_, m)| m.contains("schema-version 2")));
103 104
    assert!(problems(&format!(
…
99 unchanged lines
203 204
    assert_eq!(
204 205
        b.mounts,
205 206
        vec![
206 -
            Mount { source: MountSource::Disk, target: "/src/target".into(), size_mib: 8192 },
207 -
            Mount { source: MountSource::Tmpfs, target: "/tmp/cache".into(), size_mib: 256 },
207 +
            Mount { source: MountSource::Disk, target: "/src/target".into(), size_mib: 8192, cache: None, key: vec![] },
208 +
            Mount { source: MountSource::Tmpfs, target: "/tmp/cache".into(), size_mib: 256, cache: None, key: vec![] },
208 209
        ],
209 210
        "sorted by target"
210 211
    );
…
9 unchanged lines
220 221
}
221 222
222 223
#[test]
224 +
fn a_disk_mount_can_be_a_cache_keyed_by_files() {
225 +
    let src = |extra: &str| {
226 +
        doc(&format!(
227 +
            "job \"b\" {{\n    sandbox \"alpine\"\n    mount {{\n        source \"ephemeral://disk\"\n        target \"/src/target\"\n        size \"8GiB\"\n{extra}    }}\n    exec \"cargo\" \"build\"\n}}\n"
228 +
        ))
229 +
    };
230 +
    let plain = Pipeline::parse(&src("")).unwrap();
231 +
    let cached = Pipeline::parse(&src("        cache \"target\"\n        key \"Cargo.lock\" \"rust-toolchain.toml\"\n")).unwrap();
232 +
    let m = &cached.jobs["b"].mounts[0];
233 +
    assert_eq!((m.cache.as_deref(), m.key.clone()), (Some("target"), vec!["Cargo.lock".to_string(), "rust-toolchain.toml".to_string()]));
234 +
    assert_ne!(cached.hash(), plain.hash(), "a cache is part of the job");
235 +
    let keyless = Pipeline::parse(&src("        cache \"target\"\n")).unwrap();
236 +
    assert_ne!(keyless.hash(), cached.hash());
237 +
    let ps = problems(&src("        cache \"no spaces\"\n"));
238 +
    assert!(ps.iter().any(|(_, m)| m.contains("cache name")), "{ps:?}");
239 +
    let ps = problems(&src("        key \"Cargo.lock\"\n"));
240 +
    assert!(ps.iter().any(|(_, m)| m.contains("goes with `cache`")), "{ps:?}");
241 +
    let ps = problems(&src("        cache \"t\"\n        key \"../x\"\n"));
242 +
    assert!(ps.iter().any(|(_, m)| m.contains("path in the repository")), "{ps:?}");
243 +
    let tmpfs = doc("job \"b\" {\n    sandbox \"alpine\"\n    mount {\n        source \"ephemeral://tmpfs\"\n        target \"/tmp/c\"\n        size \"64MiB\"\n        cache \"c\"\n    }\n    exec \"true\"\n}\n");
244 +
    assert!(problems(&tmpfs).iter().any(|(_, m)| m.contains("only an ephemeral://disk")));
245 +
}
246 +
247 +
#[test]
223 248
fn mounts_are_checked() {
224 249
    let job = |mount: &str| doc(&format!("job \"b\" {{\n    sandbox \"alpine\"\n    mount {{\n{mount}\n    }}\n    exec \"true\"\n}}\n"));
225 250
    let has = |src: String, want: &str| {
…
25 unchanged lines
251 276
}
252 277
253 278
#[test]
254 -
fn loopback_is_in_m0_and_egress_is_not() {
279 +
fn loopback_and_allow_listed_egress() {
255 280
    let src = |net: &str| {
256 -
        format!("schema-version 1\nsandbox \"s\" {{\n  rootfs \"cas://blob/{ROOTFS}\"\n  network \"{net}\"\n}}\njob \"a\" {{ sandbox \"s\"; exec \"true\"; }}\n")
281 +
        format!("schema-version 1\nsandbox \"s\" {{\n  rootfs \"cas://blob/{ROOTFS}\"\n  network {net}\n}}\njob \"a\" {{ sandbox \"s\"; exec \"true\"; }}\n")
257 282
    };
258 -
    let p = Pipeline::parse(&src("loopback")).unwrap();
283 +
    let p = Pipeline::parse(&src("\"loopback\"")).unwrap();
259 284
    assert_eq!(p.sandboxes["s"].network, Network::Loopback);
260 -
    assert_ne!(p.hash(), Pipeline::parse(&src("isolated")).unwrap().hash(), "the network is part of the sandbox");
261 -
    assert!(problems(&src("egress")).iter().any(|(l, m)| *l == 4 && m.contains("not in M0")));
285 +
    assert_ne!(p.hash(), Pipeline::parse(&src("\"isolated\"")).unwrap().hash(), "the network is part of the sandbox");
286 +
287 +
    // Egress names its hosts (oxygenflow#55), sorted; the sandbox then needs
288 +
    // a runner offering the egress label.
289 +
    let p =
290 +
        Pipeline::parse(&src("\"egress\" { allow \"registry.opentofu.org\" \"*.googleapis.com\"; allow \"*.googleapis.com\"; }")).unwrap();
291 +
    let s = &p.sandboxes["s"];
292 +
    assert_eq!(s.network, Network::Egress);
293 +
    assert_eq!(s.egress_allow, ["*.googleapis.com", "registry.opentofu.org"]);
294 +
    assert_eq!(s.runs_on, [crate::EGRESS_LABEL]);
295 +
    let other = Pipeline::parse(&src("\"egress\" { allow \"registry.opentofu.org\"; }")).unwrap();
296 +
    assert_ne!(p.hash(), other.hash(), "the allow-list is part of the sandbox");
297 +
298 +
    let has = |net: &str, want: &str| {
299 +
        let ps = problems(&src(net));
300 +
        assert!(ps.iter().any(|(_, m)| m.contains(want)), "{net}: {ps:?}");
301 +
    };
302 +
    has("\"egress\"", "needs a block");
303 +
    has("\"egress\" { }", "at least one `allow");
304 +
    has("\"egress\" { allow \"Example.COM\"; }", "lower-case host name");
305 +
    has("\"egress\" { allow \"localhost\"; }", "lower-case host name");
306 +
    has("\"egress\" { allow \"example.com:8443\"; }", "lower-case host name");
307 +
    has("\"egress\" { deny \"example.com\"; }", "unknown egress setting");
308 +
    has("\"loopback\" { allow \"example.com\"; }", "takes no block");
309 +
    let syscall = format!(
310 +
        "schema-version 1\nsandbox \"s\" {{\n  runtime \"syscall\"\n  rootfs \"cas://blob/{ROOTFS}\"\n  network \"egress\" {{ allow \"example.com\"; }}\n}}\njob \"a\" {{ sandbox \"s\"; exec \"true\"; }}\n"
311 +
    );
312 +
    assert!(problems(&syscall).iter().any(|(_, m)| m.contains("needs runtime \"microvm\"")));
313 +
}
314 +
315 +
#[test]
316 +
fn egress_patterns_match_names_and_names_under_a_domain() {
317 +
    let allow = vec!["*.googleapis.com".to_string(), "registry.opentofu.org".to_string()];
318 +
    for ok in ["storage.googleapis.com", "a.b.googleapis.com", "REGISTRY.opentofu.org.", "registry.opentofu.org"] {
319 +
        assert!(crate::egress_allows(&allow, ok), "{ok}");
320 +
    }
321 +
    for no in ["googleapis.com", "evilgoogleapis.com", "googleapis.com.evil.org", "opentofu.org", "x.registry.opentofu.org", ""] {
322 +
        assert!(!crate::egress_allows(&allow, no), "{no}");
323 +
    }
262 324
}
263 325
264 326
#[test]
…
117 unchanged lines
382 444
        run_number: 3,
383 445
        env: [("CC".to_string(), "clang".to_string()), ("FLOW_REPO".to_string(), "spoofed".to_string())].into_iter().collect(),
384 446
        secrets: vec![],
447 +
        inputs: vec![],
448 +
        expected_ms: Some(17_700),
449 +
        checkpoint: None,
385 450
    };
386 451
    let env = job.with_flow_env(&claim.flow_env()).env;
387 452
    assert_eq!(env["CC"], "clang", "the hub's variables reach the job");
…
14 unchanged lines
402 467
    };
403 468
    let names: Vec<String> = old.flow_env().into_iter().map(|(k, _)| k).collect();
404 469
    assert_eq!(names, ["FLOW_REPO", "FLOW_HEADS", "FLOW_JOB"]);
470 +
    // The expected time goes with the claim, and an older hub's claim
471 +
    // without it still reads (#76).
472 +
    let json = serde_json::to_value(&old).unwrap();
473 +
    assert_eq!(json["expected_ms"], 17_700);
474 +
    let mut without = json.clone();
475 +
    without.as_object_mut().unwrap().remove("expected_ms");
476 +
    assert_eq!(serde_json::from_value::<crate::api::Claim>(without).unwrap().expected_ms, None);
405 477
}
406 478
407 479
#[test]
…
113 unchanged lines
521 593
    assert!(none.iter().any(|(_, m)| m.contains("needs one or more labels")), "{none:?}");
522 594
}
523 595
596 +
/// oxygenhub#244: a pipeline may lower its jobs' priority, for all of them
597 +
/// or one by one, never raise it.
598 +
#[test]
599 +
fn a_pipeline_lowers_its_jobs_priority_and_never_raises_it() {
600 +
    let jobs = "job \"nightly\" {\n    sandbox \"alpine\"\n    exec \"true\"\n}\njob \"check\" {\n    sandbox \"alpine\"\n    priority \"normal\"\n    exec \"true\"\n}\n";
601 +
    let p = Pipeline::parse(&format!("priority \"low\"\n{}", doc(jobs))).unwrap();
602 +
    assert_eq!(p.jobs["nightly"].priority.as_deref(), Some("low"), "the pipeline's");
603 +
    assert_eq!(p.jobs["check"].priority.as_deref(), Some("normal"), "its own");
604 +
    let one = Pipeline::parse(&doc("job \"a\" {\n    sandbox \"alpine\"\n    priority \"low\"\n    exec \"true\"\n}\n")).unwrap();
605 +
    assert_eq!(one.jobs["a"].priority.as_deref(), Some("low"));
606 +
    // None given: serialised as before, so its jobs' hashes stay.
607 +
    let plain = Pipeline::parse(&doc("job \"x\" {\n    sandbox \"alpine\"\n    exec \"true\"\n}\n")).unwrap();
608 +
    assert!(!serde_json::to_string(&plain.jobs["x"]).unwrap().contains("priority"));
609 +
    for high in ["high", "urgent"] {
610 +
        let bad = problems(&doc(&format!("job \"x\" {{\n    sandbox \"alpine\"\n    priority \"{high}\"\n    exec \"true\"\n}}\n")));
611 +
        assert!(bad.iter().any(|(_, m)| m.contains("not raise them")), "{high}: {bad:?}");
612 +
    }
613 +
    let odd = problems(&doc("job \"x\" {\n    sandbox \"alpine\"\n    priority \"soon\"\n    exec \"true\"\n}\n"));
614 +
    assert!(odd.iter().any(|(_, m)| m.contains("expected `low` or `normal`")), "{odd:?}");
615 +
}
616 +
524 617
/// oxygenflow#23: `timeout` on a job, or on its sandbox for every job in
525 618
/// it, as a duration from a minute to a day.
526 619
#[test]
…
54 unchanged lines
581 674
        assert!(e.contains(why), "{bad:?}: {e}");
582 675
    }
583 676
}
677 +
678 +
#[test]
679 +
fn a_job_takes_an_artifact_of_a_job_it_depends_on() {
680 +
    let jobs = |apply: &str| {
681 +
        doc(&format!(
682 +
            "job \"plan\" {{\n    sandbox \"alpine\"\n    output {{ artifact \"tfplan\" {{ path \"/src/out/tfplan\"; }}; report \"r\" {{ path \"/src/out/r\"; }}; }}\n    exec \"plan\"\n}}\njob \"apply\" {{\n    sandbox \"alpine\"\n{apply}    exec \"apply\"\n}}\n"
683 +
        ))
684 +
    };
685 +
    let ok = jobs("    depends-on \"plan\"\n    input { artifact \"tfplan\" from \"plan\"; path \"/src/tfplan\"; }\n");
686 +
    let p = Pipeline::parse(&ok).unwrap();
687 +
    assert_eq!(p.jobs["apply"].inputs, [crate::Input { artifact: "tfplan".into(), from: "plan".into(), path: "/src/tfplan".into() }]);
688 +
    let without = Pipeline::parse(&jobs("    depends-on \"plan\"\n")).unwrap();
689 +
    assert_ne!(p.hash(), without.hash(), "inputs are part of the job");
690 +
691 +
    let has = |apply: &str, want: &str| {
692 +
        let ps = problems(&jobs(apply));
693 +
        assert!(ps.iter().any(|(_, m)| m.contains(want)), "{apply}: {ps:?}");
694 +
    };
695 +
    has("    input { artifact \"tfplan\" from \"plan\"; path \"/src/tfplan\"; }\n", "add depends-on \"plan\"");
696 +
    has("    depends-on \"plan\"\n    input { artifact \"nope\" from \"plan\"; path \"/src/x\"; }\n", "declares no artifact \"nope\"");
697 +
    has("    depends-on \"plan\"\n    input { artifact \"r\" from \"plan\"; path \"/src/x\"; }\n", "is a report");
698 +
    has("    input { artifact \"tfplan\" from \"ghost\"; path \"/src/x\"; }\n", "which is not declared");
699 +
    has("    depends-on \"plan\"\n    input { artifact \"tfplan\" from \"plan\"; path \"/etc/x\"; }\n", "absolute path under /src");
700 +
    has("    depends-on \"plan\"\n    input { artifact \"tfplan\" \"plan\"; path \"/src/x\"; }\n", "artifact \"tfplan\" from \"plan\"");
701 +
    has("    depends-on \"plan\"\n    input { artifact \"tfplan\" from \"plan\"; }\n", "needs a `path`");
702 +
    has(
703 +
        "    depends-on \"plan\"\n    input { artifact \"tfplan\" from \"plan\"; path \"/src/x\"; }\n    input { artifact \"tfplan\" from \"plan\"; path \"/src/x\"; }\n",
704 +
        "two inputs are put at",
705 +
    );
706 +
}
707 +
708 +
#[test]
709 +
fn a_job_can_wait_for_approval() {
710 +
    let job = |approval: &str| doc(&format!("job \"apply\" {{\n    sandbox \"alpine\"\n{approval}    exec \"apply\"\n}}\n"));
711 +
    let p = Pipeline::parse(&job("    approval { by \"maintain\"; others-only #true; expires \"2h\"; }\n")).unwrap();
712 +
    assert_eq!(
713 +
        p.jobs["apply"].approval,
714 +
        Some(crate::Approval { by: "maintain".into(), others_only: true, expires_s: 7200, sealed: false })
715 +
    );
716 +
    let plain = Pipeline::parse(&job("    approval\n")).unwrap();
717 +
    assert_eq!(
718 +
        plain.jobs["apply"].approval,
719 +
        Some(crate::Approval { by: "maintain".into(), others_only: false, expires_s: 86400, sealed: false }),
720 +
        "defaults"
721 +
    );
722 +
    // Sealed (oxygenhub#235): on by default for a job with an environment,
723 +
    // as it says otherwise.
724 +
    let sealed = |extra: &str| Pipeline::parse(&job(extra)).unwrap().jobs["apply"].approval.clone().unwrap().sealed;
725 +
    assert!(sealed("    environment \"prod\"\n    approval\n"));
726 +
    assert!(!sealed("    environment \"prod\"\n    approval { sealed #false; }\n"));
727 +
    assert!(sealed("    approval { sealed #true; }\n"));
728 +
    assert_ne!(Pipeline::parse(&job("    approval { sealed #true; }\n")).unwrap().hash(), plain.hash(), "part of the job");
729 +
    assert_eq!(Pipeline::parse(&job("")).unwrap().jobs["apply"].approval, None);
730 +
    assert_ne!(p.hash(), plain.hash(), "the approval is part of the job");
731 +
    let has = |a: &str, want: &str| {
732 +
        let ps = problems(&job(a));
733 +
        assert!(ps.iter().any(|(_, m)| m.contains(want)), "{a}: {ps:?}");
734 +
    };
735 +
    has("    approval { by \"read\"; }\n", "expected \"write\", \"maintain\" or \"admin\"");
736 +
    has("    approval { others-only \"yes\"; }\n", "#true or #false");
737 +
    has("    approval { expires \"800h\"; }\n", "from 1m to 720h");
738 +
    has("    approval { when \"later\"; }\n", "unknown approval setting");
739 +
    has("    approval { sealed \"yes\"; }\n", "#true or #false");
740 +
}
741 +
742 +
#[test]
743 +
fn a_job_names_the_environment_it_deploys_to() {
744 +
    let job = |e: &str| doc(&format!("job \"apply\" {{\n    sandbox \"alpine\"\n{e}    exec \"apply\"\n}}\n"));
745 +
    let p = Pipeline::parse(&job("    environment \"prod\"\n")).unwrap();
746 +
    assert_eq!(p.jobs["apply"].environment.as_deref(), Some("prod"));
747 +
    assert_ne!(p.hash(), Pipeline::parse(&job("")).unwrap().hash(), "part of the job");
748 +
    assert!(problems(&job("    environment \"Prod env\"\n")).iter().any(|(_, m)| m.contains("1 to 32 of a-z 0-9 -")));
749 +
}
750 +
751 +
/// oxygenflow#75: a WASM sandbox needs no rootfs and has no network; its
752 +
/// job's exec names the module.
753 +
#[test]
754 +
fn a_wasm_sandbox_runs_a_module() {
755 +
    let src = "schema-version 1\nsandbox \"w\" {\n  runtime \"wasm\"\n  memory-limit \"512MiB\"\n}\njob \"t\" {\n  sandbox \"w\"\n  exec \"target/wasm32-wasip1/debug/deps/*.wasm\" \"--quiet\"\n}\n";
756 +
    let p = Pipeline::parse(src).unwrap_or_else(|ps| panic!("{:?}", ps.0));
757 +
    let sb = &p.sandboxes["w"];
758 +
    assert_eq!((sb.runtime, sb.rootfs.as_str(), sb.network, sb.memory_mib), (Runtime::Wasm, "", Network::Isolated, 512));
759 +
    let networked =
760 +
        "schema-version 1\nsandbox \"w\" {\n  runtime \"wasm\"\n  network \"loopback\"\n}\njob \"t\" { sandbox \"w\"; exec \"a.wasm\"; }\n";
761 +
    assert!(problems(networked).iter().any(|(_, m)| m.contains("has no network")));
762 +
}
763 +
764 +
#[test]
765 +
fn a_finish_says_where_its_time_went_and_an_older_one_still_reads() {
766 +
    let old = r#"{"exit_code":0,"duration_ms":5,"receipt":null,"error":null}"#;
767 +
    let mut f: crate::api::Finish = serde_json::from_str(old).unwrap();
768 +
    assert!(f.phases.is_empty());
769 +
    assert!(!serde_json::to_string(&f).unwrap().contains("phases"), "absent when there are none");
770 +
    f.phases.insert("prepare".into(), 1200);
771 +
    f.phases.insert("run".into(), 3400);
772 +
    let back: crate::api::Finish = serde_json::from_str(&serde_json::to_string(&f).unwrap()).unwrap();
773 +
    assert_eq!(back.phases["run"], 3400);
774 +
}
modified crates/flow-init/Cargo.toml +3 −0
…
8 unchanged lines
9 9
libc = "0.2"
10 10
serde.workspace = true
11 11
serde_json.workspace = true
12 +
13 +
[dev-dependencies]
14 +
tempfile.workspace = true
added crates/flow-init/src/egress.rs +80 −0
1 +
//! The guest's half of allow-listed egress (oxygenflow#55): a proxy on
2 +
//! 127.0.0.1:3128 that relays each connection, unread, over vsock to the
3 +
//! runner (host CID 2), which decides what may pass. The VM has no network
4 +
//! device; this is its only way out.
5 +
6 +
use std::fs::File;
7 +
use std::io::{Read, Write};
8 +
use std::net::{TcpListener, TcpStream};
9 +
use std::os::fd::{AsRawFd, FromRawFd};
10 +
11 +
/// Where the job's tools reach the proxy, and the vsock port behind it.
12 +
const LISTEN: &str = "127.0.0.1:3128";
13 +
const VSOCK_PORT: u32 = 3128;
14 +
15 +
/// Start relaying in the background (before the job starts, so its first
16 +
/// connection finds the proxy listening).
17 +
pub fn start() -> Result<(), String> {
18 +
    let listener = TcpListener::bind(LISTEN).map_err(|e| format!("egress proxy on {LISTEN}: {e}"))?;
19 +
    std::thread::spawn(move || {
20 +
        for conn in listener.incoming().flatten() {
21 +
            std::thread::spawn(move || relay(conn));
22 +
        }
23 +
    });
24 +
    Ok(())
25 +
}
26 +
27 +
fn vsock_to_host(port: u32) -> std::io::Result<File> {
28 +
    let fd = unsafe { libc::socket(libc::AF_VSOCK, libc::SOCK_STREAM | libc::SOCK_CLOEXEC, 0) };
29 +
    if fd < 0 {
30 +
        return Err(std::io::Error::last_os_error());
31 +
    }
32 +
    // Owned from here, so it is closed on every path.
33 +
    let file = unsafe { File::from_raw_fd(fd) };
34 +
    let mut addr: libc::sockaddr_vm = unsafe { std::mem::zeroed() };
35 +
    addr.svm_family = libc::AF_VSOCK as libc::sa_family_t;
36 +
    addr.svm_cid = libc::VMADDR_CID_HOST;
37 +
    addr.svm_port = port;
38 +
    let rc = unsafe {
39 +
        libc::connect(
40 +
            fd,
41 +
            &addr as *const libc::sockaddr_vm as *const libc::sockaddr,
42 +
            std::mem::size_of::<libc::sockaddr_vm>() as libc::socklen_t,
43 +
        )
44 +
    };
45 +
    if rc != 0 {
46 +
        return Err(std::io::Error::last_os_error());
47 +
    }
48 +
    Ok(file)
49 +
}
50 +
51 +
fn relay(tcp: TcpStream) {
52 +
    let Ok(host) = vsock_to_host(VSOCK_PORT) else {
53 +
        let mut tcp = tcp;
54 +
        let _ = tcp.write_all(b"HTTP/1.1 502 Bad Gateway\r\nConnection: close\r\n\r\nthe runner's egress proxy did not answer\n");
55 +
        return;
56 +
    };
57 +
    let (Ok(mut tcp_r), Ok(mut host_w)) = (tcp.try_clone(), host.try_clone()) else { return };
58 +
    let up = std::thread::spawn(move || {
59 +
        copy(&mut tcp_r, &mut host_w);
60 +
        unsafe { libc::shutdown(host_w.as_raw_fd(), libc::SHUT_WR) };
61 +
    });
62 +
    let (mut host_r, mut tcp_w) = (host, tcp);
63 +
    copy(&mut host_r, &mut tcp_w);
64 +
    let _ = tcp_w.shutdown(std::net::Shutdown::Write);
65 +
    let _ = up.join();
66 +
}
67 +
68 +
fn copy(from: &mut impl Read, to: &mut impl Write) {
69 +
    let mut buf = [0u8; 16 * 1024];
70 +
    loop {
71 +
        match from.read(&mut buf) {
72 +
            Ok(0) | Err(_) => return,
73 +
            Ok(n) => {
74 +
                if to.write_all(&buf[..n]).is_err() {
75 +
                    return;
76 +
                }
77 +
            }
78 +
        }
79 +
    }
80 +
}
modified crates/flow-init/src/job.rs +3 −2
…
69 unchanged lines
70 70
        req.ifr_name[i] = *b as libc::c_char;
71 71
    }
72 72
    let ok = unsafe {
73 -
        libc::ioctl(fd, libc::SIOCGIFFLAGS, &mut req) == 0 && {
73 +
        // `as _`: the request's type differs between libcs (Android's is c_int).
74 +
        libc::ioctl(fd, libc::SIOCGIFFLAGS as _, &mut req) == 0 && {
74 75
            req.ifr_ifru.ifru_flags |= libc::IFF_UP as libc::c_short;
75 -
            libc::ioctl(fd, libc::SIOCSIFFLAGS, &req) == 0
76 +
            libc::ioctl(fd, libc::SIOCSIFFLAGS as _, &req) == 0
76 77
        }
77 78
    };
78 79
    let err = std::io::Error::last_os_error();
…
58 unchanged lines
added crates/flow-init/src/lib.rs +5 −0
1 +
//! The parts of the guest init the runner shares: pruning a build cache
2 +
//! before it is kept (oxygenflow#72), done in the guest for a microVM's
3 +
//! disks and on the runner for the syscall tier's directories.
4 +
5 +
pub mod prune;
modified crates/flow-init/src/main.rs +71 −4
…
6 unchanged lines
7 7
//!
8 8
//! Frames, one per line on the console: `\x1eO ` stdout, `\x1eE ` stderr,
9 9
//! `\x1eX <code>` the exit code (128 + signal when killed), `\x1eF <text>`
10 -
//! a failure of init itself.
10 +
//! a failure of init itself, `\x1eC <json>` a build cache pruned before
11 +
//! the guest powers off (oxygenflow#72): `{"target": ..., "pruned":
12 +
//! flow_init::prune::Pruned}`, sent before the exit code.
11 13
14 +
use flow_init::prune;
12 15
use serde::Deserialize;
13 16
use std::ffi::CString;
14 17
use std::io::{Read, Write};
15 18
use std::os::fd::FromRawFd;
16 19
use std::sync::{Arc, Mutex};
17 20
21 +
mod egress;
18 22
mod job;
19 23
use job::{dev_links, job_env, loopback_up, resolve_program, PATH};
20 24
…
6 unchanged lines
27 31
    /// `network "loopback"`: bring up `lo` (the guest has no other device).
28 32
    #[serde(default)]
29 33
    loopback: bool,
34 +
    /// `network "egress"` (oxygenflow#55): `lo` comes up and a proxy on
35 +
    /// 127.0.0.1:3128 relays over vsock to the runner.
36 +
    #[serde(default)]
37 +
    egress: bool,
30 38
    /// Secrets to write before the job starts (spec §9): each at `path`
31 39
    /// under `/run/secrets`, 0400 and the job's user's, on a tmpfs.
32 40
    #[serde(default)]
…
15 unchanged lines
48 56
    size_mib: u32,
49 57
    #[serde(default)]
50 58
    device: Option<String>,
59 +
    /// A build cache's disk (oxygenflow#72): mounted `strictatime`, so
60 +
    /// what the job reads is seen, and after a passing job pruned to this
61 +
    /// percentage of its size before the guest powers off.
62 +
    #[serde(default)]
63 +
    prune: Option<u8>,
64 +
    /// Paths on the disk, relative to it, that pruning leaves: the job's
65 +
    /// outputs, which the runner reads after the guest stops.
66 +
    #[serde(default)]
67 +
    keep: Vec<String>,
51 68
}
52 69
53 70
fn main() {
…
102 unchanged lines
156 173
        match (m.kind.as_str(), &m.device) {
157 174
            ("tmpfs", _) => mount("tmpfs", &target, "tmpfs", nosuid, &format!("size={}m,uid={uid},gid={gid},mode=755", m.size_mib))?,
158 175
            ("disk", Some(dev)) => {
159 -
                mount(dev, &target, "ext4", nosuid, "")?;
176 +
                let atime = if m.prune.is_some() { libc::MS_STRICTATIME | libc::MS_LAZYTIME } else { 0 };
177 +
                mount(dev, &target, "ext4", nosuid | atime, "")?;
160 178
                chown(&target, uid, gid)?;
161 179
            }
162 180
            (kind, _) => return Err(format!("unknown mount kind {kind}")),
…
22 unchanged lines
185 203
                .map_err(|e| format!("chmod {}: {e}", s.path))?;
186 204
        }
187 205
    }
188 -
    if job.loopback {
206 +
    if job.loopback || job.egress {
189 207
        loopback_up()?;
190 208
    }
209 +
    if job.egress {
210 +
        egress::start()?;
211 +
    }
191 212
    let root = cstr("/newroot")?;
192 213
    if unsafe { libc::chroot(root.as_ptr()) } != 0 || std::env::set_current_dir("/src").is_err() {
193 214
        return Err(format!("enter the rootfs: {}", std::io::Error::last_os_error()));
194 215
    }
195 216
217 +
    // What the job reads or writes on a cache disk from now on is what
218 +
    // this run used.
219 +
    let started = job.mounts.iter().any(|m| m.prune.is_some()).then(prune::start_clock);
196 220
    let argv: Vec<CString> = job.argv.iter().map(|a| cstr(a)).collect::<Result<_, _>>()?;
197 221
    // glibc's execvpe searches the caller's PATH, and PID 1 has none of its
198 222
    // own, so the program is found here, in the job's PATH, before forking.
…
53 unchanged lines
252 276
    for r in readers {
253 277
        let _ = r.join();
254 278
    }
255 -
    Ok(if libc::WIFEXITED(status) { libc::WEXITSTATUS(status) } else { 128 + libc::WTERMSIG(status) })
279 +
    let code = if libc::WIFEXITED(status) { libc::WEXITSTATUS(status) } else { 128 + libc::WTERMSIG(status) };
280 +
    // A failing job's disks are thrown away: only a passing one's are
281 +
    // worth pruning.
282 +
    if let (Some(started), 0) = (started, code) {
283 +
        prune_caches(console, &job.mounts, started);
284 +
    }
285 +
    Ok(code)
286 +
}
287 +
288 +
/// Prune each cache disk and say so to the host. Whatever the job left
289 +
/// running stops first, so nothing changes the disks meanwhile.
290 +
fn prune_caches(console: &Mutex<std::io::Stdout>, mounts: &[MountSpec], started: std::time::SystemTime) {
291 +
    unsafe {
292 +
        libc::kill(-1, libc::SIGKILL);
293 +
        let mut status = 0;
294 +
        while libc::waitpid(-1, &mut status, 0) > 0 {}
295 +
    }
296 +
    for m in mounts {
297 +
        let Some(percent) = m.prune else { continue };
298 +
        let root = std::path::Path::new(&m.target);
299 +
        let pruned = match prune::usage(root) {
300 +
            Ok((used, size)) => {
301 +
                let mut p = prune::prune(root, started, &m.keep, size, used, percent).unwrap_or_else(|e| prune::Pruned {
302 +
                    size,
303 +
                    percent,
304 +
                    before: used,
305 +
                    after: used,
306 +
                    unpruned: Some(format!("pruning failed: {e}")),
307 +
                    ..Default::default()
308 +
                });
309 +
                // What is left as the file system counts it.
310 +
                if p.freed > 0 {
311 +
                    unsafe { libc::sync() };
312 +
                    if let Ok((now, _)) = prune::usage(root) {
313 +
                        p.after = now;
314 +
                    }
315 +
                }
316 +
                p
317 +
            }
318 +
            Err(e) => prune::Pruned { percent, unpruned: Some(format!("statvfs {}: {e}", m.target)), ..Default::default() },
319 +
        };
320 +
        let msg = serde_json::json!({ "target": m.target, "pruned": pruned });
321 +
        frame(console, b'C', msg.to_string().as_bytes());
322 +
    }
256 323
}
257 324
258 325
/// Frame a stream line by line; a long line is cut at 16 KiB.
…
23 unchanged lines
added crates/flow-init/src/prune.rs +405 −0
1 +
//! Pruning a build cache before it is kept (oxygenflow#72, spec §5.4).
2 +
//!
3 +
//! A kept disk starts the next run, so without pruning it carries every
4 +
//! build's files: each crate's artifacts under every hash a lockfile or a
5 +
//! feature set ever gave it. Before a passing job's cache disk goes back
6 +
//! into the pool, the files this run did not use are removed, oldest
7 +
//! first, until what is on the disk is at most a share of its size (the
8 +
//! cap). The microVM's guest init prunes its disks before it powers off,
9 +
//! since only it can see inside them; the runner prunes the syscall
10 +
//! tier's directories itself. Both use this module.
11 +
//!
12 +
//! What the run used is told by the files' times against when the job
13 +
//! started: written (mtime), changed (ctime) or read (atime). Reads are
14 +
//! seen only where the file system records them: the guest mounts a cache
15 +
//! disk `strictatime`; on the runner, where a directory sits on whatever
16 +
//! file system the runner has, each file's atime is set to the epoch
17 +
//! before the job ([`forget_reads`]), so even `relatime` records its
18 +
//! first read. A file system mounted `noatime` records none, and then
19 +
//! nothing is pruned.
20 +
//!
21 +
//! A build tool does not read everything it relies on. Cargo, finding a
22 +
//! crate fresh, reads its fingerprint and only looks at its rlib and
23 +
//! binaries, so their times say nothing. Every name cargo gives a unit's
24 +
//! files carries the unit's hash (`.fingerprint/foo-<hash>/`,
25 +
//! `deps/libfoo-<hash>.rlib`, `build/foo-<hash>/`), so a file stays when
26 +
//! a file this run used carries one of the hashes in its path: a fresh
27 +
//! crate's fingerprint keeps its artifacts. A file with another link
28 +
//! that stays, or under a path the job's outputs name, stays too. What
29 +
//! goes is what this run's build did not touch, which a later build needs
30 +
//! only if an older lockfile comes back, and then cargo rebuilds it.
31 +
32 +
use serde::{Deserialize, Serialize};
33 +
#[cfg(unix)]
34 +
use std::collections::{HashMap, HashSet};
35 +
#[cfg(unix)]
36 +
use std::os::unix::fs::MetadataExt;
37 +
use std::path::Path;
38 +
#[cfg(unix)]
39 +
use std::path::PathBuf;
40 +
use std::time::{Duration, SystemTime};
41 +
42 +
/// The share of a cache disk a kept cache may fill when no runner says
43 +
/// otherwise: room for this run's build and as much again.
44 +
pub const DEFAULT_PERCENT: u8 = 60;
45 +
46 +
/// What pruning a cache found and did, in bytes of blocks on its disk.
47 +
#[derive(Serialize, Deserialize, Debug, Clone, Default, PartialEq, Eq)]
48 +
pub struct Pruned {
49 +
    /// The disk's size, and the share of it the cache may fill.
50 +
    pub size: u64,
51 +
    pub percent: u8,
52 +
    /// What was on the disk before pruning, and after.
53 +
    pub before: u64,
54 +
    pub after: u64,
55 +
    /// What went: bytes and files.
56 +
    pub freed: u64,
57 +
    pub files: u64,
58 +
    /// Why nothing could be pruned, when so (no access times, or an
59 +
    /// error): only the size then decides.
60 +
    #[serde(default, skip_serializing_if = "Option::is_none")]
61 +
    pub unpruned: Option<String>,
62 +
}
63 +
64 +
impl Pruned {
65 +
    /// The most the cache may keep: `percent` of `size`.
66 +
    pub fn cap(&self) -> u64 {
67 +
        cap(self.size, self.percent)
68 +
    }
69 +
70 +
    /// Whether what is left may be kept.
71 +
    pub fn fits(&self) -> bool {
72 +
        self.after <= self.cap()
73 +
    }
74 +
}
75 +
76 +
fn cap(size: u64, percent: u8) -> u64 {
77 +
    (size as u128 * percent.min(100) as u128 / 100) as u64
78 +
}
79 +
80 +
/// Now, as a job's start to compare file times against. File systems
81 +
/// stamp times from a clock that may lag this one by a tick, so the
82 +
/// moments either side of it are left to pass: nothing done before the
83 +
/// call (such as [`forget_reads`]) can look as though it came after.
84 +
pub fn start_clock() -> SystemTime {
85 +
    std::thread::sleep(Duration::from_millis(20));
86 +
    let t = SystemTime::now();
87 +
    std::thread::sleep(Duration::from_millis(20));
88 +
    t
89 +
}
90 +
91 +
#[cfg(unix)]
92 +
/// Whether the file system holding `path` records reads (not `noatime`).
93 +
pub fn records_reads(path: &Path) -> bool {
94 +
    let Ok(c) = std::ffi::CString::new(path.as_os_str().as_encoded_bytes()) else { return false };
95 +
    let mut st: libc::statvfs = unsafe { std::mem::zeroed() };
96 +
    if unsafe { libc::statvfs(c.as_ptr(), &mut st) } != 0 {
97 +
        return false;
98 +
    }
99 +
    // Only Linux's statvfs has a `noatime` flag: elsewhere (Haiku, FreeBSD)
100 +
    // reads are taken as recorded, and the access-time check decides.
101 +
    #[cfg(any(target_os = "linux", target_os = "android"))]
102 +
    return st.f_flag & libc::ST_NOATIME == 0;
103 +
    #[cfg(not(any(target_os = "linux", target_os = "android")))]
104 +
    true
105 +
}
106 +
107 +
#[cfg(unix)]
108 +
/// The blocks in use on the file system mounted at `path`, and its size
109 +
/// (as `df` counts them).
110 +
pub fn usage(path: &Path) -> std::io::Result<(u64, u64)> {
111 +
    let c = std::ffi::CString::new(path.as_os_str().as_encoded_bytes())?;
112 +
    let mut st: libc::statvfs = unsafe { std::mem::zeroed() };
113 +
    if unsafe { libc::statvfs(c.as_ptr(), &mut st) } != 0 {
114 +
        return Err(std::io::Error::last_os_error());
115 +
    }
116 +
    let unit = st.f_frsize as u64;
117 +
    Ok(((st.f_blocks as u64).saturating_sub(st.f_bfree as u64) * unit, st.f_blocks as u64 * unit))
118 +
}
119 +
120 +
#[cfg(unix)]
121 +
/// Set every regular file's atime under `root` to the epoch, its mtime
122 +
/// kept, so the job's first read of each is recorded even on `relatime`
123 +
/// (which records a read only of a file read no later than it changed).
124 +
/// Links are not followed. The count of files.
125 +
pub fn forget_reads(root: &Path) -> std::io::Result<u64> {
126 +
    let mut n = 0;
127 +
    let mut err = None;
128 +
    walk(root, &mut |p, md| {
129 +
        if !md.file_type().is_file() {
130 +
            return;
131 +
        }
132 +
        let Ok(c) = std::ffi::CString::new(p.as_os_str().as_encoded_bytes()) else { return };
133 +
        let times = [libc::timespec { tv_sec: 0, tv_nsec: 0 }, libc::timespec { tv_sec: 0, tv_nsec: libc::UTIME_OMIT }];
134 +
        if unsafe { libc::utimensat(libc::AT_FDCWD, c.as_ptr(), times.as_ptr(), libc::AT_SYMLINK_NOFOLLOW) } == 0 {
135 +
            n += 1;
136 +
        } else if err.is_none() {
137 +
            err = Some(std::io::Error::last_os_error());
138 +
        }
139 +
    })?;
140 +
    match err {
141 +
        Some(e) => Err(e),
142 +
        None => Ok(n),
143 +
    }
144 +
}
145 +
146 +
#[cfg(unix)]
147 +
/// Each entry under `root` (not `root` itself) with its metadata, parents
148 +
/// before children; links are not followed, nor other file systems
149 +
/// entered.
150 +
fn walk(root: &Path, f: &mut dyn FnMut(&Path, &std::fs::Metadata)) -> std::io::Result<()> {
151 +
    let dev = std::fs::symlink_metadata(root)?.dev();
152 +
    let mut dirs = vec![root.to_path_buf()];
153 +
    while let Some(d) = dirs.pop() {
154 +
        let Ok(entries) = std::fs::read_dir(&d) else { continue };
155 +
        for e in entries.flatten() {
156 +
            let p = e.path();
157 +
            let Ok(md) = std::fs::symlink_metadata(&p) else { continue };
158 +
            if md.dev() != dev {
159 +
                continue;
160 +
            }
161 +
            f(&p, &md);
162 +
            if md.is_dir() {
163 +
                dirs.push(p);
164 +
            }
165 +
        }
166 +
    }
167 +
    Ok(())
168 +
}
169 +
170 +
#[cfg(unix)]
171 +
/// The unit hashes in `rel`: a 16-digit lowercase hex word after a `-`
172 +
/// in any of its names, as cargo names a unit's files.
173 +
fn hashes(rel: &Path) -> impl Iterator<Item = String> + '_ {
174 +
    rel.components().filter_map(|c| c.as_os_str().to_str()).flat_map(|name| {
175 +
        name.split('-').skip(1).filter_map(|w| {
176 +
            let w = w.split('.').next().unwrap_or("");
177 +
            (w.len() == 16 && w.bytes().all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b))).then(|| w.to_string())
178 +
        })
179 +
    })
180 +
}
181 +
182 +
#[cfg(unix)]
183 +
fn nanos(s: i64, ns: i64) -> i128 {
184 +
    s as i128 * 1_000_000_000 + ns as i128
185 +
}
186 +
187 +
#[cfg(unix)]
188 +
/// One file, however many names it has.
189 +
struct File {
190 +
    paths: Vec<PathBuf>,
191 +
    links: u64,
192 +
    bytes: u64,
193 +
    /// When it was last used as far as can be told: the later of its
194 +
    /// atime and mtime.
195 +
    last: i128,
196 +
    used: bool,
197 +
}
198 +
199 +
#[cfg(unix)]
200 +
/// Prune the cache at `root`, a disk of `size` bytes with `before` in
201 +
/// use, down to `percent` of its size, removing only regular files this
202 +
/// run did not use (see the module's notes), those used longest ago
203 +
/// first. `started` is when the job started ([`start_clock`]); `keep` are
204 +
/// paths relative to `root` (the job's outputs) that stay whatever their
205 +
/// times. Under the cap nothing is touched. `after` is `before` less what
206 +
/// was freed.
207 +
pub fn prune(root: &Path, started: SystemTime, keep: &[String], size: u64, before: u64, percent: u8) -> std::io::Result<Pruned> {
208 +
    let mut out = Pruned { size, percent, before, after: before, ..Pruned::default() };
209 +
    if before <= out.cap() {
210 +
        return Ok(out);
211 +
    }
212 +
    let since = started.duration_since(SystemTime::UNIX_EPOCH).map(|d| d.as_nanos() as i128).unwrap_or(0);
213 +
    let keep: Vec<PathBuf> = keep.iter().map(|k| PathBuf::from(k.trim_start_matches('/'))).filter(|k| !k.as_os_str().is_empty()).collect();
214 +
    let mut files: HashMap<(u64, u64), File> = HashMap::new();
215 +
    walk(root, &mut |p, md| {
216 +
        if !md.file_type().is_file() {
217 +
            return;
218 +
        }
219 +
        let rel = p.strip_prefix(root).unwrap_or(p);
220 +
        let used = [nanos(md.atime(), md.atime_nsec()), nanos(md.mtime(), md.mtime_nsec()), nanos(md.ctime(), md.ctime_nsec())]
221 +
            .into_iter()
222 +
            .any(|t| t >= since)
223 +
            || keep.iter().any(|k| rel.starts_with(k));
224 +
        let f = files.entry((md.dev(), md.ino())).or_insert_with(|| File {
225 +
            paths: Vec::new(),
226 +
            links: md.nlink(),
227 +
            bytes: md.blocks() * 512,
228 +
            last: nanos(md.atime(), md.atime_nsec()).max(nanos(md.mtime(), md.mtime_nsec())),
229 +
            used: false,
230 +
        });
231 +
        f.paths.push(p.to_path_buf());
232 +
        f.used |= used;
233 +
    })?;
234 +
    // A hash any used file carries keeps every file carrying it.
235 +
    let live: HashSet<String> =
236 +
        files.values().filter(|f| f.used).flat_map(|f| f.paths.iter().flat_map(|p| hashes(p.strip_prefix(root).unwrap_or(p)))).collect();
237 +
    let mut gone: Vec<File> = files
238 +
        .into_values()
239 +
        .filter(|f| {
240 +
            !f.used
241 +
                // A link from outside the cache: removing it frees nothing.
242 +
                && f.links == f.paths.len() as u64
243 +
                && !f.paths.iter().any(|p| hashes(p.strip_prefix(root).unwrap_or(p)).any(|h| live.contains(&h)))
244 +
        })
245 +
        .collect();
246 +
    gone.sort_by_key(|f| f.last);
247 +
    let mut emptied: HashSet<PathBuf> = HashSet::new();
248 +
    for f in gone {
249 +
        if out.after <= out.cap() {
250 +
            break;
251 +
        }
252 +
        let mut removed = true;
253 +
        for p in &f.paths {
254 +
            if std::fs::remove_file(p).is_err() {
255 +
                removed = false;
256 +
            } else if let Some(parent) = p.parent() {
257 +
                emptied.insert(parent.to_path_buf());
258 +
            }
259 +
        }
260 +
        if removed {
261 +
            out.after = out.after.saturating_sub(f.bytes);
262 +
            out.freed += f.bytes;
263 +
            out.files += 1;
264 +
        }
265 +
    }
266 +
    // Directories pruning emptied go too, deepest first; never `root`.
267 +
    let mut dirs: Vec<PathBuf> = emptied.into_iter().collect();
268 +
    dirs.sort_by_key(|d| std::cmp::Reverse(d.components().count()));
269 +
    for d in dirs {
270 +
        let mut d = d.as_path();
271 +
        while d != root && d.starts_with(root) && std::fs::remove_dir(d).is_ok() {
272 +
            match d.parent() {
273 +
                Some(p) => d = p,
274 +
                None => break,
275 +
            }
276 +
        }
277 +
    }
278 +
    Ok(out)
279 +
}
280 +
281 +
// Elsewhere (a Windows runner, which has no microVM tier and so no cache
282 +
// disks to prune) nothing records reads, so nothing is pruned.
283 +
284 +
#[cfg(not(unix))]
285 +
pub fn records_reads(_path: &Path) -> bool {
286 +
    false
287 +
}
288 +
289 +
#[cfg(not(unix))]
290 +
pub fn usage(_path: &Path) -> std::io::Result<(u64, u64)> {
291 +
    Err(std::io::Error::new(std::io::ErrorKind::Unsupported, "cache disks are pruned only on Linux"))
292 +
}
293 +
294 +
#[cfg(not(unix))]
295 +
pub fn forget_reads(_root: &Path) -> std::io::Result<u64> {
296 +
    Ok(0)
297 +
}
298 +
299 +
#[cfg(not(unix))]
300 +
pub fn prune(_root: &Path, _started: SystemTime, _keep: &[String], _size: u64, _before: u64, _percent: u8) -> std::io::Result<Pruned> {
301 +
    Err(std::io::Error::new(std::io::ErrorKind::Unsupported, "cache disks are pruned only on Linux"))
302 +
}
303 +
304 +
#[cfg(all(test, unix))]
305 +
mod tests {
306 +
    use super::*;
307 +
308 +
    /// Times around a job that started at `T`, ahead of the clock: a file
309 +
    /// whose times are set is changed (ctime) now, before it.
310 +
    const T: u64 = 4_000_000_000;
311 +
312 +
    fn at(secs: u64) -> SystemTime {
313 +
        SystemTime::UNIX_EPOCH + Duration::from_secs(secs)
314 +
    }
315 +
316 +
    /// A file of `kib` KiB at `rel`, last read and written at `secs`.
317 +
    fn file(root: &Path, rel: &str, kib: usize, secs: u64) -> PathBuf {
318 +
        let p = root.join(rel);
319 +
        std::fs::create_dir_all(p.parent().unwrap()).unwrap();
320 +
        let bytes: Vec<u8> = (0..kib << 10).map(|i: usize| (i.wrapping_mul(2_654_435_761) >> 13) as u8).collect();
321 +
        std::fs::write(&p, bytes).unwrap();
322 +
        let f = std::fs::File::options().write(true).open(&p).unwrap();
323 +
        f.set_times(std::fs::FileTimes::new().set_accessed(at(secs)).set_modified(at(secs))).unwrap();
324 +
        p
325 +
    }
326 +
327 +
    fn bytes(p: &Path) -> u64 {
328 +
        std::fs::metadata(p).unwrap().blocks() * 512
329 +
    }
330 +
331 +
    #[test]
332 +
    fn hashes_are_cargos_unit_hashes() {
333 +
        let h = |p: &str| hashes(Path::new(p)).collect::<Vec<_>>();
334 +
        assert_eq!(h("debug/deps/libdep-8f2b19826c505967.rlib"), ["8f2b19826c505967"]);
335 +
        assert_eq!(h("debug/.fingerprint/dep-8f2b19826c505967/lib-dep"), ["8f2b19826c505967"]);
336 +
        assert_eq!(h("debug/build/my-crate-0123456789abcdef/out/x.o"), ["0123456789abcdef"]);
337 +
        assert!(h("debug/incremental/app-2dadnjnzlrswc/s-hndl5uwdpw-1uaho4m.lock").is_empty());
338 +
        assert!(h("debug/app").is_empty() && h("debug/deps/x-0123456789ABCDEF").is_empty() && h("x-0123456789abcde").is_empty());
339 +
    }
340 +
341 +
    #[test]
342 +
    fn under_the_cap_nothing_is_touched() {
343 +
        let dir = tempfile::tempdir().unwrap();
344 +
        let old = file(dir.path(), "debug/deps/libold-1111111111111111.rlib", 64, 1_000);
345 +
        let p = prune(dir.path(), at(T), &[], 1 << 30, 64 << 10, 60).unwrap();
346 +
        assert!(old.exists() && p.fits() && p.freed == 0 && p.after == p.before, "{p:?}");
347 +
    }
348 +
349 +
    /// A cargo target as a run leaves it: this lockfile's units used this
350 +
    /// run (one rebuilt, one fresh: only its fingerprint read), an older
351 +
    /// lockfile's not, and an output the run did not touch.
352 +
    #[test]
353 +
    fn what_this_run_did_not_use_goes_oldest_first_and_its_units_stay_whole() {
354 +
        let dir = tempfile::tempdir().unwrap();
355 +
        let root = dir.path();
356 +
        let app = file(root, "debug/deps/app-aaaaaaaaaaaaaaaa", 256, T + 100);
357 +
        let app_fp = file(root, "debug/.fingerprint/app-aaaaaaaaaaaaaaaa/bin-app", 1, T + 100);
358 +
        let dep_rlib = file(root, "debug/deps/libdep-bbbbbbbbbbbbbbbb.rlib", 256, T - 5_000);
359 +
        let dep_fp = file(root, "debug/.fingerprint/dep-bbbbbbbbbbbbbbbb/lib-dep", 1, T - 5_000);
360 +
        std::fs::File::open(&dep_fp).unwrap().set_times(std::fs::FileTimes::new().set_accessed(at(T + 50))).unwrap();
361 +
        let script = file(root, "debug/build/dep-bbbbbbbbbbbbbbbb/out/gen.rs", 64, T - 5_000);
362 +
        // Uplifted: another name of the rebuilt binary.
363 +
        std::fs::hard_link(&app, root.join("debug/app")).unwrap();
364 +
        // An older lockfile's units, untouched, the oldest first to go.
365 +
        let oldest = file(root, "debug/deps/libdep-cccccccccccccccc.rlib", 256, T - 9_000);
366 +
        let oldest_fp = file(root, "debug/.fingerprint/dep-cccccccccccccccc/lib-dep", 1, T - 8_999);
367 +
        let older = file(root, "debug/deps/libdep-dddddddddddddddd.rlib", 256, T - 8_000);
368 +
        // A declared output, older still, that the run did not touch.
369 +
        let output = file(root, "release/tool", 128, T - 10_000);
370 +
        let all = [&app, &app_fp, &dep_rlib, &dep_fp, &script, &oldest, &oldest_fp, &older, &output];
371 +
        let before: u64 = all.iter().map(|p| bytes(p)).sum();
372 +
        // Room for all but the oldest unit.
373 +
        let want = before - bytes(&oldest) - bytes(&oldest_fp);
374 +
        let size = (want * 100).div_ceil(60);
375 +
        let p = prune(root, at(T), &["/release/tool".into()], size, before, 60).unwrap();
376 +
        assert!(p.fits(), "{p:?}");
377 +
        assert!(
378 +
            !oldest.exists() && !root.join("debug/.fingerprint/dep-cccccccccccccccc").exists(),
379 +
            "the oldest unit went, its emptied directory too"
380 +
        );
381 +
        assert!(older.exists(), "and pruning stopped under the cap: {p:?}");
382 +
        for kept in [&app, &app_fp, &dep_rlib, &dep_fp, &script, &output, &root.join("debug/app")] {
383 +
            assert!(kept.exists(), "{} stays", kept.display());
384 +
        }
385 +
        assert_eq!((p.files, p.freed, p.after), (2, p.before - want, want));
386 +
        // With no room at all: still only what the run did not use goes.
387 +
        let p = prune(root, at(T), &["release/tool".into()], 1, p.after, 60).unwrap();
388 +
        assert!(!older.exists() && !p.fits() && p.files == 1, "{p:?}");
389 +
        for kept in [&app, &dep_rlib, &dep_fp, &script, &output] {
390 +
            assert!(kept.exists(), "{} stays", kept.display());
391 +
        }
392 +
        assert!(root.join("debug/deps").is_dir(), "a directory still holding files stays");
393 +
    }
394 +
395 +
    #[test]
396 +
    fn reads_are_forgotten_so_the_next_is_seen() {
397 +
        let dir = tempfile::tempdir().unwrap();
398 +
        let p = file(dir.path(), "a/b", 1, 5_000);
399 +
        std::os::unix::fs::symlink("/etc/passwd", dir.path().join("link")).unwrap();
400 +
        assert_eq!(forget_reads(dir.path()).unwrap(), 1);
401 +
        let md = std::fs::metadata(&p).unwrap();
402 +
        assert_eq!((md.atime(), md.mtime()), (0, 5_000), "the mtime is kept");
403 +
        assert!(std::fs::symlink_metadata(dir.path().join("link")).unwrap().atime() > 0, "a link is not followed");
404 +
    }
405 +
}
modified crates/flow/Cargo.toml +24 −0
…
4 unchanged lines
5 5
license.workspace = true
6 6
description = "OxygenFlow: check pipelines, run them locally, and run jobs for OxygenHub"
7 7
8 +
[lib]
9 +
name = "flow"
10 +
path = "src/lib.rs"
11 +
8 12
[[bin]]
9 13
name = "flow"
10 14
path = "src/main.rs"
…
1 unchanged lines
12 16
[dependencies]
13 17
flow-core.workspace = true
14 18
jet.workspace = true
19 +
flow-init.workspace = true
15 20
oxy-core.workspace = true
16 21
serde.workspace = true
17 22
serde_json.workspace = true
…
5 unchanged lines
23 28
ureq = { version = "2", features = ["json"] }
24 29
ignore = "0.4"
25 30
libc = "0.2"
31 +
rustls = { version = "0.23", default-features = false, features = ["ring", "std", "tls12"] }
32 +
rcgen = "0.14"
33 +
# The WASM tier (oxygenflow#75): Cranelift-compiled WASI preview 1 modules.
34 +
wasmtime = { version = "=49.0.4", optional = true, default-features = false, features = ["cranelift", "runtime", "std", "parallel-compilation"] }
35 +
wasmtime-wasi = { version = "=49.0.4", optional = true, default-features = false, features = ["p1"] }
36 +
37 +
# A Windows runner's free disk, version, and renaming a running flow.exe
38 +
# (oxygenflow#83);
39 +
# already in the tree through wasmtime.
40 +
[target.'cfg(windows)'.dependencies]
41 +
windows-sys = { version = "0.61", features = ["Win32_Foundation", "Win32_Storage_FileSystem", "Win32_System_SystemInformation", "Wdk_System_SystemServices"] }
42 +
43 +
[features]
44 +
default = ["wasm"]
45 +
# The WASM tier: off for systems wasmtime does not build or run on (Haiku,
46 +
# oxygenflow#91).
47 +
wasm = ["dep:wasmtime", "dep:wasmtime-wasi"]
26 48
27 49
[dev-dependencies]
28 50
tempfile.workspace = true
51 +
# Test modules written as text.
52 +
wat = "1"
modified crates/flow/build.rs +5 −1
…
2 unchanged lines
3 3
//! building, else "local build".
4 4
fn main() {
5 5
    println!("cargo:rerun-if-env-changed=OXY_BUILD");
6 -
    println!("cargo:rerun-if-env-changed=FLOW_HEADS");
6 +
    // The heads name the build only when OXY_BUILD does not: a CI job that
7 +
    // sets it stays fresh from one state to the next (oxygenflow#52).
8 +
    if !std::env::var("OXY_BUILD").is_ok_and(|b| !b.trim().is_empty()) {
9 +
        println!("cargo:rerun-if-env-changed=FLOW_HEADS");
10 +
    }
7 11
    let build = std::env::var("OXY_BUILD")
8 12
        .ok()
9 13
        .map(|b| b.trim().to_string())
…
11 unchanged lines
added crates/flow/src/cache.rs +965 −0
1 +
//! The build cache (oxygenflow#28, spec §5.4): a disk mount that names a
2 +
//! `cache` is started from a disk kept by an earlier passing job, rather
3 +
//! than empty, so a build carries its compiled dependencies forward.
4 +
//!
5 +
//! Each runner keeps a pool under `<home>/cache`, by repository, job,
6 +
//! cache name and bookmark, each disk named by its key: the BLAKE3 of the
7 +
//! image, the job, the cache name and the content of its key files. A job
8 +
//! takes, in order:
9 +
//!
10 +
//! 1. its bookmark's disk with the same key (moved: no copying);
11 +
//! 2. its bookmark's newest disk (moved; cargo rebuilds what changed);
12 +
//! 3. another bookmark's disk with the same key, then its newest (copied,
13 +
//!    so a proposal never takes main's cache away);
14 +
//! 4. else an empty disk, as without a cache.
15 +
//!
16 +
//! A passing job's disk goes back into its bookmark's place; a failing
17 +
//! job's is thrown away, so nothing it wrote reaches a later job. The
18 +
//! receipt names what the job started from. The pool is kept under its
19 +
//! size limit by dropping the disks used longest ago.
20 +
//!
21 +
//! A disk is in the [`Form`] its tier mounts (oxygenflow#70): a microVM's
22 +
//! is an ext4 image, `<key>.ext4`; the syscall tier's is the host
23 +
//! directory it binds at the mount's target, `<key>.tree`, taken and kept
24 +
//! the same way. A directory goes into the pool with nothing in it
25 +
//! writable by others or set-id, and the pool itself is the runner's
26 +
//! alone.
27 +
//!
28 +
//! Beside each disk, `<key>.sources` (`<key>.tree-sources` for a
29 +
//! directory) lists the sources it was built from (oxygenflow#52): each
30 +
//! file's content hash and the mtime it had. A job that takes the disk
31 +
//! gives every file whose content is unchanged that mtime back, so a build
32 +
//! tool that judges freshness by time (cargo) rebuilds only what changed;
33 +
//! a changed file keeps the time of its checkout, newer than anything on
34 +
//! the disk, so it is always rebuilt.
35 +
//!
36 +
//! A passing job's disk is pruned before it is kept (oxygenflow#72; see
37 +
//! [`flow_init::prune`]): over [`flow_init::prune::DEFAULT_PERCENT`] of
38 +
//! its size (the runner's `cache_prune_percent`), the files the job did
39 +
//! not use go, oldest first, and a cache still over it is not kept, so a
40 +
//! kept target never grows until the job's disk is full. A microVM's
41 +
//! guest prunes its image before it stops (the runner cannot look inside
42 +
//! an ext4 image without mounting it); the runner prunes a directory
43 +
//! itself, having set its files' atimes to the epoch before the job
44 +
//! ([`forget_reads`]) so that every read is seen. The job's log says what
45 +
//! was kept and pruned.
46 +
47 +
use anyhow::{Context, Result};
48 +
use flow_core::pipeline::Mount;
49 +
use flow_init::prune::Pruned;
50 +
use std::path::{Path, PathBuf};
51 +
use std::time::SystemTime;
52 +
53 +
/// The pool's default size limit.
54 +
pub const DEFAULT_GB: u64 = 64;
55 +
56 +
/// The sources a cached disk was built from: path to content hash and
57 +
/// mtime, in whole seconds (what the workspace image keeps).
58 +
pub type Sources = std::collections::BTreeMap<String, (String, i64)>;
59 +
60 +
/// What a job started a cached disk from.
61 +
pub struct Taken {
62 +
    /// For the log and the receipt.
63 +
    pub how: String,
64 +
    /// The sources it was built from, when they were kept with it.
65 +
    pub sources: Option<Sources>,
66 +
}
67 +
68 +
/// How a tier holds a disk mount's files (oxygenflow#70).
69 +
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
70 +
pub enum Form {
71 +
    /// An ext4 image of the mount's size, attached to a microVM.
72 +
    Image,
73 +
    /// A host directory, bound into the syscall sandbox.
74 +
    Dir,
75 +
}
76 +
77 +
impl Form {
78 +
    fn ext(self) -> &'static str {
79 +
        match self {
80 +
            Form::Image => "ext4",
81 +
            Form::Dir => "tree",
82 +
        }
83 +
    }
84 +
}
85 +
86 +
/// Which form a kept entry of the pool is in, by its name.
87 +
fn form_of(p: &Path) -> Option<Form> {
88 +
    match p.extension()?.to_str()? {
89 +
        "ext4" => Some(Form::Image),
90 +
        "tree" => Some(Form::Dir),
91 +
        _ => None,
92 +
    }
93 +
}
94 +
95 +
fn sources_of(disk: &Path) -> PathBuf {
96 +
    match form_of(disk) {
97 +
        Some(Form::Dir) => disk.with_extension("tree-sources"),
98 +
        _ => disk.with_extension("sources"),
99 +
    }
100 +
}
101 +
102 +
fn read_sources(disk: &Path) -> Option<Sources> {
103 +
    serde_json::from_slice(&std::fs::read(sources_of(disk)).ok()?).ok()
104 +
}
105 +
106 +
/// How a passing job's cache disk is pruned before it is kept
107 +
/// (oxygenflow#72).
108 +
pub struct Prune<'a> {
109 +
    /// When the job started ([`flow_init::prune::start_clock`]): what it
110 +
    /// read or wrote since is what it used.
111 +
    pub started: SystemTime,
112 +
    /// The share of the disk the kept cache may fill.
113 +
    pub percent: u8,
114 +
    /// What the guest did to a microVM's image, when it said.
115 +
    pub guest: Option<&'a Pruned>,
116 +
}
117 +
118 +
/// Where a job's cached disks come from and go back to.
119 +
pub struct Pool {
120 +
    root: PathBuf,
121 +
    limit: u64,
122 +
}
123 +
124 +
/// Which job: the pool's coordinates other than the key, and the form its
125 +
/// tier keeps a disk in.
126 +
pub struct Ctx<'a> {
127 +
    pub repo: &'a str,
128 +
    pub bookmark: &'a str,
129 +
    pub job: &'a str,
130 +
    pub image: &'a str,
131 +
    pub form: Form,
132 +
}
133 +
134 +
use flow_core::api::segment;
135 +
136 +
/// Held while the pool changes: jobs run side by side, and a directory,
137 +
/// unlike an image, can be half moved or half removed while another job
138 +
/// copies it.
139 +
static CHANGING: std::sync::Mutex<()> = std::sync::Mutex::new(());
140 +
141 +
fn changing() -> std::sync::MutexGuard<'static, ()> {
142 +
    CHANGING.lock().unwrap_or_else(|p| p.into_inner())
143 +
}
144 +
145 +
impl Pool {
146 +
    pub fn new(home_root: &Path, gb: Option<u64>) -> Pool {
147 +
        Pool { root: home_root.join("cache"), limit: gb.unwrap_or(DEFAULT_GB).saturating_mul(1 << 30) }
148 +
    }
149 +
150 +
    fn cache_dir(&self, ctx: &Ctx, name: &str) -> PathBuf {
151 +
        self.root.join(segment(ctx.repo)).join(segment(ctx.job)).join(segment(name))
152 +
    }
153 +
154 +
    /// A mount's key: the image, job, cache name and its key files'
155 +
    /// content hashes (`hash_of` gives a file's, `None` when absent).
156 +
    pub fn key(ctx: &Ctx, m: &Mount, hash_of: impl Fn(&str) -> Option<String>) -> String {
157 +
        let mut h = blake3::Hasher::new();
158 +
        for part in [ctx.image, ctx.job, m.cache.as_deref().unwrap_or("")] {
159 +
            h.update(part.as_bytes());
160 +
            h.update(b"\n");
161 +
        }
162 +
        for k in &m.key {
163 +
            h.update(k.as_bytes());
164 +
            h.update(b"=");
165 +
            h.update(hash_of(k).unwrap_or_else(|| "absent".into()).as_bytes());
166 +
            h.update(b"\n");
167 +
        }
168 +
        h.finalize().to_hex()[..32].to_string()
169 +
    }
170 +
171 +
    /// Start `dest` from the pool for mount `m`: an image of `m.size_mib`,
172 +
    /// or a directory, as `ctx.form` says; `dest` must not exist yet. What
173 +
    /// it started from, or `None` (the caller makes an empty disk).
174 +
    ///
175 +
    /// A disk this job's bookmark kept is moved out of the pool, so a job
176 +
    /// of the same slot running beside this one can never take it too.
177 +
    pub fn take(&self, ctx: &Ctx, m: &Mount, key: &str, dest: &Path) -> Option<Taken> {
178 +
        let name = m.cache.as_deref()?;
179 +
        let _changing = changing();
180 +
        let dir = self.cache_dir(ctx, name);
181 +
        let size = m.size_mib as u64 * (1 << 20);
182 +
        let form = ctx.form;
183 +
        let fits = |p: &Path| match form {
184 +
            Form::Image => std::fs::metadata(p).is_ok_and(|md| md.len() == size),
185 +
            Form::Dir => std::fs::symlink_metadata(p).is_ok_and(|md| md.is_dir()),
186 +
        };
187 +
        let ext = form.ext();
188 +
        let mine = dir.join(segment(ctx.bookmark));
189 +
        let exact = mine.join(format!("{key}.{ext}"));
190 +
        // A moved disk takes its sources with it.
191 +
        if fits(&exact) {
192 +
            let sources = read_sources(&exact);
193 +
            if std::fs::rename(&exact, dest).is_ok() {
194 +
                let _ = std::fs::remove_file(sources_of(&exact));
195 +
                return Some(Taken { how: format!("cache {name}: this bookmark's, same key"), sources });
196 +
            }
197 +
        }
198 +
        if let Some(p) = newest(&mine, form, |p| fits(p)) {
199 +
            let sources = read_sources(&p);
200 +
            if std::fs::rename(&p, dest).is_ok() {
201 +
                let _ = std::fs::remove_file(sources_of(&p));
202 +
                return Some(Taken { how: format!("cache {name}: this bookmark's, an older key"), sources });
203 +
            }
204 +
        }
205 +
        let others: Vec<PathBuf> =
206 +
            std::fs::read_dir(&dir).into_iter().flatten().flatten().map(|e| e.path()).filter(|p| p.is_dir() && *p != mine).collect();
207 +
        let pick = others
208 +
            .iter()
209 +
            .map(|d| d.join(format!("{key}.{ext}")))
210 +
            .filter(|p| fits(p))
211 +
            .max_by_key(|p| modified(p))
212 +
            .map(|p| (p, "same key"))
213 +
            .or_else(|| {
214 +
                others.iter().filter_map(|d| newest(d, form, |p| fits(p))).max_by_key(|p| modified(p)).map(|p| (p, "an older key"))
215 +
            });
216 +
        if let Some((p, which)) = pick {
217 +
            let from = p.parent().and_then(|d| d.file_name()).map(|n| n.to_string_lossy().into_owned()).unwrap_or_default();
218 +
            if copy(form, &p, dest).is_ok() {
219 +
                return Some(Taken {
220 +
                    how: format!("cache {name}: copied from another bookmark's ({from}), {which}"),
221 +
                    sources: read_sources(&p),
222 +
                });
223 +
            }
224 +
        }
225 +
        None
226 +
    }
227 +
228 +
    /// Keep a passing job's disk for mount `m` under its key, then keep
229 +
    /// the pool under its limit.
230 +
    pub fn keep(&self, ctx: &Ctx, m: &Mount, key: &str, disk: &Path, sources: &Sources) -> Result<()> {
231 +
        let Some(name) = m.cache.as_deref() else { return Ok(()) };
232 +
        let _changing = changing();
233 +
        // Whatever the job left, nothing in the pool is anyone else's to
234 +
        // write or runs as the runner (oxygenflow#70).
235 +
        if ctx.form == Form::Dir {
236 +
            seal(disk).with_context(|| format!("seal {}", disk.display()))?;
237 +
        }
238 +
        let mine = self.cache_dir(ctx, name).join(segment(ctx.bookmark));
239 +
        crate::portable::create_private_dir(&mine)?;
240 +
        crate::portable::set_mode(&self.root, 0o700)?;
241 +
        let dest = mine.join(format!("{key}.{}", ctx.form.ext()));
242 +
        if ctx.form == Form::Dir {
243 +
            // Another job of this slot kept one since: this one is newer.
244 +
            crate::workspace::remove_tree(&dest)?;
245 +
        }
246 +
        if std::fs::rename(disk, &dest).is_err() {
247 +
            copy(ctx.form, disk, &dest).with_context(|| format!("keep {}", disk.display()))?;
248 +
        }
249 +
        // Moved disks keep their old time: mark it used now.
250 +
        std::fs::File::open(&dest)?.set_modified(SystemTime::now())?;
251 +
        let sidecar = sources_of(&dest);
252 +
        let tmp = sidecar.with_extension("sources-part");
253 +
        let _ = std::fs::remove_file(&tmp);
254 +
        {
255 +
            use std::io::Write;
256 +
            crate::portable::private(std::fs::OpenOptions::new().write(true).create_new(true))
257 +
                .open(&tmp)?
258 +
                .write_all(&serde_json::to_vec(sources)?)?;
259 +
        }
260 +
        std::fs::rename(&tmp, sidecar)?;
261 +
        // One disk per key is enough; older keys of this bookmark go once
262 +
        // the pool is over its limit.
263 +
        self.evict();
264 +
        Ok(())
265 +
    }
266 +
267 +
    /// After the job: a passing job's disk is pruned and kept, unless it
268 +
    /// is still over its cap; a failing one's is thrown away, so nothing
269 +
    /// it wrote reaches a later job. For a kept or refused cache, what to
270 +
    /// say in the job's log.
271 +
    #[allow(clippy::too_many_arguments)]
272 +
    pub fn settle(
273 +
        &self,
274 +
        ctx: &Ctx,
275 +
        m: &Mount,
276 +
        key: &str,
277 +
        disk: &Path,
278 +
        sources: &Sources,
279 +
        passed: bool,
280 +
        prune: &Prune,
281 +
    ) -> Result<Option<String>> {
282 +
        if passed {
283 +
            let Some(name) = m.cache.as_deref() else { return Ok(None) };
284 +
            let pruned = pruned(ctx.form, m, disk, prune)?;
285 +
            let line = describe(name, &pruned);
286 +
            if pruned.fits() {
287 +
                self.keep(ctx, m, key, disk, sources)?;
288 +
            } else {
289 +
                discard(ctx.form, disk)?;
290 +
            }
291 +
            return Ok(Some(line));
292 +
        }
293 +
        discard(ctx.form, disk)?;
294 +
        Ok(None)
295 +
    }
296 +
297 +
    /// The repository and job slots holding at least one disk, as
298 +
    /// [`flow_core::api::cache_slot`] names them, for the claim
299 +
    /// (oxygenflow#51), each with when its newest disk was kept (Unix ms):
300 +
    /// a disk's time is set when it is kept, so the newest is the freshest
301 +
    /// build (oxygenflow#54). Images and directories alike, so a job goes
302 +
    /// where its cache is whichever tier keeps it.
303 +
    pub fn slot_times(&self) -> std::collections::BTreeMap<String, u64> {
304 +
        let mut out = std::collections::BTreeMap::new();
305 +
        for repo in std::fs::read_dir(&self.root).into_iter().flatten().flatten() {
306 +
            for job in std::fs::read_dir(repo.path()).into_iter().flatten().flatten() {
307 +
                let mut newest: Option<SystemTime> = None;
308 +
                // `<job>/<cache>/<bookmark>/<disk>`
309 +
                kept(&job.path(), 2, &mut |p, _| {
310 +
                    let t = modified(p);
311 +
                    newest = Some(newest.map_or(t, |n| n.max(t)));
312 +
                });
313 +
                if let Some(t) = newest {
314 +
                    let ms = t.duration_since(SystemTime::UNIX_EPOCH).map(|d| d.as_millis() as u64).unwrap_or(0);
315 +
                    out.insert(format!("{}/{}", repo.file_name().to_string_lossy(), job.file_name().to_string_lossy()), ms);
316 +
                }
317 +
            }
318 +
        }
319 +
        out
320 +
    }
321 +
322 +
    /// Drop the disks used longest ago until the pool's blocks fit: an
323 +
    /// image's blocks, or everything in a directory. The caller holds
324 +
    /// [`changing`].
325 +
    fn evict(&self) -> u64 {
326 +
        let mut disks: Vec<(SystemTime, u64, PathBuf)> = Vec::new();
327 +
        // `<repo>/<job>/<cache>/<bookmark>/<disk>`
328 +
        kept(&self.root, 4, &mut |p, form| {
329 +
            if let Ok(md) = std::fs::symlink_metadata(p) {
330 +
                let bytes = match form {
331 +
                    Form::Image => crate::portable::disk_bytes(&md),
332 +
                    Form::Dir => tree_bytes(p),
333 +
                };
334 +
                disks.push((md.modified().unwrap_or(SystemTime::UNIX_EPOCH), bytes, p.to_path_buf()));
335 +
            }
336 +
        });
337 +
        let mut total: u64 = disks.iter().map(|d| d.1).sum();
338 +
        disks.sort();
339 +
        let mut freed = 0;
340 +
        for (_, bytes, p) in disks {
341 +
            if total <= self.limit {
342 +
                break;
343 +
            }
344 +
            let gone = match form_of(&p) {
345 +
                Some(Form::Dir) => crate::workspace::remove_tree(&p),
346 +
                _ => std::fs::remove_file(&p),
347 +
            };
348 +
            if gone.is_ok() {
349 +
                let _ = std::fs::remove_file(sources_of(&p));
350 +
                total -= bytes;
351 +
                freed += bytes;
352 +
            }
353 +
        }
354 +
        freed
355 +
    }
356 +
}
357 +
358 +
/// Throw a disk away.
359 +
fn discard(form: Form, disk: &Path) -> Result<()> {
360 +
    let gone = match form {
361 +
        Form::Dir => crate::workspace::remove_tree(disk),
362 +
        Form::Image => std::fs::remove_file(disk).or_else(|e| if e.kind() == std::io::ErrorKind::NotFound { Ok(()) } else { Err(e) }),
363 +
    };
364 +
    gone.with_context(|| format!("discard {}", disk.display()))
365 +
}
366 +
367 +
/// Before a syscall job: set the atime of every file of a directory it
368 +
/// took from the pool to the epoch, so that whatever the file system's
369 +
/// atime rule, the job's first read of each is seen when the directory is
370 +
/// pruned (oxygenflow#72). A microVM's guest mounts its disk
371 +
/// `strictatime` instead.
372 +
pub fn forget_reads(form: Form, disk: &Path) -> Result<()> {
373 +
    if form == Form::Dir && disk.is_dir() {
374 +
        flow_init::prune::forget_reads(disk).with_context(|| format!("set the atimes in {}", disk.display()))?;
375 +
    }
376 +
    Ok(())
377 +
}
378 +
379 +
/// Prune a passing job's disk, or say why it was not: a directory here,
380 +
/// an image by what its guest said (else only its size is measured).
381 +
fn pruned(form: Form, m: &Mount, disk: &Path, prune: &Prune) -> Result<Pruned> {
382 +
    let size = m.size_mib as u64 * (1 << 20);
383 +
    let percent = prune.percent;
384 +
    let unpruned = |before: u64, why: String| Pruned { size, percent, before, after: before, unpruned: Some(why), ..Pruned::default() };
385 +
    Ok(match form {
386 +
        Form::Dir => {
387 +
            let before = tree_bytes(disk);
388 +
            if !flow_init::prune::records_reads(disk) {
389 +
                unpruned(before, "this runner's file system records no reads (noatime), so what the job used cannot be told".into())
390 +
            } else {
391 +
                flow_init::prune::prune(disk, prune.started, &[], size, before, percent)
392 +
                    .unwrap_or_else(|e| unpruned(tree_bytes(disk), format!("pruning failed: {e}")))
393 +
            }
394 +
        }
395 +
        Form::Image => match prune.guest.filter(|g| g.size > 0) {
396 +
            Some(g) => g.clone(),
397 +
            None => {
398 +
                let (before, size) = image_usage(disk)?;
399 +
                let why = prune.guest.and_then(|g| g.unpruned.clone()).unwrap_or_else(|| "the guest did not prune it".into());
400 +
                Pruned { size, ..unpruned(before, why) }
401 +
            }
402 +
        },
403 +
    })
404 +
}
405 +
406 +
/// What an ext4 image holds and its size, as its superblock counts them.
407 +
fn image_usage(image: &Path) -> Result<(u64, u64)> {
408 +
    let out = std::process::Command::new("dumpe2fs").arg("-h").arg(image).output().context("run dumpe2fs (e2fsprogs)")?;
409 +
    if !out.status.success() {
410 +
        anyhow::bail!("dumpe2fs {}: {}", image.display(), String::from_utf8_lossy(&out.stderr).trim());
411 +
    }
412 +
    superblock_usage(&String::from_utf8_lossy(&out.stdout)).with_context(|| format!("dumpe2fs {}: no block counts", image.display()))
413 +
}
414 +
415 +
fn superblock_usage(header: &str) -> Option<(u64, u64)> {
416 +
    let field = |name: &str| -> Option<u64> {
417 +
        header.lines().find_map(|l| l.strip_prefix(name)?.strip_prefix(':')).and_then(|v| v.trim().parse().ok())
418 +
    };
419 +
    let (count, free, block) = (field("Block count")?, field("Free blocks")?, field("Block size")?);
420 +
    Some((count.saturating_sub(free) * block, count * block))
421 +
}
422 +
423 +
/// The job log's line for a passing job's cache: what was kept, or why
424 +
/// not (oxygenflow#72).
425 +
pub fn describe(name: &str, p: &Pruned) -> String {
426 +
    use crate::images::human;
427 +
    let cap = format!("{}% of its {} disk", p.percent, human(p.size));
428 +
    let pruned = format!("{} in {} file{} this run did not use", human(p.freed), p.files, if p.files == 1 { "" } else { "s" });
429 +
    if p.fits() && p.files > 0 {
430 +
        format!("cache {name}: kept {} of {} (pruned {pruned}; the cap is {cap})", human(p.after), human(p.before))
431 +
    } else if p.fits() {
432 +
        format!("cache {name}: kept {}, under its cap of {cap}", human(p.after))
433 +
    } else {
434 +
        let after = if p.files > 0 { format!("{} after pruning {pruned}", human(p.after)) } else { human(p.after) };
435 +
        let why = p.unpruned.as_ref().map(|u| format!(" ({u})")).unwrap_or_default();
436 +
        format!(
437 +
            "cache {name}: not kept: {after} is over its cap of {} ({cap}){why}; the next run starts empty, and a larger `size` on the mount gives the cache room",
438 +
            human(p.cap())
439 +
        )
440 +
    }
441 +
}
442 +
443 +
/// Where a job's `n`th disk mount is in its scratch directory, in the form
444 +
/// its tier mounts it: `microvm::run` attaches `mount-N.ext4`, and
445 +
/// `syscall::run` binds the directory `mount-N`.
446 +
pub fn disk_path(form: Form, scratch: &Path, n: usize) -> PathBuf {
447 +
    match form {
448 +
        Form::Image => scratch.join(format!("mount-{n}.ext4")),
449 +
        Form::Dir => crate::syscall::disk_dir(scratch, n),
450 +
    }
451 +
}
452 +
453 +
fn modified(p: &Path) -> SystemTime {
454 +
    std::fs::symlink_metadata(p).and_then(|m| m.modified()).unwrap_or(SystemTime::UNIX_EPOCH)
455 +
}
456 +
457 +
fn newest(dir: &Path, form: Form, ok: impl Fn(&Path) -> bool) -> Option<PathBuf> {
458 +
    std::fs::read_dir(dir).ok()?.flatten().map(|e| e.path()).filter(|p| form_of(p) == Some(form) && ok(p)).max_by_key(|p| modified(p))
459 +
}
460 +
461 +
/// Each kept image or directory `depth` levels of directories below `dir`,
462 +
/// with its form; never inside a kept directory.
463 +
fn kept(dir: &Path, depth: usize, f: &mut dyn FnMut(&Path, Form)) {
464 +
    for e in std::fs::read_dir(dir).into_iter().flatten().flatten() {
465 +
        let p = e.path();
466 +
        let Ok(t) = e.file_type() else { continue };
467 +
        if depth == 0 {
468 +
            match form_of(&p) {
469 +
                Some(Form::Image) if t.is_file() => f(&p, Form::Image),
470 +
                Some(Form::Dir) if t.is_dir() => f(&p, Form::Dir),
471 +
                _ => {}
472 +
            }
473 +
        } else if t.is_dir() {
474 +
            kept(&p, depth - 1, f);
475 +
        }
476 +
    }
477 +
}
478 +
479 +
/// The blocks a directory's files take, each file once however many links
480 +
/// it has; links are not followed.
481 +
fn tree_bytes(dir: &Path) -> u64 {
482 +
    fn walk(p: &Path, seen: &mut std::collections::HashSet<(u64, u64)>) -> u64 {
483 +
        let Ok(md) = std::fs::symlink_metadata(p) else { return 0 };
484 +
        if crate::portable::shared_identity(&md).is_some_and(|id| !seen.insert(id)) {
485 +
            return 0;
486 +
        }
487 +
        let mut bytes = crate::portable::disk_bytes(&md);
488 +
        if md.is_dir() {
489 +
            for e in std::fs::read_dir(p).into_iter().flatten().flatten() {
490 +
                bytes += walk(&e.path(), seen);
491 +
            }
492 +
        }
493 +
        bytes
494 +
    }
495 +
    walk(dir, &mut Default::default())
496 +
}
497 +
498 +
/// Make a directory a job left fit for the pool: nothing in it writable by
499 +
/// group or others, nothing set-id or sticky, and everything readable by
500 +
/// the runner (directories searchable and writable too, so it can be
501 +
/// copied and removed). Links are not followed. The job ran as the
502 +
/// runner's user, so all of it is the runner's to change; times are kept.
503 +
fn seal(p: &Path) -> std::io::Result<()> {
504 +
    let md = std::fs::symlink_metadata(p)?;
505 +
    if md.file_type().is_symlink() {
506 +
        return Ok(());
507 +
    }
508 +
    let mode = crate::portable::mode(&md);
509 +
    let want = (mode & !0o7022) | if md.is_dir() { 0o700 } else { 0o400 };
510 +
    if want != mode {
511 +
        crate::portable::set_mode(p, want)?;
512 +
    }
513 +
    if md.is_dir() {
514 +
        for e in std::fs::read_dir(p)? {
515 +
            seal(&e?.path())?;
516 +
        }
517 +
    }
518 +
    Ok(())
519 +
}
520 +
521 +
fn copy(form: Form, from: &Path, to: &Path) -> Result<()> {
522 +
    match form {
523 +
        Form::Image => sparse_copy(from, to),
524 +
        Form::Dir => tree_copy(from, to),
525 +
    }
526 +
}
527 +
528 +
/// Copy a sparse disk without writing its holes.
529 +
fn sparse_copy(from: &Path, to: &Path) -> Result<()> {
530 +
    let tmp = to.with_extension("part");
531 +
    let st = std::process::Command::new("cp").arg("--sparse=always").arg(from).arg(&tmp).status().context("run cp")?;
532 +
    if !st.success() {
533 +
        let _ = std::fs::remove_file(&tmp);
534 +
        anyhow::bail!("cp {} failed", from.display());
535 +
    }
536 +
    std::fs::rename(&tmp, to)?;
537 +
    Ok(())
538 +
}
539 +
540 +
/// Copy a directory with its modes and times (a build tool judges by
541 +
/// them), sharing blocks where the filesystem can.
542 +
fn tree_copy(from: &Path, to: &Path) -> Result<()> {
543 +
    let tmp = to.with_extension("tree-part");
544 +
    crate::workspace::remove_tree(&tmp)?;
545 +
    let st = std::process::Command::new("cp")
546 +
        .args(["-a", "--reflink=auto", "--no-target-directory"])
547 +
        .arg(from)
548 +
        .arg(&tmp)
549 +
        .status()
550 +
        .context("run cp")?;
551 +
    if !st.success() {
552 +
        let _ = crate::workspace::remove_tree(&tmp);
553 +
        anyhow::bail!("cp {} failed", from.display());
554 +
    }
555 +
    std::fs::rename(&tmp, to).inspect_err(|_| {
556 +
        let _ = crate::workspace::remove_tree(&tmp);
557 +
    })?;
558 +
    Ok(())
559 +
}
560 +
561 +
#[cfg(all(test, unix))]
562 +
mod tests {
563 +
    use super::*;
564 +
    use flow_core::pipeline::MountSource;
565 +
    use std::os::unix::fs::{MetadataExt, PermissionsExt};
566 +
567 +
    /// Pruning that never prunes: the cap is the whole disk.
568 +
    fn keep_all() -> Prune<'static> {
569 +
        Prune { started: SystemTime::now(), percent: 100, guest: None }
570 +
    }
571 +
572 +
    fn mount(key: &[&str]) -> Mount {
573 +
        Mount {
574 +
            source: MountSource::Disk,
575 +
            target: "/src/target".into(),
576 +
            size_mib: 1,
577 +
            cache: Some("target".into()),
578 +
            key: key.iter().map(|s| s.to_string()).collect(),
579 +
        }
580 +
    }
581 +
582 +
    fn disk(path: &Path, fill: &[u8]) {
583 +
        let f = std::fs::File::create(path).unwrap();
584 +
        f.set_len(1 << 20).unwrap();
585 +
        std::os::unix::fs::FileExt::write_at(&f, fill, 0).unwrap();
586 +
    }
587 +
588 +
    fn first(path: &Path) -> Vec<u8> {
589 +
        std::fs::read(path).unwrap()[..4].to_vec()
590 +
    }
591 +
592 +
    #[test]
593 +
    fn a_kept_disk_starts_the_next_job_of_its_bookmark_then_others_copy_it() {
594 +
        let home = tempfile::tempdir().unwrap();
595 +
        let work = tempfile::tempdir().unwrap();
596 +
        let pool = Pool::new(home.path(), None);
597 +
        let m = mount(&["Cargo.lock"]);
598 +
        let main = Ctx { repo: "andy/hub", bookmark: "main", job: "test", image: "img", form: Form::Image };
599 +
        let key = Pool::key(&main, &m, |_| Some("lock-1".into()));
600 +
        // Nothing yet: the caller makes an empty disk.
601 +
        assert!(pool.take(&main, &m, &key, &work.path().join("a.ext4")).is_none());
602 +
        // A passing job's disk is kept and moved into the next job's place.
603 +
        disk(&work.path().join("a.ext4"), b"MAIN");
604 +
        let built: Sources = [("src/lib.rs".to_string(), ("h1".to_string(), 1_000))].into();
605 +
        pool.keep(&main, &m, &key, &work.path().join("a.ext4"), &built).unwrap();
606 +
        assert!(!work.path().join("a.ext4").exists(), "moved, not copied");
607 +
        let taken = pool.take(&main, &m, &key, &work.path().join("b.ext4")).unwrap();
608 +
        assert!(taken.how.contains("same key"), "{}", taken.how);
609 +
        assert_eq!(taken.sources.as_ref(), Some(&built), "its sources come with it (#52)");
610 +
        assert_eq!(first(&work.path().join("b.ext4")), b"MAIN");
611 +
        pool.keep(&main, &m, &key, &work.path().join("b.ext4"), &built).unwrap();
612 +
        // A changed key takes the bookmark's newest disk all the same.
613 +
        let key2 = Pool::key(&main, &m, |_| Some("lock-2".into()));
614 +
        assert_ne!(key, key2);
615 +
        let taken = pool.take(&main, &m, &key2, &work.path().join("c.ext4")).unwrap();
616 +
        assert!(taken.how.contains("older key"), "{}", taken.how);
617 +
        assert_eq!(taken.sources.as_ref(), Some(&built));
618 +
        pool.keep(&main, &m, &key2, &work.path().join("c.ext4"), &built).unwrap();
619 +
        assert_eq!(
620 +
            pool.slot_times().into_keys().collect::<Vec<_>>(),
621 +
            vec![flow_core::api::cache_slot("andy/hub", "test")],
622 +
            "what the claim reports (#51)"
623 +
        );
624 +
        // With when it was last kept (#54): just now.
625 +
        let now = SystemTime::now().duration_since(SystemTime::UNIX_EPOCH).unwrap().as_millis() as u64;
626 +
        let kept = pool.slot_times()[&flow_core::api::cache_slot("andy/hub", "test")];
627 +
        assert!(now.abs_diff(kept) < 60_000, "kept {kept}, now {now}");
628 +
        // Another bookmark copies it, leaving main's in place.
629 +
        let feature = Ctx { bookmark: "feature", ..main };
630 +
        let taken = pool.take(&feature, &m, &key2, &work.path().join("d.ext4")).unwrap();
631 +
        assert!(taken.how.contains("copied from another bookmark"), "{}", taken.how);
632 +
        assert_eq!(taken.sources.as_ref(), Some(&built), "a copy reads them and leaves them");
633 +
        assert_eq!(first(&work.path().join("d.ext4")), b"MAIN");
634 +
        let taken = pool.take(&main, &m, &key2, &work.path().join("e.ext4")).unwrap();
635 +
        assert!(taken.how.contains("this bookmark's") && taken.sources.is_some(), "main's is still there, with its sources");
636 +
        // Another job or a different size never shares it.
637 +
        let other_job = Ctx { job: "build", ..main };
638 +
        assert!(pool.take(&other_job, &m, &key2, &work.path().join("f.ext4")).is_none());
639 +
        let mut bigger = mount(&["Cargo.lock"]);
640 +
        bigger.size_mib = 2;
641 +
        assert!(pool.take(&feature, &bigger, &key2, &work.path().join("g.ext4")).is_none());
642 +
    }
643 +
644 +
    #[test]
645 +
    fn the_key_follows_the_image_the_job_and_the_key_files() {
646 +
        let ctx = Ctx { repo: "r", bookmark: "main", job: "test", image: "img-1", form: Form::Image };
647 +
        let m = mount(&["Cargo.lock", "rust-toolchain.toml"]);
648 +
        let k = Pool::key(&ctx, &m, |p| Some(format!("{p}-v1")));
649 +
        assert_eq!(k, Pool::key(&ctx, &m, |p| Some(format!("{p}-v1"))));
650 +
        assert_ne!(k, Pool::key(&ctx, &m, |p| Some(format!("{p}-v2"))), "a key file changed");
651 +
        assert_ne!(k, Pool::key(&Ctx { image: "img-2", ..ctx }, &m, |p| Some(format!("{p}-v1"))), "another image");
652 +
        assert_ne!(k, Pool::key(&ctx, &m, |_| None), "a key file missing");
653 +
    }
654 +
655 +
    #[test]
656 +
    fn the_pool_drops_the_disks_used_longest_ago() {
657 +
        let home = tempfile::tempdir().unwrap();
658 +
        let work = tempfile::tempdir().unwrap();
659 +
        let pool = Pool { root: home.path().join("cache"), limit: 0 };
660 +
        let m = mount(&[]);
661 +
        let ctx = Ctx { repo: "r", bookmark: "main", job: "j", image: "i", form: Form::Image };
662 +
        let full = |p: &Path| {
663 +
            let f = std::fs::File::create(p).unwrap();
664 +
            std::os::unix::fs::FileExt::write_at(&f, &vec![7u8; 1 << 20], 0).unwrap();
665 +
        };
666 +
        full(&work.path().join("a.ext4"));
667 +
        // Over a limit of nothing: kept, then dropped at once.
668 +
        pool.keep(&ctx, &m, "k1", &work.path().join("a.ext4"), &Sources::new()).unwrap();
669 +
        assert!(pool.take(&ctx, &m, "k1", &work.path().join("b.ext4")).is_none());
670 +
        let roomy = Pool { root: home.path().join("cache"), limit: 1 << 30 };
671 +
        full(&work.path().join("c.ext4"));
672 +
        roomy.keep(&ctx, &m, "k2", &work.path().join("c.ext4"), &Sources::new()).unwrap();
673 +
        assert!(roomy.take(&ctx, &m, "k2", &work.path().join("d.ext4")).is_some(), "under the limit it stays");
674 +
    }
675 +
676 +
    /// A directory as the syscall tier leaves one: `marker` in
677 +
    /// `debug/marker`, and a file of `kib` KiB.
678 +
    fn tree(path: &Path, marker: &[u8], kib: usize) {
679 +
        std::fs::create_dir_all(path.join("debug")).unwrap();
680 +
        std::fs::write(path.join("debug/marker"), marker).unwrap();
681 +
        // Not all one byte, so a compressing filesystem stores it whole.
682 +
        let bytes: Vec<u8> = (0..kib << 10).map(|i: usize| (i.wrapping_mul(2_654_435_761) >> 13) as u8).collect();
683 +
        std::fs::write(path.join("debug/blob"), bytes).unwrap();
684 +
    }
685 +
686 +
    fn dir_ctx<'a>(bookmark: &'a str) -> Ctx<'a> {
687 +
        Ctx { repo: "andy/flow", bookmark, job: "test", image: "img", form: Form::Dir }
688 +
    }
689 +
690 +
    /// oxygenflow#70: the syscall tier's directories are taken and kept as
691 +
    /// a microVM's images are, and never mixed with them.
692 +
    #[test]
693 +
    fn a_kept_directory_starts_the_next_syscall_job_and_others_copy_it() {
694 +
        let home = tempfile::tempdir().unwrap();
695 +
        let work = tempfile::tempdir().unwrap();
696 +
        let pool = Pool::new(home.path(), None);
697 +
        let m = mount(&["Cargo.lock"]);
698 +
        let main = dir_ctx("main");
699 +
        let key = Pool::key(&main, &m, |_| Some("lock-1".into()));
700 +
        let w = |n: &str| work.path().join(n);
701 +
        assert!(pool.take(&main, &m, &key, &w("a")).is_none(), "nothing kept yet");
702 +
        // A passing job's directory goes into the pool, times and all.
703 +
        tree(&w("a"), b"FIRST", 4);
704 +
        // Named like an image: the pool never looks inside a directory.
705 +
        std::fs::write(w("a/debug/inner.ext4"), b"not a disk").unwrap();
706 +
        let old = SystemTime::UNIX_EPOCH + std::time::Duration::from_secs(1_000_000);
707 +
        std::fs::File::open(w("a/debug/marker")).unwrap().set_modified(old).unwrap();
708 +
        let built: Sources = [("src/lib.rs".to_string(), ("h1".to_string(), 1_000))].into();
709 +
        pool.settle(&main, &m, &key, &w("a"), &built, true, &keep_all()).unwrap();
710 +
        assert!(!w("a").exists(), "moved, not copied");
711 +
        assert_eq!(
712 +
            pool.slot_times().into_keys().collect::<Vec<_>>(),
713 +
            vec![flow_core::api::cache_slot("andy/flow", "test")],
714 +
            "the claim reports it like an image (#51)"
715 +
        );
716 +
        // The next job of its bookmark moves it out: a job of the same
717 +
        // slot beside it finds nothing to take.
718 +
        let taken = pool.take(&main, &m, &key, &w("b")).unwrap();
719 +
        assert!(taken.how.contains("this bookmark's, same key"), "{}", taken.how);
720 +
        assert_eq!(taken.sources.as_ref(), Some(&built), "its sources come with it (#52)");
721 +
        assert_eq!(std::fs::read(w("b/debug/marker")).unwrap(), b"FIRST");
722 +
        assert_eq!(modified(&w("b/debug/marker")), old, "its files keep their times");
723 +
        assert_eq!(std::fs::read(w("b/debug/inner.ext4")).unwrap(), b"not a disk");
724 +
        assert!(pool.take(&main, &m, &key, &w("c")).is_none(), "taken once");
725 +
        assert!(pool.slot_times().is_empty());
726 +
        // Kept again, and under a changed key the newest is taken all the same.
727 +
        std::fs::write(w("b/debug/marker"), b"SECOND").unwrap();
728 +
        pool.settle(&main, &m, &key, &w("b"), &built, true, &keep_all()).unwrap();
729 +
        let key2 = Pool::key(&main, &m, |_| Some("lock-2".into()));
730 +
        let taken = pool.take(&main, &m, &key2, &w("d")).unwrap();
731 +
        assert!(taken.how.contains("older key"), "{}", taken.how);
732 +
        assert_eq!(std::fs::read(w("d/debug/marker")).unwrap(), b"SECOND");
733 +
        pool.settle(&main, &m, &key2, &w("d"), &built, true, &keep_all()).unwrap();
734 +
        // Another bookmark copies it, with its times, and main keeps its own.
735 +
        let feature = dir_ctx("feature");
736 +
        let taken = pool.take(&feature, &m, &key2, &w("e")).unwrap();
737 +
        assert!(taken.how.contains("copied from another bookmark's (main-"), "{}", taken.how);
738 +
        assert_eq!(taken.sources.as_ref(), Some(&built));
739 +
        assert_eq!(std::fs::read(w("e/debug/marker")).unwrap(), b"SECOND");
740 +
        assert!(pool.take(&main, &m, &key2, &w("f")).is_some(), "main's is still there");
741 +
        assert_eq!(modified(&w("e/debug/marker")), modified(&w("f/debug/marker")), "a copy keeps the times");
742 +
        pool.settle(&main, &m, &key2, &w("f"), &built, true, &keep_all()).unwrap();
743 +
        // A microVM job of the same slot never takes a directory, nor a
744 +
        // syscall job an image.
745 +
        let image = Ctx { form: Form::Image, ..main };
746 +
        assert!(pool.take(&image, &m, &key2, &w("g.ext4")).is_none());
747 +
        disk(&w("h.ext4"), b"DISK");
748 +
        pool.keep(&image, &m, &key2, &w("h.ext4"), &built).unwrap();
749 +
        let taken = pool.take(&main, &m, &key2, &w("i")).unwrap();
750 +
        assert!(w("i/debug/marker").is_file(), "{}", taken.how);
751 +
        assert!(pool.take(&main, &m, &key2, &w("j")).is_none(), "the image is not a directory");
752 +
        assert!(pool.take(&image, &m, &key2, &w("k.ext4")).is_some(), "and is still there for a microVM");
753 +
    }
754 +
755 +
    #[test]
756 +
    fn a_failing_jobs_directory_is_thrown_away() {
757 +
        let home = tempfile::tempdir().unwrap();
758 +
        let work = tempfile::tempdir().unwrap();
759 +
        let pool = Pool::new(home.path(), None);
760 +
        let m = mount(&[]);
761 +
        let main = dir_ctx("main");
762 +
        tree(&work.path().join("a"), b"GOOD", 1);
763 +
        pool.settle(&main, &m, "k", &work.path().join("a"), &Sources::new(), true, &keep_all()).unwrap();
764 +
        let taken = pool.take(&main, &m, "k", &work.path().join("b"));
765 +
        assert!(taken.is_some());
766 +
        // The job that took it fails after writing to it: not kept, gone.
767 +
        std::fs::write(work.path().join("b/debug/marker"), b"POISON").unwrap();
768 +
        pool.settle(&main, &m, "k", &work.path().join("b"), &Sources::new(), false, &keep_all()).unwrap();
769 +
        assert!(!work.path().join("b").exists());
770 +
        assert!(pool.take(&main, &m, "k", &work.path().join("c")).is_none(), "nothing it wrote reaches a later job");
771 +
        assert!(pool.slot_times().is_empty());
772 +
        // A failed image is thrown away the same way.
773 +
        let image = Ctx { form: Form::Image, ..main };
774 +
        disk(&work.path().join("d.ext4"), b"BAD!");
775 +
        pool.settle(&image, &m, "k", &work.path().join("d.ext4"), &Sources::new(), false, &keep_all()).unwrap();
776 +
        assert!(!work.path().join("d.ext4").exists() && pool.slot_times().is_empty());
777 +
    }
778 +
779 +
    #[test]
780 +
    fn directories_are_evicted_by_their_size_oldest_first() {
781 +
        let home = tempfile::tempdir().unwrap();
782 +
        let work = tempfile::tempdir().unwrap();
783 +
        // Room for one 512 KiB directory, not two.
784 +
        let pool = Pool { root: home.path().join("cache"), limit: 800 << 10 };
785 +
        let m = mount(&[]);
786 +
        let (main, feature) = (dir_ctx("main"), dir_ctx("feature"));
787 +
        tree(&work.path().join("a"), b"OLD", 512);
788 +
        pool.keep(&main, &m, "k1", &work.path().join("a"), &Sources::new()).unwrap();
789 +
        let kept = pool.cache_dir(&main, "target").join(segment("main")).join("k1.tree");
790 +
        assert!(kept.is_dir(), "under the limit it stays");
791 +
        assert!(tree_bytes(&kept) >= 512 << 10, "a directory counts what is in it");
792 +
        // Older than the next one, however fast this test runs.
793 +
        std::fs::File::open(&kept).unwrap().set_modified(SystemTime::UNIX_EPOCH + std::time::Duration::from_secs(1)).unwrap();
794 +
        tree(&work.path().join("b"), b"NEW", 512);
795 +
        pool.keep(&feature, &m, "k1", &work.path().join("b"), &Sources::new()).unwrap();
796 +
        assert!(!kept.exists() && !kept.with_extension("tree-sources").exists(), "the one used longest ago goes, sources and all");
797 +
        let taken = pool.take(&main, &m, "k1", &work.path().join("c")).unwrap();
798 +
        assert!(taken.how.contains("bookmark's (feature-"), "{}", taken.how);
799 +
        assert_eq!(std::fs::read(work.path().join("c/debug/marker")).unwrap(), b"NEW");
800 +
    }
801 +
802 +
    /// oxygenflow#70: the job wrote what it liked; the pool keeps nothing
803 +
    /// others may write or that runs as the runner, and is the runner's.
804 +
    #[test]
805 +
    fn nothing_kept_is_writable_by_others_or_set_id() {
806 +
        let home = tempfile::tempdir().unwrap();
807 +
        let work = tempfile::tempdir().unwrap();
808 +
        let pool = Pool::new(home.path(), None);
809 +
        let m = mount(&[]);
810 +
        let main = dir_ctx("main");
811 +
        let a = work.path().join("a");
812 +
        tree(&a, b"X", 1);
813 +
        let mode = |p: &Path, m: u32| std::fs::set_permissions(p, std::fs::Permissions::from_mode(m)).unwrap();
814 +
        std::fs::write(a.join("open"), b"").unwrap();
815 +
        mode(&a.join("open"), 0o777);
816 +
        std::fs::write(a.join("suid"), b"").unwrap();
817 +
        mode(&a.join("suid"), 0o4755);
818 +
        std::fs::write(a.join("debug/sealed"), b"secret").unwrap();
819 +
        mode(&a.join("debug/sealed"), 0o000);
820 +
        std::os::unix::fs::symlink("/etc/passwd", a.join("link")).unwrap();
821 +
        let when = modified(&a.join("debug/marker"));
822 +
        mode(&a.join("debug"), 0o1777);
823 +
        mode(&a, 0o500);
824 +
        pool.keep(&main, &m, "k", &a, &Sources::new()).unwrap();
825 +
        let root = home.path().join("cache");
826 +
        assert_eq!(std::fs::metadata(&root).unwrap().mode() & 0o777, 0o700, "the pool is the runner's alone");
827 +
        let mut seen = 0;
828 +
        fn each(p: &Path, f: &mut dyn FnMut(&Path, &std::fs::Metadata)) {
829 +
            let md = std::fs::symlink_metadata(p).unwrap();
830 +
            f(p, &md);
831 +
            if md.is_dir() {
832 +
                for e in std::fs::read_dir(p).unwrap() {
833 +
                    each(&e.unwrap().path(), f);
834 +
                }
835 +
            }
836 +
        }
837 +
        each(&root, &mut |p, md| {
838 +
            seen += 1;
839 +
            if !md.file_type().is_symlink() {
840 +
                assert_eq!(md.mode() & 0o7022, 0, "{} is {:o}", p.display(), md.mode());
841 +
            }
842 +
        });
843 +
        assert!(seen >= 10, "{seen}");
844 +
        // Still readable, with its times, so the next job builds on it.
845 +
        let taken = pool.take(&main, &m, "k", &work.path().join("b"));
846 +
        assert!(taken.is_some());
847 +
        let b = work.path().join("b");
848 +
        assert_eq!(std::fs::read(b.join("debug/sealed")).unwrap(), b"secret");
849 +
        assert_eq!(std::fs::metadata(b.join("open")).unwrap().mode() & 0o777, 0o755);
850 +
        assert_eq!(std::fs::read_link(b.join("link")).unwrap(), Path::new("/etc/passwd"), "a link is kept, not followed");
851 +
        assert_eq!(modified(&b.join("debug/marker")), when);
852 +
    }
853 +
    /// A file of `kib` KiB at `rel` under `root`, last read and written
854 +
    /// at `secs` (ahead of the clock, so the time it was changed, now,
855 +
    /// is before a job starting at [`T`]).
856 +
    fn aged(root: &Path, rel: &str, kib: usize, secs: u64) -> PathBuf {
857 +
        let p = root.join(rel);
858 +
        std::fs::create_dir_all(p.parent().unwrap()).unwrap();
859 +
        let bytes: Vec<u8> = (0..kib << 10).map(|i: usize| (i.wrapping_mul(2_654_435_761) >> 13) as u8).collect();
860 +
        std::fs::write(&p, bytes).unwrap();
861 +
        let t = SystemTime::UNIX_EPOCH + std::time::Duration::from_secs(secs);
862 +
        std::fs::File::options()
863 +
            .write(true)
864 +
            .open(&p)
865 +
            .unwrap()
866 +
            .set_times(std::fs::FileTimes::new().set_accessed(t).set_modified(t))
867 +
            .unwrap();
868 +
        p
869 +
    }
870 +
871 +
    const T: u64 = 4_000_000_000;
872 +
873 +
    fn at_t() -> SystemTime {
874 +
        SystemTime::UNIX_EPOCH + std::time::Duration::from_secs(T)
875 +
    }
876 +
877 +
    /// oxygenflow#72: a syscall job's directory is pruned of what the job
878 +
    /// did not use before it is kept, and the log says so; one still over
879 +
    /// its cap is not kept at all.
880 +
    #[test]
881 +
    fn a_directory_is_pruned_of_what_the_job_did_not_use_before_it_is_kept() {
882 +
        let home = tempfile::tempdir().unwrap();
883 +
        let work = tempfile::tempdir().unwrap();
884 +
        let pool = Pool::new(home.path(), None);
885 +
        let main = dir_ctx("main");
886 +
        let mut m = mount(&[]);
887 +
        m.size_mib = 8;
888 +
        let a = work.path().join("a");
889 +
        // This run's build, and an older lockfile's that it did not touch.
890 +
        aged(&a, "debug/deps/libnew-aaaaaaaaaaaaaaaa.rlib", 2 << 10, T + 10);
891 +
        aged(&a, "debug/deps/libold-bbbbbbbbbbbbbbbb.rlib", 3 << 10, T - 1_000);
892 +
        let prune = Prune { started: at_t(), percent: 50, guest: None };
893 +
        let line = pool.settle(&main, &m, "k", &a, &Sources::new(), true, &prune).unwrap().unwrap();
894 +
        assert_eq!(
895 +
            line,
896 +
            "cache target: kept 2 MiB of 5 MiB (pruned 3 MiB in 1 file this run did not use; the cap is 50% of its 8 MiB disk)"
897 +
        );
898 +
        let b = work.path().join("b");
899 +
        assert!(pool.take(&main, &m, "k", &b).unwrap().how.contains("same key"));
900 +
        assert!(b.join("debug/deps/libnew-aaaaaaaaaaaaaaaa.rlib").is_file() && !b.join("debug/deps/libold-bbbbbbbbbbbbbbbb.rlib").exists());
901 +
        // Before the next job its reads are forgotten, its writes not.
902 +
        forget_reads(Form::Dir, &b).unwrap();
903 +
        let md = std::fs::metadata(b.join("debug/deps/libnew-aaaaaaaaaaaaaaaa.rlib")).unwrap();
904 +
        assert_eq!((md.atime(), md.mtime()), (0, (T + 10) as i64));
905 +
        // This run's own files over the cap: nothing it used goes, and the
906 +
        // cache is not kept, so the next run starts empty.
907 +
        aged(&b, "debug/deps/libbig-cccccccccccccccc.rlib", 3 << 10, T + 20);
908 +
        let line = pool.settle(&main, &m, "k", &b, &Sources::new(), true, &prune).unwrap().unwrap();
909 +
        assert!(
910 +
            line.starts_with("cache target: not kept: 5 MiB is over its cap of 4 MiB (50% of its 8 MiB disk); the next run starts empty"),
911 +
            "{line}"
912 +
        );
913 +
        assert!(!b.exists() && pool.take(&main, &m, "k", &work.path().join("c")).is_none());
914 +
    }
915 +
916 +
    /// oxygenflow#72: a microVM's image is kept or not by what its guest
917 +
    /// said; without word from it, by what its superblock says is in use.
918 +
    #[test]
919 +
    fn an_image_is_kept_by_what_its_guest_pruned_or_else_by_its_size() {
920 +
        let home = tempfile::tempdir().unwrap();
921 +
        let work = tempfile::tempdir().unwrap();
922 +
        let pool = Pool::new(home.path(), None);
923 +
        let ctx = Ctx { repo: "r", bookmark: "main", job: "j", image: "i", form: Form::Image };
924 +
        let mut m = mount(&[]);
925 +
        m.size_mib = 16;
926 +
        let fits = Pruned { size: 16 << 20, percent: 60, before: 12 << 20, after: 6 << 20, freed: 6 << 20, files: 40, unpruned: None };
927 +
        let over = Pruned { after: 11 << 20, ..fits.clone() };
928 +
        disk(&work.path().join("a.ext4"), b"KEEP");
929 +
        let line = pool
930 +
            .settle(&ctx, &m, "k", &work.path().join("a.ext4"), &Sources::new(), true, &Prune { guest: Some(&fits), ..keep_all() })
931 +
            .unwrap()
932 +
            .unwrap();
933 +
        assert_eq!(
934 +
            line,
935 +
            "cache target: kept 6 MiB of 12 MiB (pruned 6 MiB in 40 files this run did not use; the cap is 60% of its 16 MiB disk)"
936 +
        );
937 +
        assert!(pool.take(&ctx, &m, "k", &work.path().join("b.ext4")).is_none(), "a 1 MiB disk is not the mount's 16 MiB");
938 +
        disk(&work.path().join("c.ext4"), b"OVER");
939 +
        let line = pool
940 +
            .settle(&ctx, &m, "k", &work.path().join("c.ext4"), &Sources::new(), true, &Prune { guest: Some(&over), ..keep_all() })
941 +
            .unwrap()
942 +
            .unwrap();
943 +
        assert!(line.starts_with("cache target: not kept: 11 MiB after pruning 6 MiB in 40 files"), "{line}");
944 +
        assert!(!work.path().join("c.ext4").exists());
945 +
        // No word from the guest: an empty file system is well under.
946 +
        let img = work.path().join("d.ext4");
947 +
        crate::workspace::empty_ext4(&img, 16, "scratch").unwrap();
948 +
        let line = pool.settle(&ctx, &m, "k", &img, &Sources::new(), true, &Prune { percent: 60, ..keep_all() }).unwrap().unwrap();
949 +
        assert!(line.starts_with("cache target: kept ") && line.contains("under its cap of 60% of its 16 MiB disk"), "{line}");
950 +
        assert!(pool.take(&ctx, &m, "k", &work.path().join("e.ext4")).is_some());
951 +
        // ... and one nearly all in use is not kept, saying why.
952 +
        let line = pool
953 +
            .settle(&ctx, &m, "k", &work.path().join("e.ext4"), &Sources::new(), true, &Prune { percent: 1, ..keep_all() })
954 +
            .unwrap()
955 +
            .unwrap();
956 +
        assert!(line.contains("not kept") && line.contains("(the guest did not prune it)"), "{line}");
957 +
    }
958 +
959 +
    #[test]
960 +
    fn the_superblock_says_what_is_in_use() {
961 +
        let h = "Filesystem volume name:   scratch\nBlock count:              4096\nFree blocks:              1024\nBlock size:               4096\n";
962 +
        assert_eq!(superblock_usage(h), Some((3072 * 4096, 4096 * 4096)));
963 +
        assert_eq!(superblock_usage("Block count: 1\n"), None);
964 +
    }
965 +
}
added crates/flow/src/checkpoint.rs +212 −0
1 +
//! Checkpoints (oxygenflow#77, docs/handover.md §7): the directories a job
2 +
//! declares with `checkpoint "/src/…"`, saved when its runner hands it back
3 +
//! and restored before the next attempt runs the same exec, so a tool that
4 +
//! skips finished work (cargo, make) carries on where it stopped.
5 +
//!
6 +
//! The format is the runner's own, so CI builds need no new crate: a magic
7 +
//! line, then entries (a kind byte, the path under /src, mode, modification
8 +
//! time and, for a file, its bytes). Files keep their times, since cargo's
9 +
//! fingerprints read them. Links are left out. Unpacking accepts only paths
10 +
//! under the job's own checkpoints, without `.` or `..`.
11 +
12 +
use anyhow::{bail, Context, Result};
13 +
use flow_core::pipeline::{Job, MountSource};
14 +
use std::path::{Path, PathBuf};
15 +
use std::time::{Duration, SystemTime, UNIX_EPOCH};
16 +
17 +
/// At most this much is saved: the hub keeps it as it keeps an artifact.
18 +
pub const MAX_BYTES: usize = 512 << 20;
19 +
20 +
const MAGIC: &[u8] = b"OXYFLOW-CHECKPOINT 1\n";
21 +
const DIR: u8 = 0;
22 +
const FILE: u8 = 1;
23 +
24 +
/// What was saved or restored.
25 +
#[derive(Debug, Default, PartialEq, Eq)]
26 +
pub struct Counts {
27 +
    pub files: usize,
28 +
    pub dirs: usize,
29 +
    pub bytes: u64,
30 +
}
31 +
32 +
/// Where checkpoint `path` (under /src) is on this host, for a job of a
33 +
/// directory tier (WASM, syscall): in a disk mount's directory when a mount
34 +
/// holds it, else in the job's tree. `None` for one under a tmpfs, which
35 +
/// the syscall tier keeps in memory.
36 +
pub fn host_dir(path: &str, job: &Job, tree: &Path, scratch: &Path, wasm: bool) -> Option<PathBuf> {
37 +
    let under = |t: &str| path == t || path.starts_with(&format!("{}/", t.trim_end_matches('/')));
38 +
    let mut disks = 0;
39 +
    let mut best: Option<(usize, PathBuf)> = None;
40 +
    for (i, m) in job.mounts.iter().enumerate() {
41 +
        let dir = match m.source {
42 +
            MountSource::Disk => {
43 +
                disks += 1;
44 +
                Some(crate::syscall::disk_dir(scratch, disks - 1))
45 +
            }
46 +
            MountSource::Tmpfs if wasm => Some(scratch.join(format!("tmpfs-{i}"))),
47 +
            MountSource::Tmpfs => None,
48 +
        };
49 +
        if under(&m.target) && best.as_ref().is_none_or(|(len, _)| m.target.len() > *len) {
50 +
            let rest = path[m.target.trim_end_matches('/').len()..].trim_start_matches('/');
51 +
            best = Some((m.target.len(), dir.map(|d| d.join(rest)).unwrap_or_default()));
52 +
        }
53 +
    }
54 +
    match best {
55 +
        Some((_, d)) if d.as_os_str().is_empty() => None,
56 +
        Some((_, d)) => Some(d),
57 +
        None => Some(tree.join(path.trim_start_matches("/src").trim_start_matches('/'))),
58 +
    }
59 +
}
60 +
61 +
/// Pack `dirs` (each a checkpoint path and where it is on this host).
62 +
pub fn pack(dirs: &[(String, PathBuf)]) -> Result<(Vec<u8>, Counts)> {
63 +
    let mut out = MAGIC.to_vec();
64 +
    let mut n = Counts::default();
65 +
    for (path, host) in dirs {
66 +
        if !host.is_dir() {
67 +
            continue;
68 +
        }
69 +
        let mut stack = vec![(host.clone(), path.trim_end_matches('/').to_string())];
70 +
        while let Some((at, name)) = stack.pop() {
71 +
            let meta = std::fs::symlink_metadata(&at)?;
72 +
            if meta.is_dir() {
73 +
                entry(&mut out, DIR, &name, &meta, &[]);
74 +
                n.dirs += 1;
75 +
                let mut children: Vec<_> = std::fs::read_dir(&at)?.collect::<std::io::Result<_>>()?;
76 +
                children.sort_by_key(|e| e.file_name());
77 +
                for c in children.into_iter().rev() {
78 +
                    let Some(child) = c.file_name().to_str().map(String::from) else { continue };
79 +
                    stack.push((c.path(), format!("{name}/{child}")));
80 +
                }
81 +
            } else if meta.is_file() {
82 +
                let bytes = std::fs::read(&at).with_context(|| format!("read {}", at.display()))?;
83 +
                n.files += 1;
84 +
                n.bytes += bytes.len() as u64;
85 +
                entry(&mut out, FILE, &name, &meta, &bytes);
86 +
            }
87 +
            if out.len() > MAX_BYTES {
88 +
                bail!("the checkpoint is over {} MiB", MAX_BYTES >> 20);
89 +
            }
90 +
        }
91 +
    }
92 +
    Ok((out, n))
93 +
}
94 +
95 +
fn entry(out: &mut Vec<u8>, kind: u8, path: &str, meta: &std::fs::Metadata, bytes: &[u8]) {
96 +
    let since = meta.modified().ok().and_then(|t| t.duration_since(UNIX_EPOCH).ok()).unwrap_or_default();
97 +
    out.push(kind);
98 +
    out.extend((path.len() as u32).to_le_bytes());
99 +
    out.extend(path.as_bytes());
100 +
    out.extend(crate::portable::mode(meta).to_le_bytes());
101 +
    out.extend(since.as_secs().to_le_bytes());
102 +
    out.extend(since.subsec_nanos().to_le_bytes());
103 +
    out.extend((bytes.len() as u64).to_le_bytes());
104 +
    out.extend(bytes);
105 +
}
106 +
107 +
/// Restore a checkpoint: each entry at `place(checkpoint)` joined with the
108 +
/// rest of its path, for the job's own `checkpoints` only.
109 +
pub fn unpack(bytes: &[u8], checkpoints: &[String], place: impl Fn(&str) -> Option<PathBuf>) -> Result<Counts> {
110 +
    let Some(mut rest) = bytes.strip_prefix(MAGIC) else { bail!("not a checkpoint") };
111 +
    let mut n = Counts::default();
112 +
    let take = |rest: &mut &[u8], k: usize| -> Result<Vec<u8>> {
113 +
        if rest.len() < k {
114 +
            bail!("the checkpoint is cut short");
115 +
        }
116 +
        let (a, b) = rest.split_at(k);
117 +
        *rest = b;
118 +
        Ok(a.to_vec())
119 +
    };
120 +
    while !rest.is_empty() {
121 +
        let kind = take(&mut rest, 1)?[0];
122 +
        let len = u32::from_le_bytes(take(&mut rest, 4)?.try_into().unwrap()) as usize;
123 +
        let path = String::from_utf8(take(&mut rest, len)?).context("a path that is not UTF-8")?;
124 +
        let mode = u32::from_le_bytes(take(&mut rest, 4)?.try_into().unwrap());
125 +
        let secs = u64::from_le_bytes(take(&mut rest, 8)?.try_into().unwrap());
126 +
        let nanos = u32::from_le_bytes(take(&mut rest, 4)?.try_into().unwrap());
127 +
        let size = u64::from_le_bytes(take(&mut rest, 8)?.try_into().unwrap()) as usize;
128 +
        let data = take(&mut rest, size)?;
129 +
        // Under one of the job's checkpoints, without . or .. on the way.
130 +
        let Some(root) = checkpoints.iter().map(|c| c.trim_end_matches('/')).find(|c| path == *c || path.starts_with(&format!("{c}/")))
131 +
        else {
132 +
            bail!("{path} is not under the job's checkpoints");
133 +
        };
134 +
        let below = path[root.len()..].trim_start_matches('/');
135 +
        if below.split('/').any(|p| p == "." || p == ".." || (p.is_empty() && !below.is_empty())) {
136 +
            bail!("{path}: . or .. in a checkpoint path");
137 +
        }
138 +
        let Some(base) = place(root) else { continue };
139 +
        let at = if below.is_empty() { base } else { base.join(below) };
140 +
        match kind {
141 +
            DIR => {
142 +
                std::fs::create_dir_all(&at)?;
143 +
                n.dirs += 1;
144 +
            }
145 +
            FILE => {
146 +
                if let Some(parent) = at.parent() {
147 +
                    std::fs::create_dir_all(parent)?;
148 +
                }
149 +
                // Never through a link already there.
150 +
                if std::fs::symlink_metadata(&at).is_ok_and(|m| !m.is_file()) {
151 +
                    bail!("{path}: something other than a file is in the way");
152 +
                }
153 +
                std::fs::write(&at, &data)?;
154 +
                let f = std::fs::File::options().write(true).open(&at)?;
155 +
                crate::portable::set_mode(&at, mode & 0o777)?;
156 +
                f.set_modified(SystemTime::UNIX_EPOCH + Duration::new(secs, nanos))?;
157 +
                n.files += 1;
158 +
                n.bytes += data.len() as u64;
159 +
            }
160 +
            other => bail!("{path}: unknown entry kind {other}"),
161 +
        }
162 +
    }
163 +
    Ok(n)
164 +
}
165 +
166 +
#[cfg(test)]
167 +
mod tests {
168 +
    use super::*;
169 +
170 +
    #[test]
171 +
    fn a_checkpoint_comes_back_with_its_times_and_only_where_it_belongs() {
172 +
        let from = tempfile::tempdir().unwrap();
173 +
        let target = from.path().join("target");
174 +
        std::fs::create_dir_all(target.join("debug/deps")).unwrap();
175 +
        std::fs::write(target.join("debug/deps/libx.rlib"), b"built").unwrap();
176 +
        std::fs::write(target.join("debug/.fingerprint"), b"fp").unwrap();
177 +
        let old = SystemTime::UNIX_EPOCH + Duration::new(1_700_000_000, 123_000_000);
178 +
        std::fs::File::options().write(true).open(target.join("debug/deps/libx.rlib")).unwrap().set_modified(old).unwrap();
179 +
        let (bytes, n) = pack(&[("/src/target".into(), target.clone())]).unwrap();
180 +
        assert_eq!((n.files, n.dirs, n.bytes), (2, 3, 7));
181 +
182 +
        let to = tempfile::tempdir().unwrap();
183 +
        let place = |root: &str| (root == "/src/target").then(|| to.path().join("t"));
184 +
        let back = unpack(&bytes, &["/src/target".into()], place).unwrap();
185 +
        assert_eq!(back, n);
186 +
        let restored = to.path().join("t/debug/deps/libx.rlib");
187 +
        assert_eq!(std::fs::read(&restored).unwrap(), b"built");
188 +
        assert_eq!(std::fs::metadata(&restored).unwrap().modified().unwrap(), old, "cargo reads the times");
189 +
190 +
        // Not the job's checkpoint, or a path that climbs out: refused.
191 +
        assert!(unpack(&bytes, &["/src/other".into()], place).is_err());
192 +
        let mut evil = MAGIC.to_vec();
193 +
        let meta = std::fs::metadata(target.join("debug/.fingerprint")).unwrap();
194 +
        entry(&mut evil, FILE, "/src/target/../escape", &meta, b"x");
195 +
        assert!(unpack(&evil, &["/src/target".into()], place).is_err());
196 +
        assert!(!to.path().join("escape").exists());
197 +
        assert!(unpack(b"junk", &["/src/target".into()], place).is_err());
198 +
    }
199 +
200 +
    #[test]
201 +
    fn a_checkpoint_under_a_disk_mount_is_in_the_mount_s_directory() {
202 +
        let p = flow_core::Pipeline::parse(
203 +
            "schema-version 1\nsandbox \"w\" {\n    runtime \"wasm\"\n}\njob \"j\" {\n    sandbox \"w\"\n    mount {\n        source \"ephemeral://disk\"\n        target \"/src/target\"\n        size \"1GiB\"\n    }\n    checkpoint \"/src/target\" \"/src/state\"\n    exec \"x.wasm\"\n}\n",
204 +
        )
205 +
        .unwrap();
206 +
        let job = &p.jobs["j"];
207 +
        assert_eq!(job.checkpoints, ["/src/state", "/src/target"]);
208 +
        let (tree, scratch) = (Path::new("/w/tree"), Path::new("/w"));
209 +
        assert_eq!(host_dir("/src/target", job, tree, scratch, true), Some(crate::syscall::disk_dir(scratch, 0)));
210 +
        assert_eq!(host_dir("/src/state", job, tree, scratch, true), Some(PathBuf::from("/w/tree/state")));
211 +
    }
212 +
}
added crates/flow/src/deps.rs +129 −0
1 +
//! `flow deps check` (#101): whether every crate a `Cargo.lock` needs is
2 +
//! vendored in the build image at that version. Jobs run with no network,
3 +
//! so a crate the image lacks fails the run with "no matching package";
4 +
//! this says so before the push, naming the versions the image does have.
5 +
6 +
use std::collections::{BTreeMap, BTreeSet};
7 +
use std::path::Path;
8 +
9 +
/// A crate at a version.
10 +
pub type Crate = (String, String);
11 +
12 +
/// The packages of a `Cargo.lock` that come from somewhere (a registry or
13 +
/// git): the ones `cargo vendor` copies. A workspace's own members have no
14 +
/// `source` and are left out.
15 +
pub fn from_lock(text: &str) -> Result<BTreeSet<Crate>, String> {
16 +
    let lock: toml::Value = text.parse().map_err(|e| format!("not a Cargo.lock: {e}"))?;
17 +
    let mut out = BTreeSet::new();
18 +
    for p in lock.get("package").and_then(|p| p.as_array()).into_iter().flatten() {
19 +
        let (Some(name), Some(version)) = (p.get("name").and_then(|v| v.as_str()), p.get("version").and_then(|v| v.as_str())) else {
20 +
            continue;
21 +
        };
22 +
        if p.get("source").is_some() {
23 +
            out.insert((name.to_string(), version.to_string()));
24 +
        }
25 +
    }
26 +
    Ok(out)
27 +
}
28 +
29 +
/// A crate list as `flow deps list` and the image's `crates.txt` write it:
30 +
/// `NAME VERSION` a line; blank lines and `#` comments are skipped.
31 +
pub fn from_list(text: &str) -> BTreeSet<Crate> {
32 +
    text.lines()
33 +
        .map(str::trim)
34 +
        .filter(|l| !l.is_empty() && !l.starts_with('#'))
35 +
        .filter_map(|l| {
36 +
            let mut w = l.split_whitespace();
37 +
            Some((w.next()?.to_string(), w.next()?.to_string()))
38 +
        })
39 +
        .collect()
40 +
}
41 +
42 +
/// The crates of a `cargo vendor` directory (the image's `/opt/vendor`):
43 +
/// each subdirectory's `Cargo.toml` names its package and version.
44 +
pub fn from_vendor(dir: &Path) -> Result<BTreeSet<Crate>, String> {
45 +
    let entries = std::fs::read_dir(dir).map_err(|e| format!("{}: {e}", dir.display()))?;
46 +
    let mut out = BTreeSet::new();
47 +
    for e in entries.flatten() {
48 +
        let manifest = e.path().join("Cargo.toml");
49 +
        let Ok(text) = std::fs::read_to_string(&manifest) else { continue };
50 +
        let Ok(v) = text.parse::<toml::Value>() else { continue };
51 +
        let pkg = v.get("package");
52 +
        let field = |k: &str| pkg.and_then(|p| p.get(k)).and_then(|x| x.as_str()).map(str::to_string);
53 +
        if let (Some(n), Some(ver)) = (field("name"), field("version")) {
54 +
            out.insert((n, ver));
55 +
        }
56 +
    }
57 +
    Ok(out)
58 +
}
59 +
60 +
/// What `need` lacks from `have`, each with the versions of that crate
61 +
/// `have` does hold (empty when it holds none).
62 +
pub fn missing(need: &BTreeSet<Crate>, have: &BTreeSet<Crate>) -> Vec<(Crate, Vec<String>)> {
63 +
    let mut versions: BTreeMap<&str, Vec<String>> = BTreeMap::new();
64 +
    for (n, v) in have {
65 +
        versions.entry(n.as_str()).or_default().push(v.clone());
66 +
    }
67 +
    need.iter().filter(|c| !have.contains(*c)).map(|c| (c.clone(), versions.get(c.0.as_str()).cloned().unwrap_or_default())).collect()
68 +
}
69 +
70 +
/// A crate list in the form [`from_list`] reads.
71 +
pub fn render_list(crates: &BTreeSet<Crate>) -> String {
72 +
    crates.iter().map(|(n, v)| format!("{n} {v}\n")).collect()
73 +
}
74 +
75 +
#[cfg(test)]
76 +
mod tests {
77 +
    use super::*;
78 +
79 +
    const LOCK: &str = r#"
80 +
version = 4
81 +
82 +
[[package]]
83 +
name = "mine"
84 +
version = "0.1.0"
85 +
dependencies = ["rustls"]
86 +
87 +
[[package]]
88 +
name = "rustls"
89 +
version = "0.23.45"
90 +
source = "registry+https://github.com/rust-lang/crates.io-index"
91 +
92 +
[[package]]
93 +
name = "tokio-rustls"
94 +
version = "0.26.5"
95 +
source = "registry+https://github.com/rust-lang/crates.io-index"
96 +
97 +
[[package]]
98 +
name = "deranged"
99 +
version = "0.5.8"
100 +
source = "registry+https://github.com/rust-lang/crates.io-index"
101 +
"#;
102 +
103 +
    #[test]
104 +
    fn names_what_the_image_lacks_and_what_it_has_instead() {
105 +
        let need = from_lock(LOCK).unwrap();
106 +
        assert_eq!(need.len(), 3, "the workspace's own package is not vendored");
107 +
        let have = from_list("# the image\nrustls 0.23.45\ntokio-rustls 0.26.4\n\n");
108 +
        let m = missing(&need, &have);
109 +
        assert_eq!(
110 +
            m,
111 +
            vec![(("deranged".into(), "0.5.8".into()), vec![]), (("tokio-rustls".into(), "0.26.5".into()), vec!["0.26.4".into()]),]
112 +
        );
113 +
        assert!(missing(&need, &need).is_empty());
114 +
        assert_eq!(from_list(&render_list(&need)), need);
115 +
        assert!(from_lock("not = [toml").is_err());
116 +
    }
117 +
118 +
    #[test]
119 +
    fn reads_a_vendor_directory() {
120 +
        let dir = tempfile::tempdir().unwrap();
121 +
        for (d, n, v) in [("rustls-0.23.45", "rustls", "0.23.45"), ("pre-1.0.0-rc.1", "pre", "1.0.0-rc.1")] {
122 +
            std::fs::create_dir(dir.path().join(d)).unwrap();
123 +
            std::fs::write(dir.path().join(d).join("Cargo.toml"), format!("[package]\nname = \"{n}\"\nversion = \"{v}\"\n")).unwrap();
124 +
        }
125 +
        std::fs::create_dir(dir.path().join("junk")).unwrap();
126 +
        let got = from_vendor(dir.path()).unwrap();
127 +
        assert_eq!(got, from_list("rustls 0.23.45\npre 1.0.0-rc.1\n"));
128 +
    }
129 +
}
added crates/flow/src/egress.rs +328 −0
1 +
//! Allow-listed egress for a microVM job (oxygenflow#55). The VM has no
2 +
//! network device: `flow-init` serves a proxy on the guest's 127.0.0.1
3 +
//! and relays each connection over vsock to this end, a Unix socket that
4 +
//! Firecracker connects for the guest. Here each `CONNECT host:443` is
5 +
//! checked against the sandbox's allow-list, the host is resolved (the
6 +
//! guest never resolves), any address that is not public is refused, the
7 +
//! decision goes into the job log, and the bytes are relayed untouched (no
8 +
//! TLS is opened here).
9 +
10 +
#[cfg(unix)]
11 +
use std::io::{Read, Write};
12 +
use std::net::IpAddr;
13 +
#[cfg(unix)]
14 +
use std::net::{SocketAddr, TcpStream, ToSocketAddrs};
15 +
#[cfg(unix)]
16 +
use std::os::unix::net::{UnixListener, UnixStream};
17 +
use std::path::Path;
18 +
#[cfg(unix)]
19 +
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
20 +
use std::sync::mpsc;
21 +
#[cfg(unix)]
22 +
use std::sync::Arc;
23 +
#[cfg(unix)]
24 +
use std::time::Duration;
25 +
26 +
/// The vsock port the guest's relay connects to on the host (CID 2).
27 +
pub const VSOCK_PORT: u32 = 3128;
28 +
/// Where the job's tools find the proxy inside the guest.
29 +
pub const GUEST_PROXY: &str = "http://127.0.0.1:3128";
30 +
/// The only port relayed: TLS.
31 +
const PORT: u16 = 443;
32 +
#[cfg(unix)]
33 +
/// Connections at once, per job.
34 +
const MAX_OPEN: usize = 64;
35 +
#[cfg(unix)]
36 +
const CONNECT_TIMEOUT: Duration = Duration::from_secs(10);
37 +
#[cfg(unix)]
38 +
/// The longest request head read.
39 +
const MAX_HEAD: usize = 8 * 1024;
40 +
41 +
/// What a CONNECT asked for, or why it was refused.
42 +
#[derive(Debug, PartialEq, Eq)]
43 +
pub enum Ask {
44 +
    Connect { host: String, port: u16 },
45 +
    Refused(String),
46 +
}
47 +
48 +
/// Read a request head: `CONNECT host:port HTTP/1.x`, and only that.
49 +
pub fn parse_head(head: &str) -> Ask {
50 +
    let line = head.lines().next().unwrap_or("");
51 +
    let mut parts = line.split_whitespace();
52 +
    let (method, target, version) = (parts.next().unwrap_or(""), parts.next().unwrap_or(""), parts.next().unwrap_or(""));
53 +
    if method != "CONNECT" {
54 +
        return Ask::Refused(format!("only https is relayed (CONNECT), not {method:?}"));
55 +
    }
56 +
    if !version.starts_with("HTTP/1.") {
57 +
        return Ask::Refused("not an HTTP/1 request".into());
58 +
    }
59 +
    let Some((host, port)) = target.rsplit_once(':') else { return Ask::Refused(format!("CONNECT {target:?} names no port")) };
60 +
    let Ok(port) = port.parse::<u16>() else { return Ask::Refused(format!("CONNECT {target:?} has no valid port")) };
61 +
    let host = host.trim_start_matches('[').trim_end_matches(']').trim_end_matches('.').to_ascii_lowercase();
62 +
    if host.is_empty() {
63 +
        return Ask::Refused("CONNECT names no host".into());
64 +
    }
65 +
    Ask::Connect { host, port }
66 +
}
67 +
68 +
/// Whether the job may go to `host:port`: the port is 443, the host is a
69 +
/// name (an address is refused: the allow-list names hosts) the list allows.
70 +
pub fn decide(allow: &[String], host: &str, port: u16) -> Result<(), String> {
71 +
    if port != PORT {
72 +
        return Err(format!("only port {PORT} is relayed"));
73 +
    }
74 +
    if host.parse::<IpAddr>().is_ok() {
75 +
        return Err("an address, not a name: the allow-list names hosts".into());
76 +
    }
77 +
    if !flow_core::egress_allows(allow, host) {
78 +
        return Err("not in the sandbox's egress allow-list".into());
79 +
    }
80 +
    Ok(())
81 +
}
82 +
83 +
/// Whether an address is on the public internet: not loopback, private,
84 +
/// shared (CGNAT), link-local (cloud metadata among them), multicast,
85 +
/// unspecified or documentation, nor an IPv6 local range or a mapped form.
86 +
pub fn public(ip: IpAddr) -> bool {
87 +
    match ip {
88 +
        IpAddr::V4(v4) => {
89 +
            let o = v4.octets();
90 +
            !(v4.is_loopback()
91 +
                || v4.is_private()
92 +
                || v4.is_link_local()
93 +
                || v4.is_multicast()
94 +
                || v4.is_broadcast()
95 +
                || v4.is_unspecified()
96 +
                || v4.is_documentation()
97 +
                || o[0] == 0
98 +
                || (o[0] == 100 && (o[1] & 0xc0) == 64)
99 +
                || (o[0] == 192 && o[1] == 0 && o[2] == 0)
100 +
                || o[0] >= 240)
101 +
        }
102 +
        IpAddr::V6(v6) => {
103 +
            if let Some(v4) = v6.to_ipv4_mapped() {
104 +
                return public(IpAddr::V4(v4));
105 +
            }
106 +
            let s = v6.segments();
107 +
            !(v6.is_loopback()
108 +
                || v6.is_unspecified()
109 +
                || v6.is_multicast()
110 +
                || (s[0] & 0xfe00) == 0xfc00
111 +
                || (s[0] & 0xffc0) == 0xfe80
112 +
                || (s[0] == 0x64 && s[1] == 0xff9b)
113 +
                || (s[0] == 0x2001 && s[1] == 0x0db8))
114 +
        }
115 +
    }
116 +
}
117 +
118 +
#[cfg(unix)]
119 +
/// The proxy for one job: listening until dropped.
120 +
pub struct Proxy {
121 +
    stop: Arc<AtomicBool>,
122 +
    path: std::path::PathBuf,
123 +
}
124 +
125 +
#[cfg(unix)]
126 +
impl Drop for Proxy {
127 +
    fn drop(&mut self) {
128 +
        self.stop.store(true, Ordering::SeqCst);
129 +
        // Wake the accept loop so it sees the stop.
130 +
        let _ = UnixStream::connect(&self.path);
131 +
        let _ = std::fs::remove_file(&self.path);
132 +
    }
133 +
}
134 +
135 +
#[cfg(unix)]
136 +
/// Listen on `socket` (Firecracker's `uds_path` and `_3128`) for the
137 +
/// guest's connections; each decision is sent to `log` as a line.
138 +
pub fn start(socket: &Path, allow: Vec<String>, log: mpsc::Sender<String>) -> std::io::Result<Proxy> {
139 +
    let _ = std::fs::remove_file(socket);
140 +
    let listener = UnixListener::bind(socket)?;
141 +
    let stop = Arc::new(AtomicBool::new(false));
142 +
    let open = Arc::new(AtomicUsize::new(0));
143 +
    let allow = Arc::new(allow);
144 +
    let s = stop.clone();
145 +
    std::thread::spawn(move || {
146 +
        for conn in listener.incoming() {
147 +
            if s.load(Ordering::SeqCst) {
148 +
                break;
149 +
            }
150 +
            let Ok(conn) = conn else { continue };
151 +
            if open.load(Ordering::SeqCst) >= MAX_OPEN {
152 +
                let _ = refuse(conn, "too many connections at once");
153 +
                let _ = log.send(format!("egress: refused a connection: {MAX_OPEN} already open"));
154 +
                continue;
155 +
            }
156 +
            open.fetch_add(1, Ordering::SeqCst);
157 +
            let (allow, log, open) = (allow.clone(), log.clone(), open.clone());
158 +
            std::thread::spawn(move || {
159 +
                serve(conn, &allow, &log);
160 +
                open.fetch_sub(1, Ordering::SeqCst);
161 +
            });
162 +
        }
163 +
    });
164 +
    Ok(Proxy { stop, path: socket.to_path_buf() })
165 +
}
166 +
167 +
#[cfg(unix)]
168 +
fn refuse(mut conn: UnixStream, why: &str) -> std::io::Result<()> {
169 +
    let body = format!("{why}\n");
170 +
    write!(conn, "HTTP/1.1 403 Forbidden\r\nContent-Type: text/plain\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", body.len())
171 +
}
172 +
173 +
#[cfg(unix)]
174 +
/// Read up to the blank line that ends a request head.
175 +
fn read_head(conn: &mut UnixStream) -> std::io::Result<String> {
176 +
    let mut head = Vec::new();
177 +
    let mut byte = [0u8; 1];
178 +
    while !head.ends_with(b"\r\n\r\n") {
179 +
        if head.len() >= MAX_HEAD || conn.read(&mut byte)? == 0 {
180 +
            break;
181 +
        }
182 +
        head.push(byte[0]);
183 +
    }
184 +
    Ok(String::from_utf8_lossy(&head).into_owned())
185 +
}
186 +
187 +
#[cfg(unix)]
188 +
fn serve(mut conn: UnixStream, allow: &[String], log: &mpsc::Sender<String>) {
189 +
    let _ = conn.set_read_timeout(Some(Duration::from_secs(30)));
190 +
    let head = match read_head(&mut conn) {
191 +
        Ok(h) => h,
192 +
        Err(_) => return,
193 +
    };
194 +
    let (host, port) = match parse_head(&head) {
195 +
        Ask::Connect { host, port } => (host, port),
196 +
        Ask::Refused(why) => {
197 +
            let _ = log.send(format!("egress: refused: {why}"));
198 +
            let _ = refuse(conn, &why);
199 +
            return;
200 +
        }
201 +
    };
202 +
    if let Err(why) = decide(allow, &host, port) {
203 +
        let _ = log.send(format!("egress: refused {host}:{port}: {why}"));
204 +
        let _ = refuse(conn, &format!("{host}:{port}: {why}"));
205 +
        return;
206 +
    }
207 +
    let addrs: Vec<SocketAddr> = match (host.as_str(), port).to_socket_addrs() {
208 +
        Ok(a) => a.collect(),
209 +
        Err(e) => {
210 +
            let _ = log.send(format!("egress: refused {host}:{port}: it could not be resolved ({e})"));
211 +
            let _ = refuse(conn, &format!("{host} could not be resolved"));
212 +
            return;
213 +
        }
214 +
    };
215 +
    if let Some(bad) = addrs.iter().find(|a| !public(a.ip())) {
216 +
        let why = format!("it resolves to {}, which is not a public address", bad.ip());
217 +
        let _ = log.send(format!("egress: refused {host}:{port}: {why}"));
218 +
        let _ = refuse(conn, &format!("{host}: {why}"));
219 +
        return;
220 +
    }
221 +
    let out = addrs.iter().find_map(|a| TcpStream::connect_timeout(a, CONNECT_TIMEOUT).ok());
222 +
    let Some(out) = out else {
223 +
        let _ = log.send(format!("egress: {host}:{port} is allowed but did not answer"));
224 +
        let _ = conn.write_all(b"HTTP/1.1 502 Bad Gateway\r\nConnection: close\r\n\r\n");
225 +
        return;
226 +
    };
227 +
    let _ = log.send(format!("egress: allowed {host}:{port}"));
228 +
    if conn.write_all(b"HTTP/1.1 200 Connection established\r\n\r\n").is_err() {
229 +
        return;
230 +
    }
231 +
    let _ = conn.set_read_timeout(None);
232 +
    relay(conn, out);
233 +
}
234 +
235 +
#[cfg(unix)]
236 +
/// Copy both ways until either side closes.
237 +
fn relay(guest: UnixStream, out: TcpStream) {
238 +
    let (Ok(mut guest_r), Ok(mut out_w)) = (guest.try_clone(), out.try_clone()) else { return };
239 +
    let up = std::thread::spawn(move || {
240 +
        let _ = std::io::copy(&mut guest_r, &mut out_w);
241 +
        let _ = out_w.shutdown(std::net::Shutdown::Write);
242 +
    });
243 +
    let (mut out_r, mut guest_w) = (out, guest);
244 +
    let _ = std::io::copy(&mut out_r, &mut guest_w);
245 +
    let _ = guest_w.shutdown(std::net::Shutdown::Write);
246 +
    let _ = up.join();
247 +
}
248 +
249 +
/// The egress proxy serves a microVM's vsock, which only Linux has.
250 +
#[cfg(not(unix))]
251 +
pub struct Proxy;
252 +
253 +
#[cfg(not(unix))]
254 +
pub fn start(_socket: &Path, _allow: Vec<String>, _log: mpsc::Sender<String>) -> std::io::Result<Proxy> {
255 +
    Err(std::io::Error::new(std::io::ErrorKind::Unsupported, "egress serves microVMs, which run only on Linux"))
256 +
}
257 +
258 +
#[cfg(all(test, unix))]
259 +
mod tests {
260 +
    use super::*;
261 +
262 +
    fn allow() -> Vec<String> {
263 +
        vec!["*.googleapis.com".into(), "registry.opentofu.org".into()]
264 +
    }
265 +
266 +
    #[test]
267 +
    fn only_connect_to_an_allowed_name_on_443_is_let_through() {
268 +
        assert_eq!(
269 +
            parse_head("CONNECT storage.googleapis.com:443 HTTP/1.1\r\nHost: x\r\n\r\n"),
270 +
            Ask::Connect { host: "storage.googleapis.com".into(), port: 443 }
271 +
        );
272 +
        assert!(matches!(parse_head("GET http://example.com/ HTTP/1.1\r\n\r\n"), Ask::Refused(w) if w.contains("only https")));
273 +
        assert!(matches!(parse_head("CONNECT example.com HTTP/1.1\r\n\r\n"), Ask::Refused(w) if w.contains("no port")));
274 +
        assert_eq!(decide(&allow(), "storage.googleapis.com", 443), Ok(()));
275 +
        assert_eq!(decide(&allow(), "registry.opentofu.org", 443), Ok(()));
276 +
        assert!(decide(&allow(), "example.com", 443).unwrap_err().contains("allow-list"));
277 +
        assert!(decide(&allow(), "registry.opentofu.org", 22).unwrap_err().contains("only port 443"));
278 +
        assert!(decide(&allow(), "142.250.1.1", 443).unwrap_err().contains("names hosts"));
279 +
    }
280 +
281 +
    #[test]
282 +
    fn private_addresses_are_not_public() {
283 +
        for ip in [
284 +
            "127.0.0.1",
285 +
            "10.1.2.3",
286 +
            "192.168.1.1",
287 +
            "172.16.0.1",
288 +
            "169.254.169.254",
289 +
            "100.64.0.1",
290 +
            "0.0.0.0",
291 +
            "::1",
292 +
            "fd00::1",
293 +
            "fe80::1",
294 +
            "::ffff:10.0.0.1",
295 +
        ] {
296 +
            assert!(!public(ip.parse().unwrap()), "{ip}");
297 +
        }
298 +
        for ip in ["142.250.1.1", "2a00:1450:4009::200e"] {
299 +
            assert!(public(ip.parse().unwrap()), "{ip}");
300 +
        }
301 +
    }
302 +
303 +
    #[test]
304 +
    fn the_proxy_refuses_and_logs_a_host_not_allowed() {
305 +
        let dir = tempfile::tempdir().unwrap();
306 +
        let sock = dir.path().join("v.sock_3128");
307 +
        let (tx, rx) = mpsc::channel();
308 +
        let proxy = start(&sock, allow(), tx).unwrap();
309 +
        let ask = |head: &str| {
310 +
            let mut c = UnixStream::connect(&sock).unwrap();
311 +
            c.write_all(head.as_bytes()).unwrap();
312 +
            let mut answer = String::new();
313 +
            c.read_to_string(&mut answer).unwrap();
314 +
            answer
315 +
        };
316 +
        let a = ask("CONNECT example.com:443 HTTP/1.1\r\n\r\n");
317 +
        assert!(a.starts_with("HTTP/1.1 403") && a.contains("not in the sandbox's egress allow-list"), "{a}");
318 +
        assert_eq!(
319 +
            rx.recv_timeout(Duration::from_secs(5)).unwrap(),
320 +
            "egress: refused example.com:443: not in the sandbox's egress allow-list"
321 +
        );
322 +
        let a = ask("GET http://storage.googleapis.com/ HTTP/1.1\r\n\r\n");
323 +
        assert!(a.starts_with("HTTP/1.1 403"), "{a}");
324 +
        assert!(rx.recv_timeout(Duration::from_secs(5)).unwrap().contains("only https"));
325 +
        drop(proxy);
326 +
        assert!(!sock.exists(), "the socket goes with the job");
327 +
    }
328 +
}
modified crates/flow/src/facts.rs +66 −9
…
16 unchanged lines
17 17
        load_x100: load_x100(),
18 18
        speed,
19 19
        speed_class: speed.map(Facts::class_of),
20 -
        kernel: std::fs::read_to_string("/proc/sys/kernel/osrelease").ok().map(|s| s.trim().to_string()),
20 +
        kernel: kernel(),
21 21
    }
22 22
}
23 23
24 -
/// Free space for an unprivileged writer on the file system holding `path`, in MiB.
24 +
/// The kernel's release, as uname says. Not read from /proc: an Android
25 +
/// app may not read it there, and is refused (and logged) every claim.
26 +
#[cfg(unix)]
27 +
pub fn kernel() -> Option<String> {
28 +
    let mut u: libc::utsname = unsafe { std::mem::zeroed() };
29 +
    // SAFETY: `u` is a writable utsname.
30 +
    if unsafe { libc::uname(&mut u) } != 0 {
31 +
        return None;
32 +
    }
33 +
    // SAFETY: uname writes NUL-terminated strings.
34 +
    let release = unsafe { std::ffi::CStr::from_ptr(u.release.as_ptr()) };
35 +
    Some(release.to_string_lossy().into_owned())
36 +
}
37 +
38 +
/// Windows' version (`Windows 10.0.26100`), from `RtlGetVersion`, which
39 +
/// unlike `GetVersionEx` does not depend on the program's manifest.
40 +
#[cfg(windows)]
41 +
pub fn kernel() -> Option<String> {
42 +
    use windows_sys::Win32::System::SystemInformation::OSVERSIONINFOW;
43 +
    // SAFETY: an all-zero OSVERSIONINFOW is valid; its size is set before the call.
44 +
    let mut v: OSVERSIONINFOW = unsafe { std::mem::zeroed() };
45 +
    v.dwOSVersionInfoSize = std::mem::size_of::<OSVERSIONINFOW>() as u32;
46 +
    // SAFETY: `v` is a writable OSVERSIONINFOW with its size set.
47 +
    let status = unsafe { windows_sys::Wdk::System::SystemServices::RtlGetVersion(&mut v) };
48 +
    (status == 0).then(|| format!("Windows {}.{}.{}", v.dwMajorVersion, v.dwMinorVersion, v.dwBuildNumber))
49 +
}
50 +
51 +
/// Free space for an unprivileged writer on the file system holding `path`,
52 +
/// in MiB.
53 +
#[cfg(unix)]
25 54
pub fn free_disk_mib(path: &std::path::Path) -> Option<u64> {
26 55
    use std::os::unix::ffi::OsStrExt;
27 56
    let c = std::ffi::CString::new(path.as_os_str().as_bytes()).ok()?;
…
5 unchanged lines
33 62
    Some((s.f_bavail as u64).saturating_mul(s.f_frsize as u64) >> 20)
34 63
}
35 64
65 +
#[cfg(windows)]
66 +
pub fn free_disk_mib(path: &std::path::Path) -> Option<u64> {
67 +
    use std::os::windows::ffi::OsStrExt;
68 +
    let wide: Vec<u16> = path.as_os_str().encode_wide().chain(Some(0)).collect();
69 +
    let mut free: u64 = 0;
70 +
    // SAFETY: `wide` is a NUL-terminated path and `free` a writable u64;
71 +
    // the totals it may also fill are not asked for.
72 +
    let ok = unsafe {
73 +
        windows_sys::Win32::Storage::FileSystem::GetDiskFreeSpaceExW(wide.as_ptr(), &mut free, std::ptr::null_mut(), std::ptr::null_mut())
74 +
    };
75 +
    (ok != 0).then_some(free >> 20)
76 +
}
77 +
36 78
/// The disk the files under `dir` take (their blocks, not their sizes: a
37 79
/// job's disks are sparse), in MiB; links are not followed.
38 80
pub fn allocated_mib(dir: &std::path::Path) -> Option<u64> {
39 -
    use std::os::unix::fs::MetadataExt;
40 81
    let mut total: u64 = 0;
41 82
    let mut stack = vec![dir.to_path_buf()];
42 83
    while let Some(d) = stack.pop() {
…
2 unchanged lines
45 86
            if m.is_dir() {
46 87
                stack.push(e.path());
47 88
            } else {
48 -
                total = total.saturating_add(m.blocks() * 512);
89 +
                total = total.saturating_add(crate::portable::disk_bytes(&m));
49 90
            }
50 91
        }
51 92
    }
…
7 unchanged lines
59 100
    Some(kib / 1024)
60 101
}
61 102
62 -
/// The 1-minute load average from /proc/loadavg, times 100.
103 +
/// The 1-minute load average, times 100, from sysinfo (the kernel's own
104 +
/// numbers behind /proc/loadavg, which an Android app may not read); not
105 +
/// measured on Windows.
106 +
#[cfg(not(any(target_os = "linux", target_os = "android")))]
63 107
pub fn load_x100() -> Option<u32> {
64 -
    let text = std::fs::read_to_string("/proc/loadavg").ok()?;
65 -
    let first: f64 = text.split_whitespace().next()?.parse().ok()?;
66 -
    Some((first * 100.0).round() as u32)
108 +
    None
109 +
}
110 +
111 +
#[cfg(any(target_os = "linux", target_os = "android"))]
112 +
pub fn load_x100() -> Option<u32> {
113 +
    let mut i: libc::sysinfo = unsafe { std::mem::zeroed() };
114 +
    // SAFETY: `i` is a writable sysinfo.
115 +
    if unsafe { libc::sysinfo(&mut i) } != 0 {
116 +
        return None;
117 +
    }
118 +
    // Fixed point with 16 fraction bits (SI_LOAD_SHIFT).
119 +
    Some(((i.loads[0] as f64) / 65536.0 * 100.0).round() as u32)
67 120
}
68 121
69 122
/// The benchmark, one thread: BLAKE3 over 256 MiB (spec §11.1), and a sort
…
30 unchanged lines
100 153
    #[test]
101 154
    fn the_cheap_facts_are_read_and_classes_double() {
102 155
        assert!(free_disk_mib(std::path::Path::new("/")).is_some());
103 -
        assert!(load_x100().is_some());
156 +
        // The same numbers /proc gives, read without it (an Android app may not).
157 +
        let proc_load: f64 = std::fs::read_to_string("/proc/loadavg").unwrap().split_whitespace().next().unwrap().parse().unwrap();
158 +
        let load = load_x100().unwrap() as f64 / 100.0;
159 +
        assert!((load - proc_load).abs() < 1.0 + proc_load / 4.0, "sysinfo {load}, /proc {proc_load}");
160 +
        assert_eq!(kernel().unwrap(), std::fs::read_to_string("/proc/sys/kernel/osrelease").unwrap().trim());
104 161
        assert!(peak_rss_mib(std::process::id()).is_some_and(|m| m > 0));
105 162
        let dir = tempfile::tempdir().unwrap();
106 163
        let f = std::fs::File::create(dir.path().join("sparse")).unwrap();
…
8 unchanged lines
modified crates/flow/src/home.rs +122 −7
…
39 unchanged lines
40 40
/// Where `flow runner register` keeps the runner's hub and credential.
41 41
pub const RUNNER_TOML: &str = "runner.toml";
42 42
43 +
/// The hub's CA certificate as `flow runner register --ca` keeps it (#98).
44 +
pub const CA_PEM: &str = "hub-ca.pem";
45 +
43 46
#[derive(Clone, Debug, Deserialize)]
44 47
pub struct RunnerConfig {
45 48
    pub hub: String,
…
11 unchanged lines
57 60
    pub memory_mib: Option<u32>,
58 61
    #[serde(default)]
59 62
    pub cpus: Option<u32>,
63 +
    /// Sharing images with other runners (docs/jetstream.md §8.1): absent,
64 +
    /// the runner neither serves nor asks peers.
65 +
    #[serde(default)]
66 +
    pub swarm: Option<SwarmConfig>,
67 +
    /// The build cache's size limit in GiB (#28; default 64).
68 +
    #[serde(default)]
69 +
    pub cache_gb: Option<u64>,
70 +
    /// The share of its disk, in percent, a kept build cache may fill
71 +
    /// (oxygenflow#72; default 60): over it, the files the job did not use
72 +
    /// are pruned, oldest first, and a cache still over it is not kept.
73 +
    #[serde(default)]
74 +
    pub cache_prune_percent: Option<u8>,
75 +
    /// Let microVM jobs reach the hosts their sandbox allows, through this
76 +
    /// runner (oxygenflow#55); off unless set. The runner then offers the
77 +
    /// `egress` label, which such jobs need.
78 +
    #[serde(default)]
79 +
    pub egress: bool,
80 +
    /// Sign each job's receipt with a fresh key the hub certifies for that
81 +
    /// job, just before signing (keyless, oxygenhub#241), rather than with
82 +
    /// the runner's own key; off unless set. When the hub cannot certify,
83 +
    /// the runner's key signs, and the job's log says why.
84 +
    #[serde(default)]
85 +
    pub keyless: bool,
86 +
    /// Serve this runner's metrics at this address, e.g. `127.0.0.1:9464`
87 +
    /// (oxygenflow#35); off unless set.
88 +
    #[serde(default)]
89 +
    pub metrics: Option<String>,
90 +
    /// A bearer token a scrape of the metrics must send.
91 +
    #[serde(default)]
92 +
    pub metrics_token: Option<String>,
93 +
    /// `[runner.update]`: whether this runner takes updates (#85).
94 +
    #[serde(default)]
95 +
    pub update: Option<UpdateConfig>,
96 +
}
97 +
98 +
/// `[runner.update]` in flow.toml (docs/jetstream.md §8.2): the runner's
99 +
/// last word on updates, whatever the hub's policy says.
100 +
#[derive(Clone, Debug, Deserialize)]
101 +
pub struct UpdateConfig {
102 +
    /// `false` refuses every update, so a site can hold a machine back.
103 +
    #[serde(default = "yes")]
104 +
    pub allow: bool,
105 +
}
106 +
107 +
fn yes() -> bool {
108 +
    true
60 109
}
61 110
111 +
impl RunnerConfig {
112 +
    /// The labels offered to the hub: the configured ones, and `egress`
113 +
    /// when egress is on.
114 +
    pub fn offered_labels(&self) -> Vec<String> {
115 +
        let mut l = self.labels.clone();
116 +
        if self.egress && !l.iter().any(|x| x == flow_core::EGRESS_LABEL) {
117 +
            l.push(flow_core::EGRESS_LABEL.to_string());
118 +
        }
119 +
        l
120 +
    }
121 +
}
122 +
123 +
/// `[runner.swarm]` in flow.toml, or `[swarm]` in runner.toml.
124 +
#[derive(Clone, Debug, Deserialize)]
125 +
pub struct SwarmConfig {
126 +
    /// Where peers reach this runner, e.g. `0.0.0.0:7401`.
127 +
    pub listen: String,
128 +
    /// The address peers are given (default: `listen`).
129 +
    #[serde(default)]
130 +
    pub advertise: Option<String>,
131 +
    /// Peers are only ever in the same site.
132 +
    #[serde(default)]
133 +
    pub site: String,
134 +
    /// The most this runner sends to peers, MiB a second (default 50).
135 +
    #[serde(default)]
136 +
    pub upload_mib_s: Option<u32>,
137 +
}
138 +
62 139
#[derive(Clone, Debug, Default, Deserialize)]
63 140
pub struct Config {
64 141
    pub microvm: Option<MicrovmConfig>,
…
28 unchanged lines
93 170
        let registered = root.join(RUNNER_TOML);
94 171
        if config.runner.is_none() && registered.is_file() {
95 172
            let text = std::fs::read_to_string(&registered).with_context(|| format!("read {}", registered.display()))?;
96 -
            config.runner = Some(toml::from_str(&text).with_context(|| format!("parse {}", registered.display()))?);
173 +
            let mut runner: RunnerConfig = toml::from_str(&text).with_context(|| format!("parse {}", registered.display()))?;
174 +
            // A relative CA there is the home's (#98): register keeps its
175 +
            // copy there, and the service may start in any directory.
176 +
            runner.ca = runner.ca.map(|ca| if ca.is_absolute() { ca } else { root.join(ca) });
177 +
            config.runner = Some(runner);
97 178
        }
98 179
        for d in ["cas", "work", "repos"] {
99 180
            std::fs::create_dir_all(root.join(d)).with_context(|| format!("create {}", root.join(d).display()))?;
…
20 unchanged lines
120 201
        self.root.join("cas").join(format!("{hash}.json"))
121 202
    }
122 203
123 -
    /// The architecture an image was imported for, when its manifest says.
204 +
    /// The architecture an image is for: as its manifest records, else
205 +
    /// read from the image's own shell (spec §11.2) and recorded, so an
206 +
    /// image fetched or imported without one is read once and a runner
207 +
    /// declines what it cannot run instead of failing it. `None` when the
208 +
    /// image is not here or names no architecture.
124 209
    pub fn image_arch(&self, hash: &str) -> Option<String> {
125 -
        let text = std::fs::read_to_string(self.image_manifest(hash)).ok()?;
126 -
        let v: serde_json::Value = serde_json::from_str(&text).ok()?;
127 -
        v["arch"].as_str().map(String::from)
210 +
        let path = self.image_manifest(hash);
211 +
        let mut v: serde_json::Value =
212 +
            std::fs::read_to_string(&path).ok().and_then(|t| serde_json::from_str(&t).ok()).unwrap_or_else(|| serde_json::json!({}));
213 +
        if let Some(a) = v["arch"].as_str() {
214 +
            return Some(a.to_string());
215 +
        }
216 +
        let image = self.rootfs(hash);
217 +
        if v.get("arch_read_ms").is_some() || !image.is_file() {
218 +
            return None;
219 +
        }
220 +
        let arch = jet::rootfs::arch_of_file(&image);
221 +
        let o = v.as_object_mut()?;
222 +
        if let Some(a) = arch {
223 +
            o.insert("arch".into(), a.into());
224 +
        }
225 +
        let now = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map(|d| d.as_millis() as u64).unwrap_or(0);
226 +
        o.insert("arch_read_ms".into(), now.into());
227 +
        let tmp = path.with_extension("json.part");
228 +
        if std::fs::write(&tmp, v.to_string()).is_ok() {
229 +
            let _ = std::fs::rename(&tmp, &path);
230 +
        }
231 +
        arch.map(String::from)
128 232
    }
129 233
130 234
    /// Every image with a known architecture (spec §11.2).
…
98 unchanged lines
229 333
        let tmp = self.root.join("runner.key.tmp");
230 334
        {
231 335
            use std::io::Write;
232 -
            use std::os::unix::fs::OpenOptionsExt;
233 -
            let mut f = std::fs::OpenOptions::new().write(true).create(true).truncate(true).mode(0o600).open(&tmp)?;
336 +
            let mut f = crate::portable::private(std::fs::OpenOptions::new().write(true).create(true).truncate(true)).open(&tmp)?;
234 337
            f.write_all(&secret.encode(&identity.hash()))?;
235 338
            f.sync_all()?;
236 339
        }
…
20 unchanged lines
257 360
        let sig = b.secret.sign(oxy_core::identity::Domain::Attest, &b.identity.hash(), &oxy_core::hash::ObjectRef([1; 32]));
258 361
        oxy_core::identity::verify(&a.identity, &sig, oxy_core::identity::Domain::Attest, &oxy_core::hash::ObjectRef([1; 32])).unwrap();
259 362
    }
363 +
364 +
    /// #98: runner.toml's relative CA is the home's, wherever flow starts.
365 +
    #[test]
366 +
    fn a_registered_runners_relative_ca_is_in_its_home() {
367 +
        let dir = tempfile::tempdir().unwrap();
368 +
        std::fs::write(dir.path().join(RUNNER_TOML), "hub = \"https://hub\"\ntoken = \"t\"\nca = \"hub-ca.pem\"\n").unwrap();
369 +
        let home = Home::locate(Some(dir.path())).unwrap();
370 +
        assert_eq!(home.config.runner.unwrap().ca, Some(dir.path().join("hub-ca.pem")));
371 +
        std::fs::write(dir.path().join(RUNNER_TOML), "hub = \"https://hub\"\ntoken = \"t\"\nca = \"/etc/ssl/hub.pem\"\n").unwrap();
372 +
        let home = Home::locate(Some(dir.path())).unwrap();
373 +
        assert_eq!(home.config.runner.unwrap().ca, Some(PathBuf::from("/etc/ssl/hub.pem")));
374 +
    }
260 375
}
modified crates/flow/src/images.rs +117 −14
…
14 unchanged lines
15 15
16 16
/// Requests in flight at once.
17 17
const PARALLEL: usize = 8;
18 +
/// Chunk names per `missing` question.
19 +
const ASK: usize = 20_000;
18 20
19 21
/// A hub's image endpoints, with a token.
20 22
pub struct Client {
23 +
    root: String,
21 24
    base: String,
22 25
    token: String,
23 26
    agent: ureq::Agent,
…
3 unchanged lines
27 30
pub struct Moved {
28 31
    pub chunks: usize,
29 32
    pub bytes: u64,
33 +
    /// Of `bytes`, those that came from peers rather than the hub.
34 +
    pub peer_bytes: u64,
30 35
    pub elapsed: Duration,
31 36
}
32 37
…
1 unchanged lines
34 39
    pub fn new(hub: &str, token: &str, ca: Option<&Path>) -> Result<Client> {
35 40
        let tls = oxy_core::protocol::tls_client_config(ca).map_err(|e| anyhow::anyhow!("{e}"))?;
36 41
        let agent = ureq::AgentBuilder::new().tls_config(tls).timeout(Duration::from_secs(120)).build();
37 -
        Ok(Client { base: format!("{}/api/v1", hub.trim_end_matches('/')), token: token.into(), agent })
42 +
        let root = hub.trim_end_matches('/').to_string();
43 +
        Ok(Client { base: format!("{root}/api/v1"), root, token: token.into(), agent })
38 44
    }
39 45
40 46
    fn auth(&self) -> String {
…
24 unchanged lines
65 71
        }
66 72
    }
67 73
68 -
    fn chunk(&self, hash: &str) -> Result<Vec<u8>> {
74 +
    /// The hub's identity (`/.well-known/oxyhub-key`): the key its swarm
75 +
    /// tickets are signed with.
76 +
    pub fn hub_identity(&self) -> Result<oxy_core::identity::Identity> {
77 +
        let url = format!("{}/.well-known/oxyhub-key", self.root);
78 +
        let r = self.agent.get(&url).call().map_err(|e| Self::fail(&url, e))?;
79 +
        let mut out = Vec::new();
80 +
        std::io::Read::read_to_end(&mut r.into_reader(), &mut out)?;
81 +
        oxy_core::identity::Identity::decode(&out).map_err(|e| anyhow::anyhow!("{url}: not a hub identity: {e}"))
82 +
    }
83 +
84 +
    /// Tell the hub's tracker this runner is fetching image `hash`, or has
85 +
    /// stopped: the peers to ask and the ticket they admit.
86 +
    pub fn announce(&self, hash: &str, state: &str) -> Result<flow_core::api::Announced> {
87 +
        let url = format!("{}/flow/swarm/announce", self.base);
88 +
        let body = flow_core::api::Announce { image: hash.into(), state: state.into() };
89 +
        Ok(self.agent.post(&url).set("Authorization", &self.auth()).send_json(&body).map_err(|e| Self::fail(&url, e))?.into_json()?)
90 +
    }
91 +
92 +
    pub(crate) fn chunk(&self, hash: &str) -> Result<Vec<u8>> {
69 93
        let url = format!("{}/flow/images/chunks/{hash}", self.base);
70 94
        let r = self.agent.get(&url).set("Authorization", &self.auth()).call().map_err(|e| Self::fail(&url, e))?;
71 95
        let mut out = Vec::new();
…
14 unchanged lines
86 110
            bail!("{} holds an image named {}, not {}: the file was changed", path.display(), &m.hash[..12], &hash[..12]);
87 111
        }
88 112
        let mut missing: HashSet<String> = HashSet::new();
89 -
        for ask in m.needed().chunks(50_000) {
113 +
        // 20,000 names is about 1.3 MB of JSON: under a hub's 2 MB default
114 +
        // body limit, which refused 50,000 at a time (oxygenhub#147).
115 +
        for ask in m.needed().chunks(ASK) {
90 116
            let url = format!("{}/flow/images/missing", self.base);
91 117
            let r: serde_json::Value = self
92 118
                .agent
…
34 unchanged lines
127 153
            return Err(e);
128 154
        }
129 155
        let url = format!("{}/flow/images/{hash}", self.base);
130 -
        self.agent.put(&url).set("Authorization", &self.auth()).send_json(&m).map_err(|e| Self::fail(&url, e))?;
156 +
        // The hub reads the whole image before it answers, which on a busy
157 +
        // disk takes far longer than the agent's 120 s (oxygenhub#341).
158 +
        self.agent
159 +
            .put(&url)
160 +
            .timeout(Duration::from_secs(3600))
161 +
            .set("Authorization", &self.auth())
162 +
            .send_json(&m)
163 +
            .map_err(|e| Self::fail(&url, e))?;
131 164
        let bytes = *bytes.lock().unwrap();
132 -
        Ok(Moved { chunks: count, bytes, elapsed: start.elapsed() })
165 +
        Ok(Moved { chunks: count, bytes, peer_bytes: 0, elapsed: start.elapsed() })
133 166
    }
134 167
135 168
    /// Fetch image `hash` into `home`: the chunks its cache lacks, the
…
6 unchanged lines
142 175
            bail!("the hub's manifest for {} names {}", &hash[..12], &m.hash[..12]);
143 176
        }
144 177
        let cache = chunk_cache(home);
178 +
        save_manifest(&home.root, &m)?;
145 179
        let wanted: Vec<String> = m.needed().into_iter().filter(|h| !cache_path(&cache, h).is_file()).map(String::from).collect();
146 180
        let (bytes, failed) = (Mutex::new(0u64), Mutex::new(None::<anyhow::Error>));
147 181
        let next = Mutex::new(wanted.iter());
…
26 unchanged lines
174 208
        if let Some(e) = failed.into_inner().unwrap() {
175 209
            return Err(e);
176 210
        }
177 -
        let out = home.rootfs(hash);
178 -
        std::fs::create_dir_all(out.parent().expect("the cas directory"))?;
179 -
        images::assemble(&m, &out, |c| Ok(std::fs::read(cache_path(&cache, c.hash()))?))
180 -
            .with_context(|| format!("write image {}", &hash[..12]))?;
181 -
        if let Some(arch) = &m.arch {
182 -
            let manifest = serde_json::json!({ "arch": arch, "imported_ms": std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map(|d| d.as_millis() as u64).unwrap_or(0), "from": "hub" });
183 -
            std::fs::write(home.image_manifest(hash), serde_json::to_vec(&manifest)?)?;
211 +
        write_image(home, &m)?;
212 +
        Ok(Some(Moved { chunks: wanted.len(), bytes: bytes.into_inner().unwrap(), peer_bytes: 0, elapsed: start.elapsed() }))
213 +
    }
214 +
}
215 +
216 +
/// Write image `m` into `home` from the chunk cache, checked against its
217 +
/// hash, and record its architecture.
218 +
pub(crate) fn write_image(home: &Home, m: &Manifest) -> Result<()> {
219 +
    let cache = chunk_cache(home);
220 +
    let out = home.rootfs(&m.hash);
221 +
    std::fs::create_dir_all(out.parent().expect("the cas directory"))?;
222 +
    images::assemble(m, &out, |c| Ok(std::fs::read(cache_path(&cache, c.hash()))?))
223 +
        .with_context(|| format!("write image {}", &m.hash[..12]))?;
224 +
    if let Some(arch) = &m.arch {
225 +
        let manifest = serde_json::json!({ "arch": arch, "imported_ms": std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map(|d| d.as_millis() as u64).unwrap_or(0), "from": "hub" });
226 +
        std::fs::write(home.image_manifest(&m.hash), serde_json::to_vec(&manifest)?)?;
227 +
    }
228 +
    Ok(())
229 +
}
230 +
231 +
impl Client {
232 +
    /// Tell the hub which images this home holds, and send the manifest of
233 +
    /// each it keeps none for (oxygenflow#94, jetstream.md §8.1.1): the
234 +
    /// swarm can then carry an image nobody pushed. A manifest not kept here
235 +
    /// yet is made by reading the image once, and kept. How many were sent;
236 +
    /// none from a hub that predates it.
237 +
    pub fn offer_held(&self, home: &Home) -> Result<usize> {
238 +
        let images = home.images();
239 +
        if images.is_empty() {
240 +
            return Ok(0);
184 241
        }
185 -
        Ok(Some(Moved { chunks: wanted.len(), bytes: bytes.into_inner().unwrap(), elapsed: start.elapsed() }))
242 +
        let url = format!("{}/flow/images/held", self.base);
243 +
        let wanted: flow_core::api::WantedManifests =
244 +
            match self.agent.post(&url).set("Authorization", &self.auth()).send_json(flow_core::api::HeldImages { images }) {
245 +
                Ok(r) => r.into_json()?,
246 +
                // A hub that predates it: no such route (404), or a route
247 +
                // that takes other methods only (405).
248 +
                Err(ureq::Error::Status(404 | 405, _)) => return Ok(0),
249 +
                Err(e) => return Err(Self::fail(&url, e)),
250 +
            };
251 +
        let mut sent = 0;
252 +
        for hash in wanted.wanted.iter().filter(|h| images::is_hash(h)) {
253 +
            let path = home.rootfs(hash);
254 +
            if !path.is_file() {
255 +
                continue;
256 +
            }
257 +
            let kept = std::fs::read(manifest_path(&home.root, hash)).ok().and_then(|b| serde_json::from_slice::<Manifest>(&b).ok());
258 +
            let m = match kept.filter(|m| m.hash == *hash) {
259 +
                Some(m) => m,
260 +
                None => {
261 +
                    let m = images::scan_file(&path, home.image_arch(hash), |_, _| Ok(()))?;
262 +
                    if m.hash != *hash {
263 +
                        continue;
264 +
                    }
265 +
                    save_manifest(&home.root, &m)?;
266 +
                    m
267 +
                }
268 +
            };
269 +
            let url = format!("{}/flow/images/{hash}/held", self.base);
270 +
            self.agent.put(&url).set("Authorization", &self.auth()).send_json(&m).map_err(|e| Self::fail(&url, e))?;
271 +
            sent += 1;
272 +
        }
273 +
        Ok(sent)
186 274
    }
187 275
}
188 276
277 +
/// Where an image's chunk manifest is kept (`manifests/<hash>.json`), so
278 +
/// the runner can serve its chunks to peers out of the image itself.
279 +
pub(crate) fn manifest_path(root: &Path, hash: &str) -> PathBuf {
280 +
    root.join("manifests").join(format!("{hash}.json"))
281 +
}
282 +
283 +
pub(crate) fn save_manifest(root: &Path, m: &Manifest) -> Result<()> {
284 +
    let p = manifest_path(root, &m.hash);
285 +
    std::fs::create_dir_all(p.parent().expect("the manifests directory"))?;
286 +
    let tmp = p.with_extension("part");
287 +
    std::fs::write(&tmp, serde_json::to_vec(m)?)?;
288 +
    std::fs::rename(&tmp, &p)?;
289 +
    Ok(())
290 +
}
291 +
189 292
fn send_batch(batch: &[(String, Vec<u8>)], send: &(dyn Fn(&str, &[u8]) -> Result<()> + Sync), failed: &Mutex<Option<anyhow::Error>>) {
190 293
    std::thread::scope(|s| {
191 294
        for part in batch.chunks(batch.len().div_ceil(PARALLEL).max(1)) {
…
14 unchanged lines
206 309
    home.root.join("chunks")
207 310
}
208 311
209 -
fn cache_path(cache: &Path, hash: &str) -> PathBuf {
312 +
pub(crate) fn cache_path(cache: &Path, hash: &str) -> PathBuf {
210 313
    cache.join(&hash[..2]).join(hash)
211 314
}
212 315
…
8 unchanged lines
added crates/flow/src/lib.rs +35 −0
1 +
//! OxygenFlow's runner and tiers as a library: the `flow` command is built
2 +
//! on it, and so is the Android app's runner (oxygenhub-android#8), which
3 +
//! runs the WASM tier in its own process (docs/handover.md §10).
4 +
5 +
pub mod cache;
6 +
pub mod checkpoint;
7 +
pub mod cpio;
8 +
pub mod deps;
9 +
pub mod egress;
10 +
pub mod facts;
11 +
/// The job-starting code `flow-init` uses in a microVM, shared with the
12 +
/// syscall tier.
13 +
#[cfg(any(target_os = "linux", target_os = "android"))]
14 +
#[path = "../../flow-init/src/job.rs"]
15 +
pub mod guest;
16 +
pub mod home;
17 +
pub mod images;
18 +
pub mod metrics;
19 +
pub mod microvm;
20 +
pub mod outputs;
21 +
pub mod portable;
22 +
pub mod runner;
23 +
pub mod swarm;
24 +
// Linux's (and Android's) namespaces, Landlock, seccomp and cgroups:
25 +
// elsewhere (Windows, Haiku, FreeBSD) the tier says why it is not offered.
26 +
#[cfg(any(target_os = "linux", target_os = "android"))]
27 +
pub mod syscall;
28 +
#[cfg(not(any(target_os = "linux", target_os = "android")))]
29 +
#[path = "syscall_none.rs"]
30 +
pub mod syscall;
31 +
pub mod tier;
32 +
pub mod update;
33 +
pub mod wait;
34 +
pub mod wasm;
35 +
pub mod workspace;
modified crates/flow/src/main.rs +210 −44
1 1
//! `flow`: check pipelines, run them locally, and run jobs for OxygenHub
2 2
//! (docs/spec.md, docs/m0.md).
3 3
4 -
mod cpio;
5 -
mod facts;
6 -
/// The job-starting code `flow-init` uses in a microVM, shared with the
7 -
/// syscall tier.
8 -
#[path = "../../flow-init/src/job.rs"]
9 -
mod guest;
10 -
mod home;
11 -
mod images;
12 -
mod microvm;
13 -
mod outputs;
14 -
mod runner;
15 -
mod syscall;
16 -
mod wait;
17 -
mod workspace;
4 +
use flow::{facts, home, images, microvm, runner, syscall, wait, wasm, workspace};
18 5
19 6
use anyhow::{bail, Context, Result};
20 7
use clap::{Parser, Subcommand};
…
36 unchanged lines
57 44
    /// Measure this machine as a runner does: its speed score (100 on the
58 45
    /// reference machine) and class, free disk and load (spec §11.1)
59 46
    Bench,
47 +
    /// Ask this home's running runner to drain: claim nothing more, finish
48 +
    /// and report what it holds, then stop (as SIGTERM does on Linux; on
49 +
    /// Windows, the way to stop a runner without losing its job)
50 +
    Drain,
51 +
    /// Install a new flow build in place of this one (docs/jetstream.md
52 +
    /// §8.2): check it, keep this one as flow.prev, and have the running
53 +
    /// runner leave with notice so its unit starts the new build. A build
54 +
    /// that does not reach the hub goes back to flow.prev by itself
55 +
    Update {
56 +
        /// The new flow binary
57 +
        #[arg(long)]
58 +
        from: PathBuf,
59 +
        /// Seconds the running runner has to finish what it holds; jobs
60 +
        /// that will not finish in time are handed back with their checkpoints
61 +
        #[arg(long, default_value_t = 600)]
62 +
        notice: u64,
63 +
        /// Check the build and say what would happen, changing nothing
64 +
        #[arg(long)]
65 +
        dry_run: bool,
66 +
        /// Install a build older than this one
67 +
        #[arg(long)]
68 +
        allow_older: bool,
69 +
        /// The installed binary to replace (default: this one)
70 +
        #[arg(long)]
71 +
        exe: Option<PathBuf>,
72 +
    },
60 73
}
61 74
62 75
#[derive(Subcommand)]
76 +
enum DepsCmd {
77 +
    /// Whether every crate a Cargo.lock needs is in the image at that
78 +
    /// version; exit 1 naming each missing one and the versions it has
79 +
    Check {
80 +
        /// The lock file (default: Cargo.lock)
81 +
        #[arg(default_value = "Cargo.lock")]
82 +
        lock: PathBuf,
83 +
        #[command(flatten)]
84 +
        image: DepsSource,
85 +
    },
86 +
    /// Print the image's crates as `NAME VERSION` lines (its crates.txt)
87 +
    List {
88 +
        #[command(flatten)]
89 +
        image: DepsSource,
90 +
    },
91 +
}
92 +
93 +
/// Where the image's crate set comes from: its crate list (the crates.txt
94 +
/// published with the image), a vendor directory, or the lock files it is
95 +
/// built from; inside a job, /opt/vendor.
96 +
#[derive(clap::Args)]
97 +
#[group(multiple = false)]
98 +
struct DepsSource {
99 +
    /// The image's crate list (crates.txt, written by images/rust/build.sh)
100 +
    #[arg(long, value_name = "FILE")]
101 +
    list: Option<PathBuf>,
102 +
    /// A cargo vendor directory (default inside a job: /opt/crates.txt, else /opt/vendor)
103 +
    #[arg(long, value_name = "DIR")]
104 +
    vendor: Option<PathBuf>,
105 +
    /// The lock files the image vendors (oxygen's, oxygenhub's,
106 +
    /// oxygenflow's), as images/rust/build.sh does (repeatable)
107 +
    #[arg(long = "image-lock", value_name = "LOCK")]
108 +
    locks: Vec<PathBuf>,
109 +
}
110 +
111 +
impl DepsSource {
112 +
    fn crates(&self) -> Result<BTreeSet<flow::deps::Crate>> {
113 +
        let read = |p: &Path| std::fs::read_to_string(p).with_context(|| p.display().to_string());
114 +
        if let Some(f) = &self.list {
115 +
            return Ok(flow::deps::from_list(&read(f)?));
116 +
        }
117 +
        if !self.locks.is_empty() {
118 +
            let mut all = BTreeSet::new();
119 +
            for l in &self.locks {
120 +
                all.extend(flow::deps::from_lock(&read(l)?).map_err(|e| anyhow::anyhow!("{}: {e}", l.display()))?);
121 +
            }
122 +
            return Ok(all);
123 +
        }
124 +
        // Inside a job: the image's own list, else its vendor directory.
125 +
        if self.vendor.is_none() && Path::new("/opt/crates.txt").is_file() {
126 +
            return Ok(flow::deps::from_list(&read(Path::new("/opt/crates.txt"))?));
127 +
        }
128 +
        let dir = self.vendor.clone().unwrap_or_else(|| PathBuf::from("/opt/vendor"));
129 +
        if self.vendor.is_none() && !dir.is_dir() {
130 +
            bail!("say what the image holds: --list crates.txt (published with the image), --vendor DIR, or --image-lock for each lock file it is built from; inside a job, /opt/vendor is read");
131 +
        }
132 +
        flow::deps::from_vendor(&dir).map_err(|e| anyhow::anyhow!(e))
133 +
    }
134 +
}
135 +
136 +
#[derive(Subcommand)]
63 137
enum Cmd {
64 138
    /// Validate a pipeline and print its jobs and hash
65 139
    Check {
…
2 unchanged lines
68 142
    },
69 143
    /// Print a file's BLAKE3, for pinning in flow.toml
70 144
    Hash { file: PathBuf },
145 +
    /// The crates the build image vendors, for jobs with no network (#101)
146 +
    Deps {
147 +
        #[command(subcommand)]
148 +
        cmd: DepsCmd,
149 +
    },
71 150
    /// Rootfs images for sandboxes
72 151
    Rootfs {
73 152
        #[command(subcommand)]
…
45 unchanged lines
119 198
    /// Check that this machine can run the syscall tier
120 199
    #[command(name = "__sandbox-probe", hide = true)]
121 200
    SandboxProbe,
201 +
    /// Rename a program that cannot rename its own file (Windows; #83)
202 +
    #[command(name = "__rename", hide = true)]
203 +
    Rename { from: PathBuf, to: PathBuf },
204 +
    /// The WASM tier's helper, started by the runner for each job
205 +
    #[command(name = "__wasm", hide = true)]
206 +
    Wasm { spec: PathBuf },
122 207
}
123 208
124 209
#[derive(Subcommand)]
…
13 unchanged lines
138 223
    /// List the images this home holds
139 224
    List,
140 225
    /// Send an image to a hub, so its runners fetch it instead of having it
141 -
    /// copied to them: only the chunks the hub lacks travel. Uses
142 -
    /// $OXY_TOKEN (a token that may write repositories) and $OXY_CA
226 +
    /// copied to them: only the chunks the hub lacks travel. Signs in as
227 +
    /// `oh` is signed in to that hub, or with $OXY_TOKEN and $OXY_CA
143 228
    Push {
144 229
        /// The image's hash, as pipelines pin it (cas://blob/HASH)
145 230
        hash: String,
146 -
        /// The hub (default: the runner's, from runner.toml or flow.toml)
231 +
        /// The hub (default: $OH_HUB, else the runner's, from runner.toml or flow.toml)
147 232
        #[arg(long)]
148 233
        hub: Option<String>,
149 234
    },
150 235
    /// Fetch an image a hub holds into this home, as a runner does for a
151 -
    /// job whose image it lacks. Uses $OXY_TOKEN and $OXY_CA
236 +
    /// job whose image it lacks. Signs in as `oh` is, or with $OXY_TOKEN
152 237
    Pull {
153 238
        hash: String,
154 239
        #[arg(long)]
…
13 unchanged lines
168 253
    match cli.cmd {
169 254
        Cmd::Sandbox { spec } => std::process::exit(syscall::sandbox_main(&spec)),
170 255
        Cmd::SandboxProbe => std::process::exit(syscall::probe_main()),
256 +
        Cmd::Rename { from, to } => {
257 +
            flow::portable::rename(&from, &to).with_context(|| format!("renaming {} to {}", from.display(), to.display()))?;
258 +
            Ok(())
259 +
        }
260 +
        Cmd::Wasm { spec } => std::process::exit(wasm::wasm_main(&spec)),
171 261
        Cmd::Check { file } => check(&file.unwrap_or_else(|| PathBuf::from(PIPELINE_PATH))),
262 +
        Cmd::Deps { cmd: DepsCmd::List { image } } => {
263 +
            print!("{}", flow::deps::render_list(&image.crates()?));
264 +
            Ok(())
265 +
        }
266 +
        Cmd::Deps { cmd: DepsCmd::Check { lock, image } } => {
267 +
            let text = std::fs::read_to_string(&lock).with_context(|| lock.display().to_string())?;
268 +
            let need = flow::deps::from_lock(&text).map_err(|e| anyhow::anyhow!("{}: {e}", lock.display()))?;
269 +
            let missing = flow::deps::missing(&need, &image.crates()?);
270 +
            if missing.is_empty() {
271 +
                println!("deps: all {} crates of {} are in the image", need.len(), lock.display());
272 +
                return Ok(());
273 +
            }
274 +
            eprintln!("deps: {} of {} crates of {} are not in the image:", missing.len(), need.len(), lock.display());
275 +
            for ((name, version), has) in &missing {
276 +
                let instead = match has.as_slice() {
277 +
                    [] => "the image has no version of it".to_string(),
278 +
                    vs => format!("the image has {}: cargo update -p {name}@{version} --precise {}", vs.join(", "), vs.last().unwrap()),
279 +
                };
280 +
                eprintln!("  {name} {version}  ({instead})");
281 +
            }
282 +
            eprintln!("pin a version the image has, or rebuild the image with this workspace's lock (images/rust/build.sh)");
283 +
            std::process::exit(1);
284 +
        }
172 285
        Cmd::Hash { file } => {
173 286
            println!("{}", home::hash_file(&file)?);
174 287
            Ok(())
…
62 unchanged lines
237 350
            let f = facts::now(&home, Some(speed));
238 351
            println!("speed {speed} (class {}), benchmark {ms} ms", flow_core::api::Facts::class_of(speed));
239 352
            println!("free disk {} MiB at {}", f.free_disk_mib.unwrap_or(0), home.root.display());
240 -
            println!("load {:.2}", f.load_x100.unwrap_or(0) as f64 / 100.0);
353 +
            match f.load_x100 {
354 +
                Some(l) => println!("load {:.2}", l as f64 / 100.0),
355 +
                None => println!("load not measured here"),
356 +
            }
241 357
            if let Some(k) = f.kernel {
242 358
                println!("kernel {k}");
243 359
            }
…
20 unchanged lines
264 380
            println!("registered as {} (runner {}); start it with `flow runner`", got.name, got.id);
265 381
            Ok(())
266 382
        }
383 +
        Cmd::Runner { cmd: Some(RunnerCmd::Drain) } => {
384 +
            let home = Home::locate(cli.home.as_deref())?;
385 +
            if flow::update::ask_to_drain(&home.root)? {
386 +
                println!("asked the runner to drain: it claims nothing more and stops once its jobs are done and reported");
387 +
            } else {
388 +
                println!("no runner of this home is running; the next one starts afresh");
389 +
            }
390 +
            Ok(())
391 +
        }
392 +
        Cmd::Runner { cmd: Some(RunnerCmd::Update { from, notice, dry_run, allow_older, exe }) } => {
393 +
            let home = Home::locate(cli.home.as_deref())?;
394 +
            let exe = match exe {
395 +
                Some(e) => e,
396 +
                None => std::env::current_exe().context("finding this flow binary")?,
397 +
            };
398 +
            let done = flow::update::install(&home, &exe, &from, Duration::from_secs(notice), dry_run, allow_older)?;
399 +
            if dry_run {
400 +
                println!(
401 +
                    "flow {} checks out: it would replace flow {} at {} (dry run: nothing changed)",
402 +
                    done.to,
403 +
                    done.from,
404 +
                    done.exe.display()
405 +
                );
406 +
                return Ok(());
407 +
            }
408 +
            println!("flow {} installed at {}; flow {} kept as {}", done.to, done.exe.display(), done.from, flow::update::PREV);
409 +
            match done.told {
410 +
                Some(pid) => println!("the runner (pid {pid}) leaves within {notice} s, then its unit starts flow {}", done.to),
411 +
                None => println!(
412 +
                    "no runner of this home was told (none runs, or one from before runner updates, which does not listen): \
413 +
                     restart its unit, e.g. `systemctl restart --no-block flow-runner`; it drains and starts flow {}",
414 +
                    done.to
415 +
                ),
416 +
            }
417 +
            Ok(())
418 +
        }
267 419
        Cmd::Runner { cmd: None } => {
268 420
            let home = Home::locate(cli.home.as_deref())?;
421 +
            // A new build on trial that keeps failing to start goes back to
422 +
            // the one before, which then runs in its place (#85).
423 +
            if let flow::update::Start::RolledBack(p) = flow::update::on_start(&home.root)? {
424 +
                #[cfg(unix)]
425 +
                {
426 +
                    use std::os::unix::process::CommandExt;
427 +
                    let err = std::process::Command::new(&p.exe).args(std::env::args_os().skip(1)).exec();
428 +
                    bail!("starting flow {} again after the rollback: {err}", p.from);
429 +
                }
430 +
                // Windows has no exec: run the restored build and end with it.
431 +
                #[cfg(not(unix))]
432 +
                {
433 +
                    let status = std::process::Command::new(&p.exe)
434 +
                        .args(std::env::args_os().skip(1))
435 +
                        .status()
436 +
                        .with_context(|| format!("starting flow {} again after the rollback", p.from))?;
437 +
                    std::process::exit(status.code().unwrap_or(1));
438 +
                }
439 +
            }
269 440
            // The runner's oxy state (identity, user config) stays in its home.
270 441
            if std::env::var_os("OXY_HOME").is_none() {
271 442
                std::env::set_var("OXY_HOME", home.root.join("oxy"));
…
25 unchanged lines
297 468
    }
298 469
}
299 470
300 -
/// The hub image commands talk to: `--hub`, else the runner's; with
301 -
/// $OXY_TOKEN, else the runner's own token.
471 +
/// The hub image commands talk to: `--hub`, else $OH_HUB, else the
472 +
/// runner's. The token: $OXY_TOKEN, else the login `oh auth login` (or
473 +
/// `oxy auth login`) saved for that hub in Oxygen's credential store
474 +
/// (oxygenflow#44), else the runner's own.
302 475
fn image_client(home: &Home, hub: Option<String>) -> Result<images::Client> {
303 476
    let runner = home.config.runner.clone();
304 -
    let hub = hub.or_else(|| runner.as_ref().map(|r| r.hub.clone())).context("name the hub with --hub https://HUB")?;
477 +
    let hub = hub
478 +
        .or_else(|| std::env::var("OH_HUB").ok().filter(|h| !h.is_empty()))
479 +
        .or_else(|| runner.as_ref().map(|r| r.hub.clone()))
480 +
        .context("name the hub with --hub https://HUB (or set OH_HUB)")?;
481 +
    let (saved_token, saved_ca) = oxy_core::credentials::Credentials::user().lookup(&hub);
305 482
    let token = std::env::var("OXY_TOKEN")
306 483
        .ok()
484 +
        .filter(|t| !t.is_empty())
485 +
        .or(saved_token)
307 486
        .or_else(|| runner.as_ref().map(|r| r.token.clone()))
308 -
        .context("set OXY_TOKEN to a token of yours on the hub")?;
309 -
    let ca = std::env::var_os("OXY_CA").map(PathBuf::from).or_else(|| runner.as_ref().and_then(|r| r.ca.clone()));
487 +
        .with_context(|| format!("no token for {hub}: sign in with `oh auth login --hub {hub}`, or set OXY_TOKEN"))?;
488 +
    let ca = std::env::var_os("OXY_CA").map(PathBuf::from).or(saved_ca).or_else(|| runner.as_ref().and_then(|r| r.ca.clone()));
310 489
    images::Client::new(&hub, &token, ca.as_deref())
311 490
}
312 491
…
48 unchanged lines
361 540
362 541
/// The architecture an ELF file was built for, by its header.
363 542
fn elf_arch(path: &Path) -> Option<&'static str> {
364 -
    let bytes = std::fs::read(path).ok()?;
365 -
    if bytes.len() < 20 || &bytes[..4] != b"\x7fELF" {
366 -
        return None;
367 -
    }
368 -
    let little = bytes[5] == 1;
369 -
    let machine = if little { u16::from_le_bytes([bytes[18], bytes[19]]) } else { u16::from_be_bytes([bytes[18], bytes[19]]) };
370 -
    Some(match machine {
371 -
        0x3E => "x86_64",
372 -
        0xB7 => "aarch64",
373 -
        0xF3 => "riscv64",
374 -
        0x03 => "x86",
375 -
        0x28 => "arm",
376 -
        0x15 => "powerpc64",
377 -
        0x16 => "s390x",
378 -
        _ => return None,
379 -
    })
543 +
    let mut head = [0u8; 20];
544 +
    std::io::Read::read_exact(&mut std::fs::File::open(path).ok()?, &mut head).ok()?;
545 +
    jet::rootfs::elf_arch(&head)
380 546
}
381 547
382 548
/// The architecture of an unpacked rootfs: its shell's, or the first
…
119 unchanged lines
502 668
            continue;
503 669
        }
504 670
        let sb = &p.sandboxes[&job.sandbox];
505 -
        if sb.runtime == Runtime::Wasm {
506 -
            bail!("job {}: runtime {} is not available in this build yet", job.name, sb.runtime.as_str());
507 -
        }
508 671
        println!("── {} ({})", job.name, job.exec.join(" "));
509 672
        let js = scratch.join(&job.name);
510 673
        std::fs::create_dir_all(&js)?;
…
17 unchanged lines
528 691
            })
529 692
            .collect::<Result<Vec<_>>>()?;
530 693
        // Each job starts from the same files: jobs do not share writes.
531 -
        let out = if sb.runtime == Runtime::Syscall {
694 +
        let out = if sb.runtime == Runtime::Wasm {
695 +
            workspace::copy_tree(&scratch.join("tree"), &js.join("tree"))?;
696 +
            wasm::run(sb, job, &js.join("tree"), &js, timeout, &mut |r: Record| print_record(&r))?
697 +
        } else if sb.runtime == Runtime::Syscall {
532 698
            workspace::copy_tree(&scratch.join("tree"), &js.join("tree"))?;
533 699
            let group = format!("local-{}-{}", std::process::id(), js.file_name().unwrap_or_default().to_string_lossy());
534 700
            syscall::run(home, sb, job, &inputs, &js.join("tree"), &js, timeout, &mut |r: Record| print_record(&r), &group)?
…
56 unchanged lines
added crates/flow/src/metrics.rs +149 −0
1 +
//! The runner's own metrics (oxygenflow#35, oxygenhub#212): a Prometheus
2 +
//! text page on an address of the runner's choosing, off unless
3 +
//! `metrics = "127.0.0.1:9464"` is set under `[runner]`. With
4 +
//! `metrics_token`, a scrape must send it as a bearer token.
5 +
6 +
use std::io::{BufRead, BufReader, Write};
7 +
use std::sync::atomic::{AtomicU64, Ordering::Relaxed};
8 +
use std::time::Instant;
9 +
10 +
/// Jobs this runner claimed.
11 +
pub static CLAIMS: AtomicU64 = AtomicU64::new(0);
12 +
/// Jobs declined before running (the preflight said no).
13 +
pub static DECLINED: AtomicU64 = AtomicU64::new(0);
14 +
/// Finished jobs by result.
15 +
pub static PASSED: AtomicU64 = AtomicU64::new(0);
16 +
pub static FAILED: AtomicU64 = AtomicU64::new(0);
17 +
pub static ERRORS: AtomicU64 = AtomicU64::new(0);
18 +
pub static CANCELLED: AtomicU64 = AtomicU64::new(0);
19 +
/// Milliseconds finished jobs ran, together.
20 +
pub static JOB_MS: AtomicU64 = AtomicU64::new(0);
21 +
/// Jobs running now.
22 +
pub static RUNNING: AtomicU64 = AtomicU64::new(0);
23 +
24 +
/// Count a job's end: by exit code, its error, and how long it ran.
25 +
pub fn finished(exit_code: Option<i32>, error: Option<&str>, duration_ms: u64, cancelled: bool) {
26 +
    let counter = if cancelled {
27 +
        &CANCELLED
28 +
    } else if error.is_some() {
29 +
        &ERRORS
30 +
    } else if exit_code == Some(0) {
31 +
        &PASSED
32 +
    } else {
33 +
        &FAILED
34 +
    };
35 +
    counter.fetch_add(1, Relaxed);
36 +
    JOB_MS.fetch_add(duration_ms, Relaxed);
37 +
}
38 +
39 +
/// The page.
40 +
pub fn render(name: &str, started: Instant) -> String {
41 +
    let n = name.replace('\\', "\\\\").replace('"', "\\\"");
42 +
    let mut out = String::new();
43 +
    let mut family = |metric: &str, help: &str, kind: &str, samples: &[(String, u64)]| {
44 +
        out.push_str(&format!("# HELP {metric} {help}\n# TYPE {metric} {kind}\n"));
45 +
        for (labels, v) in samples {
46 +
            let sep = if labels.is_empty() { "" } else { "," };
47 +
            out.push_str(&format!("{metric}{{runner=\"{n}\"{sep}{labels}}} {v}\n"));
48 +
        }
49 +
    };
50 +
    family("flow_runner_build_info", "This runner's version.", "gauge", &[(format!("version=\"{}\"", env!("CARGO_PKG_VERSION")), 1)]);
51 +
    family("flow_runner_uptime_seconds", "Seconds since the runner started.", "gauge", &[(String::new(), started.elapsed().as_secs())]);
52 +
    family("flow_runner_claims_total", "Jobs claimed from the hub.", "counter", &[(String::new(), CLAIMS.load(Relaxed))]);
53 +
    family("flow_runner_declined_total", "Claimed jobs declined before running.", "counter", &[(String::new(), DECLINED.load(Relaxed))]);
54 +
    family(
55 +
        "flow_runner_jobs_finished_total",
56 +
        "Finished jobs by result.",
57 +
        "counter",
58 +
        &[
59 +
            ("result=\"passed\"".into(), PASSED.load(Relaxed)),
60 +
            ("result=\"failed\"".into(), FAILED.load(Relaxed)),
61 +
            ("result=\"error\"".into(), ERRORS.load(Relaxed)),
62 +
            ("result=\"cancelled\"".into(), CANCELLED.load(Relaxed)),
63 +
        ],
64 +
    );
65 +
    family(
66 +
        "flow_runner_job_seconds_total",
67 +
        "Seconds finished jobs ran, together.",
68 +
        "counter",
69 +
        &[(String::new(), JOB_MS.load(Relaxed) / 1000)],
70 +
    );
71 +
    family("flow_runner_jobs_running", "Jobs running now.", "gauge", &[(String::new(), RUNNING.load(Relaxed))]);
72 +
    out
73 +
}
74 +
75 +
/// Serve the page at `addr` on a thread of its own; a failure to listen is
76 +
/// said and the runner goes on without it.
77 +
pub fn serve(addr: &str, token: Option<String>, name: String) {
78 +
    let listener = match std::net::TcpListener::bind(addr) {
79 +
        Ok(l) => l,
80 +
        Err(e) => {
81 +
            eprintln!("flow runner {name}: not serving metrics on {addr}: {e}");
82 +
            return;
83 +
        }
84 +
    };
85 +
    eprintln!("flow runner {name}: metrics on http://{addr}/metrics");
86 +
    let started = Instant::now();
87 +
    std::thread::spawn(move || {
88 +
        for stream in listener.incoming().flatten() {
89 +
            let _ = answer(stream, token.as_deref(), &name, started);
90 +
        }
91 +
    });
92 +
}
93 +
94 +
fn answer(mut stream: std::net::TcpStream, token: Option<&str>, name: &str, started: Instant) -> std::io::Result<()> {
95 +
    stream.set_read_timeout(Some(std::time::Duration::from_secs(5)))?;
96 +
    let mut reader = BufReader::new(stream.try_clone()?);
97 +
    let mut line = String::new();
98 +
    reader.read_line(&mut line)?;
99 +
    let path = line.split_whitespace().nth(1).unwrap_or("").to_string();
100 +
    let mut authorized = token.is_none();
101 +
    loop {
102 +
        let mut h = String::new();
103 +
        if reader.read_line(&mut h)? == 0 || h.trim().is_empty() {
104 +
            break;
105 +
        }
106 +
        if let (Some(t), Some((k, v))) = (token, h.split_once(':')) {
107 +
            if k.trim().eq_ignore_ascii_case("authorization") && v.trim() == format!("Bearer {t}") {
108 +
                authorized = true;
109 +
            }
110 +
        }
111 +
    }
112 +
    let (status, body) = match (path.as_str(), authorized) {
113 +
        ("/metrics", true) => ("200 OK", render(name, started)),
114 +
        ("/metrics", false) => ("401 Unauthorized", "a bearer token is needed\n".to_string()),
115 +
        _ => ("404 Not Found", "not found\n".to_string()),
116 +
    };
117 +
    write!(
118 +
        stream,
119 +
        "HTTP/1.1 {status}\r\nContent-Type: text/plain; version=0.0.4\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
120 +
        body.len()
121 +
    )
122 +
}
123 +
124 +
#[cfg(test)]
125 +
mod tests {
126 +
    use super::*;
127 +
    use std::io::Read;
128 +
129 +
    #[test]
130 +
    fn a_scrape_needs_the_token_and_reads_the_counters() {
131 +
        let l = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
132 +
        let addr = l.local_addr().unwrap().to_string();
133 +
        drop(l);
134 +
        serve(&addr, Some("t0k".into()), "earth-1".into());
135 +
        CLAIMS.fetch_add(1, Relaxed);
136 +
        finished(Some(0), None, 2500, false);
137 +
        let get = |auth: &str| {
138 +
            let mut s = std::net::TcpStream::connect(&addr).unwrap();
139 +
            write!(s, "GET /metrics HTTP/1.1\r\nHost: x\r\n{auth}\r\n").unwrap();
140 +
            let mut body = String::new();
141 +
            s.read_to_string(&mut body).unwrap();
142 +
            body
143 +
        };
144 +
        assert!(get("").starts_with("HTTP/1.1 401"));
145 +
        let page = get("Authorization: Bearer t0k\r\n");
146 +
        assert!(page.starts_with("HTTP/1.1 200"), "{page}");
147 +
        assert!(page.contains("flow_runner_claims_total{runner=\"earth-1\"} ") && page.contains("result=\"passed\"}"), "{page}");
148 +
    }
149 +
}
modified crates/flow/src/microvm.rs +127 −10
…
21 unchanged lines
22 22
    pub cancelled: bool,
23 23
    /// The most memory the sandbox held, in MiB (oxygenflow#41).
24 24
    pub peak_memory_mib: Option<u64>,
25 +
    /// Each cache disk the guest pruned before it stopped, by its target
26 +
    /// (oxygenflow#72); the syscall tier's are pruned by the runner.
27 +
    pub pruned: Vec<(String, flow_init::prune::Pruned)>,
25 28
}
26 29
27 30
/// Where a job's records go, and whether it is still wanted: asked a few
…
55 unchanged lines
83 86
    let rootfs = home.verified(&rootfs, &sandbox.rootfs)?;
84 87
    let environment = environment(home, sandbox, job)?;
85 88
86 -
    let mut spec = guest_spec(job);
89 +
    let percent = home.config.runner.as_ref().and_then(|r| r.cache_prune_percent).unwrap_or(flow_init::prune::DEFAULT_PERCENT);
90 +
    let mut spec = guest_spec(job, percent);
87 91
    spec["loopback"] = serde_json::Value::Bool(sandbox.network == flow_core::Network::Loopback);
92 +
    // Allow-listed egress (oxygenflow#55): a vsock device and the runner's
93 +
    // proxy behind it; the guest relays 127.0.0.1:3128 to it.
94 +
    let egress = sandbox.network == flow_core::Network::Egress;
95 +
    let (egress_tx, egress_rx) = mpsc::channel::<String>();
96 +
    let mut _proxy = None;
97 +
    if egress {
98 +
        if !home.config.runner.as_ref().is_some_and(|r| r.egress) {
99 +
            bail!("this runner does not allow network egress (a runner allows it with `egress = true` under [runner] in flow.toml)");
100 +
        }
101 +
        _proxy = Some(
102 +
            crate::egress::start(&scratch.join(format!("v.sock_{}", crate::egress::VSOCK_PORT)), sandbox.egress_allow.clone(), egress_tx)
103 +
                .context("start the egress proxy")?,
104 +
        );
105 +
        spec["egress"] = serde_json::Value::Bool(true);
106 +
        add_proxy_env(&mut spec);
107 +
        sink.record(Record {
108 +
            t: 0,
109 +
            stream: "flow".into(),
110 +
            text: format!("egress: through this runner, to {} only", sandbox.egress_allow.join(", ")),
111 +
        });
112 +
    }
88 113
    // Secrets ride in the guest's job file, which only flow-init reads,
89 114
    // outside the root the job is confined to; it writes each as a file.
90 115
    if !secrets.is_empty() {
…
5 unchanged lines
96 121
    let mut disks = Vec::new();
97 122
    for (i, m) in job.mounts.iter().filter(|m| m.source == MountSource::Disk).enumerate() {
98 123
        let path = scratch.join(format!("mount-{i}.ext4"));
99 -
        crate::workspace::empty_ext4(&path, m.size_mib, "scratch")?;
124 +
        // A disk already in place came from the build cache (#28).
125 +
        if !path.exists() {
126 +
            crate::workspace::empty_ext4(&path, m.size_mib, "scratch")?;
127 +
        }
100 128
        disks.push(
101 129
            serde_json::json!({ "drive_id": format!("mount{i}"), "path_on_host": path, "is_root_device": false, "is_read_only": false }),
102 130
        );
…
22 unchanged lines
125 153
        "drives": drives,
126 154
        "machine-config": { "vcpu_count": sandbox.cpus, "mem_size_mib": sandbox.memory_mib },
127 155
    });
156 +
    let mut vm = vm;
157 +
    if egress {
158 +
        // Relative to the VMM's working directory (the scratch directory),
159 +
        // which keeps the socket path short.
160 +
        vm["vsock"] = serde_json::json!({ "guest_cid": 3, "uds_path": "v.sock" });
161 +
    }
128 162
    let vm_path = scratch.join("vm.json");
129 163
    std::fs::write(&vm_path, serde_json::to_vec_pretty(&vm)?)?;
130 164
…
22 unchanged lines
153 187
    // The VMM's peak resident memory is the guest memory it touched: read
154 188
    // while it runs, since it is gone once it exits.
155 189
    let mut peak_memory_mib = None;
190 +
    let mut pruned = Vec::new();
156 191
    loop {
157 192
        if let Some(m) = crate::facts::peak_rss_mib(child.id()) {
158 193
            peak_memory_mib = Some(m);
159 194
        }
195 +
        while let Ok(line) = egress_rx.try_recv() {
196 +
            sink.record(Record { t: t(&started), stream: "flow".into(), text: line });
197 +
        }
160 198
        match rx.recv_timeout(Duration::from_millis(200)) {
161 199
            Ok((from_console, line)) => {
162 200
                let text = String::from_utf8_lossy(&line).trim_end_matches('\r').to_string();
…
8 unchanged lines
171 209
                        "E" => sink.record(rec("err", body)),
172 210
                        "X" => exit_code = body.trim().parse().ok(),
173 211
                        "F" => sink.record(rec("flow", &format!("guest init failed: {body}"))),
212 +
                        "C" => match pruned_frame(body) {
213 +
                            Some(p) => pruned.push(p),
214 +
                            None => sink.record(rec("console", &text)),
215 +
                        },
174 216
                        _ => sink.record(rec("console", &text)),
175 217
                    }
176 218
                } else if !text.is_empty() {
…
36 unchanged lines
213 255
        timed_out,
214 256
        cancelled,
215 257
        peak_memory_mib,
258 +
        pruned,
216 259
    })
217 260
}
218 261
262 +
/// A `\x1eC` frame: a cache disk the guest pruned (oxygenflow#72).
263 +
fn pruned_frame(body: &str) -> Option<(String, flow_init::prune::Pruned)> {
264 +
    #[derive(serde::Deserialize)]
265 +
    struct Frame {
266 +
        target: String,
267 +
        pruned: flow_init::prune::Pruned,
268 +
    }
269 +
    let f: Frame = serde_json::from_str(body).ok()?;
270 +
    Some((f.target, f.pruned))
271 +
}
272 +
273 +
/// The proxy variables a job with egress gets, in both spellings tools
274 +
/// read, unless the job sets them itself.
275 +
fn add_proxy_env(spec: &mut serde_json::Value) {
276 +
    let Some(env) = spec["env"].as_array_mut() else { return };
277 +
    let set: Vec<String> = env.iter().filter_map(|kv| kv[0].as_str().map(String::from)).collect();
278 +
    for (k, v) in [
279 +
        ("HTTPS_PROXY", crate::egress::GUEST_PROXY),
280 +
        ("https_proxy", crate::egress::GUEST_PROXY),
281 +
        ("HTTP_PROXY", crate::egress::GUEST_PROXY),
282 +
        ("http_proxy", crate::egress::GUEST_PROXY),
283 +
        ("NO_PROXY", "localhost,127.0.0.1"),
284 +
        ("no_proxy", "localhost,127.0.0.1"),
285 +
    ] {
286 +
        if !set.iter().any(|s| s == k) {
287 +
            env.push(serde_json::json!([k, v]));
288 +
        }
289 +
    }
290 +
}
291 +
219 292
/// What `flow-init` reads from `/job.json`: argv, env, and the mounts
220 -
/// with the device each disk mount is attached as (vdc onwards).
221 -
pub fn guest_spec(job: &Job) -> serde_json::Value {
293 +
/// with the device each disk mount is attached as (vdc onwards). A cache
294 +
/// disk is pruned to `percent` of its size after a passing job
295 +
/// (oxygenflow#72), leaving the job's outputs on it.
296 +
pub fn guest_spec(job: &Job, percent: u8) -> serde_json::Value {
222 297
    let mut next_disk = b'c';
223 298
    let mounts: Vec<serde_json::Value> = job
224 299
        .mounts
…
3 unchanged lines
228 303
            MountSource::Disk => {
229 304
                let dev = format!("/dev/vd{}", next_disk as char);
230 305
                next_disk += 1;
231 -
                serde_json::json!({ "kind": "disk", "target": m.target, "size_mib": m.size_mib, "device": dev })
306 +
                let mut disk = serde_json::json!({ "kind": "disk", "target": m.target, "size_mib": m.size_mib, "device": dev });
307 +
                if m.cache.is_some() {
308 +
                    let prefix = format!("{}/", m.target);
309 +
                    let keep: Vec<&str> = job.outputs.iter().filter_map(|o| o.path.strip_prefix(&prefix)).collect();
310 +
                    disk["prune"] = percent.into();
311 +
                    disk["keep"] = keep.into();
312 +
                }
313 +
                disk
232 314
            }
233 315
        })
234 316
        .collect();
…
25 unchanged lines
260 342
    use flow_core::Mount;
261 343
262 344
    #[test]
345 +
    fn a_job_with_egress_gets_the_proxy_variables_unless_it_sets_them() {
346 +
        let mut spec = serde_json::json!({ "env": [["HTTPS_PROXY", "http://mine:1"]] });
347 +
        super::add_proxy_env(&mut spec);
348 +
        let env = spec["env"].as_array().unwrap();
349 +
        let get = |k: &str| env.iter().find(|kv| kv[0] == k).map(|kv| kv[1].as_str().unwrap().to_string());
350 +
        assert_eq!(get("HTTPS_PROXY").as_deref(), Some("http://mine:1"), "the job's own is kept");
351 +
        assert_eq!(get("https_proxy").as_deref(), Some("http://127.0.0.1:3128"));
352 +
        assert_eq!(get("NO_PROXY").as_deref(), Some("localhost,127.0.0.1"));
353 +
    }
354 +
355 +
    #[test]
263 356
    fn guest_spec_numbers_disks_after_the_workspace() {
264 357
        let mut job: Job = serde_json::from_value(serde_json::json!({
265 358
            "name": "t", "sandbox": "s", "scope": [], "depends_on": [], "env": {"A": "1"}, "exec": ["true"]
266 359
        }))
267 360
        .unwrap();
268 -
        assert_eq!(guest_spec(&job)["mounts"], serde_json::json!([]), "no mounts");
361 +
        assert_eq!(guest_spec(&job, 60)["mounts"], serde_json::json!([]), "no mounts");
269 362
        job.mounts = vec![
270 -
            Mount { source: MountSource::Disk, target: "/src/target".into(), size_mib: 8192 },
271 -
            Mount { source: MountSource::Tmpfs, target: "/tmp/cache".into(), size_mib: 256 },
272 -
            Mount { source: MountSource::Disk, target: "/tmp/z".into(), size_mib: 64 },
363 +
            Mount { source: MountSource::Disk, target: "/src/target".into(), size_mib: 8192, cache: None, key: vec![] },
364 +
            Mount { source: MountSource::Tmpfs, target: "/tmp/cache".into(), size_mib: 256, cache: None, key: vec![] },
365 +
            Mount { source: MountSource::Disk, target: "/tmp/z".into(), size_mib: 64, cache: None, key: vec![] },
273 366
        ];
274 -
        let spec = guest_spec(&job);
367 +
        let spec = guest_spec(&job, 60);
275 368
        assert_eq!(spec["mounts"][0], serde_json::json!({"kind": "disk", "target": "/src/target", "size_mib": 8192, "device": "/dev/vdc"}));
276 369
        assert_eq!(spec["mounts"][1], serde_json::json!({"kind": "tmpfs", "target": "/tmp/cache", "size_mib": 256}));
277 370
        assert_eq!(spec["mounts"][2]["device"], "/dev/vdd");
278 371
        assert_eq!(spec["env"], serde_json::json!([["A", "1"]]));
279 372
    }
373 +
374 +
    /// oxygenflow#72: a cache disk is pruned in the guest, leaving the
375 +
    /// job's outputs on it; a plain disk is not, and the guest's report
376 +
    /// comes back as a frame.
377 +
    #[test]
378 +
    fn a_cache_disk_is_pruned_in_the_guest_but_its_outputs_stay() {
379 +
        let mut job: Job = serde_json::from_value(serde_json::json!({
380 +
            "name": "t", "sandbox": "s", "scope": [], "depends_on": [], "env": {}, "exec": ["true"],
381 +
            "outputs": [{"name": "bin", "path": "/src/target/release/tool"}, {"name": "w", "path": "/src/w"}]
382 +
        }))
383 +
        .unwrap();
384 +
        job.mounts = vec![
385 +
            Mount { source: MountSource::Disk, target: "/src/target".into(), size_mib: 8192, cache: Some("t".into()), key: vec![] },
386 +
            Mount { source: MountSource::Disk, target: "/tmp/z".into(), size_mib: 64, cache: None, key: vec![] },
387 +
        ];
388 +
        let spec = guest_spec(&job, 45);
389 +
        assert_eq!((spec["mounts"][0]["prune"].as_u64(), &spec["mounts"][0]["keep"]), (Some(45), &serde_json::json!(["release/tool"])));
390 +
        assert!(spec["mounts"][1].get("prune").is_none() && spec["mounts"][1].get("keep").is_none());
391 +
        let (target, p) =
392 +
            pruned_frame(r#"{"target":"/src/target","pruned":{"size":100,"percent":60,"before":90,"after":50,"freed":40,"files":3}}"#)
393 +
                .unwrap();
394 +
        assert_eq!((target.as_str(), p.after, p.cap(), p.fits(), p.unpruned), ("/src/target", 50, 60, true, None));
395 +
        assert!(pruned_frame("not json").is_none());
396 +
    }
280 397
}
modified crates/flow/src/outputs.rs +48 −9
…
34 unchanged lines
35 35
/// [`MAX_ARTIFACT_BYTES`]. The job wrote that tree, so no link is
36 36
/// followed on the way: each directory is opened with `O_NOFOLLOW`, and
37 37
/// so is the file.
38 +
#[cfg(unix)]
38 39
pub fn read_dir_file(base: &Path, inner: &str) -> Result<Vec<u8>> {
39 40
    use std::ffi::CString;
40 -
    use std::io::Read;
41 41
    use std::os::fd::FromRawFd;
42 -
    let parts: Vec<&str> = inner.split('/').filter(|p| !p.is_empty()).collect();
43 -
    if parts.is_empty() || parts.iter().any(|p| *p == "." || *p == "..") {
44 -
        bail!("{inner} is not a clean path");
45 -
    }
42 +
    let parts = clean_parts(inner)?;
46 43
    let c = |s: &str| CString::new(s).context("a path holds NUL");
47 44
    let mut dir = unsafe { libc::open(c(&base.to_string_lossy())?.as_ptr(), libc::O_DIRECTORY | libc::O_RDONLY | libc::O_CLOEXEC) };
48 45
    if dir < 0 {
…
15 unchanged lines
64 61
    if fd < 0 {
65 62
        bail!("{inner} is not a regular file in the job's files ({err})");
66 63
    }
67 -
    let file = unsafe { std::fs::File::from_raw_fd(fd) };
64 +
    read_open(unsafe { std::fs::File::from_raw_fd(fd) }, inner)
65 +
}
66 +
67 +
/// [`read_dir_file`] on Windows (oxygenflow#83), where a WASM job's files
68 +
/// are a directory: each step is checked to be a directory and not a link
69 +
/// (nor a junction), then the file to be a regular file, before it is read.
70 +
#[cfg(not(unix))]
71 +
pub fn read_dir_file(base: &Path, inner: &str) -> Result<Vec<u8>> {
72 +
    let parts = clean_parts(inner)?;
73 +
    let mut at = base.to_path_buf();
74 +
    let (last, dirs) = parts.split_last().unwrap();
75 +
    for d in dirs {
76 +
        at.push(d);
77 +
        let md = std::fs::symlink_metadata(&at).with_context(|| format!("{inner}: {d} is not in the job's files"))?;
78 +
        if !md.is_dir() || md.file_type().is_symlink() {
79 +
            bail!("{inner}: {d} is not a directory in the job's files");
80 +
        }
81 +
    }
82 +
    at.push(last);
83 +
    let md = std::fs::symlink_metadata(&at).with_context(|| format!("{inner} is not in the job's files"))?;
84 +
    if !md.is_file() || md.file_type().is_symlink() {
85 +
        bail!("{inner} is not a regular file in the job's files (it may be missing or a link)");
86 +
    }
87 +
    read_open(std::fs::File::open(&at)?, inner)
88 +
}
89 +
90 +
/// `inner`'s parts, refusing an empty path, `.` and `..`.
91 +
fn clean_parts(inner: &str) -> Result<Vec<&str>> {
92 +
    let parts: Vec<&str> = inner.split('/').filter(|p| !p.is_empty()).collect();
93 +
    if parts.is_empty() || parts.iter().any(|p| *p == "." || *p == "..") {
94 +
        bail!("{inner} is not a clean path");
95 +
    }
96 +
    Ok(parts)
97 +
}
98 +
99 +
/// Read an opened artifact file, at most [`MAX_ARTIFACT_BYTES`].
100 +
fn read_open(file: std::fs::File, inner: &str) -> Result<Vec<u8>> {
101 +
    use std::io::Read;
68 102
    let meta = file.metadata()?;
69 103
    if !meta.is_file() {
70 104
        bail!("{inner} is not a regular file in the job's files (it may be missing or a link)");
…
148 unchanged lines
219 253
            secrets: vec![],
220 254
            identities: vec![],
221 255
            mounts: vec![
222 -
                Mount { source: MountSource::Tmpfs, target: "/src/cache".into(), size_mib: 64 },
223 -
                Mount { source: MountSource::Disk, target: "/src/target".into(), size_mib: 1024 },
224 -
                Mount { source: MountSource::Disk, target: "/src/work".into(), size_mib: 1024 },
256 +
                Mount { source: MountSource::Tmpfs, target: "/src/cache".into(), size_mib: 64, cache: None, key: vec![] },
257 +
                Mount { source: MountSource::Disk, target: "/src/target".into(), size_mib: 1024, cache: None, key: vec![] },
258 +
                Mount { source: MountSource::Disk, target: "/src/work".into(), size_mib: 1024, cache: None, key: vec![] },
225 259
            ],
226 260
            outputs: vec![],
261 +
            inputs: vec![],
262 +
            approval: None,
263 +
            environment: None,
227 264
            runs_on: vec![],
228 265
            timeout_s: None,
229 266
            on: None,
267 +
            priority: None,
268 +
            checkpoints: Vec::new(),
230 269
        }
231 270
    }
232 271
…
81 unchanged lines
added crates/flow/src/portable.rs +138 −0
1 +
//! What differs between the systems a runner runs on (oxygenflow#83): file
2 +
//! modes, file identities and links are Unix's. On Windows, where a runner
3 +
//! offers the WASM tier only, modes are not kept (files read as 0644 and
4 +
//! directories as 0755), each file is its own, and sizes are lengths.
5 +
6 +
use std::fs::{Metadata, OpenOptions};
7 +
use std::io;
8 +
use std::path::Path;
9 +
10 +
/// Set `p`'s permission bits; nothing on Windows.
11 +
pub fn set_mode(p: &Path, mode: u32) -> io::Result<()> {
12 +
    #[cfg(unix)]
13 +
    {
14 +
        use std::os::unix::fs::PermissionsExt;
15 +
        std::fs::set_permissions(p, std::fs::Permissions::from_mode(mode))
16 +
    }
17 +
    #[cfg(not(unix))]
18 +
    {
19 +
        let _ = (p, mode);
20 +
        Ok(())
21 +
    }
22 +
}
23 +
24 +
/// A file's permission bits (`0o7777` of its mode).
25 +
pub fn mode(md: &Metadata) -> u32 {
26 +
    #[cfg(unix)]
27 +
    {
28 +
        use std::os::unix::fs::MetadataExt;
29 +
        md.mode() & 0o7777
30 +
    }
31 +
    #[cfg(not(unix))]
32 +
    {
33 +
        let base = if md.is_dir() { 0o755 } else { 0o644 };
34 +
        if md.permissions().readonly() {
35 +
            base & !0o222
36 +
        } else {
37 +
            base
38 +
        }
39 +
    }
40 +
}
41 +
42 +
/// Open a new file readable and writable by its owner only.
43 +
pub fn private(o: &mut OpenOptions) -> &mut OpenOptions {
44 +
    #[cfg(unix)]
45 +
    {
46 +
        use std::os::unix::fs::OpenOptionsExt;
47 +
        o.mode(0o600)
48 +
    }
49 +
    #[cfg(not(unix))]
50 +
    {
51 +
        o
52 +
    }
53 +
}
54 +
55 +
/// Make `p` and its parents, the last only its owner's.
56 +
pub fn create_private_dir(p: &Path) -> io::Result<()> {
57 +
    #[cfg(unix)]
58 +
    {
59 +
        std::os::unix::fs::DirBuilderExt::mode(std::fs::DirBuilder::new().recursive(true), 0o700).create(p)
60 +
    }
61 +
    #[cfg(not(unix))]
62 +
    {
63 +
        std::fs::create_dir_all(p)
64 +
    }
65 +
}
66 +
67 +
/// The disk a file takes: its blocks on Unix, its length elsewhere.
68 +
pub fn disk_bytes(md: &Metadata) -> u64 {
69 +
    #[cfg(unix)]
70 +
    {
71 +
        use std::os::unix::fs::MetadataExt;
72 +
        md.blocks() * 512
73 +
    }
74 +
    #[cfg(not(unix))]
75 +
    {
76 +
        md.len()
77 +
    }
78 +
}
79 +
80 +
/// For a file with more than one name, what names it once (device, inode);
81 +
/// `None` for a file with one name, a directory, or anywhere but Unix.
82 +
pub fn shared_identity(md: &Metadata) -> Option<(u64, u64)> {
83 +
    #[cfg(unix)]
84 +
    {
85 +
        use std::os::unix::fs::MetadataExt;
86 +
        (md.nlink() > 1 && !md.is_dir()).then(|| (md.dev(), md.ino()))
87 +
    }
88 +
    #[cfg(not(unix))]
89 +
    {
90 +
        let _ = md;
91 +
        None
92 +
    }
93 +
}
94 +
95 +
/// Make `link` a symbolic link to `target` (a file link on Windows, which
96 +
/// needs Developer Mode or the right to make them).
97 +
pub fn symlink(target: &Path, link: &Path) -> io::Result<()> {
98 +
    #[cfg(unix)]
99 +
    {
100 +
        std::os::unix::fs::symlink(target, link)
101 +
    }
102 +
    #[cfg(windows)]
103 +
    {
104 +
        std::os::windows::fs::symlink_file(target, link)
105 +
    }
106 +
}
107 +
108 +
/// Rename `from` to `to`, which must not exist. On Windows through
109 +
/// `MoveFileExW`, which renames a program that is running, trying again
110 +
/// while another program (a virus scanner) holds the file.
111 +
pub fn rename(from: &Path, to: &Path) -> io::Result<()> {
112 +
    #[cfg(windows)]
113 +
    {
114 +
        use std::os::windows::ffi::OsStrExt;
115 +
        let wide = |p: &Path| p.as_os_str().encode_wide().chain(Some(0)).collect::<Vec<u16>>();
116 +
        let (f, t) = (wide(from), wide(to));
117 +
        // A virus scanner holds a new program for a while after it is
118 +
        // written (a sharing violation, 32, or access denied, 5): try again
119 +
        // for half a minute.
120 +
        let mut tries = 0;
121 +
        loop {
122 +
            // SAFETY: both are NUL-terminated wide paths.
123 +
            if unsafe { windows_sys::Win32::Storage::FileSystem::MoveFileExW(f.as_ptr(), t.as_ptr(), 0) } != 0 {
124 +
                return Ok(());
125 +
            }
126 +
            let e = io::Error::last_os_error();
127 +
            tries += 1;
128 +
            if tries >= 30 || !matches!(e.raw_os_error(), Some(32) | Some(5)) {
129 +
                return Err(e);
130 +
            }
131 +
            std::thread::sleep(std::time::Duration::from_secs(1));
132 +
        }
133 +
    }
134 +
    #[cfg(not(windows))]
135 +
    {
136 +
        std::fs::rename(from, to)
137 +
    }
138 +
}
modified crates/flow/src/runner.rs +1411 −116
…
9 unchanged lines
10 10
use flow_core::{scope_tree_hash, Receipt};
11 11
use oxy_core::hash::ChangeHash;
12 12
use oxy_core::repo::Repo;
13 -
use std::path::PathBuf;
13 +
use std::path::{Path, PathBuf};
14 14
use std::time::{Duration, Instant};
15 15
16 16
/// A request the hub refused (4xx). On a job's endpoints, 409 means the
…
30 unchanged lines
47 47
        error: Some(CANCELLED.into()),
48 48
        peak_disk_mib: None,
49 49
        peak_memory_mib: None,
50 +
        phases: Default::default(),
50 51
    }
51 52
}
52 53
…
19 unchanged lines
72 73
73 74
    fn post_with(&self, path: &str, body: &impl serde::Serialize, delays: &[Duration]) -> Result<Option<ureq::Response>> {
74 75
        let url = format!("{}/api/v1{path}", self.base);
75 -
        let mut delays = delays.iter();
76 -
        loop {
77 -
            match self.agent.post(&url).set("Authorization", &format!("Bearer {}", self.token)).send_json(body) {
78 -
                Ok(r) if r.status() == 204 => return Ok(None),
79 -
                Ok(r) => return Ok(Some(r)),
80 -
                Err(ureq::Error::Status(code, r)) => {
81 -
                    let text = r.into_string().unwrap_or_default();
82 -
                    match delays.next() {
83 -
                        Some(d) if code >= 500 => std::thread::sleep(*d),
84 -
                        _ if code < 500 => return Err(anyhow::Error::new(Refused { code, text }).context(url)),
85 -
                        _ => bail!("{url}: HTTP {code}: {text}"),
86 -
                    }
87 -
                }
88 -
                Err(e) => bail!("{url}: {e}"),
89 -
            }
90 -
        }
76 +
        let r = retrying(&url, delays, false, || {
77 +
            self.agent.post(&url).set("Authorization", &format!("Bearer {}", self.token)).send_json(body).map_err(Box::new)
78 +
        })?;
79 +
        Ok(if r.status() == 204 { None } else { Some(r) })
80 +
    }
81 +
82 +
    /// POST what a job leaves behind (oxygenflow#49): through a hub that is
83 +
    /// away for a few minutes, as while a deploy restarts it, so a job's
84 +
    /// declines and uploads outlast it.
85 +
    fn post_patiently(&self, path: &str, body: &impl serde::Serialize) -> Result<Option<ureq::Response>> {
86 +
        let url = format!("{}/api/v1{path}", self.base);
87 +
        let r = retrying(&url, &PATIENT_DELAYS, true, || {
88 +
            self.agent.post(&url).set("Authorization", &format!("Bearer {}", self.token)).send_json(body).map_err(Box::new)
89 +
        })?;
90 +
        Ok(if r.status() == 204 { None } else { Some(r) })
91 91
    }
92 92
93 93
    pub fn claim(&self, req: &ClaimRequest) -> Result<Option<Claim>> {
…
5 unchanged lines
99 99
100 100
    /// Hand job `job` back, saying why this runner will not run it.
101 101
    pub fn decline(&self, job: i64, reason: &str) -> Result<()> {
102 -
        self.post(&format!("/flow/jobs/{job}/decline"), &Decline { reason: reason.into() })?;
102 +
        self.post_patiently(&format!("/flow/jobs/{job}/decline"), &Decline { reason: reason.into() })?;
103 +
        Ok(())
104 +
    }
105 +
106 +
    /// Hand back a job this runner holds as it leaves (oxygenflow#76): the
107 +
    /// hub queues it again ahead of new work for another runner.
108 +
    pub fn release(&self, job: i64, reason: &str) -> Result<()> {
109 +
        self.post_patiently(&format!("/flow/jobs/{job}/release"), &Decline { reason: reason.into() })?;
103 110
        Ok(())
104 111
    }
105 112
113 +
    /// Say this runner is there while it claims nothing: its slots are full,
114 +
    /// or it is draining (oxygenhub#243). Once, no retries: the next comes
115 +
    /// soon. A hub without the endpoint (404) is not an error.
116 +
    /// A ten-minute certificate for `identity`, asked with a job's token
117 +
    /// (keyless, oxygenhub#241), and the identity of the hub's key that
118 +
    /// issued it, from `/.well-known/oxyhub-key`.
119 +
    pub fn certify(
120 +
        &self,
121 +
        job_token: &str,
122 +
        identity: &oxy_core::identity::Identity,
123 +
        proof: &oxy_core::identity::Signature,
124 +
    ) -> Result<(oxy_core::cert::SignedCertificate, oxy_core::identity::Identity)> {
125 +
        use oxy_core::dsse::{base64, base64_decode};
126 +
        let body = serde_json::json!({
127 +
            "identity": base64(&identity.encode()),
128 +
            "proof": base64(&oxy_core::cbor::encode(&proof.to_value())),
129 +
            "scope": "seal",
130 +
        });
131 +
        let url = format!("{}/api/v1/keyless/certify", self.base);
132 +
        let r = self
133 +
            .agent
134 +
            .post(&url)
135 +
            .set("Authorization", &format!("Bearer {job_token}"))
136 +
            .send_json(body)
137 +
            .map_err(|e| anyhow::anyhow!("asking the hub to certify the key: {e}"))?;
138 +
        let v: serde_json::Value = r.into_json()?;
139 +
        let cert = base64_decode(v["certificate"].as_str().unwrap_or(""))
140 +
            .context("the hub's answer has no certificate")
141 +
            .and_then(|b| oxy_core::cert::SignedCertificate::decode(&b).map_err(|e| anyhow::anyhow!("{e}")))?;
142 +
        let mut bytes = Vec::new();
143 +
        std::io::Read::read_to_end(
144 +
            &mut self.agent.get(&format!("{}/.well-known/oxyhub-key", self.base)).call()?.into_reader(),
145 +
            &mut bytes,
146 +
        )?;
147 +
        let issuer = oxy_core::identity::Identity::decode(&bytes)?;
148 +
        if issuer.hash() != cert.cert.issuer || cert.verify(&issuer).is_err() {
149 +
            bail!("the certificate is not signed by the hub's published key");
150 +
        }
151 +
        if cert.cert.key_id() != identity.hash() {
152 +
            bail!("the hub certified another key");
153 +
        }
154 +
        Ok((cert, issuer))
155 +
    }
156 +
157 +
    pub fn heartbeat(&self) -> Result<()> {
158 +
        match self.post_with("/flow/runners/heartbeat", &serde_json::json!({}), &[]) {
159 +
            Ok(_) => Ok(()),
160 +
            Err(e) if format!("{e:#}").contains("404") => Ok(()),
161 +
            Err(e) => Err(e),
162 +
        }
163 +
    }
164 +
165 +
    /// The update policy for this runner and the build it should run
166 +
    /// (oxygenhub#322); `None` from a hub without runner updates.
167 +
    pub fn update_answer(&self) -> Result<Option<flow_core::api::UpdateAnswer>> {
168 +
        let url = format!("{}/api/v1/flow/runner/update", self.base);
169 +
        match self.agent.get(&url).set("Authorization", &format!("Bearer {}", self.token)).call() {
170 +
            Ok(r) => Ok(Some(r.into_json()?)),
171 +
            Err(ureq::Error::Status(404, _)) => Ok(None),
172 +
            Err(e) => Err(anyhow::anyhow!("{url}: {e}")),
173 +
        }
174 +
    }
175 +
176 +
    /// Tell the hub what this runner did about its update.
177 +
    pub fn report_update(&self, note: &str) -> Result<()> {
178 +
        self.post_with("/flow/runner/update", &flow_core::api::UpdateReport { note: note.to_string() }, &[])?;
179 +
        Ok(())
180 +
    }
181 +
182 +
    /// Download an offered build to `to`, refusing it unless its BLAKE3
183 +
    /// is the one the hub attests and it is the size it said.
184 +
    pub fn download_build(&self, offer: &flow_core::api::UpdateOffer, to: &Path) -> Result<()> {
185 +
        use std::io::{Read, Write};
186 +
        let url = format!("{}{}", self.base, offer.url);
187 +
        let r = self.agent.get(&url).timeout(Duration::from_secs(1800)).call().map_err(|e| anyhow::anyhow!("{url}: {e}"))?;
188 +
        let part = to.with_extension("partial");
189 +
        let mut out = std::fs::File::create(&part).with_context(|| format!("writing {}", part.display()))?;
190 +
        let (mut hasher, mut n, mut buf) = (blake3::Hasher::new(), 0u64, vec![0u8; 1 << 16]);
191 +
        let mut body = r.into_reader().take(offer.size + 1);
192 +
        loop {
193 +
            let k = body.read(&mut buf)?;
194 +
            if k == 0 {
195 +
                break;
196 +
            }
197 +
            hasher.update(&buf[..k]);
198 +
            out.write_all(&buf[..k])?;
199 +
            n += k as u64;
200 +
        }
201 +
        drop(out);
202 +
        let got = hasher.finalize().to_hex().to_string();
203 +
        if n != offer.size || got != offer.blake3 {
204 +
            let _ = std::fs::remove_file(&part);
205 +
            bail!("{} does not match what the hub attests ({n} bytes, BLAKE3 {}…)", offer.asset, &got[..12]);
206 +
        }
207 +
        std::fs::rename(&part, to)?;
208 +
        Ok(())
209 +
    }
210 +
106 211
    pub fn log(&self, job: i64, records: Vec<Record>) -> Result<()> {
107 212
        self.post(&format!("/flow/jobs/{job}/log"), &LogBatch { records })?;
108 213
        Ok(())
…
3 unchanged lines
112 217
    /// a blob of the job's repository and answers with its ObjectRef.
113 218
    pub fn artifact(&self, job: i64, name: &str, bytes: &[u8]) -> Result<String> {
114 219
        let url = format!("{}/api/v1/flow/jobs/{job}/artifacts/{name}", self.base);
115 -
        let mut delays = RETRY_DELAYS.iter();
116 -
        loop {
117 -
            let req = self
118 -
                .agent
220 +
        let r = retrying(&url, &PATIENT_DELAYS, true, || {
221 +
            self.agent
222 +
                .post(&url)
223 +
                .timeout(Duration::from_secs(600))
224 +
                .set("Authorization", &format!("Bearer {}", self.token))
225 +
                .set("Content-Type", "application/octet-stream")
226 +
                .send_bytes(bytes)
227 +
                .map_err(Box::new)
228 +
        })?;
229 +
        let v: serde_json::Value = r.into_json()?;
230 +
        v["blob"].as_str().map(String::from).context("the hub named no blob")
231 +
    }
232 +
233 +
    /// Keep a checkpoint of job `job` (oxygenflow#77) as its runner hands
234 +
    /// it back: the hub answers with its blob, which the next attempt's
235 +
    /// claim names.
236 +
    pub fn save_checkpoint(&self, job: i64, bytes: &[u8]) -> Result<String> {
237 +
        let url = format!("{}/api/v1/flow/jobs/{job}/checkpoint", self.base);
238 +
        let r = retrying(&url, &PATIENT_DELAYS, true, || {
239 +
            self.agent
119 240
                .post(&url)
120 241
                .timeout(Duration::from_secs(600))
121 242
                .set("Authorization", &format!("Bearer {}", self.token))
122 -
                .set("Content-Type", "application/octet-stream");
123 -
            match req.send_bytes(bytes) {
124 -
                Ok(r) => {
125 -
                    let v: serde_json::Value = r.into_json()?;
126 -
                    return v["blob"].as_str().map(String::from).context("the hub named no blob");
127 -
                }
128 -
                Err(ureq::Error::Status(code, r)) => {
129 -
                    let text = r.into_string().unwrap_or_default();
130 -
                    match delays.next() {
131 -
                        Some(d) if code >= 500 => std::thread::sleep(*d),
132 -
                        _ => bail!("{url}: HTTP {code}: {text}"),
133 -
                    }
134 -
                }
135 -
                Err(e) => bail!("{url}: {e}"),
136 -
            }
243 +
                .set("Content-Type", "application/octet-stream")
244 +
                .send_bytes(bytes)
245 +
                .map_err(Box::new)
246 +
        })?;
247 +
        let v: serde_json::Value = r.into_json()?;
248 +
        v["blob"].as_str().map(String::from).context("the hub named no blob")
249 +
    }
250 +
251 +
    /// The checkpoint an earlier attempt of job `job` saved (oxygenflow#77),
252 +
    /// which the caller checks against the blob its claim names.
253 +
    pub fn checkpoint(&self, job: i64) -> Result<Vec<u8>> {
254 +
        let url = format!("{}/api/v1/flow/jobs/{job}/checkpoint", self.base);
255 +
        let r = retrying(&url, &PATIENT_DELAYS, false, || {
256 +
            self.agent
257 +
                .get(&url)
258 +
                .timeout(Duration::from_secs(600))
259 +
                .set("Authorization", &format!("Bearer {}", self.token))
260 +
                .call()
261 +
                .map_err(Box::new)
262 +
        })?;
263 +
        let mut bytes = Vec::new();
264 +
        std::io::Read::read_to_end(&mut std::io::Read::take(r.into_reader(), crate::checkpoint::MAX_BYTES as u64 + 1), &mut bytes)?;
265 +
        Ok(bytes)
266 +
    }
267 +
268 +
    /// Input `index` of job `job` (oxygenflow#58): the bytes of the
269 +
    /// producing job's artifact, which the caller checks against the blob the
270 +
    /// claim names.
271 +
    pub fn input(&self, job: i64, index: usize) -> Result<Vec<u8>> {
272 +
        let url = format!("{}/api/v1/flow/jobs/{job}/inputs/{index}", self.base);
273 +
        let r = retrying(&url, &PATIENT_DELAYS, false, || {
274 +
            self.agent
275 +
                .get(&url)
276 +
                .timeout(Duration::from_secs(600))
277 +
                .set("Authorization", &format!("Bearer {}", self.token))
278 +
                .call()
279 +
                .map_err(Box::new)
280 +
        })?;
281 +
        let mut bytes = Vec::new();
282 +
        let mut reader = r.into_reader();
283 +
        std::io::Read::read_to_end(&mut std::io::Read::take(&mut reader, MAX_INPUT_BYTES as u64 + 1), &mut bytes)?;
284 +
        if bytes.len() > MAX_INPUT_BYTES {
285 +
            bail!("larger than {} MiB", MAX_INPUT_BYTES >> 20);
137 286
        }
287 +
        Ok(bytes)
138 288
    }
139 289
140 290
    /// Upload file `path` of report `name` of job `job` (oxygenhub#127);
…
1 unchanged lines
142 292
    /// report's manifest follows through [`Hub::artifact`].
143 293
    pub fn report_file(&self, job: i64, name: &str, path: &str, bytes: &[u8]) -> Result<String> {
144 294
        let url = format!("{}/api/v1/flow/jobs/{job}/reports/{name}/files", self.base);
145 -
        let mut delays = RETRY_DELAYS.iter();
146 -
        loop {
147 -
            let req = self
148 -
                .agent
295 +
        let r = retrying(&url, &PATIENT_DELAYS, true, || {
296 +
            self.agent
149 297
                .post(&url)
150 298
                .timeout(Duration::from_secs(600))
151 299
                .set("Authorization", &format!("Bearer {}", self.token))
152 300
                .set("Content-Type", "application/octet-stream")
153 -
                .set("X-Flow-Report-Path", path);
154 -
            match req.send_bytes(bytes) {
155 -
                Ok(r) => {
156 -
                    let v: serde_json::Value = r.into_json()?;
157 -
                    return v["blob"].as_str().map(String::from).context("the hub named no blob");
158 -
                }
159 -
                Err(ureq::Error::Status(code, r)) => {
160 -
                    let text = r.into_string().unwrap_or_default();
161 -
                    match delays.next() {
162 -
                        Some(d) if code >= 500 => std::thread::sleep(*d),
163 -
                        _ => bail!("{url}: HTTP {code}: {text}"),
164 -
                    }
165 -
                }
166 -
                Err(e) => bail!("{url}: {e}"),
167 -
            }
168 -
        }
301 +
                .set("X-Flow-Report-Path", path)
302 +
                .send_bytes(bytes)
303 +
                .map_err(Box::new)
304 +
        })?;
305 +
        let v: serde_json::Value = r.into_json()?;
306 +
        v["blob"].as_str().map(String::from).context("the hub named no blob")
169 307
    }
170 308
171 309
    /// Report a job's end until the hub has it (oxygenhub#71): while the hub
…
90 unchanged lines
262 400
        got.credential, got.name
263 401
    );
264 402
    if let Some(ca) = &ca {
265 -
        text.push_str(&format!("ca = {:?}\n", ca.display().to_string()));
403 +
        let kept = keep_ca(&home.root, ca)?;
404 +
        text.push_str(&format!("ca = {:?}\n", kept.display().to_string()));
266 405
    }
267 406
    if !labels.is_empty() {
268 407
        text.push_str(&format!("labels = {labels:?}\n"));
…
1 unchanged lines
270 409
    let path = home.root.join(crate::home::RUNNER_TOML);
271 410
    {
272 411
        use std::io::Write;
273 -
        use std::os::unix::fs::OpenOptionsExt;
274 412
        let tmp = path.with_extension("toml.tmp");
275 -
        let mut f = std::fs::OpenOptions::new().write(true).create(true).truncate(true).mode(0o600).open(&tmp)?;
413 +
        let mut f = crate::portable::private(std::fs::OpenOptions::new().write(true).create(true).truncate(true)).open(&tmp)?;
276 414
        f.write_all(text.as_bytes())?;
277 415
        f.sync_all()?;
278 416
        std::fs::rename(&tmp, &path)?;
…
1 unchanged lines
280 418
    Ok(got)
281 419
}
282 420
421 +
/// Copy the hub's CA certificate `ca` into the Flow home `root` (#98) and
422 +
/// give the copy's absolute path, for runner.toml: a service starts in
423 +
/// another directory, and the file given may move or go.
424 +
fn keep_ca(root: &Path, ca: &Path) -> Result<PathBuf> {
425 +
    let kept = root.join(crate::home::CA_PEM);
426 +
    let same = matches!((std::fs::canonicalize(ca), std::fs::canonicalize(&kept)), (Ok(a), Ok(b)) if a == b);
427 +
    if !same {
428 +
        std::fs::copy(ca, &kept).with_context(|| format!("copy {} into {}", ca.display(), root.display()))?;
429 +
    }
430 +
    Ok(std::fs::canonicalize(&kept)?)
431 +
}
432 +
283 433
/// Waits before trying a hub request again after a server error.
284 434
const RETRY_DELAYS: [Duration; 3] = [Duration::from_secs(1), Duration::from_secs(2), Duration::from_secs(4)];
285 435
436 +
/// Waits for what a job leaves behind (oxygenflow#49): three minutes in
437 +
/// all, longer than a hub takes to restart.
438 +
/// Check an input's bytes against the blob the producing job's receipt
439 +
/// names, then write them at its path under `tree` (the workspace's `/src`).
440 +
fn place_input(tree: &Path, i: &flow_core::api::ArtifactInput, bytes: &[u8]) -> Result<[u8; 32]> {
441 +
    let Some(rel) = i.path.strip_prefix("/src/").filter(|r| r.split('/').all(|p| !p.is_empty() && p != "." && p != ".." && p != ".oxy"))
442 +
    else {
443 +
        bail!("path {:?} is not a clean path under /src", i.path);
444 +
    };
445 +
    let blob = Repo::blob_ref(bytes);
446 +
    if blob.to_hex() != i.blob {
447 +
        bail!("it is not what {}'s receipt names (blob {}, not {})", i.from, &blob.to_hex()[..12], &i.blob[..i.blob.len().min(12)]);
448 +
    }
449 +
    let dest = tree.join(rel);
450 +
    if let Some(dir) = dest.parent() {
451 +
        std::fs::create_dir_all(dir)?;
452 +
    }
453 +
    std::fs::write(&dest, bytes)?;
454 +
    Ok(blob.0)
455 +
}
456 +
457 +
/// The largest input a job takes: the hub's artifact limit.
458 +
pub const MAX_INPUT_BYTES: usize = 512 << 20;
459 +
460 +
const PATIENT_DELAYS: [Duration; 15] = {
461 +
    let mut d = [Duration::from_secs(15); 15];
462 +
    d[0] = Duration::from_secs(1);
463 +
    d[1] = Duration::from_secs(2);
464 +
    d[2] = Duration::from_secs(4);
465 +
    d[3] = Duration::from_secs(8);
466 +
    d
467 +
};
468 +
469 +
/// Send a request until the hub answers, waiting `delays` in turn after a
470 +
/// server error (5xx: nothing was committed) and, when `connection`, after
471 +
/// the hub could not be reached at all. A refusal (4xx) is final, as a
472 +
/// [`Refused`] the caller can look at.
473 +
fn retrying(
474 +
    url: &str,
475 +
    delays: &[Duration],
476 +
    connection: bool,
477 +
    mut send: impl FnMut() -> Result<ureq::Response, Box<ureq::Error>>,
478 +
) -> Result<ureq::Response> {
479 +
    let mut delays = delays.iter();
480 +
    loop {
481 +
        let why = match send().map_err(|e| *e) {
482 +
            Ok(r) => return Ok(r),
483 +
            Err(ureq::Error::Status(code, r)) if code < 500 => {
484 +
                let text = r.into_string().unwrap_or_default();
485 +
                return Err(anyhow::Error::new(Refused { code, text }).context(url.to_string()));
486 +
            }
487 +
            Err(ureq::Error::Status(code, r)) => format!("HTTP {code}: {}", r.into_string().unwrap_or_default()),
488 +
            Err(e) if connection => e.to_string(),
489 +
            Err(e) => bail!("{url}: {e}"),
490 +
        };
491 +
        match delays.next() {
492 +
            Some(d) => std::thread::sleep(*d),
493 +
            None => bail!("{url}: {why}"),
494 +
        }
495 +
    }
496 +
}
497 +
286 498
/// What the runner may run at once (flow.toml `[runner]`): at most
287 499
/// `max_jobs` jobs, whose sandboxes together fit `memory_mib` and `cpus`.
288 500
#[derive(Debug, Default, PartialEq, Eq)]
…
46 unchanged lines
335 547
    let mut missing = Vec::new();
336 548
    let kvm = std::fs::OpenOptions::new().read(true).write(true).open("/dev/kvm");
337 549
    match (kvm, home.config.microvm.is_some()) {
550 +
        _ if !cfg!(target_os = "linux") => missing.push("microvm: the tier runs only on Linux (Firecracker and KVM)".to_string()),
338 551
        (Ok(_), true) => have.push("microvm".to_string()),
339 552
        (Ok(_), false) => missing.push("microvm: flow.toml has no [microvm] section".to_string()),
340 553
        (Err(e), _) => missing.push(format!(
…
4 unchanged lines
345 558
        Ok(()) => have.push("syscall".to_string()),
346 559
        Err(e) => missing.push(format!("syscall: {e}")),
347 560
    }
561 +
    // wasmtime runs on every machine a build with it does (oxygenflow#75).
562 +
    if crate::wasm::AVAILABLE {
563 +
        have.push("wasm".to_string());
564 +
    } else {
565 +
        missing.push("wasm: this build has no WASM tier".to_string());
566 +
    }
348 567
    if have.is_empty() {
349 568
        return Err(format!("no isolation tier can run here: {}", missing.join("; ")));
350 569
    }
…
3 unchanged lines
354 573
/// Fetch `claim`'s image from the hub when this home lacks it
355 574
/// (oxygenflow#39), saying so in the job's log. A hub that does not hold it
356 575
/// leaves it missing, and the preflight declines the job as before.
357 -
fn fetch_image(home: &Home, hub: &Hub, cfg: &RunnerConfig, claim: &Claim) {
576 +
fn fetch_image(home: &Home, hub: &Hub, cfg: &RunnerConfig, swarm: Option<&crate::swarm::Swarm>, claim: &Claim) -> Option<u64> {
358 577
    let hash = &claim.sandbox.rootfs;
359 578
    if hash.is_empty() || home.rootfs(hash).is_file() || !jet::images::is_hash(hash) {
360 -
        return;
579 +
        return None;
361 580
    }
362 581
    let note = |text: String| {
363 582
        eprintln!("job {}: {text}", claim.job_id);
…
3 unchanged lines
367 586
    static FETCHING: std::sync::Mutex<()> = std::sync::Mutex::new(());
368 587
    let _one = FETCHING.lock().unwrap_or_else(|p| p.into_inner());
369 588
    if home.rootfs(hash).is_file() {
370 -
        return;
589 +
        return None;
371 590
    }
372 591
    let client = match crate::images::Client::new(&cfg.hub, &cfg.token, cfg.ca.as_deref()) {
373 592
        Ok(c) => c,
374 -
        Err(e) => return note(format!("image {}: cannot reach the hub: {e:#}", &hash[..12])),
593 +
        Err(e) => {
594 +
            note(format!("image {}: cannot reach the hub: {e:#}", &hash[..12]));
595 +
            return None;
596 +
        }
375 597
    };
376 -
    note(format!("image {} is not on this runner: fetching it from the hub", &hash[..12]));
377 -
    match client.fetch(home, hash) {
598 +
    let fetching = Instant::now();
599 +
    let fetched = match swarm {
600 +
        Some(sw) => {
601 +
            note(format!("image {} is not on this runner: fetching it from its peers and the hub", &hash[..12]));
602 +
            sw.fetch(home, hash)
603 +
        }
604 +
        None => {
605 +
            note(format!("image {} is not on this runner: fetching it from the hub", &hash[..12]));
606 +
            client.fetch(home, hash)
607 +
        }
608 +
    };
609 +
    let took = fetching.elapsed().as_millis() as u64;
610 +
    match fetched {
378 611
        Ok(Some(m)) => note(format!(
379 -
            "image {}: fetched {} chunks, {} in {:.1} s",
612 +
            "image {}: fetched {} chunks, {}{} in {:.1} s",
380 613
            &hash[..12],
381 614
            m.chunks,
382 615
            crate::images::human(m.bytes),
616 +
            if swarm.is_some() { format!(" ({} from peers)", crate::images::human(m.peer_bytes)) } else { String::new() },
383 617
            m.elapsed.as_secs_f64()
384 618
        )),
385 619
        Ok(None) => note(format!("image {}: the hub does not hold it either", &hash[..12])),
386 620
        Err(e) => note(format!("image {}: fetch failed: {e:#}", &hash[..12])),
387 621
    }
622 +
    Some(took)
388 623
}
389 624
390 625
/// Why this runner cannot run `claim` now (spec §11.5), checked before
…
16 unchanged lines
407 642
                return Some(format!("the syscall tier cannot run here: {e}"));
408 643
            }
409 644
        }
410 -
        flow_core::Runtime::Wasm => return Some("this runner has no wasm tier".into()),
645 +
        // No image and no architecture: only the memory matters.
646 +
        flow_core::Runtime::Wasm => {
647 +
            return available_memory_mib()
648 +
                .filter(|free| *free < u64::from(sb.memory_mib))
649 +
                .map(|free| format!("only {free} MiB of memory is free here; the sandbox needs {} MiB", sb.memory_mib));
650 +
        }
411 651
    }
412 652
    sandbox_problem(home, sb, std::env::consts::ARCH, available_memory_mib())
413 653
}
…
21 unchanged lines
435 675
    None
436 676
}
437 677
678 +
/// The memory to claim with: the budget's room, never more than the host
679 +
/// has free now. An idle runner's budget is unlimited (a lone job may use
680 +
/// it all), so without this it was offered sandboxes it then declined for
681 +
/// want of memory, one after another, having fetched their images first
682 +
/// (an 8 GiB job on a 7 GB machine).
683 +
fn claim_memory(room: Option<u32>, host_free_mib: Option<u64>) -> Option<u32> {
684 +
    match (room, host_free_mib) {
685 +
        (Some(r), Some(h)) => Some(r.min(u32::try_from(h).unwrap_or(u32::MAX))),
686 +
        (None, Some(h)) => Some(u32::try_from(h).unwrap_or(u32::MAX)),
687 +
        (room, None) => room,
688 +
    }
689 +
}
690 +
438 691
/// `MemAvailable` from /proc/meminfo, in MiB.
439 692
fn available_memory_mib() -> Option<u64> {
440 693
    let info = std::fs::read_to_string("/proc/meminfo").ok()?;
…
2 unchanged lines
443 696
}
444 697
445 698
pub fn serve(home: &Home) -> Result<()> {
699 +
    drain_on_stop();
700 +
    crate::update::note_runner(&home.root);
701 +
    // `flow runner update` swapped a new build in and asks this one to
702 +
    // leave with the notice it gave (#85).
703 +
    #[cfg(unix)]
704 +
    {
705 +
        let handler = on_update as extern "C" fn(libc::c_int) as libc::sighandler_t;
706 +
        unsafe { libc::signal(libc::SIGUSR1, handler) };
707 +
    }
708 +
    if let Some(p) = crate::update::pending(&home.root) {
709 +
        // On trial: a build that does not reach the hub in time goes back.
710 +
        let root = home.root.clone();
711 +
        std::thread::spawn(move || {
712 +
            std::thread::sleep(crate::update::PROVE_WITHIN);
713 +
            if crate::update::pending(&root).is_some_and(|now| now.to == p.to && now.at_ms == p.at_ms) {
714 +
                let why = format!("did not reach the hub within {} minutes", crate::update::PROVE_WITHIN.as_secs() / 60);
715 +
                if crate::update::roll_back(&root, &p, &why).is_ok() {
716 +
                    std::process::exit(1);
717 +
                }
718 +
            }
719 +
        });
720 +
    }
721 +
    let served = serve_until_drained(home);
722 +
    crate::update::forget_runner(&home.root);
723 +
    // Left for a new build: on Windows, end as a failure, so a Task
724 +
    // Scheduler task set to restart on failure starts it (systemd's
725 +
    // Restart=always starts it after a clean exit).
726 +
    if cfg!(windows) && served.is_ok() && LEFT_FOR_UPDATE.load(std::sync::atomic::Ordering::SeqCst) {
727 +
        eprintln!("flow runner: left for the new build; exiting 75 so it is started");
728 +
        std::process::exit(75);
729 +
    }
730 +
    served
731 +
}
732 +
733 +
/// Set once the runner leaves for a new build.
734 +
static LEFT_FOR_UPDATE: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
735 +
736 +
/// Set by SIGUSR1: a new build is in place; leave for it.
737 +
static UPDATE: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
738 +
739 +
#[cfg(unix)]
740 +
extern "C" fn on_update(_: libc::c_int) {
741 +
    UPDATE.store(true, std::sync::atomic::Ordering::SeqCst);
742 +
}
743 +
744 +
/// The runner as a library, as the Android app runs it (oxygenhub-android#8):
745 +
/// no signal handlers; [`stop`] drains it, and this returns once the jobs
746 +
/// it holds are done and reported. It may be started again afterwards.
747 +
pub fn serve_embedded(home: &Home) -> Result<()> {
748 +
    STOPS.store(0, std::sync::atomic::Ordering::SeqCst);
749 +
    *LEAVING.lock().unwrap_or_else(|p| p.into_inner()) = None;
750 +
    serve_until_drained(home)
751 +
}
752 +
753 +
/// A departure ([`leave_within`]): why, and when the runner must be gone.
754 +
#[derive(Clone, Debug)]
755 +
struct Leave {
756 +
    reason: String,
757 +
    by: Instant,
758 +
}
759 +
760 +
/// Set once the runner is leaving: a job whose expected rest fits before
761 +
/// the deadline finishes; the others are stopped and handed back
762 +
/// (oxygenflow#76).
763 +
static LEAVING: std::sync::Mutex<Option<Leave>> = std::sync::Mutex::new(None);
764 +
765 +
fn leaving() -> Option<String> {
766 +
    LEAVING.lock().unwrap_or_else(|p| p.into_inner()).as_ref().map(|l| l.reason.clone())
767 +
}
768 +
769 +
/// Ask a runner started with [`serve_embedded`] to leave now: claim nothing
770 +
/// more, stop the jobs it holds and hand them back to the hub, which queues
771 +
/// them ahead of new work for another runner (oxygenflow#76), then return.
772 +
/// For a departure with no time to finish: a phone off the allowed network.
773 +
pub fn leave(reason: &str) {
774 +
    leave_within(reason, Duration::ZERO);
775 +
}
776 +
777 +
/// [`leave`], with `notice`: a job the hub expects to finish within it
778 +
/// (the claim's `expected_ms`, less the time it has run) finishes and is
779 +
/// reported; the others, and any still running when the notice is up, are
780 +
/// handed back. For a phone taken off charge, a spot reclaim.
781 +
pub fn leave_within(reason: &str, notice: Duration) {
782 +
    *LEAVING.lock().unwrap_or_else(|p| p.into_inner()) = Some(Leave { reason: reason.to_string(), by: Instant::now() + notice });
783 +
    stop();
784 +
}
785 +
786 +
/// Ask a runner started with [`serve_embedded`] to drain: claim nothing
787 +
/// more, finish and report what it holds, then return (oxygenflow#76's
788 +
/// drain, for a phone taken off charge or off the allowed network).
789 +
pub fn stop() {
790 +
    STOPS.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
791 +
}
792 +
793 +
fn serve_until_drained(home: &Home) -> Result<()> {
446 794
    let cfg = home.config.runner.clone().context("flow.toml has no [runner] section")?;
447 795
    // A runner offers only what it can run: a machine without KVM claims
448 796
    // nothing rather than taking MicroVM jobs and failing them (#36).
…
5 unchanged lines
454 802
    let name = cfg.name.clone().unwrap_or_else(|| "flow".into());
455 803
    let budget = Budget::of(&cfg);
456 804
    let key = home.runner_key(&name)?;
805 +
    // The swarm (docs/jetstream.md §8.1): a runner that cannot serve its
806 +
    // peers still runs jobs, fetching images from the hub alone.
807 +
    let swarm = crate::swarm::Swarm::start(home, &cfg).unwrap_or_else(|e| {
808 +
        eprintln!("flow runner: not joining the swarm: {e:#}");
809 +
        None
810 +
    });
457 811
    eprintln!(
458 812
        "flow runner {name}: polling {} (up to {} jobs at once, tiers {}), signing receipts as {}",
459 813
        cfg.hub,
…
1 unchanged lines
461 815
        runtimes.join(", "),
462 816
        key.identity.hash().to_hex()
463 817
    );
818 +
    if let Some(addr) = &cfg.metrics {
819 +
        crate::metrics::serve(addr, cfg.metrics_token.clone(), name.clone());
820 +
    }
821 +
    // The manifests of images held here that the hub keeps none of, so
822 +
    // peers in this runner's site can fetch them without anyone pushing
823 +
    // them (oxygenflow#94): at start and every ten minutes, which also
824 +
    // catches an image imported while it runs. Only a runner that serves
825 +
    // its peers has anything to offer.
826 +
    if swarm.is_some() {
827 +
        let (root, cfg) = (home.root.clone(), cfg.clone());
828 +
        std::thread::spawn(move || loop {
829 +
            let offered = Home::locate(Some(&root))
830 +
                .and_then(|home| crate::images::Client::new(&cfg.hub, &cfg.token, cfg.ca.as_deref())?.offer_held(&home));
831 +
            match offered {
832 +
                Ok(0) => {}
833 +
                Ok(n) => eprintln!("flow runner: sent the hub the manifests of {n} image(s) only runners hold"),
834 +
                Err(e) => eprintln!("flow runner: offering held images: {e:#}"),
835 +
            }
836 +
            std::thread::sleep(Duration::from_secs(600));
837 +
        });
838 +
    }
464 839
    let used = std::sync::Mutex::new(Used::default());
465 840
    let mut idle = Duration::from_secs(2);
466 841
    let pending = report_pending(home);
…
20 unchanged lines
487 862
        };
488 863
        let mut speed = measure();
489 864
        let mut benched = Instant::now();
865 +
        let mut said_draining = false;
866 +
        // When the hub last heard from us (oxygenhub#243): a claim, or a
867 +
        // heartbeat while there is no room to claim.
868 +
        let mut heard = Instant::now();
869 +
        let beat = |heard: &mut Instant| {
870 +
            if heard.elapsed() >= HUB_HEARTBEAT {
871 +
                if let Err(e) = hub.heartbeat() {
872 +
                    eprintln!("flow runner {name}: heartbeat: {e:#}");
873 +
                }
874 +
                *heard = Instant::now();
875 +
            }
876 +
        };
877 +
        let mut on_trial = crate::update::pending(&home.root).is_some();
878 +
        // Runner updates (oxygenhub#322): asked of the hub once it has
879 +
        // answered a claim, then every 15 minutes.
880 +
        let mut update_asked: Option<Instant> = None;
881 +
        let mut update_said = String::new();
490 882
        loop {
883 +
            if !stopping() && crate::update::drain_requested(&home.root) {
884 +
                eprintln!("flow runner {name}: asked to drain (flow runner drain)");
885 +
                stop();
886 +
            }
887 +
            if UPDATE.swap(false, std::sync::atomic::Ordering::SeqCst) || crate::update::requested(&home.root) {
888 +
                let p = crate::update::pending(&home.root);
889 +
                let notice = Duration::from_secs(p.as_ref().map_or(0, |p| p.notice_secs));
890 +
                let to = p.map_or_else(|| "a new build".to_string(), |p| format!("flow {}", p.to));
891 +
                eprintln!("flow runner {name}: updating to {to}: leaving within {} s", notice.as_secs());
892 +
                LEFT_FOR_UPDATE.store(true, std::sync::atomic::Ordering::SeqCst);
893 +
                leave_within(&format!("updating to {to}"), notice);
894 +
            }
895 +
            // Draining (oxygenflow#50): told to stop, claim nothing more and
896 +
            // stop once every job held is done and reported.
897 +
            if stopping() {
898 +
                let (jobs, holding) = (used.lock().unwrap().jobs, held.lock().unwrap().len());
899 +
                if jobs == 0 && holding == 0 {
900 +
                    eprintln!("flow runner {name}: drained, stopping");
901 +
                    break;
902 +
                }
903 +
                if !said_draining {
904 +
                    eprintln!("flow runner {name}: draining: claiming nothing more, stopping when the {jobs} job(s) held are done (stop again to stop now)");
905 +
                    said_draining = true;
906 +
                }
907 +
                beat(&mut heard);
908 +
                std::thread::sleep(Duration::from_millis(500));
909 +
                continue;
910 +
            }
491 911
            if used.lock().unwrap().jobs == 0 && benched.elapsed() >= Duration::from_secs(15 * 60) {
492 912
                speed = measure();
493 913
                benched = Instant::now();
494 914
            }
495 915
            let room = budget.room(*used.lock().unwrap());
496 916
            let Some((free_memory_mib, free_cpus)) = room else {
917 +
                beat(&mut heard);
497 918
                std::thread::sleep(Duration::from_millis(500));
498 919
                continue;
499 920
            };
921 +
            heard = Instant::now();
922 +
            let cache_times = crate::cache::Pool::new(&home.root, cfg.cache_gb).slot_times();
500 923
            let req = ClaimRequest {
501 924
                runner: name.clone(),
502 925
                runtimes: runtimes.clone(),
503 -
                free_memory_mib,
926 +
                free_memory_mib: claim_memory(free_memory_mib, available_memory_mib()),
504 927
                free_cpus,
505 928
                info: Some(RunnerInfo {
506 929
                    version: env!("OXY_VERSION_LONG").into(),
507 -
                    ..RunnerInfo::here(cfg.labels.clone(), budget.max_jobs as u32, cfg.memory_mib, cfg.cpus)
930 +
                    ..RunnerInfo::here(cfg.offered_labels(), budget.max_jobs as u32, cfg.memory_mib, cfg.cpus)
508 931
                }),
509 932
                // Read each time: an image imported while the runner runs counts.
510 933
                images: Some(home.images()),
…
1 unchanged lines
512 935
                holding: Some(held.lock().unwrap().iter().copied().collect()),
513 936
                fetches_images: true,
514 937
                facts: Some(crate::facts::now(home, Some(speed))),
938 +
                swarm: swarm.as_ref().map(|s| s.addr.clone()),
939 +
                caches: Some(cache_times.keys().cloned().collect()),
940 +
                cache_times: Some(cache_times),
515 941
            };
516 -
            match hub.claim(&req) {
942 +
            let answer = hub.claim(&req);
943 +
            if on_trial && answer.is_ok() {
944 +
                crate::update::proven(&home.root);
945 +
                on_trial = false;
946 +
            }
947 +
            if answer.is_ok() && update_asked.is_none_or(|t| t.elapsed() >= UPDATE_EVERY) {
948 +
                update_asked = Some(Instant::now());
949 +
                if let Some(note) = crate::update::take_note(&home.root) {
950 +
                    let _ = hub.report_update(&note);
951 +
                }
952 +
                consider_update(home, &hub, swarm.as_ref(), &mut update_said);
953 +
            }
954 +
            match answer {
517 955
                Ok(Some(claim)) => {
518 956
                    idle = Duration::from_secs(2);
957 +
                    crate::metrics::CLAIMS.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
519 958
                    let (mem, cpus) = (claim.sandbox.memory_mib, claim.sandbox.cpus);
520 959
                    {
521 960
                        let mut u = used.lock().unwrap();
…
2 unchanged lines
524 963
                        u.cpus += cpus;
525 964
                    }
526 965
                    held.lock().unwrap().insert(claim.job_id);
527 -
                    let (home, hub, cfg, used, key, held) = (home, &hub, &cfg, &used, &key, &held);
966 +
                    let (home, hub, cfg, used, key, held, swarm) = (home, &hub, &cfg, &used, &key, &held, swarm.as_ref());
528 967
                    scope.spawn(move || {
529 968
                        let id = claim.job_id;
530 -
                        fetch_image(home, hub, cfg, &claim);
969 +
                        let image_ms = fetch_image(home, hub, cfg, swarm, &claim);
531 970
                        if let Some(why) = preflight(home, &claim) {
532 971
                            eprintln!("job {id}: declined: {why}");
972 +
                            crate::metrics::DECLINED.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
533 973
                            if let Err(e) = hub.decline(id, &why) {
534 974
                                eprintln!("job {id}: decline: {e:#}");
535 975
                            }
…
10 unchanged lines
546 986
                            claim.job.name,
547 987
                            claim.heads.first().map(|h| &h[..12]).unwrap_or("-")
548 988
                        );
549 -
                        let finish = run_claim(home, hub, cfg, key, &claim).unwrap_or_else(|e| Finish {
989 +
                        crate::metrics::RUNNING.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
990 +
                        let _shown = Running::start(id, format!("{} #{} {}", claim.repo, claim.run_number, claim.job.name));
991 +
                        let finish = run_claim(home, hub, cfg, key, &claim, image_ms).unwrap_or_else(|e| Finish {
550 992
                            exit_code: None,
551 993
                            duration_ms: 0,
552 994
                            receipt: None,
…
1 unchanged lines
554 996
                            error: Some(format!("{e:#}")),
555 997
                            peak_disk_mib: None,
556 998
                            peak_memory_mib: None,
999 +
                            phases: Default::default(),
557 1000
                        });
558 1001
                        eprintln!(
559 1002
                            "job {id}: exit {:?}{}",
560 1003
                            finish.exit_code,
561 1004
                            finish.error.as_deref().map(|e| format!(" ({e})")).unwrap_or_default()
562 1005
                        );
563 -
                        if finish.error.as_deref() != Some(CANCELLED) {
564 -
                            report_finish(home, hub, id, &finish);
1006 +
                        crate::metrics::RUNNING.fetch_sub(1, std::sync::atomic::Ordering::Relaxed);
1007 +
                        let cancelled = finish.error.as_deref() == Some(CANCELLED);
1008 +
                        crate::metrics::finished(finish.exit_code, finish.error.as_deref(), finish.duration_ms, cancelled);
1009 +
                        match leaving() {
1010 +
                            // Stopped as the runner leaves: another runner takes it.
1011 +
                            Some(why) if cancelled => {
1012 +
                                eprintln!("job {id}: handed back: {why}");
1013 +
                                if let Err(e) = hub.release(id, &why) {
1014 +
                                    eprintln!("job {id}: handing back: {e:#}; the hub requeues it once this runner's lease lapses");
1015 +
                                }
1016 +
                            }
1017 +
                            _ if !cancelled => report_finish(home, hub, id, &finish),
1018 +
                            _ => {}
565 1019
                        }
566 1020
                        held.lock().unwrap().remove(&id);
567 1021
                        let mut u = used.lock().unwrap();
…
10 unchanged lines
578 1032
                }
579 1033
            }
580 1034
        }
1035 +
        Ok(())
581 1036
    })
582 1037
}
583 1038
1039 +
/// Pack and keep a handed-back job's checkpoints (oxygenflow#77); what
1040 +
/// happened goes to its log, and a failure only means the next attempt
1041 +
/// starts again.
1042 +
#[allow(clippy::too_many_arguments)]
1043 +
fn save_checkpoint(hub: &Hub, id: i64, job: &flow_core::Job, scratch: &Path, dir_tier: bool, wasm: bool, mut say: impl FnMut(String)) {
1044 +
    if !dir_tier {
1045 +
        say("checkpoint: only the WASM and syscall tiers save one yet; the next attempt starts again".into());
1046 +
        return;
1047 +
    }
1048 +
    let dirs: Vec<(String, PathBuf)> = job
1049 +
        .checkpoints
1050 +
        .iter()
1051 +
        .filter_map(|c| crate::checkpoint::host_dir(c, job, &scratch.join("tree"), scratch, wasm).map(|d| (c.clone(), d)))
1052 +
        .collect();
1053 +
    match crate::checkpoint::pack(&dirs).and_then(|(bytes, n)| Ok((hub.save_checkpoint(id, &bytes)?, n))) {
1054 +
        Ok((blob, n)) => {
1055 +
            say(format!("checkpoint {}: {} files, {} bytes, kept for the next attempt", &blob[..12.min(blob.len())], n.files, n.bytes))
1056 +
        }
1057 +
        Err(e) => say(format!("checkpoint: not kept ({e:#}); the next attempt starts again")),
1058 +
    }
1059 +
}
1060 +
1061 +
/// How many times the runner was told to stop (SIGTERM or SIGINT).
1062 +
static STOPS: std::sync::atomic::AtomicUsize = std::sync::atomic::AtomicUsize::new(0);
1063 +
1064 +
extern "C" fn on_stop(_: libc::c_int) {
1065 +
    // A second stop means now: only async-signal-safe calls here.
1066 +
    if STOPS.fetch_add(1, std::sync::atomic::Ordering::SeqCst) >= 1 {
1067 +
        unsafe { libc::_exit(143) };
1068 +
    }
1069 +
}
1070 +
1071 +
/// Drain instead of dying on the first SIGTERM or SIGINT (oxygenflow#50),
1072 +
/// so restarting a runner never kills the job it runs: the unit's
1073 +
/// `KillMode=mixed` sends the signal to the runner alone, not its microVMs.
1074 +
fn drain_on_stop() {
1075 +
    let handler = on_stop as extern "C" fn(libc::c_int) as libc::sighandler_t;
1076 +
    unsafe {
1077 +
        libc::signal(libc::SIGTERM, handler);
1078 +
        libc::signal(libc::SIGINT, handler);
1079 +
    }
1080 +
}
1081 +
1082 +
fn stopping() -> bool {
1083 +
    STOPS.load(std::sync::atomic::Ordering::SeqCst) > 0
1084 +
}
1085 +
584 1086
/// Batches records to the hub every second (or every 200 lines). A quiet
585 1087
/// job's sink still asks the hub every few seconds whether the job runs,
586 1088
/// with an empty batch; once the hub answers 409 the job is not wanted.
…
3 unchanged lines
590 1092
    buf: Vec<Record>,
591 1093
    last: Instant,
592 1094
    gone: bool,
1095 +
    /// Set when the runner leaves ([`LEAVING`]): the job stops, to be
1096 +
    /// handed back, unless its expected rest fits the notice.
1097 +
    leaving: &'a std::sync::Mutex<Option<Leave>>,
1098 +
    /// When the job started, and how long the hub expects it to take.
1099 +
    started: Instant,
1100 +
    expected: Option<Duration>,
593 1101
}
594 1102
1103 +
/// The most log lines sent at once, as a busy job's sink flushes them.
1104 +
const BATCH: usize = 200;
1105 +
1106 +
/// The most log lines kept while the hub cannot be reached.
1107 +
const MAX_UNSENT: usize = 50_000;
1108 +
595 1109
/// How often a quiet job asks whether it was cancelled.
596 1110
const HEARTBEAT: Duration = Duration::from_secs(5);
597 1111
…
3 unchanged lines
601 1115
    }
602 1116
603 1117
    fn cancelled(&mut self) -> bool {
1118 +
        let leave = self.leaving.lock().unwrap_or_else(|p| p.into_inner()).clone();
1119 +
        if let Some(l) = leave {
1120 +
            // Finished if what is left of it fits the notice; else handed
1121 +
            // back as the runner leaves, what the job said so far first.
1122 +
            let now = Instant::now();
1123 +
            let fits = self.expected.is_some_and(|e| now + e.saturating_sub(self.started.elapsed()) <= l.by);
1124 +
            if !fits || now >= l.by {
1125 +
                self.flush();
1126 +
                return true;
1127 +
            }
1128 +
        }
604 1129
        if !self.gone && self.last.elapsed() >= HEARTBEAT {
605 1130
            self.send();
606 1131
        }
…
14 unchanged lines
621 1146
            r.text.truncate(cut);
622 1147
        }
623 1148
        self.buf.push(r);
624 -
        if self.buf.len() >= 200 || self.last.elapsed() >= Duration::from_secs(1) {
1149 +
        if self.buf.len() >= BATCH || self.last.elapsed() >= Duration::from_secs(1) {
625 1150
            self.flush();
626 1151
        }
627 1152
    }
…
5 unchanged lines
633 1158
        self.last = Instant::now();
634 1159
    }
635 1160
636 -
    /// Send what is buffered, even nothing.
1161 +
    /// Send what is buffered, even nothing, in batches of at most
1162 +
    /// `BATCH` lines.
637 1163
    fn send(&mut self) {
638 -
        if self.gone {
1164 +
        loop {
1165 +
            if self.gone {
1166 +
                return;
1167 +
            }
1168 +
            let n = self.buf.len().min(BATCH);
1169 +
            let batch: Vec<Record> = self.buf.drain(..n).collect();
1170 +
            match self.hub.log(self.job, batch.clone()) {
1171 +
                Ok(()) => {}
1172 +
                Err(e) if job_gone(&e) => self.gone = true,
1173 +
                Err(e) => {
1174 +
                    // The hub is away (a restart, say): keep the lines for
1175 +
                    // the next batch, up to a bound, so the log loses none
1176 +
                    // (oxygenflow#49).
1177 +
                    eprintln!("job {}: log: {e:#}; kept for the next batch", self.job);
1178 +
                    let mut kept = batch;
1179 +
                    kept.append(&mut self.buf);
1180 +
                    let over = kept.len().saturating_sub(MAX_UNSENT);
1181 +
                    if over > 0 {
1182 +
                        kept.drain(..over);
1183 +
                        kept.insert(
1184 +
                            0,
1185 +
                            Record {
1186 +
                                t: 0,
1187 +
                                stream: "flow".into(),
1188 +
                                text: format!("({over} log lines lost while the hub could not be reached)"),
1189 +
                            },
1190 +
                        );
1191 +
                    }
1192 +
                    self.buf = kept;
1193 +
                    break;
1194 +
                }
1195 +
            }
1196 +
            if self.buf.is_empty() {
1197 +
                break;
1198 +
            }
1199 +
        }
1200 +
        self.last = Instant::now();
1201 +
    }
1202 +
}
1203 +
1204 +
/// How often a runner with no room to claim tells the hub it is there:
1205 +
/// well inside the hub's 90 s before it shows a runner offline.
1206 +
const HUB_HEARTBEAT: Duration = Duration::from_secs(30);
1207 +
1208 +
/// How often a runner asks the hub whether it should run another build.
1209 +
const UPDATE_EVERY: Duration = Duration::from_secs(15 * 60);
1210 +
1211 +
/// Notice a runner gives itself to leave for an update the hub offered.
1212 +
const UPDATE_NOTICE: Duration = Duration::from_secs(600);
1213 +
1214 +
/// Ask the hub for this runner's update, and act on it: download the build
1215 +
/// it offers, check it against the hub's BLAKE3 and as a runner build, and
1216 +
/// install it, or for a dry run say what would happen. What it did goes to
1217 +
/// the hub once per outcome (`said`).
1218 +
fn consider_update(home: &Home, hub: &Hub, swarm: Option<&crate::swarm::Swarm>, said: &mut String) {
1219 +
    let answer = match hub.update_answer() {
1220 +
        Ok(Some(a)) => a,
1221 +
        Ok(None) => return,
1222 +
        Err(e) => {
1223 +
            eprintln!("flow runner: asking for an update: {e:#}");
639 1224
            return;
640 1225
        }
641 -
        match self.hub.log(self.job, std::mem::take(&mut self.buf)) {
642 -
            Ok(()) => {}
643 -
            Err(e) if job_gone(&e) => self.gone = true,
644 -
            Err(e) => eprintln!("job {}: log: {e:#}", self.job),
1226 +
    };
1227 +
    let Some(offer) = answer.offer else { return };
1228 +
    let mut say = |note: String| {
1229 +
        if *said != note {
1230 +
            eprintln!("flow runner: {note}");
1231 +
            if let Err(e) = hub.report_update(&note) {
1232 +
                eprintln!("flow runner: telling the hub: {e:#}");
1233 +
            }
1234 +
            *said = note;
645 1235
        }
646 -
        self.last = Instant::now();
1236 +
    };
1237 +
    if home.config.runner.as_ref().and_then(|r| r.update.as_ref()).is_some_and(|u| !u.allow) {
1238 +
        return say(format!("refuses flow {}: flow.toml says [runner.update] allow = false", offer.version));
647 1239
    }
1240 +
    let dir = home.root.join("updates");
1241 +
    let file = dir.join(format!("flow-{}", offer.version));
1242 +
    let fresh = std::fs::read(&file).is_ok_and(|b| blake3::hash(&b).to_hex().as_str() == offer.blake3);
1243 +
    if !fresh {
1244 +
        if let Err(e) =
1245 +
            std::fs::create_dir_all(&dir).map_err(anyhow::Error::from).and_then(|_| fetch_build(home, hub, swarm, &offer, &file))
1246 +
        {
1247 +
            return say(format!("flow {}: not fetched: {e:#}", offer.version));
1248 +
        }
1249 +
    }
1250 +
    let exe = match std::env::current_exe() {
1251 +
        Ok(e) => e,
1252 +
        Err(e) => return say(format!("flow {}: cannot find its own binary: {e}", offer.version)),
1253 +
    };
1254 +
    match crate::update::install(home, &exe, &file, UPDATE_NOTICE, answer.policy.dry_run, offer.older) {
1255 +
        Ok(done) if answer.policy.dry_run => say(format!("dry run: flow {} checks out here and would replace flow {}", done.to, done.from)),
1256 +
        Ok(done) => {
1257 +
            let _ = std::fs::remove_file(&file);
1258 +
            say(format!("installed flow {} (was {}); leaving within {} s to start it", done.to, done.from, UPDATE_NOTICE.as_secs()))
1259 +
        }
1260 +
        Err(e) => say(format!("flow {}: not installed: {e:#}", offer.version)),
1261 +
    }
1262 +
}
1263 +
1264 +
/// Fetch an offered build to `file`: through the swarm when this runner is
1265 +
/// in one (oxygenflow#84), as an image named by the build's BLAKE3, which
1266 +
/// the hub chunks when it first offers it, so runners of a site fetch it
1267 +
/// from each other and the hub serves it about once per site; over HTTP
1268 +
/// otherwise, or when the swarm cannot get it. Either way it must be the
1269 +
/// size and BLAKE3 the hub attests.
1270 +
fn fetch_build(
1271 +
    home: &Home,
1272 +
    hub: &Hub,
1273 +
    swarm: Option<&crate::swarm::Swarm>,
1274 +
    offer: &flow_core::api::UpdateOffer,
1275 +
    file: &Path,
1276 +
) -> Result<()> {
1277 +
    if let Some(sw) = swarm.filter(|_| jet::images::is_hash(&offer.blake3)) {
1278 +
        match sw.fetch(home, &offer.blake3) {
1279 +
            Ok(Some(m)) => {
1280 +
                let image = home.rootfs(&offer.blake3);
1281 +
                match crate::update::copy_checked(&image, file, offer.size, &offer.blake3) {
1282 +
                    Ok(()) => {
1283 +
                        eprintln!(
1284 +
                            "flow runner: fetched flow {} through the swarm: {} chunks, {} ({} from peers)",
1285 +
                            offer.version,
1286 +
                            m.chunks,
1287 +
                            crate::images::human(m.bytes),
1288 +
                            crate::images::human(m.peer_bytes)
1289 +
                        );
1290 +
                        return Ok(());
1291 +
                    }
1292 +
                    Err(e) => eprintln!("flow runner: flow {} from the swarm: {e:#}; fetching it over HTTP", offer.version),
1293 +
                }
1294 +
            }
1295 +
            Ok(None) => eprintln!("flow runner: the hub has not shared flow {} with the swarm yet; fetching it over HTTP", offer.version),
1296 +
            Err(e) => eprintln!("flow runner: flow {} through the swarm: {e:#}; fetching it over HTTP", offer.version),
1297 +
        }
1298 +
    }
1299 +
    hub.download_build(offer, file)
1300 +
}
1301 +
1302 +
/// The jobs this runner is running now, for a host to show (the Android
1303 +
/// app's card and notification, oxygenhub-android#8): by job id,
1304 +
/// "owner/name #run job".
1305 +
static RUNNING_NOW: std::sync::Mutex<std::collections::BTreeMap<i64, String>> = std::sync::Mutex::new(std::collections::BTreeMap::new());
1306 +
1307 +
/// A job shown in [`running`] while this lives.
1308 +
struct Running(i64);
1309 +
1310 +
impl Running {
1311 +
    fn start(id: i64, what: String) -> Running {
1312 +
        RUNNING_NOW.lock().unwrap_or_else(|p| p.into_inner()).insert(id, what);
1313 +
        Running(id)
1314 +
    }
1315 +
}
1316 +
1317 +
impl Drop for Running {
1318 +
    fn drop(&mut self) {
1319 +
        RUNNING_NOW.lock().unwrap_or_else(|p| p.into_inner()).remove(&self.0);
1320 +
    }
1321 +
}
1322 +
1323 +
/// What this runner is running now: "owner/name #run job" for each job, in
1324 +
/// the order claimed.
1325 +
pub fn running() -> Vec<String> {
1326 +
    RUNNING_NOW.lock().unwrap_or_else(|p| p.into_inner()).values().cloned().collect()
648 1327
}
649 1328
650 -
/// Held while a job's workspace is prepared (see `run_claim`).
1329 +
/// Held while a job takes its cache disks and images its workspace (see
1330 +
/// `run_claim`): one job at a time on the runner's disk and cache pool.
651 1331
static PREPARE: std::sync::Mutex<()> = std::sync::Mutex::new(());
652 1332
653 -
fn run_claim(home: &Home, hub: &Hub, cfg: &RunnerConfig, key: &RunnerKey, claim: &Claim) -> Result<Finish> {
1333 +
/// Held while the first clone of any repository passes its token through
1334 +
/// the process environment.
1335 +
static CLONE: std::sync::Mutex<()> = std::sync::Mutex::new(());
1336 +
1337 +
/// One lock per cached repository: fetches into it, and exports from it,
1338 +
/// never overlap, while jobs of other repositories go on. A slow repository
1339 +
/// (one kept in a bucket, say) once held every job on the runner waiting
1340 +
/// behind a single global lock.
1341 +
fn repo_lock(name: &str) -> std::sync::Arc<std::sync::Mutex<()>> {
1342 +
    static LOCKS: std::sync::OnceLock<std::sync::Mutex<std::collections::HashMap<String, std::sync::Arc<std::sync::Mutex<()>>>>> =
1343 +
        std::sync::OnceLock::new();
1344 +
    let mut locks = LOCKS.get_or_init(Default::default).lock().unwrap_or_else(|p| p.into_inner());
1345 +
    locks.entry(name.to_string()).or_default().clone()
1346 +
}
1347 +
1348 +
/// Seconds, to one decimal place, for the job's log.
1349 +
fn secs(d: std::time::Duration) -> String {
1350 +
    format!("{:.1} s", d.as_secs_f64())
1351 +
}
1352 +
1353 +
/// `image_ms`: how long fetching its image took, when the runner lacked it.
1354 +
fn run_claim(home: &Home, hub: &Hub, cfg: &RunnerConfig, key: &RunnerKey, claim: &Claim, image_ms: Option<u64>) -> Result<Finish> {
654 1355
    let started = Instant::now();
655 -
    let mut sink = LogSink { hub, job: claim.job_id, buf: Vec::new(), last: Instant::now(), gone: false };
1356 +
    let mut sink = LogSink {
1357 +
        hub,
1358 +
        job: claim.job_id,
1359 +
        buf: Vec::new(),
1360 +
        last: Instant::now(),
1361 +
        gone: false,
1362 +
        leaving: &LEAVING,
1363 +
        started: Instant::now(),
1364 +
        expected: claim.expected_ms.map(Duration::from_millis),
1365 +
    };
656 1366
    let note =
657 1367
        |sink: &mut LogSink, text: String| sink.push(Record { t: started.elapsed().as_millis() as u64, stream: "flow".into(), text });
658 1368
    let heads: Vec<ChangeHash> = claim.heads.iter().map(|h| ChangeHash::from_hex(h).context("bad head")).collect::<Result<_>>()?;
659 1369
    note(&mut sink, format!("fetching {} for state {}", claim.repo, claim.heads.iter().map(|h| &h[..12]).collect::<Vec<_>>().join("+")));
660 -
    // Preparing a workspace (the cached repositories, the export, the
661 -
    // image) is one job at a time: fetches into a cached repository must not
662 -
    // overlap, and the first clone passes its token through the process
663 -
    // environment. Only the microVMs run side by side.
664 -
    let mut preparing = Some(PREPARE.lock().unwrap_or_else(|p| p.into_inner()));
1370 +
    // Preparing a workspace: fetches into a cached repository, and exports
1371 +
    // from it, are one job at a time per repository (`repo_lock`); taking
1372 +
    // cache disks and imaging the workspace are one job at a time on the
1373 +
    // runner (`PREPARE`); a first clone, whose token goes through the process
1374 +
    // environment, one at a time (`CLONE`).
1375 +
    let main_lock = repo_lock(&claim.repo);
1376 +
    let asked = Instant::now();
1377 +
    let mut main_guard = Some(main_lock.lock().unwrap_or_else(|p| p.into_inner()));
1378 +
    if asked.elapsed().as_secs() >= 2 {
1379 +
        note(&mut sink, format!("waited {} for another job fetching {}", secs(asked.elapsed()), claim.repo));
1380 +
    }
1381 +
    let fetching = Instant::now();
665 1382
    let repo = cached_repo(home, cfg, claim)?;
1383 +
    let fetched = fetching.elapsed();
1384 +
    let mut preparing = None;
666 1385
    let scratch = home.root.join("work").join(format!("job-{}", claim.job_id));
667 1386
    let _ = std::fs::remove_dir_all(&scratch);
668 1387
    std::fs::create_dir_all(scratch.join("tree"))?;
669 1388
    let result = (|| -> Result<Finish> {
1389 +
        let exporting = Instant::now();
670 1390
        let mut files = workspace::export_state(&repo, &heads, &scratch.join("tree"))?;
1391 +
        main_guard.take();
1392 +
        note(&mut sink, format!("fetched in {}, {} files exported in {}", secs(fetched), files.len(), secs(exporting.elapsed())));
671 1393
        // Pinned subrepos (Oxygen spec §7.8), at their paths, as part of the
672 1394
        // workspace and of the scope hash.
673 1395
        for sub in &claim.subrepos {
…
9 unchanged lines
683 1405
                    sub.path
684 1406
                ),
685 1407
            );
1408 +
            let sub_lock = repo_lock(&sub.repo);
1409 +
            let asked = Instant::now();
1410 +
            let sub_guard = sub_lock.lock().unwrap_or_else(|p| p.into_inner());
1411 +
            if asked.elapsed().as_secs() >= 2 {
1412 +
                note(&mut sink, format!("waited {} for another job fetching {}", secs(asked.elapsed()), sub.repo));
1413 +
            }
686 1414
            let sub_repo = cached_repo_at(home, cfg, &sub.repo, &sub.clone_url, &claim.token)?;
687 1415
            let sub_heads: Vec<ChangeHash> =
688 1416
                sub.heads.iter().map(|h| ChangeHash::from_hex(h).context("bad subrepo head")).collect::<Result<_>>()?;
689 1417
            let dir = scratch.join("tree").join(&sub.path);
690 1418
            std::fs::create_dir_all(&dir)?;
691 1419
            let sub_files = workspace::export_state(&sub_repo, &sub_heads, &dir)?;
1420 +
            drop(sub_guard);
692 1421
            files.extend(sub_files.into_iter().map(|(p, h)| (format!("{}/{p}", sub.path), h)));
693 1422
        }
694 1423
        let scope_hash = scope_tree_hash(&claim.job.scope, files.iter().map(|(p, h)| (p.as_str(), *h)));
1424 +
        // Inputs (oxygenflow#58): each producing job's artifact, checked
1425 +
        // against the blob its receipt names, then put at its path. Not part
1426 +
        // of the scope: they are the run's, not the repository's.
1427 +
        let mut input_blobs = Vec::new();
1428 +
        for (index, i) in claim.inputs.iter().enumerate() {
1429 +
            let what = format!("input {} from {}", i.artifact, i.from);
1430 +
            let bytes = hub.input(claim.job_id, index).with_context(|| format!("{what}: fetching it"))?;
1431 +
            let blob = place_input(&scratch.join("tree"), i, &bytes).with_context(|| what.clone())?;
1432 +
            note(&mut sink, format!("{what}: {} bytes at {}, blob {} as its receipt names", bytes.len(), i.path, &i.blob[..12]));
1433 +
            input_blobs.push(blob);
1434 +
        }
695 1435
        // The syscall tier runs on the tree itself; a microVM gets an image.
696 1436
        let syscall_tier = claim.sandbox.runtime == flow_core::Runtime::Syscall;
697 -
        if !syscall_tier {
1437 +
        let wasm_tier = claim.sandbox.runtime == flow_core::Runtime::Wasm;
1438 +
        // The syscall and WASM tiers see the job's files as a directory;
1439 +
        // the MicroVM tier as an ext4 image.
1440 +
        let dir_tier = syscall_tier || wasm_tier;
1441 +
        let job = claim.job.in_sandbox(&claim.sandbox).with_flow_env(&claim.flow_env());
1442 +
        // Disk mounts that name a cache start from a kept disk (#28): a
1443 +
        // microVM takes an image already in place as it is, the syscall tier
1444 +
        // a directory (#70). Taken before the workspace is imaged: the
1445 +
        // sources the disk was built from give unchanged files their old
1446 +
        // times back (#52).
1447 +
        let asked = Instant::now();
1448 +
        preparing = Some(PREPARE.lock().unwrap_or_else(|p| p.into_inner()));
1449 +
        if asked.elapsed().as_secs() >= 2 {
1450 +
            note(&mut sink, format!("waited {} for another job imaging its workspace", secs(asked.elapsed())));
1451 +
        }
1452 +
        let pool = crate::cache::Pool::new(&home.root, cfg.cache_gb);
1453 +
        let form = if dir_tier { crate::cache::Form::Dir } else { crate::cache::Form::Image };
1454 +
        let cache_ctx =
1455 +
            crate::cache::Ctx { repo: &claim.repo, bookmark: &claim.bookmark, job: &claim.job.name, image: &claim.sandbox.rootfs, form };
1456 +
        let mut cached: Vec<(usize, String, String)> = Vec::new();
1457 +
        let mut built_from: Option<crate::cache::Sources> = None;
1458 +
        for (i, m) in job.mounts.iter().filter(|m| m.source == flow_core::pipeline::MountSource::Disk).enumerate() {
1459 +
            let Some(name) = &m.cache else { continue };
1460 +
            let key = crate::cache::Pool::key(&cache_ctx, m, |p| files.iter().find(|(f, _)| f == p).map(|(_, h)| hex::encode(h)));
1461 +
            let how = match pool.take(&cache_ctx, m, &key, &crate::cache::disk_path(form, &scratch, i)) {
1462 +
                Some(t) => {
1463 +
                    if built_from.is_none() {
1464 +
                        built_from = t.sources;
1465 +
                    }
1466 +
                    t.how
1467 +
                }
1468 +
                None => format!("cache {name}: none kept yet, an empty disk"),
1469 +
            };
1470 +
            note(&mut sink, format!("{} ({})", how, m.target));
1471 +
            // What the job reads is seen when the disk is pruned (#72).
1472 +
            if let Err(e) = crate::cache::forget_reads(form, &crate::cache::disk_path(form, &scratch, i)) {
1473 +
                note(&mut sink, format!("cache {name}: {e:#}"));
1474 +
            }
1475 +
            cached.push((i, key, how));
1476 +
        }
1477 +
        // What the job reads or writes on its cache from now on is what it
1478 +
        // used; the rest may be pruned before the cache is kept (#72).
1479 +
        let cache_started = if cached.is_empty() { std::time::SystemTime::now() } else { flow_init::prune::start_clock() };
1480 +
        let sources = if cached.is_empty() {
1481 +
            crate::cache::Sources::new()
1482 +
        } else {
1483 +
            let (sources, restored) = workspace::restore_times(&scratch.join("tree"), &files, built_from.as_ref());
1484 +
            if built_from.is_some() {
1485 +
                note(
1486 +
                    &mut sink,
1487 +
                    format!(
1488 +
                        "cache: {restored} of {} files and directories unchanged since the cached build, their times kept",
1489 +
                        sources.len()
1490 +
                    ),
1491 +
                );
1492 +
            }
1493 +
            sources
1494 +
        };
1495 +
        // An earlier attempt's checkpoint (oxygenflow#77): back in place,
1496 +
        // over the cache, before the same exec runs again; the receipt
1497 +
        // lists it among the job's inputs.
1498 +
        if let Some(blob) = &claim.checkpoint {
1499 +
            if dir_tier {
1500 +
                let bytes = hub.checkpoint(claim.job_id).context("fetching the checkpoint")?;
1501 +
                if Repo::blob_ref(&bytes).to_hex() != *blob {
1502 +
                    bail!("the checkpoint fetched is not blob {}, as the claim names", &blob[..12.min(blob.len())]);
1503 +
                }
1504 +
                let place = |root: &str| crate::checkpoint::host_dir(root, &job, &scratch.join("tree"), &scratch, wasm_tier);
1505 +
                let n = crate::checkpoint::unpack(&bytes, &job.checkpoints, place)?;
1506 +
                note(
1507 +
                    &mut sink,
1508 +
                    format!("restored checkpoint {}: {} files, {} bytes, from the attempt handed back", &blob[..12], n.files, n.bytes),
1509 +
                );
1510 +
                input_blobs.push(Repo::blob_ref(&bytes).0);
1511 +
            } else {
1512 +
                note(&mut sink, "a checkpoint was saved, but only the WASM and syscall tiers restore one yet: starting again".into());
1513 +
            }
1514 +
        }
1515 +
        if !dir_tier {
698 1516
            workspace::image(&scratch.join("tree"), &scratch.join("workspace.ext4"))?;
699 1517
            std::fs::remove_dir_all(scratch.join("tree"))?;
700 1518
        }
…
2 unchanged lines
703 1521
        if microvm::Sink::cancelled(&mut sink) {
704 1522
            return Ok(cancelled());
705 1523
        }
706 -
        note(
707 -
            &mut sink,
708 -
            format!(
709 -
                "{}: {} {}, {} MiB, rootfs {}",
710 -
                if syscall_tier { "starting a syscall sandbox" } else { "booting a microVM" },
711 -
                claim.sandbox.cpus,
712 -
                if syscall_tier { "CPU" } else { "vCPU" },
713 -
                claim.sandbox.memory_mib,
714 -
                &claim.sandbox.rootfs[..12]
715 -
            ),
716 -
        );
717 -
        let job = claim.job.in_sandbox(&claim.sandbox).with_flow_env(&claim.flow_env());
1524 +
        if wasm_tier {
1525 +
            note(&mut sink, format!("starting WASM (wasmtime {}): {} MiB", crate::wasm::WASMTIME, claim.sandbox.memory_mib));
1526 +
        } else {
1527 +
            note(
1528 +
                &mut sink,
1529 +
                format!(
1530 +
                    "{}: {} {}, {} MiB, rootfs {}",
1531 +
                    if syscall_tier { "starting a syscall sandbox" } else { "booting a microVM" },
1532 +
                    claim.sandbox.cpus,
1533 +
                    if syscall_tier { "CPU" } else { "vCPU" },
1534 +
                    claim.sandbox.memory_mib,
1535 +
                    &claim.sandbox.rootfs[..12]
1536 +
                ),
1537 +
            );
1538 +
        }
718 1539
        let timeout = Duration::from_secs(claim.timeout_s);
719 -
        let outcome = if syscall_tier {
1540 +
        // Where the time goes (oxygenflow#35): what came before the
1541 +
        // sandbox, the sandbox, and what comes after.
1542 +
        let prepare_ms = started.elapsed().as_millis() as u64;
1543 +
        let sandbox_started = Instant::now();
1544 +
        let mut outcome = if wasm_tier {
1545 +
            crate::wasm::run(&claim.sandbox, &job, &scratch.join("tree"), &scratch, timeout, &mut sink)?
1546 +
        } else if syscall_tier {
720 1547
            let group = format!("job-{}", claim.job_id);
721 1548
            crate::syscall::run(home, &claim.sandbox, &job, &claim.secrets, &scratch.join("tree"), &scratch, timeout, &mut sink, &group)?
722 1549
        } else {
723 1550
            microvm::run(home, &claim.sandbox, &job, &claim.secrets, &scratch.join("workspace.ext4"), &scratch, timeout, &mut sink)?
724 1551
        };
1552 +
        let run_ms = sandbox_started.elapsed().as_millis() as u64;
725 1553
        if outcome.cancelled {
1554 +
            // Handed back as the runner leaves: its checkpoints first, for
1555 +
            // the next attempt (oxygenflow#77).
1556 +
            if leaving().is_some() && !job.checkpoints.is_empty() {
1557 +
                save_checkpoint(hub, claim.job_id, &job, &scratch, dir_tier, wasm_tier, |t| note(&mut sink, t));
1558 +
            }
726 1559
            return Ok(cancelled());
727 1560
        }
1561 +
        // The receipt says what a cached disk started from.
1562 +
        let disk_mounts: Vec<&flow_core::pipeline::Mount> =
1563 +
            job.mounts.iter().filter(|m| m.source == flow_core::pipeline::MountSource::Disk).collect();
1564 +
        for (i, key, how) in &cached {
1565 +
            outcome.environment.parts.push((format!("cache {}", disk_mounts[*i].target), format!("{key}: {how}")));
1566 +
        }
728 1567
        // A passing job's artifacts, read from its disks and uploaded; a
729 1568
        // declared artifact that cannot be kept fails the job. A failing
730 1569
        // job keeps only its `when "always"` ones (oxygenflow#37), outside
…
4 unchanged lines
735 1574
        let wanted = claim.job.outputs.iter().filter(|o| passed || o.always);
736 1575
        for o in wanted {
737 1576
            if o.kind == flow_core::OutputKind::Report {
738 -
                let read = if syscall_tier {
1577 +
                let read = if dir_tier {
739 1578
                    let (dir, inner) = crate::outputs::locate_dir(&claim.job, &scratch.join("tree"), &scratch, &o.path);
740 1579
                    crate::outputs::read_report_dir(&dir, &inner)
741 1580
                } else {
…
18 unchanged lines
760 1599
                }
761 1600
                continue;
762 1601
            }
763 -
            let read = if syscall_tier {
1602 +
            let read = if dir_tier {
764 1603
                let (dir, inner) = crate::outputs::locate_dir(&claim.job, &scratch.join("tree"), &scratch, &o.path);
765 1604
                crate::outputs::read_dir_file(&dir, &inner)
766 1605
            } else {
…
27 unchanged lines
794 1633
            job_name: claim.job.name.clone(),
795 1634
            pipeline_hash: hex32(&claim.pipeline_hash),
796 1635
            scope_tree_hash: scope_hash,
797 -
            inputs: vec![],
1636 +
            inputs: input_blobs.clone(),
798 1637
            environment: outcome.environment.hash(),
799 1638
            outputs,
800 1639
            exit_code: code as u32,
801 1640
            duration_ms: outcome.duration_ms,
802 1641
        });
803 1642
        let signed_receipt = match &receipt {
804 -
            Some(r) => Some(hex::encode(sign_receipt(&repo, &heads, cfg, key, claim, r)?.encode())),
1643 +
            Some(r) => {
1644 +
                let mut signed = sign_receipt(&repo, &heads, cfg, key, claim, r)?;
1645 +
                if cfg.keyless {
1646 +
                    match sign_keyless(&repo, &heads, cfg, hub, claim, r) {
1647 +
                        Ok(k) => signed = k,
1648 +
                        Err(e) => {
1649 +
                            note(&mut sink, format!("receipt: signed with the runner's key; a short-lived key was not certified: {e:#}"))
1650 +
                        }
1651 +
                    }
1652 +
                }
1653 +
                Some(hex::encode(signed.encode()))
1654 +
            }
805 1655
            None => None,
806 1656
        };
1657 +
        // Measured before a cached disk leaves the scratch directory.
1658 +
        let peak_disk_mib = crate::facts::allocated_mib(&scratch);
1659 +
        // A passing job's cached disks are pruned and kept (#72); a
1660 +
        // failing job's are thrown away, so nothing it wrote reaches a
1661 +
        // later job.
1662 +
        let kept = passed && output_error.is_none();
1663 +
        for (i, key, _) in &cached {
1664 +
            let disk = crate::cache::disk_path(form, &scratch, *i);
1665 +
            let m = disk_mounts[*i];
1666 +
            let prune = crate::cache::Prune {
1667 +
                started: cache_started,
1668 +
                percent: cfg.cache_prune_percent.unwrap_or(flow_init::prune::DEFAULT_PERCENT),
1669 +
                guest: outcome.pruned.iter().find(|(t, _)| *t == m.target).map(|(_, p)| p),
1670 +
            };
1671 +
            match pool.settle(&cache_ctx, m, key, &disk, &sources, kept, &prune) {
1672 +
                Ok(Some(line)) => note(&mut sink, line),
1673 +
                Ok(None) => {}
1674 +
                Err(e) => note(&mut sink, format!("cache: not {}: {e:#}", if kept { "kept" } else { "discarded" })),
1675 +
            }
1676 +
        }
1677 +
        let total_ms = started.elapsed().as_millis() as u64;
1678 +
        let mut phases: std::collections::BTreeMap<String, u64> =
1679 +
            [("prepare", prepare_ms), ("run", run_ms), ("finish", total_ms.saturating_sub(prepare_ms + run_ms))]
1680 +
                .into_iter()
1681 +
                .map(|(k, v)| (k.to_string(), v))
1682 +
                .collect();
1683 +
        if let Some(ms) = image_ms {
1684 +
            phases.insert("image".into(), ms);
1685 +
        }
807 1686
        Ok(Finish {
1687 +
            phases,
808 1688
            exit_code: outcome.exit_code,
809 -
            duration_ms: started.elapsed().as_millis() as u64,
1689 +
            duration_ms: total_ms,
810 1690
            receipt,
811 1691
            signed_receipt,
812 1692
            error: output_error.or_else(|| outcome.timed_out.then(|| "stopped at the time limit".to_string())),
813 1693
            // What the job took (oxygenflow#41): its disks are sparse files
814 1694
            // in the scratch directory, so their blocks now are their peak.
815 -
            peak_disk_mib: crate::facts::allocated_mib(&scratch),
1695 +
            peak_disk_mib,
816 1696
            peak_memory_mib: outcome.peak_memory_mib,
817 1697
        })
818 1698
    })();
…
42 unchanged lines
861 1741
    Ok(oxy_core::receipt::SignedReceipt::sign(receipt.to_oxy(), repo.id()?, state, &run, &key.identity, &key.secret))
862 1742
}
863 1743
1744 +
/// Sign `receipt` with a fresh key the hub certifies for this job now,
1745 +
/// just before signing, so the certificate covers the moment the hub
1746 +
/// accepts it (keyless, oxygenhub#241). The key is dropped with the
1747 +
/// receipt signed: nothing long-lived signs.
1748 +
fn sign_keyless(
1749 +
    repo: &Repo,
1750 +
    heads: &[ChangeHash],
1751 +
    cfg: &RunnerConfig,
1752 +
    hub: &Hub,
1753 +
    claim: &Claim,
1754 +
    receipt: &Receipt,
1755 +
) -> Result<oxy_core::receipt::SignedReceipt> {
1756 +
    let sk = oxy_core::identity::SecretKey::generate();
1757 +
    let now = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map(|d| d.as_millis() as u64).unwrap_or(0);
1758 +
    let name = format!("{} job {}", cfg.name.as_deref().unwrap_or("runner"), claim.job_id);
1759 +
    let id = sk.identity(&name, "", now);
1760 +
    let proof = sk.sign_challenge(&id.hash(), &oxy_core::cert::certify_challenge(&claim.token)).map_err(|e| anyhow::anyhow!("{e}"))?;
1761 +
    let (cert, issuer) = hub.certify(&claim.token, &id, &proof)?;
1762 +
    let state = state_of(heads);
1763 +
    let run = format!("{}/{}/flow/{}", cfg.hub.trim_end_matches('/'), claim.repo, claim.run_number);
1764 +
    Ok(oxy_core::receipt::SignedReceipt::sign(receipt.to_oxy(), repo.id()?, state, &run, &id, &sk).with_certificate(cert, issuer))
1765 +
}
1766 +
864 1767
/// The ObjectRef of the state whose heads are `heads` (Oxygen spec §10.1).
865 1768
pub fn state_of(heads: &[ChangeHash]) -> oxy_core::hash::ObjectRef {
866 1769
    use oxy_core::hash::{hash_object, ObjectType};
…
73 unchanged lines
940 1843
        with_retries(&FETCH_WAITS, || repo.fetch("origin").map(|_| ()).map_err(|e| anyhow::anyhow!("fetch {clone_url}: {e}")))?;
941 1844
        return Ok(repo);
942 1845
    }
943 -
    // First job for this repository: clone with the job's token.
1846 +
    // First job for this repository: clone with the job's token, which goes
1847 +
    // through the process environment, so one clone at a time.
1848 +
    let _clone = CLONE.lock().unwrap_or_else(|p| p.into_inner());
944 1849
    std::env::set_var("OXY_TOKEN", token);
945 1850
    if let Some(ca) = &cfg.ca {
946 1851
        std::env::set_var("OXY_CA", ca);
…
14 unchanged lines
961 1866
962 1867
#[cfg(test)]
963 1868
mod tests {
1869 +
1870 +
    #[test]
1871 +
    fn running_jobs_show_while_they_run() {
1872 +
        let a = super::Running::start(-41, "andy/demo #7 test".into());
1873 +
        let b = super::Running::start(-40, "andy/demo #7 lint".into());
1874 +
        let now = super::running();
1875 +
        assert!(now.contains(&"andy/demo #7 test".to_string()) && now.contains(&"andy/demo #7 lint".to_string()));
1876 +
        drop(a);
1877 +
        assert!(!super::running().contains(&"andy/demo #7 test".to_string()));
1878 +
        drop(b);
1879 +
        assert!(!super::running().iter().any(|r| r.starts_with("andy/demo #7")));
1880 +
    }
1881 +
1882 +
    /// #98: register keeps a copy of the CA in the home, by absolute path,
1883 +
    /// and keeping the home's own copy again is a no-op.
1884 +
    #[test]
1885 +
    fn the_ca_is_kept_in_the_home_by_absolute_path() {
1886 +
        let home = tempfile::tempdir().unwrap();
1887 +
        let elsewhere = tempfile::tempdir().unwrap();
1888 +
        let given = elsewhere.path().join("ca.pem");
1889 +
        std::fs::write(&given, "PEM").unwrap();
1890 +
        let kept = super::keep_ca(home.path(), &given).unwrap();
1891 +
        assert!(kept.is_absolute() && kept.ends_with(crate::home::CA_PEM), "{kept:?}");
1892 +
        assert_eq!(std::fs::read_to_string(&kept).unwrap(), "PEM");
1893 +
        std::fs::remove_file(&given).unwrap();
1894 +
        assert_eq!(super::keep_ca(home.path(), &kept).unwrap(), kept, "the home's copy again");
1895 +
        assert_eq!(std::fs::read_to_string(&kept).unwrap(), "PEM");
1896 +
    }
1897 +
1898 +
    #[test]
1899 +
    fn an_input_is_placed_only_when_it_is_what_the_receipt_names() {
1900 +
        let dir = tempfile::tempdir().unwrap();
1901 +
        let bytes = b"the reviewed plan";
1902 +
        let blob = Repo::blob_ref(bytes).to_hex();
1903 +
        let input = |path: &str, blob: &str| flow_core::api::ArtifactInput {
1904 +
            artifact: "tfplan".into(),
1905 +
            from: "plan".into(),
1906 +
            path: path.into(),
1907 +
            blob: blob.into(),
1908 +
        };
1909 +
        let got = super::place_input(dir.path(), &input("/src/plans/tfplan", &blob), bytes).unwrap();
1910 +
        assert_eq!(hex::encode(got), blob);
1911 +
        assert_eq!(std::fs::read(dir.path().join("plans/tfplan")).unwrap(), bytes);
1912 +
        let wrong = super::place_input(dir.path(), &input("/src/other", &"0".repeat(64)), bytes).unwrap_err();
1913 +
        assert!(format!("{wrong:#}").contains("not what plan's receipt names"), "{wrong:#}");
1914 +
        assert!(!dir.path().join("other").exists(), "nothing written when it does not match");
1915 +
        for bad in ["/etc/passwd", "/src/../x", "/src/.oxy/x", "/src/"] {
1916 +
            assert!(super::place_input(dir.path(), &input(bad, &blob), bytes).is_err(), "{bad}");
1917 +
        }
1918 +
    }
1919 +
964 1920
    use super::*;
965 1921
966 1922
    #[test]
…
23 unchanged lines
990 1946
    }
991 1947
992 1948
    #[test]
1949 +
    fn a_runner_reads_an_unrecorded_images_architecture_and_declines_another() {
1950 +
        // An image fetched or imported with no recorded architecture, built
1951 +
        // for riscv64: the runner reads its shell, records it, and declines
1952 +
        // instead of running it and failing (spec §11.2, §11.5).
1953 +
        let dir = tempfile::tempdir().unwrap();
1954 +
        let home = Home::locate(Some(dir.path().join("home").as_path())).unwrap();
1955 +
        let tree = dir.path().join("tree");
1956 +
        std::fs::create_dir_all(tree.join("bin")).unwrap();
1957 +
        let mut elf = vec![0u8; 64];
1958 +
        elf[..7].copy_from_slice(b"\x7fELF\x02\x01\x01");
1959 +
        elf[18..20].copy_from_slice(&0xF3u16.to_le_bytes());
1960 +
        std::fs::write(tree.join("bin/sh"), &elf).unwrap();
1961 +
        let img = dir.path().join("rootfs.ext4");
1962 +
        let made = std::process::Command::new("mkfs.ext4").args(["-q", "-F", "-d"]).arg(&tree).arg(&img).arg("8M").status();
1963 +
        if !made.is_ok_and(|s| s.success()) {
1964 +
            eprintln!("mkfs.ext4 not found: skipped");
1965 +
            return;
1966 +
        }
1967 +
        let hash = crate::home::hash_file(&img).unwrap();
1968 +
        std::fs::create_dir_all(home.rootfs(&hash).parent().unwrap()).unwrap();
1969 +
        std::fs::rename(&img, home.rootfs(&hash)).unwrap();
1970 +
        assert!(!home.image_manifest(&hash).exists());
1971 +
        let sb: flow_core::Sandbox = serde_json::from_value(serde_json::json!({
1972 +
            "name": "s", "runtime": "syscall", "rootfs": hash, "memory_mib": 64, "cpus": 1, "network": "isolated"
1973 +
        }))
1974 +
        .unwrap();
1975 +
        assert_eq!(
1976 +
            sandbox_problem(&home, &sb, "x86_64", None).unwrap(),
1977 +
            format!("rootfs {} is for riscv64; this runner is x86_64", &hash[..12])
1978 +
        );
1979 +
        assert_eq!(sandbox_problem(&home, &sb, "riscv64", None), None, "a riscv64 runner takes it");
1980 +
        // Learned: recorded once, and reported with the runner's claims.
1981 +
        let manifest: serde_json::Value = serde_json::from_slice(&std::fs::read(home.image_manifest(&hash)).unwrap()).unwrap();
1982 +
        assert_eq!(manifest["arch"], "riscv64");
1983 +
        assert_eq!(home.image_archs().get(&hash).map(String::as_str), Some("riscv64"));
1984 +
    }
1985 +
1986 +
    #[test]
993 1987
    fn transient_network_errors_are_tried_again_and_others_are_not() {
994 1988
        assert!(transient("fetch x: Dns Failed: resolve dns name 'oxygenhub.io:443': Temporary failure in name resolution"));
995 1989
        assert!(transient("HTTP 503: the hub is restarting"));
…
26 unchanged lines
1022 2016
1023 2017
    /// A hub that answers each request with the next of `codes`, counting them.
1024 2018
    fn fake_hub(codes: Vec<u16>) -> (String, std::sync::Arc<std::sync::atomic::AtomicUsize>) {
2019 +
        fake_hub_on(std::net::TcpListener::bind("127.0.0.1:0").unwrap(), codes)
2020 +
    }
2021 +
2022 +
    /// A hub at `listener` answering each request with the next of `codes`.
2023 +
    fn fake_hub_on(listener: std::net::TcpListener, codes: Vec<u16>) -> (String, std::sync::Arc<std::sync::atomic::AtomicUsize>) {
1025 2024
        use std::io::{Read, Write};
1026 -
        let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
1027 2025
        let addr = listener.local_addr().unwrap();
1028 2026
        let seen = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0));
1029 2027
        let count = seen.clone();
…
95 unchanged lines
1125 2123
        assert_eq!(cfg.labels, ["gpu"]);
1126 2124
    }
1127 2125
2126 +
    /// A hub that answers certify with a certificate `sign_as` signs for
2127 +
    /// `cert_key`, and publishes `published` as its key: two requests.
2128 +
    fn fake_certify_hub(
2129 +
        sign_as: (oxy_core::identity::Identity, oxy_core::identity::SecretKey),
2130 +
        cert_key: oxy_core::identity::Identity,
2131 +
        published: oxy_core::identity::Identity,
2132 +
    ) -> String {
2133 +
        use std::io::{Read, Write};
2134 +
        let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
2135 +
        let addr = listener.local_addr().unwrap();
2136 +
        std::thread::spawn(move || {
2137 +
            for _ in 0..2 {
2138 +
                let Ok((mut s, _)) = listener.accept() else { return };
2139 +
                let mut req = Vec::new();
2140 +
                let mut buf = [0u8; 65536];
2141 +
                loop {
2142 +
                    let n = s.read(&mut buf).unwrap_or(0);
2143 +
                    if n == 0 {
2144 +
                        break;
2145 +
                    }
2146 +
                    req.extend_from_slice(&buf[..n]);
2147 +
                    let text = String::from_utf8_lossy(&req);
2148 +
                    if let Some(end) = text.find("\r\n\r\n") {
2149 +
                        let len = text[..end]
2150 +
                            .lines()
2151 +
                            .find_map(|l| {
2152 +
                                l.to_ascii_lowercase().strip_prefix("content-length:").map(|v| v.trim().parse::<usize>().unwrap_or(0))
2153 +
                            })
2154 +
                            .unwrap_or(0);
2155 +
                        if req.len() >= end + 4 + len {
2156 +
                            break;
2157 +
                        }
2158 +
                    }
2159 +
                }
2160 +
                let line = String::from_utf8_lossy(&req).lines().next().unwrap_or("").to_string();
2161 +
                let body: Vec<u8> = if line.contains("/keyless/certify") {
2162 +
                    let cert = oxy_core::cert::SignedCertificate::sign(
2163 +
                        oxy_core::cert::Certificate {
2164 +
                            key: cert_key.clone(),
2165 +
                            issuer: sign_as.0.hash(),
2166 +
                            subject: "repo:andy/demo:bookmark:main".into(),
2167 +
                            claims: vec![],
2168 +
                            scope: oxy_core::identity::Scope::SEAL,
2169 +
                            not_before: 1,
2170 +
                            expires: 600_001,
2171 +
                            grant: None,
2172 +
                        },
2173 +
                        &sign_as.0,
2174 +
                        &sign_as.1,
2175 +
                    );
2176 +
                    serde_json::json!({ "certificate": oxy_core::dsse::base64(&cert.encode()) }).to_string().into_bytes()
2177 +
                } else {
2178 +
                    published.encode()
2179 +
                };
2180 +
                let _ = write!(s, "HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n", body.len());
2181 +
                let _ = s.write_all(&body);
2182 +
            }
2183 +
        });
2184 +
        format!("http://{addr}")
2185 +
    }
2186 +
2187 +
    /// keyless (oxygenhub#241): the runner takes a certificate only when it
2188 +
    /// is for its fresh key and signed by the key the hub publishes.
2189 +
    #[test]
2190 +
    fn a_certificate_is_taken_only_for_this_key_from_the_hubs_key() {
2191 +
        use oxy_core::identity::SecretKey;
2192 +
        let hub_sk = SecretKey::generate();
2193 +
        let hub_id = hub_sk.identity("hub", "", 1);
2194 +
        let sk = SecretKey::generate();
2195 +
        let id = sk.identity("earth-1 job 7", "", 1);
2196 +
        let proof = sk.sign_challenge(&id.hash(), &oxy_core::cert::certify_challenge("flowjob_x")).unwrap();
2197 +
2198 +
        let base = fake_certify_hub((hub_id.clone(), hub_sk.clone()), id.clone(), hub_id.clone());
2199 +
        let (cert, issuer) = hub_at(base).certify("flowjob_x", &id, &proof).unwrap();
2200 +
        assert_eq!((cert.cert.key_id(), issuer.hash()), (id.hash(), hub_id.hash()));
2201 +
2202 +
        let mallory_sk = SecretKey::generate();
2203 +
        let mallory = mallory_sk.identity("mallory", "", 1);
2204 +
        let base = fake_certify_hub((mallory.clone(), mallory_sk), id.clone(), hub_id.clone());
2205 +
        let e = hub_at(base).certify("flowjob_x", &id, &proof).unwrap_err().to_string();
2206 +
        assert!(e.contains("not signed by the hub's published key"), "{e}");
2207 +
2208 +
        let other = SecretKey::generate().identity("other", "", 1);
2209 +
        let base = fake_certify_hub((hub_id.clone(), hub_sk), other, hub_id);
2210 +
        let e = hub_at(base).certify("flowjob_x", &id, &proof).unwrap_err().to_string();
2211 +
        assert!(e.contains("another key"), "{e}");
2212 +
    }
2213 +
1128 2214
    fn hub_at(base: String) -> Hub {
1129 2215
        Hub { base, token: "t".into(), agent: ureq::AgentBuilder::new().timeout(Duration::from_secs(5)).build() }
1130 2216
    }
1131 2217
2218 +
    /// oxygenhub#322: an offered build is kept only if it is the bytes the
2219 +
    /// hub attests; a peer or a proxy cannot change what is installed.
2220 +
    #[test]
2221 +
    fn a_build_is_kept_only_if_it_matches_the_hubs_blake3() {
2222 +
        let (body, other) = (b"#!flow build 0.3.2".to_vec(), b"#!something else!!".to_vec());
2223 +
        let serve = |bytes: Vec<u8>| {
2224 +
            let listener = std::net::TcpListener::bind("127.0.0.1:0").unwrap();
2225 +
            let addr = listener.local_addr().unwrap();
2226 +
            std::thread::spawn(move || {
2227 +
                use std::io::{Read, Write};
2228 +
                let (mut s, _) = listener.accept().unwrap();
2229 +
                let mut buf = [0u8; 4096];
2230 +
                let _ = s.read(&mut buf);
2231 +
                let _ = write!(s, "HTTP/1.1 200 OK\r\nContent-Length: {}\r\nConnection: close\r\n\r\n", bytes.len());
2232 +
                let _ = s.write_all(&bytes);
2233 +
            });
2234 +
            format!("http://{addr}")
2235 +
        };
2236 +
        let offer = flow_core::api::UpdateOffer {
2237 +
            version: "0.3.2".into(),
2238 +
            asset: "flow-linux-x86_64".into(),
2239 +
            url: "/andy/oxygenflow/asset/1/flow-linux-x86_64".into(),
2240 +
            blake3: blake3::hash(&body).to_hex().to_string(),
2241 +
            size: body.len() as u64,
2242 +
            ..Default::default()
2243 +
        };
2244 +
        let dir = tempfile::tempdir().unwrap();
2245 +
        let to = dir.path().join("flow-0.3.2");
2246 +
        hub_at(serve(body.clone())).download_build(&offer, &to).unwrap();
2247 +
        assert_eq!(std::fs::read(&to).unwrap(), body);
2248 +
        std::fs::remove_file(&to).unwrap();
2249 +
        let e = hub_at(serve(other)).download_build(&offer, &to).unwrap_err();
2250 +
        assert!(e.to_string().contains("does not match what the hub attests"), "{e:#}");
2251 +
        assert!(!to.exists() && !to.with_extension("partial").exists(), "nothing of it is left");
2252 +
    }
2253 +
1132 2254
    /// oxygenhub#71: a job's result is offered until the hub takes it,
1133 2255
    /// through server errors, and a refusal ends the trying at once.
1134 2256
    #[test]
…
6 unchanged lines
1141 2263
            error: None,
1142 2264
            peak_disk_mib: None,
1143 2265
            peak_memory_mib: None,
2266 +
            phases: Default::default(),
1144 2267
        };
1145 2268
        let quick = Duration::from_millis(10);
1146 2269
        let (base, seen) = fake_hub(vec![502, 503, 500, 500, 204]);
…
22 unchanged lines
1169 2292
        assert_eq!(seen.load(std::sync::atomic::Ordering::SeqCst), 1, "a refusal is final");
1170 2293
    }
1171 2294
2295 +
    static NOT_LEAVING: std::sync::Mutex<Option<Leave>> = std::sync::Mutex::new(None);
2296 +
2297 +
    /// oxygenflow#76: once the runner leaves, a job's sink says it is
2298 +
    /// cancelled, having sent what the job said so far, and the job is
2299 +
    /// handed back to the hub.
2300 +
    #[test]
2301 +
    fn a_leaving_runner_stops_its_job_and_hands_it_back() {
2302 +
        use microvm::Sink;
2303 +
        let leaving = std::sync::Mutex::new(None);
2304 +
        let (base, seen) = fake_hub(vec![204, 204]);
2305 +
        let hub = hub_at(base);
2306 +
        let mut sink = LogSink {
2307 +
            hub: &hub,
2308 +
            job: 7,
2309 +
            buf: Vec::new(),
2310 +
            last: Instant::now(),
2311 +
            gone: false,
2312 +
            leaving: &leaving,
2313 +
            started: Instant::now(),
2314 +
            expected: None,
2315 +
        };
2316 +
        sink.buf.push(Record { t: 1, stream: "out".into(), text: "step 1".into() });
2317 +
        assert!(!sink.cancelled());
2318 +
        *leaving.lock().unwrap() = Some(Leave { reason: "off Wi-Fi".into(), by: Instant::now() });
2319 +
        assert!(sink.cancelled());
2320 +
        assert!(sink.buf.is_empty(), "flushed before the job stops");
2321 +
        assert_eq!(seen.load(std::sync::atomic::Ordering::SeqCst), 1);
2322 +
        hub.release(7, "off Wi-Fi").unwrap();
2323 +
        assert_eq!(seen.load(std::sync::atomic::Ordering::SeqCst), 2);
2324 +
    }
2325 +
2326 +
    /// oxygenflow#76: with notice, a job the hub expects to finish within
2327 +
    /// it runs on; one that will not, or whose time is unknown, is handed
2328 +
    /// back at once, and any job still running when the notice is up.
2329 +
    #[test]
2330 +
    fn a_job_that_fits_the_notice_finishes_and_the_others_go_back() {
2331 +
        use microvm::Sink;
2332 +
        let (base, _) = fake_hub(vec![204; 8]);
2333 +
        let hub = hub_at(base);
2334 +
        let leaving = std::sync::Mutex::new(Some(Leave { reason: "off charge".into(), by: Instant::now() + Duration::from_secs(60) }));
2335 +
        let sink = |expected: Option<u64>, ran: u64| LogSink {
2336 +
            hub: &hub,
2337 +
            job: 1,
2338 +
            buf: Vec::new(),
2339 +
            last: Instant::now(),
2340 +
            gone: false,
2341 +
            leaving: &leaving,
2342 +
            started: Instant::now() - Duration::from_secs(ran),
2343 +
            expected: expected.map(Duration::from_secs),
2344 +
        };
2345 +
        assert!(!sink(Some(70), 30).cancelled(), "40 s left of it, within a minute: it finishes");
2346 +
        assert!(sink(Some(600), 30).cancelled(), "nearly ten minutes left: handed back");
2347 +
        assert!(sink(None, 0).cancelled(), "no idea how long: handed back");
2348 +
        *leaving.lock().unwrap() = Some(Leave { reason: "off charge".into(), by: Instant::now() - Duration::from_secs(1) });
2349 +
        assert!(sink(Some(70), 30).cancelled(), "the notice is up: handed back whatever was expected");
2350 +
    }
2351 +
1172 2352
    /// oxygenflow#32: a quiet job asks the hub every few seconds whether
1173 2353
    /// it still runs; a 409 marks it cancelled, and nothing more is sent.
1174 2354
    #[test]
…
3 unchanged lines
1178 2358
        let quiet = Instant::now() - HEARTBEAT - Duration::from_secs(1);
1179 2359
        let (base, seen) = fake_hub(vec![204]);
1180 2360
        let hub = hub_at(base);
1181 -
        let mut sink = LogSink { hub: &hub, job: 1, buf: Vec::new(), last: quiet, gone: false };
2361 +
        let mut sink = LogSink {
2362 +
            hub: &hub,
2363 +
            job: 1,
2364 +
            buf: Vec::new(),
2365 +
            last: quiet,
2366 +
            gone: false,
2367 +
            leaving: &NOT_LEAVING,
2368 +
            started: Instant::now(),
2369 +
            expected: None,
2370 +
        };
1182 2371
        assert!(!sink.cancelled());
1183 2372
        assert!(!sink.cancelled(), "asked again only after the heartbeat");
1184 2373
        assert_eq!(n(&seen), 1);
1185 2374
1186 2375
        let (base, seen) = fake_hub(vec![409]);
1187 2376
        let hub = hub_at(base);
1188 -
        let mut sink = LogSink { hub: &hub, job: 1, buf: Vec::new(), last: quiet, gone: false };
2377 +
        let mut sink = LogSink {
2378 +
            hub: &hub,
2379 +
            job: 1,
2380 +
            buf: Vec::new(),
2381 +
            last: quiet,
2382 +
            gone: false,
2383 +
            leaving: &NOT_LEAVING,
2384 +
            started: Instant::now(),
2385 +
            expected: None,
2386 +
        };
1189 2387
        assert!(sink.cancelled());
1190 2388
        sink.record(Record { t: 0, stream: "out".into(), text: "late".into() });
1191 2389
        sink.flush();
…
1 unchanged lines
1193 2391
        assert_eq!(n(&seen), 1, "nothing is sent for a cancelled job");
1194 2392
    }
1195 2393
2394 +
    /// oxygenflow#49: a hub that is away for a while (a deploy restarting
2395 +
    /// it) costs a job's uploads a wait, not the upload, and costs its log
2396 +
    /// no lines.
2397 +
    #[test]
2398 +
    fn a_job_outlasts_a_hub_restart() {
2399 +
        let n = |seen: &std::sync::Arc<std::sync::atomic::AtomicUsize>| seen.load(std::sync::atomic::Ordering::SeqCst);
2400 +
        // A port nobody listens on yet: the hub is restarting.
2401 +
        let port = std::net::TcpListener::bind("127.0.0.1:0").unwrap().local_addr().unwrap().port();
2402 +
        let base = format!("http://127.0.0.1:{port}");
2403 +
        let hub = hub_at(base.clone());
2404 +
2405 +
        // Log lines sent while it is away are kept, and go when it is back.
2406 +
        let mut sink = LogSink {
2407 +
            hub: &hub,
2408 +
            job: 1,
2409 +
            buf: Vec::new(),
2410 +
            last: Instant::now(),
2411 +
            gone: false,
2412 +
            leaving: &NOT_LEAVING,
2413 +
            started: Instant::now(),
2414 +
            expected: None,
2415 +
        };
2416 +
        for i in 0..450 {
2417 +
            sink.buf.push(Record { t: i, stream: "out".into(), text: format!("line {i}") });
2418 +
        }
2419 +
        sink.send();
2420 +
        assert_eq!(sink.buf.len(), 450, "nothing lost while the hub is away");
2421 +
        let (_, seen) = fake_hub_on(std::net::TcpListener::bind(("127.0.0.1", port)).unwrap(), vec![204, 204, 204]);
2422 +
        sink.send();
2423 +
        assert!(sink.buf.is_empty() && !sink.gone);
2424 +
        assert_eq!(n(&seen), 3, "in batches of 200: 200, 200 and 50");
2425 +
2426 +
        // An upload waits for the hub to come back, through refused
2427 +
        // connections and server errors alike.
2428 +
        let port = std::net::TcpListener::bind("127.0.0.1:0").unwrap().local_addr().unwrap().port();
2429 +
        let url = format!("http://127.0.0.1:{port}/x");
2430 +
        let agent = ureq::AgentBuilder::new().timeout(Duration::from_secs(5)).build();
2431 +
        let later = std::thread::spawn(move || {
2432 +
            std::thread::sleep(Duration::from_millis(150));
2433 +
            fake_hub_on(std::net::TcpListener::bind(("127.0.0.1", port)).unwrap(), vec![502, 204])
2434 +
        });
2435 +
        let quick = [Duration::from_millis(50); 20];
2436 +
        let r = retrying(&url, &quick, true, || agent.post(&url).send_string("{}").map_err(Box::new)).unwrap();
2437 +
        assert_eq!(r.status(), 204);
2438 +
        assert_eq!(n(&later.join().unwrap().1), 2, "a 502 from the proxy, then the hub");
2439 +
        // Without `connection`, a hub that cannot be reached ends it at once.
2440 +
        let port = std::net::TcpListener::bind("127.0.0.1:0").unwrap().local_addr().unwrap().port();
2441 +
        let url = format!("http://127.0.0.1:{port}/x");
2442 +
        assert!(retrying(&url, &quick, false, || agent.post(&url).send_string("{}").map_err(Box::new)).is_err());
2443 +
        assert!(PATIENT_DELAYS.iter().sum::<Duration>() >= Duration::from_secs(180), "longer than a hub takes to restart");
2444 +
    }
2445 +
2446 +
    /// oxygenflow#50: the first stop drains (the second, not called here,
2447 +
    /// exits at once).
2448 +
    #[test]
2449 +
    fn a_stop_drains_the_runner() {
2450 +
        assert!(!stopping());
2451 +
        on_stop(libc::SIGTERM);
2452 +
        assert!(stopping(), "told to stop, it claims nothing more");
2453 +
        // An embedded runner starts afresh, and stop() drains it the same way.
2454 +
        STOPS.store(0, std::sync::atomic::Ordering::SeqCst);
2455 +
        stop();
2456 +
        assert!(stopping());
2457 +
        STOPS.store(0, std::sync::atomic::Ordering::SeqCst);
2458 +
    }
2459 +
2460 +
    /// Lines kept for a hub that stays away are bounded, saying how many went.
1196 2461
    #[test]
2462 +
    fn unsent_log_lines_are_bounded() {
2463 +
        let port = std::net::TcpListener::bind("127.0.0.1:0").unwrap().local_addr().unwrap().port();
2464 +
        let hub = hub_at(format!("http://127.0.0.1:{port}"));
2465 +
        let mut sink = LogSink {
2466 +
            hub: &hub,
2467 +
            job: 1,
2468 +
            buf: Vec::new(),
2469 +
            last: Instant::now(),
2470 +
            gone: false,
2471 +
            leaving: &NOT_LEAVING,
2472 +
            started: Instant::now(),
2473 +
            expected: None,
2474 +
        };
2475 +
        sink.buf = (0..MAX_UNSENT as u64 + 10).map(|i| Record { t: i, stream: "out".into(), text: String::new() }).collect();
2476 +
        sink.send();
2477 +
        assert_eq!(sink.buf.len(), MAX_UNSENT + 1);
2478 +
        assert_eq!(sink.buf[0].text, "(10 log lines lost while the hub could not be reached)");
2479 +
        assert_eq!(sink.buf[1].t, 10, "the oldest go first");
2480 +
    }
2481 +
2482 +
    #[test]
1197 2483
    fn the_budget_admits_jobs_that_fit() {
1198 2484
        let b = Budget { max_jobs: 3, memory_mib: Some(24_576), cpus: Some(8) };
1199 2485
        let used = |jobs, memory_mib, cpus| Used { jobs, memory_mib, cpus };
…
7 unchanged lines
1207 2493
        let open = Budget { max_jobs: 2, memory_mib: None, cpus: None };
1208 2494
        assert_eq!(open.room(used(1, 8192, 4)), Some((None, None)), "no limits configured");
1209 2495
    }
2496 +
2497 +
    #[test]
2498 +
    fn a_claim_never_asks_for_more_memory_than_the_host_has_free() {
2499 +
        assert_eq!(claim_memory(None, Some(6541)), Some(6541), "idle: the host's free memory, not unlimited");
2500 +
        assert_eq!(claim_memory(Some(16_384), Some(6541)), Some(6541));
2501 +
        assert_eq!(claim_memory(Some(4096), Some(6541)), Some(4096), "the budget when it is smaller");
2502 +
        assert_eq!(claim_memory(None, None), None, "nothing known: no limit, as before");
2503 +
        assert_eq!(claim_memory(Some(4096), None), Some(4096));
2504 +
    }
1210 2505
}
added crates/flow/src/swarm.rs +1041 −0
1 +
//! The swarm (oxygenflow#47, docs/jetstream.md §8.1): a runner with a
2 +
//! `[runner.swarm]` section serves the image chunks it holds to its peers,
3 +
//! and fetches an image it lacks from its peers and the hub together.
4 +
//!
5 +
//! The server is a small HTTPS endpoint on the swarm address: a peer
6 +
//! shows the ticket its hub signed and gets a session, then asks which
7 +
//! chunks of an image this runner has and for chunks by hash. A chunk is
8 +
//! served from the chunk cache, or read out of a whole image by its
9 +
//! manifest. Every chunk is checked by the side that asked for it, so a
10 +
//! peer can only fail to help.
11 +
//!
12 +
//! Each runner makes its own TLS certificate on first use and reports its
13 +
//! BLAKE3 with its swarm address; the hub hands that on with the peer, and
14 +
//! a fetcher talks only to a peer that shows that certificate and holds
15 +
//! its key. Nobody on the network reads the chunks, tickets or sessions.
16 +
17 +
use crate::home::{Home, RunnerConfig};
18 +
use crate::images::{cache_path, chunk_cache, manifest_path, save_manifest, write_image, Client, Moved};
19 +
use anyhow::{bail, Context, Result};
20 +
use flow_core::api::{SwarmAddr, SwarmPeer};
21 +
use jet::images::{is_hash, Manifest};
22 +
use jet::swarm::{Bits, Pick, Picker, Source, Ticket};
23 +
use oxy_core::identity::Identity;
24 +
use rustls::client::danger::{HandshakeSignatureValid, ServerCertVerified, ServerCertVerifier};
25 +
use rustls::crypto::CryptoProvider;
26 +
use rustls::pki_types::{CertificateDer, PrivateKeyDer, PrivatePkcs8KeyDer, ServerName, UnixTime};
27 +
use rustls::{DigitallySignedStruct, SignatureScheme};
28 +
use std::collections::HashMap;
29 +
use std::io::{BufRead, BufReader, Read, Seek, SeekFrom, Write};
30 +
use std::net::{TcpListener, TcpStream};
31 +
use std::path::{Path, PathBuf};
32 +
use std::sync::atomic::{AtomicBool, AtomicU64, AtomicUsize, Ordering};
33 +
use std::sync::{Arc, Mutex};
34 +
use std::time::{Duration, Instant, SystemTime, UNIX_EPOCH};
35 +
36 +
/// Connections served at once; more are told 503 and come back shortly.
37 +
const MAX_CONNS: usize = 16;
38 +
/// Connections past this are closed without a word.
39 +
const MAX_OPEN: usize = 4 * MAX_CONNS;
40 +
/// Requests in flight to one peer, and to the hub.
41 +
const PEER_WORKERS: usize = 4;
42 +
const HUB_WORKERS: usize = 8;
43 +
/// How often a fetcher asks its peers what they have.
44 +
const REFRESH: Duration = Duration::from_millis(500);
45 +
/// Hub failures a fetch survives before it gives up.
46 +
const HUB_FAILURES: usize = 5;
47 +
/// The largest request body the server reads: a ticket.
48 +
const MAX_BODY: usize = 64 * 1024;
49 +
/// Names the part files of chunks two sources fetch at once apart.
50 +
static PART: AtomicU64 = AtomicU64::new(0);
51 +
52 +
fn now_ms() -> u64 {
53 +
    SystemTime::now().duration_since(UNIX_EPOCH).map(|d| d.as_millis() as u64).unwrap_or(0)
54 +
}
55 +
56 +
/// An image this runner offers: which of its needed chunks are here
57 +
/// (`None`: all of them, the image is whole) and which it is fetching now.
58 +
struct Offer {
59 +
    bits: Option<Bits>,
60 +
    fetching: Bits,
61 +
}
62 +
63 +
struct State {
64 +
    root: PathBuf,
65 +
    site: String,
66 +
    client: Client,
67 +
    /// The hub's identity, which signs tickets; fetched when first needed.
68 +
    hub: Mutex<Option<Identity>>,
69 +
    offers: Mutex<HashMap<String, Offer>>,
70 +
    /// Where a chunk lies in a whole image: the image, offset and length.
71 +
    located: Mutex<HashMap<String, (PathBuf, u64, u32)>>,
72 +
    /// Sessions: token to (peer name, expiry).
73 +
    sessions: Mutex<HashMap<String, (String, u64)>>,
74 +
    conns: AtomicUsize,
75 +
    /// The upload allowance: bytes in hand (negative: owed) and when it
76 +
    /// was last topped up.
77 +
    bucket: Mutex<(f64, Instant)>,
78 +
    rate: f64,
79 +
    tls: Arc<rustls::ServerConfig>,
80 +
}
81 +
82 +
/// This runner's place in the swarm.
83 +
pub struct Swarm {
84 +
    state: Arc<State>,
85 +
    pub addr: SwarmAddr,
86 +
}
87 +
88 +
impl Swarm {
89 +
    /// Start serving peers when the runner is configured for it.
90 +
    pub fn start(home: &Home, cfg: &RunnerConfig) -> Result<Option<Swarm>> {
91 +
        let Some(sc) = &cfg.swarm else { return Ok(None) };
92 +
        let listener = TcpListener::bind(&sc.listen).with_context(|| format!("swarm: listen on {}", sc.listen))?;
93 +
        let client = Client::new(&cfg.hub, &cfg.token, cfg.ca.as_deref())?;
94 +
        let hub = client.hub_identity().map_err(|e| eprintln!("swarm: the hub's key: {e:#}; trying again when a peer comes")).ok();
95 +
        let rate = sc.upload_mib_s.unwrap_or(50).max(1) as f64 * (1 << 20) as f64;
96 +
        let (cert, key) = certificate(&home.root)?;
97 +
        let fingerprint = blake3::hash(&cert).to_hex().to_string();
98 +
        let tls = server_tls(cert, key)?;
99 +
        let state = Arc::new(State {
100 +
            root: home.root.clone(),
101 +
            site: sc.site.clone(),
102 +
            client,
103 +
            hub: Mutex::new(hub),
104 +
            offers: Mutex::new(HashMap::new()),
105 +
            located: Mutex::new(HashMap::new()),
106 +
            sessions: Mutex::new(HashMap::new()),
107 +
            conns: AtomicUsize::new(0),
108 +
            bucket: Mutex::new((rate, Instant::now())),
109 +
            rate,
110 +
            tls,
111 +
        });
112 +
        // Whole images: offered from their manifests; one without a
113 +
        // manifest (imported here, not fetched) is scanned for one, in the
114 +
        // background and one at a time.
115 +
        let mut unscanned = Vec::new();
116 +
        for h in home.images() {
117 +
            match std::fs::read(manifest_path(&home.root, &h)).ok().and_then(|b| serde_json::from_slice::<Manifest>(&b).ok()) {
118 +
                Some(m) if m.hash == h => offer_whole(&state, &m, &home.rootfs(&h)),
119 +
                _ => unscanned.push((h.clone(), home.rootfs(&h), home.image_arch(&h))),
120 +
            }
121 +
        }
122 +
        let scanning = state.clone();
123 +
        std::thread::spawn(move || {
124 +
            for (h, path, arch) in unscanned {
125 +
                match jet::images::scan_file(&path, arch, |_, _| Ok(())) {
126 +
                    Ok(m) if m.hash == h => {
127 +
                        save_manifest(&scanning.root, &m).ok();
128 +
                        offer_whole(&scanning, &m, &path);
129 +
                    }
130 +
                    Ok(_) => eprintln!("swarm: {} does not hold the image its name says; not offering it", path.display()),
131 +
                    Err(e) => eprintln!("swarm: scan {}: {e}", path.display()),
132 +
                }
133 +
            }
134 +
        });
135 +
        let serving = state.clone();
136 +
        std::thread::spawn(move || accept(listener, serving));
137 +
        let addr = SwarmAddr { addr: sc.advertise.clone().unwrap_or_else(|| sc.listen.clone()), site: sc.site.clone(), cert: fingerprint };
138 +
        eprintln!(
139 +
            "swarm: serving image chunks on {} over TLS (certificate {}, site {:?}, up to {} MiB/s)",
140 +
            sc.listen,
141 +
            &addr.cert[..12],
142 +
            sc.site,
143 +
            sc.upload_mib_s.unwrap_or(50)
144 +
        );
145 +
        Ok(Some(Swarm { state, addr }))
146 +
    }
147 +
148 +
    /// Fetch image `hash` into `home` from the peers and the hub (§8.1):
149 +
    /// the rarest chunks from peers first, the hub for what no peer has.
150 +
    /// `Ok(None)` when the hub does not hold it.
151 +
    pub fn fetch(&self, home: &Home, hash: &str) -> Result<Option<Moved>> {
152 +
        let start = Instant::now();
153 +
        let client = &self.state.client;
154 +
        let Some(m) = client.manifest(hash)? else { return Ok(None) };
155 +
        if m.hash != hash {
156 +
            bail!("the hub's manifest for {} names {}", &hash[..12], &m.hash[..12]);
157 +
        }
158 +
        save_manifest(&home.root, &m)?;
159 +
        let needed: Vec<String> = m.needed().into_iter().map(String::from).collect();
160 +
        let cache = chunk_cache(home);
161 +
        let mut have = Bits::new(needed.len());
162 +
        for (i, h) in needed.iter().enumerate() {
163 +
            if cache_path(&cache, h).is_file() {
164 +
                have.set(i);
165 +
            }
166 +
        }
167 +
        self.state.offers.lock().unwrap().insert(hash.into(), Offer { bits: Some(have.clone()), fetching: Bits::new(needed.len()) });
168 +
        let announced = client
169 +
            .announce(hash, "fetching")
170 +
            .map_err(|e| eprintln!("swarm: announce {}: {e:#}; fetching from the hub alone", &hash[..12]))
171 +
            .ok();
172 +
        let seed = now_ms() ^ (std::process::id() as u64) << 32;
173 +
        let picker = Mutex::new(Picker::new(needed.len(), have, seed));
174 +
        let peers: Mutex<Vec<Arc<Peer>>> = Mutex::new(Vec::new());
175 +
        if let Some(a) = &announced {
176 +
            self.join(&a.peers, &a.ticket, hash, needed.len(), &peers, &picker);
177 +
        }
178 +
        let (bytes, peer_bytes, got) = (AtomicU64::new(0), AtomicU64::new(0), AtomicUsize::new(0));
179 +
        let (failed, hub_failures) = (Mutex::new(None::<anyhow::Error>), AtomicUsize::new(0));
180 +
        {
181 +
            let (picker, peers, needed, cache) = (&picker, &peers, &needed, &cache);
182 +
            let (bytes, peer_bytes, got, failed, hub_failures) = (&bytes, &peer_bytes, &got, &failed, &hub_failures);
183 +
            let (state, m) = (&self.state, &m);
184 +
            let worker = move |src: Source| loop {
185 +
                if failed.lock().unwrap().is_some() {
186 +
                    break;
187 +
                }
188 +
                let peer = match src {
189 +
                    Source::Peer(p) => Some(peers.lock().unwrap()[p].clone()),
190 +
                    Source::Hub => None,
191 +
                };
192 +
                if peer.as_ref().is_some_and(|p| !p.alive.load(Ordering::Relaxed)) {
193 +
                    break;
194 +
                }
195 +
                // Taken apart from the match, so the lock is not held
196 +
                // while waiting.
197 +
                let pick = picker.lock().unwrap().pick(src);
198 +
                let i = match pick {
199 +
                    Pick::Done => break,
200 +
                    Pick::Wait => {
201 +
                        std::thread::sleep(Duration::from_millis(20));
202 +
                        continue;
203 +
                    }
204 +
                    Pick::Chunk(i) => i,
205 +
                };
206 +
                if let Some(o) = state.offers.lock().unwrap().get_mut(&m.hash) {
207 +
                    o.fetching.set(i);
208 +
                }
209 +
                let h = &needed[i];
210 +
                let fetched = match &peer {
211 +
                    Some(p) => self.peer_chunk(p, h),
212 +
                    None => client.chunk(h),
213 +
                };
214 +
                let kept = fetched.and_then(|b| {
215 +
                    if blake3::hash(&b).to_hex().as_str() != h {
216 +
                        bail!("chunk {} came back with other bytes", &h[..12]);
217 +
                    }
218 +
                    let p = cache_path(cache, h);
219 +
                    std::fs::create_dir_all(p.parent().expect("a chunk directory"))?;
220 +
                    let tmp = p.with_extension(format!("part{}", PART.fetch_add(1, Ordering::Relaxed)));
221 +
                    std::fs::write(&tmp, &b)?;
222 +
                    std::fs::rename(&tmp, &p)?;
223 +
                    Ok(b.len() as u64)
224 +
                });
225 +
                match kept {
226 +
                    Ok(n) => {
227 +
                        if let Some(Offer { bits: Some(b), fetching }) = state.offers.lock().unwrap().get_mut(&m.hash) {
228 +
                            b.set(i);
229 +
                            fetching.clear(i);
230 +
                        }
231 +
                        picker.lock().unwrap().got(i);
232 +
                        bytes.fetch_add(n, Ordering::Relaxed);
233 +
                        got.fetch_add(1, Ordering::Relaxed);
234 +
                        if peer.is_some() {
235 +
                            peer_bytes.fetch_add(n, Ordering::Relaxed);
236 +
                        }
237 +
                    }
238 +
                    Err(e) => {
239 +
                        picker.lock().unwrap().failed(i);
240 +
                        if let Some(o) = state.offers.lock().unwrap().get_mut(&m.hash) {
241 +
                            o.fetching.clear(i);
242 +
                        }
243 +
                        match (&peer, src) {
244 +
                            // A busy peer is asked again shortly.
245 +
                            (Some(_), _) if e.is::<Busy>() => std::thread::sleep(Duration::from_millis(200)),
246 +
                            (Some(p), Source::Peer(idx)) => {
247 +
                                eprintln!("swarm: peer {} failed ({e:#}); carrying on without it", p.name);
248 +
                                p.alive.store(false, Ordering::Relaxed);
249 +
                                picker.lock().unwrap().drop_peer(idx);
250 +
                                break;
251 +
                            }
252 +
                            _ => {
253 +
                                if hub_failures.fetch_add(1, Ordering::Relaxed) + 1 >= HUB_FAILURES {
254 +
                                    *failed.lock().unwrap() = Some(e);
255 +
                                    break;
256 +
                                }
257 +
                                std::thread::sleep(Duration::from_millis(500));
258 +
                            }
259 +
                        }
260 +
                    }
261 +
                }
262 +
            };
263 +
            // An image only runners hold (jetstream.md §8.1.1): the hub
264 +
            // has its manifest but none of its chunks, so only peers are asked.
265 +
            let hub_seeds = announced.as_ref().is_none_or(|a| a.seed != Some(false));
266 +
            std::thread::scope(|s| {
267 +
                if hub_seeds {
268 +
                    for _ in 0..HUB_WORKERS {
269 +
                        s.spawn(move || worker(Source::Hub));
270 +
                    }
271 +
                }
272 +
                let mut spawned = 0;
273 +
                let mut spawn_new = || {
274 +
                    let n = peers.lock().unwrap().len();
275 +
                    for p in spawned..n {
276 +
                        for _ in 0..PEER_WORKERS {
277 +
                            s.spawn(move || worker(Source::Peer(p)));
278 +
                        }
279 +
                    }
280 +
                    spawned = n;
281 +
                };
282 +
                spawn_new();
283 +
                // Keep the peers' maps fresh, and find new peers while
284 +
                // fetching, until every chunk is here.
285 +
                let mut asked_hub = Instant::now();
286 +
                let interval = Duration::from_secs(announced.as_ref().map(|a| a.interval_s).unwrap_or(30).max(5));
287 +
                let mut last = Instant::now();
288 +
                let mut peerless = Instant::now();
289 +
                while picker.lock().unwrap().missing() > 0 && failed.lock().unwrap().is_none() {
290 +
                    std::thread::sleep(Duration::from_millis(100));
291 +
                    if !hub_seeds {
292 +
                        if peers.lock().unwrap().iter().any(|p| p.alive.load(Ordering::Relaxed)) {
293 +
                            peerless = Instant::now();
294 +
                        } else if peerless.elapsed() >= interval * 2 {
295 +
                            *failed.lock().unwrap() = Some(anyhow::anyhow!(
296 +
                                "image {} is held only by runners, and none in this site could serve it for {} s; send it to the hub with `flow rootfs push`",
297 +
                                &hash[..12],
298 +
                                (interval * 2).as_secs()
299 +
                            ));
300 +
                        }
301 +
                    }
302 +
                    if last.elapsed() < REFRESH {
303 +
                        continue;
304 +
                    }
305 +
                    last = Instant::now();
306 +
                    let now: Vec<(usize, Arc<Peer>)> =
307 +
                        peers.lock().unwrap().iter().cloned().enumerate().filter(|(_, p)| p.alive.load(Ordering::Relaxed)).collect();
308 +
                    for (idx, p) in now {
309 +
                        match self.peer_have(&p, hash, needed.len()) {
310 +
                            Ok((bits, fetching)) => picker.lock().unwrap().peer_fetching(idx, bits, fetching),
311 +
                            Err(e) if e.is::<Busy>() => {}
312 +
                            Err(_) => {
313 +
                                p.alive.store(false, Ordering::Relaxed);
314 +
                                picker.lock().unwrap().drop_peer(idx);
315 +
                            }
316 +
                        }
317 +
                    }
318 +
                    if announced.is_some() && asked_hub.elapsed() >= interval {
319 +
                        asked_hub = Instant::now();
320 +
                        if let Ok(a) = client.announce(hash, "fetching") {
321 +
                            self.join(&a.peers, &a.ticket, hash, needed.len(), peers, picker);
322 +
                            spawn_new();
323 +
                        }
324 +
                    }
325 +
                }
326 +
            });
327 +
        }
328 +
        if let Some(e) = failed.into_inner().unwrap() {
329 +
            client.announce(hash, "stopped").ok();
330 +
            return Err(e);
331 +
        }
332 +
        write_image(home, &m)?;
333 +
        offer_whole(&self.state, &m, &home.rootfs(hash));
334 +
        client.announce(hash, "done").ok();
335 +
        Ok(Some(Moved {
336 +
            chunks: got.into_inner(),
337 +
            bytes: bytes.into_inner(),
338 +
            peer_bytes: peer_bytes.into_inner(),
339 +
            elapsed: start.elapsed(),
340 +
        }))
341 +
    }
342 +
343 +
    /// Admit this runner to the peers it does not know yet: a session from
344 +
    /// each, and what each has.
345 +
    fn join(&self, found: &[SwarmPeer], ticket: &str, hash: &str, len: usize, peers: &Mutex<Vec<Arc<Peer>>>, picker: &Mutex<Picker>) {
346 +
        for sp in found {
347 +
            if peers.lock().unwrap().iter().any(|p| p.runner == sp.runner) {
348 +
                continue;
349 +
            }
350 +
            if !is_hash(&sp.cert) {
351 +
                eprintln!("swarm: peer {} gave no certificate (an older flow); not asking it", sp.name);
352 +
                continue;
353 +
            }
354 +
            let joined = hello(&sp.addr, &sp.cert, ticket).and_then(|(agent, session)| {
355 +
                let p = Peer {
356 +
                    runner: sp.runner,
357 +
                    name: sp.name.clone(),
358 +
                    base: format!("https://{}/swarm/v1", sp.addr),
359 +
                    agent,
360 +
                    session,
361 +
                    alive: AtomicBool::new(true),
362 +
                };
363 +
                let (bits, fetching) = self.peer_have(&p, hash, len)?;
364 +
                Ok((p, bits, fetching))
365 +
            });
366 +
            match joined {
367 +
                Ok((p, bits, fetching)) => {
368 +
                    let mut list = peers.lock().unwrap();
369 +
                    picker.lock().unwrap().peer_fetching(list.len(), bits, fetching);
370 +
                    list.push(Arc::new(p));
371 +
                }
372 +
                Err(e) => eprintln!("swarm: peer {} ({}): {e:#}", sp.name, sp.addr),
373 +
            }
374 +
        }
375 +
    }
376 +
377 +
    /// What a peer has of image `hash`, and what it is fetching now:
378 +
    /// nothing yet when it has not begun (the tracker may name it a moment
379 +
    /// before it starts).
380 +
    fn peer_have(&self, p: &Peer, hash: &str, len: usize) -> Result<(Bits, Bits)> {
381 +
        let url = format!("{}/images/{hash}/have", p.base);
382 +
        let r = match p.agent.get(&url).set("Authorization", &format!("Bearer {}", p.session)).call() {
383 +
            Ok(r) => r,
384 +
            Err(ureq::Error::Status(404, _)) => return Ok((Bits::new(len), Bits::new(len))),
385 +
            Err(e) => return Err(peer_error(&url, e)),
386 +
        };
387 +
        let mut body = Vec::new();
388 +
        r.into_reader().take(len as u64 / 4 + 64).read_to_end(&mut body)?;
389 +
        if body == b"complete" {
390 +
            return Ok((Bits::full(len), Bits::new(len)));
391 +
        }
392 +
        let (have, fetching) = body.split_at(len.div_ceil(8).min(body.len()));
393 +
        Ok((Bits::from_bytes(len, have), Bits::from_bytes(len, fetching)))
394 +
    }
395 +
396 +
    fn peer_chunk(&self, p: &Peer, hash: &str) -> Result<Vec<u8>> {
397 +
        let url = format!("{}/chunks/{hash}", p.base);
398 +
        let r = p.agent.get(&url).set("Authorization", &format!("Bearer {}", p.session)).call().map_err(|e| peer_error(&url, e))?;
399 +
        let mut out = Vec::new();
400 +
        r.into_reader().take(oxy_core::chunk::MAX_SIZE as u64 + 1).read_to_end(&mut out)?;
401 +
        Ok(out)
402 +
    }
403 +
}
404 +
405 +
/// A peer this fetch asks for chunks, over a connection that trusts only
406 +
/// its certificate.
407 +
struct Peer {
408 +
    runner: i64,
409 +
    name: String,
410 +
    base: String,
411 +
    agent: ureq::Agent,
412 +
    session: String,
413 +
    alive: AtomicBool,
414 +
}
415 +
416 +
/// A peer at its connection limit: ask it again shortly.
417 +
#[derive(Debug)]
418 +
struct Busy;
419 +
420 +
impl std::fmt::Display for Busy {
421 +
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
422 +
        f.write_str("the peer is busy")
423 +
    }
424 +
}
425 +
426 +
impl std::error::Error for Busy {}
427 +
428 +
fn peer_error(url: &str, e: ureq::Error) -> anyhow::Error {
429 +
    match e {
430 +
        ureq::Error::Status(503, _) => anyhow::Error::new(Busy),
431 +
        ureq::Error::Status(code, r) => anyhow::anyhow!("{url}: HTTP {code}: {}", r.into_string().unwrap_or_default().trim()),
432 +
        e => anyhow::anyhow!("{url}: {e}"),
433 +
    }
434 +
}
435 +
436 +
/// Show `ticket` to the peer at `addr`, trusting only the certificate
437 +
/// `cert`: a connection to it and a session.
438 +
fn hello(addr: &str, cert: &str, ticket: &str) -> Result<(ureq::Agent, String)> {
439 +
    let agent = peer_agent(cert)?;
440 +
    let url = format!("https://{addr}/swarm/v1/hello");
441 +
    let r: serde_json::Value = agent.post(&url).send_string(ticket).map_err(|e| peer_error(&url, e))?.into_json()?;
442 +
    let session = r["session"].as_str().map(String::from).context("the peer gave no session")?;
443 +
    Ok((agent, session))
444 +
}
445 +
446 +
fn provider() -> Arc<CryptoProvider> {
447 +
    Arc::new(rustls::crypto::ring::default_provider())
448 +
}
449 +
450 +
/// A client that trusts one certificate, by its BLAKE3, whatever name
451 +
/// the peer goes by: the one the hub said this peer has.
452 +
fn peer_agent(cert: &str) -> Result<ureq::Agent> {
453 +
    let provider = provider();
454 +
    let cfg = rustls::ClientConfig::builder_with_provider(provider.clone())
455 +
        .with_safe_default_protocol_versions()?
456 +
        .dangerous()
457 +
        .with_custom_certificate_verifier(Arc::new(Pinned { cert: cert.to_string(), provider }))
458 +
        .with_no_client_auth();
459 +
    Ok(ureq::AgentBuilder::new().tls_config(Arc::new(cfg)).timeout_connect(Duration::from_secs(2)).timeout(Duration::from_secs(20)).build())
460 +
}
461 +
462 +
/// Trusts the certificate with this BLAKE3 and no other. The handshake's
463 +
/// signature is still checked against it, so only the holder of its key
464 +
/// passes.
465 +
#[derive(Debug)]
466 +
struct Pinned {
467 +
    cert: String,
468 +
    provider: Arc<CryptoProvider>,
469 +
}
470 +
471 +
impl ServerCertVerifier for Pinned {
472 +
    fn verify_server_cert(
473 +
        &self,
474 +
        end_entity: &CertificateDer<'_>,
475 +
        _intermediates: &[CertificateDer<'_>],
476 +
        _name: &ServerName<'_>,
477 +
        _ocsp: &[u8],
478 +
        _now: UnixTime,
479 +
    ) -> std::result::Result<ServerCertVerified, rustls::Error> {
480 +
        if blake3::hash(end_entity.as_ref()).to_hex().as_str() == self.cert {
481 +
            Ok(ServerCertVerified::assertion())
482 +
        } else {
483 +
            Err(rustls::Error::General("not the peer the hub named: its certificate is another".into()))
484 +
        }
485 +
    }
486 +
487 +
    fn verify_tls12_signature(
488 +
        &self,
489 +
        message: &[u8],
490 +
        cert: &CertificateDer<'_>,
491 +
        dss: &DigitallySignedStruct,
492 +
    ) -> std::result::Result<HandshakeSignatureValid, rustls::Error> {
493 +
        rustls::crypto::verify_tls12_signature(message, cert, dss, &self.provider.signature_verification_algorithms)
494 +
    }
495 +
496 +
    fn verify_tls13_signature(
497 +
        &self,
498 +
        message: &[u8],
499 +
        cert: &CertificateDer<'_>,
500 +
        dss: &DigitallySignedStruct,
501 +
    ) -> std::result::Result<HandshakeSignatureValid, rustls::Error> {
502 +
        rustls::crypto::verify_tls13_signature(message, cert, dss, &self.provider.signature_verification_algorithms)
503 +
    }
504 +
505 +
    fn supported_verify_schemes(&self) -> Vec<SignatureScheme> {
506 +
        self.provider.signature_verification_algorithms.supported_schemes()
507 +
    }
508 +
}
509 +
510 +
/// The runner's swarm certificate and key (`swarm/cert.der`, and
511 +
/// `swarm/key.der` readable by the runner alone), made on first use and
512 +
/// kept, so its fingerprint holds across restarts.
513 +
fn certificate(root: &Path) -> Result<(Vec<u8>, Vec<u8>)> {
514 +
    let dir = root.join("swarm");
515 +
    let (cert_path, key_path) = (dir.join("cert.der"), dir.join("key.der"));
516 +
    if let (Ok(c), Ok(k)) = (std::fs::read(&cert_path), std::fs::read(&key_path)) {
517 +
        return Ok((c, k));
518 +
    }
519 +
    std::fs::create_dir_all(&dir)?;
520 +
    let made = rcgen::generate_simple_self_signed(vec!["flow-swarm".to_string()]).context("make the swarm's certificate")?;
521 +
    let (cert, key) = (made.cert.der().to_vec(), made.signing_key.serialize_der());
522 +
    {
523 +
        let mut f = crate::portable::private(std::fs::OpenOptions::new().write(true).create(true).truncate(true)).open(&key_path)?;
524 +
        f.write_all(&key)?;
525 +
    }
526 +
    std::fs::write(&cert_path, &cert)?;
527 +
    Ok((cert, key))
528 +
}
529 +
530 +
fn server_tls(cert: Vec<u8>, key: Vec<u8>) -> Result<Arc<rustls::ServerConfig>> {
531 +
    let cfg = rustls::ServerConfig::builder_with_provider(provider())
532 +
        .with_safe_default_protocol_versions()?
533 +
        .with_no_client_auth()
534 +
        .with_single_cert(vec![CertificateDer::from(cert)], PrivateKeyDer::Pkcs8(PrivatePkcs8KeyDer::from(key)))
535 +
        .context("the swarm's certificate")?;
536 +
    Ok(Arc::new(cfg))
537 +
}
538 +
539 +
/// Offer a whole image: every chunk of it can be read out of `path`.
540 +
fn offer_whole(state: &State, m: &Manifest, path: &Path) {
541 +
    let mut located = state.located.lock().unwrap();
542 +
    let mut at = 0u64;
543 +
    for c in &m.chunks {
544 +
        if !c.is_zeros() {
545 +
            located.entry(c.hash().to_string()).or_insert_with(|| (path.to_path_buf(), at, c.1));
546 +
        }
547 +
        at += c.1 as u64;
548 +
    }
549 +
    state.offers.lock().unwrap().insert(m.hash.clone(), Offer { bits: None, fetching: Bits::new(0) });
550 +
}
551 +
552 +
fn accept(listener: TcpListener, state: Arc<State>) {
553 +
    for conn in listener.incoming() {
554 +
        let Ok(conn) = conn else { continue };
555 +
        if state.conns.load(Ordering::Relaxed) >= MAX_OPEN {
556 +
            continue;
557 +
        }
558 +
        let state = state.clone();
559 +
        std::thread::spawn(move || {
560 +
            // Past the limit a peer is still answered, with 503, so it
561 +
            // comes back rather than giving up on this runner.
562 +
            let busy = state.conns.fetch_add(1, Ordering::Relaxed) >= MAX_CONNS;
563 +
            serve_tls(conn, &state, busy).ok();
564 +
            state.conns.fetch_sub(1, Ordering::Relaxed);
565 +
        });
566 +
    }
567 +
}
568 +
569 +
fn serve_tls(conn: TcpStream, state: &State, busy: bool) -> std::io::Result<()> {
570 +
    conn.set_read_timeout(Some(Duration::from_secs(30)))?;
571 +
    conn.set_write_timeout(Some(Duration::from_secs(30)))?;
572 +
    conn.set_nodelay(true)?;
573 +
    let tls = rustls::ServerConnection::new(state.tls.clone()).map_err(std::io::Error::other)?;
574 +
    serve_conn(rustls::StreamOwned::new(tls, conn), state, busy)
575 +
}
576 +
577 +
fn respond(conn: &mut impl Write, code: u16, kind: &str, body: &[u8]) -> std::io::Result<()> {
578 +
    let reason = match code {
579 +
        200 => "OK",
580 +
        400 => "Bad Request",
581 +
        401 => "Unauthorized",
582 +
        403 => "Forbidden",
583 +
        404 => "Not Found",
584 +
        503 => "Service Unavailable",
585 +
        _ => "Error",
586 +
    };
587 +
    write!(conn, "HTTP/1.1 {code} {reason}\r\nContent-Type: {kind}\r\nContent-Length: {}\r\n\r\n", body.len())?;
588 +
    conn.write_all(body)?;
589 +
    conn.flush()
590 +
}
591 +
592 +
/// One connection: requests one after another while the peer keeps it.
593 +
fn serve_conn<S: Read + Write>(conn: S, state: &State, busy: bool) -> std::io::Result<()> {
594 +
    let mut r = BufReader::new(conn);
595 +
    loop {
596 +
        let mut line = String::new();
597 +
        if r.read_line(&mut line)? == 0 {
598 +
            return Ok(());
599 +
        }
600 +
        let mut parts = line.split_whitespace();
601 +
        let (method, path) = (parts.next().unwrap_or("").to_string(), parts.next().unwrap_or("").to_string());
602 +
        let (mut length, mut auth, mut close) = (0usize, String::new(), false);
603 +
        loop {
604 +
            let mut h = String::new();
605 +
            if r.read_line(&mut h)? == 0 {
606 +
                return Ok(());
607 +
            }
608 +
            let h = h.trim_end();
609 +
            if h.is_empty() {
610 +
                break;
611 +
            }
612 +
            if let Some((k, v)) = h.split_once(':') {
613 +
                match k.trim().to_ascii_lowercase().as_str() {
614 +
                    "content-length" => length = v.trim().parse().unwrap_or(usize::MAX),
615 +
                    "authorization" => auth = v.trim().to_string(),
616 +
                    "connection" => close = v.trim().eq_ignore_ascii_case("close"),
617 +
                    _ => {}
618 +
                }
619 +
            }
620 +
        }
621 +
        if length > MAX_BODY {
622 +
            return respond(r.get_mut(), 400, "text/plain", b"too large");
623 +
        }
624 +
        let mut body = vec![0; length];
625 +
        r.read_exact(&mut body)?;
626 +
        if busy {
627 +
            return respond(r.get_mut(), 503, "text/plain", b"busy: ask again shortly");
628 +
        }
629 +
        let (code, kind, reply) = route(state, &method, &path, &auth, &body);
630 +
        if code == 200 && kind == "application/octet-stream" {
631 +
            throttle(state, reply.len());
632 +
        }
633 +
        respond(r.get_mut(), code, kind, &reply)?;
634 +
        if close {
635 +
            return Ok(());
636 +
        }
637 +
    }
638 +
}
639 +
640 +
fn route(state: &State, method: &str, path: &str, auth: &str, body: &[u8]) -> (u16, &'static str, Vec<u8>) {
641 +
    let text = |code, s: &str| (code, "text/plain", s.as_bytes().to_vec());
642 +
    if method == "POST" && path == "/swarm/v1/hello" {
643 +
        return match admit(state, &String::from_utf8_lossy(body)) {
644 +
            Ok(session) => (200, "application/json", serde_json::json!({ "session": session }).to_string().into_bytes()),
645 +
            Err(why) => text(403, &why),
646 +
        };
647 +
    }
648 +
    let session = auth.strip_prefix("Bearer ").unwrap_or("");
649 +
    let admitted = {
650 +
        let mut sessions = state.sessions.lock().unwrap();
651 +
        let now = now_ms();
652 +
        sessions.retain(|_, (_, until)| *until > now);
653 +
        sessions.contains_key(session)
654 +
    };
655 +
    if !admitted {
656 +
        return text(401, "show a ticket at /swarm/v1/hello first");
657 +
    }
658 +
    let rest = path.strip_prefix("/swarm/v1/").unwrap_or("");
659 +
    match (method, rest.split('/').collect::<Vec<_>>().as_slice()) {
660 +
        ("GET", ["images", image, "have"]) => match state.offers.lock().unwrap().get(*image) {
661 +
            Some(Offer { bits: None, .. }) => (200, "text/plain", b"complete".to_vec()),
662 +
            Some(Offer { bits: Some(b), fetching }) => (200, "application/x-have", [b.to_bytes(), fetching.to_bytes()].concat()),
663 +
            None => text(404, "this runner does not hold that image"),
664 +
        },
665 +
        ("GET", ["chunks", chunk]) if is_hash(chunk) => match read_chunk(state, chunk) {
666 +
            Some(b) => (200, "application/octet-stream", b),
667 +
            None => text(404, "no such chunk here"),
668 +
        },
669 +
        _ => text(404, "not a swarm path"),
670 +
    }
671 +
}
672 +
673 +
/// Admit a peer that shows a ticket from this runner's hub, for this
674 +
/// site: a session until the ticket expires.
675 +
fn admit(state: &State, ticket: &str) -> std::result::Result<String, String> {
676 +
    let hub = {
677 +
        let mut hub = state.hub.lock().unwrap();
678 +
        if hub.is_none() {
679 +
            *hub = state.client.hub_identity().ok();
680 +
        }
681 +
        hub.clone().ok_or("this runner cannot reach its hub to check tickets")?
682 +
    };
683 +
    let t = Ticket::open(ticket, &hub, now_ms())?;
684 +
    if t.site != state.site {
685 +
        return Err(format!("this runner is in site {:?}, the ticket's runner in {:?}", state.site, t.site));
686 +
    }
687 +
    let session = random_token().map_err(|e| format!("make a session: {e}"))?;
688 +
    state.sessions.lock().unwrap().insert(session.clone(), (t.name, t.expires_ms));
689 +
    Ok(session)
690 +
}
691 +
692 +
fn random_token() -> std::io::Result<String> {
693 +
    let mut b = [0u8; 32];
694 +
    std::fs::File::open("/dev/urandom")?.read_exact(&mut b)?;
695 +
    Ok(hex::encode(b))
696 +
}
697 +
698 +
fn read_chunk(state: &State, hash: &str) -> Option<Vec<u8>> {
699 +
    if let Ok(b) = std::fs::read(cache_path(&state.root.join("chunks"), hash)) {
700 +
        return Some(b);
701 +
    }
702 +
    let (path, at, len) = state.located.lock().unwrap().get(hash).cloned()?;
703 +
    let mut f = std::fs::File::open(path).ok()?;
704 +
    f.seek(SeekFrom::Start(at)).ok()?;
705 +
    let mut b = vec![0; len as usize];
706 +
    f.read_exact(&mut b).ok()?;
707 +
    Some(b)
708 +
}
709 +
710 +
/// Hold back until sending `n` more bytes keeps within the upload rate.
711 +
fn throttle(state: &State, n: usize) {
712 +
    let wait = {
713 +
        let mut b = state.bucket.lock().unwrap();
714 +
        let now = Instant::now();
715 +
        b.0 = (b.0 + now.duration_since(b.1).as_secs_f64() * state.rate).min(state.rate);
716 +
        b.1 = now;
717 +
        b.0 -= n as f64;
718 +
        if b.0 < 0.0 {
719 +
            -b.0 / state.rate
720 +
        } else {
721 +
            0.0
722 +
        }
723 +
    };
724 +
    if wait > 0.0 {
725 +
        std::thread::sleep(Duration::from_secs_f64(wait));
726 +
    }
727 +
}
728 +
729 +
#[cfg(test)]
730 +
mod tests {
731 +
    use super::*;
732 +
    use oxy_core::identity::SecretKey;
733 +
734 +
    /// A hub that holds one image and runs the tracker: it serves the
735 +
    /// manifest and chunks (counting the chunks), and answers announces
736 +
    /// with every other runner as a peer and a ticket for the caller,
737 +
    /// known by its token.
738 +
    struct FakeHub {
739 +
        url: String,
740 +
        chunks_served: Arc<AtomicUsize>,
741 +
        runners: Arc<Mutex<Vec<(String, SwarmPeer)>>>,
742 +
    }
743 +
744 +
    fn fake_hub(m: Manifest, image: Vec<u8>, site: &str) -> FakeHub {
745 +
        hub_with(m, image, site, Duration::ZERO, true)
746 +
    }
747 +
748 +
    /// A hub that takes `delay` over each chunk.
749 +
    fn slow_hub(m: Manifest, image: Vec<u8>, site: &str, delay: Duration) -> FakeHub {
750 +
        hub_with(m, image, site, delay, true)
751 +
    }
752 +
753 +
    /// A hub that keeps the image's manifest but none of its chunks, as for
754 +
    /// an image only runners hold (jetstream.md §8.1.1).
755 +
    fn peer_only_hub(m: Manifest, image: Vec<u8>, site: &str) -> FakeHub {
756 +
        hub_with(m, image, site, Duration::ZERO, false)
757 +
    }
758 +
759 +
    fn hub_with(m: Manifest, image: Vec<u8>, site: &str, delay: Duration, seeds: bool) -> FakeHub {
760 +
        let sk = SecretKey::generate();
761 +
        let id = sk.identity("hub", "hub@example", 1);
762 +
        let key = id.hash();
763 +
        let mut chunks = HashMap::new();
764 +
        let mut at = 0usize;
765 +
        for c in &m.chunks {
766 +
            chunks.insert(c.hash().to_string(), image[at..at + c.len()].to_vec());
767 +
            at += c.len();
768 +
        }
769 +
        let listener = TcpListener::bind("127.0.0.1:0").unwrap();
770 +
        let url = format!("http://{}", listener.local_addr().unwrap());
771 +
        let served = Arc::new(AtomicUsize::new(0));
772 +
        let runners: Arc<Mutex<Vec<(String, SwarmPeer)>>> = Arc::new(Mutex::new(Vec::new()));
773 +
        let (count, known, site) = (served.clone(), runners.clone(), site.to_string());
774 +
        let shared = Arc::new((m, chunks, id, key, sk));
775 +
        std::thread::spawn(move || {
776 +
            for conn in listener.incoming() {
777 +
                let Ok(conn) = conn else { continue };
778 +
                let (shared, count, known, site) = (shared.clone(), count.clone(), known.clone(), site.clone());
779 +
                std::thread::spawn(move || {
780 +
                    let (m, chunks, id, key, sk) = &*shared;
781 +
                    let mut out = conn.try_clone().unwrap();
782 +
                    let mut r = BufReader::new(conn);
783 +
                    loop {
784 +
                        let mut line = String::new();
785 +
                        if r.read_line(&mut line).unwrap_or(0) == 0 {
786 +
                            return;
787 +
                        }
788 +
                        let path = line.split_whitespace().nth(1).unwrap_or("").to_string();
789 +
                        let (mut len, mut token) = (0, String::new());
790 +
                        loop {
791 +
                            let mut h = String::new();
792 +
                            r.read_line(&mut h).unwrap();
793 +
                            let h = h.trim_end().to_string();
794 +
                            if h.is_empty() {
795 +
                                break;
796 +
                            }
797 +
                            let lower = h.to_ascii_lowercase();
798 +
                            if let Some(v) = lower.strip_prefix("content-length:") {
799 +
                                len = v.trim().parse().unwrap();
800 +
                            }
801 +
                            if let Some(v) = h.strip_prefix("Authorization: Bearer ") {
802 +
                                token = v.to_string();
803 +
                            }
804 +
                        }
805 +
                        let mut body = vec![0; len];
806 +
                        r.read_exact(&mut body).unwrap();
807 +
                        let (code, reply): (u16, Vec<u8>) = if path == "/.well-known/oxyhub-key" {
808 +
                            (200, id.encode())
809 +
                        } else if path == format!("/api/v1/flow/images/{}", m.hash) {
810 +
                            (200, serde_json::to_vec(m).unwrap())
811 +
                        } else if let Some(c) = path.strip_prefix("/api/v1/flow/images/chunks/").filter(|_| seeds) {
812 +
                            count.fetch_add(1, Ordering::Relaxed);
813 +
                            std::thread::sleep(delay);
814 +
                            (200, chunks[c].clone())
815 +
                        } else if path == "/api/v1/flow/swarm/announce" {
816 +
                            let runners = known.lock().unwrap();
817 +
                            let me = runners.iter().find(|(t, _)| *t == token).map(|(_, p)| p.clone()).unwrap();
818 +
                            let ticket = Ticket {
819 +
                                hub: key.to_hex(),
820 +
                                runner: me.runner,
821 +
                                name: me.name.clone(),
822 +
                                site: site.clone(),
823 +
                                expires_ms: now_ms() + 60_000,
824 +
                            };
825 +
                            let peers: Vec<SwarmPeer> = runners.iter().filter(|(t, _)| *t != token).map(|(_, p)| p.clone()).collect();
826 +
                            let a = flow_core::api::Announced {
827 +
                                ticket: ticket.seal(key, sk),
828 +
                                peers,
829 +
                                interval_s: 5,
830 +
                                seed: (!seeds).then_some(false),
831 +
                            };
832 +
                            (200, serde_json::to_vec(&a).unwrap())
833 +
                        } else {
834 +
                            (404, b"no".to_vec())
835 +
                        };
836 +
                        let head = format!("HTTP/1.1 {code} X\r\nContent-Length: {}\r\n\r\n", reply.len());
837 +
                        if out.write_all(head.as_bytes()).and_then(|_| out.write_all(&reply)).is_err() {
838 +
                            return;
839 +
                        }
840 +
                    }
841 +
                });
842 +
            }
843 +
        });
844 +
        FakeHub { url, chunks_served: served, runners }
845 +
    }
846 +
847 +
    fn free_port() -> u16 {
848 +
        TcpListener::bind("127.0.0.1:0").unwrap().local_addr().unwrap().port()
849 +
    }
850 +
851 +
    /// A runner home in `dir` with a swarm on `port`, known to `hub` by
852 +
    /// `token` as runner `id`.
853 +
    fn runner(dir: &Path, hub: &FakeHub, id: i64, token: &str, port: u16, site: &str) -> (Home, Swarm) {
854 +
        let toml = format!(
855 +
            "hub = \"{}\"\ntoken = \"{token}\"\nname = \"r{id}\"\n[swarm]\nlisten = \"127.0.0.1:{port}\"\nsite = \"{site}\"\n",
856 +
            hub.url
857 +
        );
858 +
        std::fs::write(dir.join(crate::home::RUNNER_TOML), toml).unwrap();
859 +
        let home = Home::locate(Some(dir)).unwrap();
860 +
        let cfg = home.config.runner.clone().unwrap();
861 +
        let swarm = Swarm::start(&home, &cfg).unwrap().unwrap();
862 +
        hub.runners.lock().unwrap().push((
863 +
            token.into(),
864 +
            SwarmPeer {
865 +
                runner: id,
866 +
                name: format!("r{id}"),
867 +
                addr: format!("127.0.0.1:{port}"),
868 +
                cert: swarm.addr.cert.clone(),
869 +
                complete: true,
870 +
            },
871 +
        ));
872 +
        (home, swarm)
873 +
    }
874 +
875 +
    fn image() -> Vec<u8> {
876 +
        // 3 MiB of varied bytes, with a run of zeros the swarm never moves.
877 +
        let mut x = 0x9e3779b97f4a7c15u64;
878 +
        let mut b: Vec<u8> = (0..3 << 20)
879 +
            .map(|_| {
880 +
                x ^= x << 13;
881 +
                x ^= x >> 7;
882 +
                x ^= x << 17;
883 +
                x as u8
884 +
            })
885 +
            .collect();
886 +
        b[1 << 20..(1 << 20) + (300 << 10)].fill(0);
887 +
        b
888 +
    }
889 +
890 +
    #[test]
891 +
    fn a_runner_fetches_an_image_from_its_peer_and_not_the_hub() {
892 +
        let bytes = image();
893 +
        let m = jet::images::scan(&bytes[..], Some("x86_64".into()), |_, _| Ok(())).unwrap();
894 +
        let hub = fake_hub(m.clone(), bytes.clone(), "lab");
895 +
        let (a_dir, b_dir) = (tempfile::tempdir().unwrap(), tempfile::tempdir().unwrap());
896 +
        // Runner A holds the image, imported rather than fetched: its swarm
897 +
        // scans it for a manifest before offering it.
898 +
        std::fs::create_dir_all(a_dir.path().join("cas")).unwrap();
899 +
        std::fs::write(a_dir.path().join("cas").join(format!("{}.ext4", m.hash)), &bytes).unwrap();
900 +
        let (_a_home, a) = runner(a_dir.path(), &hub, 1, "ta", free_port(), "lab");
901 +
        let deadline = Instant::now() + Duration::from_secs(20);
902 +
        while !a.state.offers.lock().unwrap().contains_key(&m.hash) {
903 +
            assert!(Instant::now() < deadline, "runner A never offered its image");
904 +
            std::thread::sleep(Duration::from_millis(20));
905 +
        }
906 +
        let (b_home, b) = runner(b_dir.path(), &hub, 2, "tb", free_port(), "lab");
907 +
        let moved = b.fetch(&b_home, &m.hash).unwrap().unwrap();
908 +
        assert_eq!(std::fs::read(b_home.rootfs(&m.hash)).unwrap(), bytes, "the image came back whole");
909 +
        assert_eq!(moved.peer_bytes, moved.bytes, "every chunk came from the peer");
910 +
        assert_eq!(hub.chunks_served.load(Ordering::Relaxed), 0, "the hub sent no chunks");
911 +
        assert_eq!(moved.chunks, m.needed().len());
912 +
        // B now offers the whole image too, read out of its rootfs.
913 +
        assert!(matches!(b.state.offers.lock().unwrap().get(&m.hash), Some(Offer { bits: None, .. })));
914 +
    }
915 +
916 +
    #[test]
917 +
    fn an_image_only_runners_hold_comes_from_a_peer_and_fails_plainly_without_one() {
918 +
        let bytes = image();
919 +
        let m = jet::images::scan(&bytes[..], Some("x86_64".into()), |_, _| Ok(())).unwrap();
920 +
        let hub = peer_only_hub(m.clone(), bytes.clone(), "lab");
921 +
        // No peer yet: no hub to fall back on, so the fetch says why.
922 +
        let lone = tempfile::tempdir().unwrap();
923 +
        let (lone_home, lone_runner) = runner(lone.path(), &hub, 3, "tc", free_port(), "lab");
924 +
        let e = lone_runner.fetch(&lone_home, &m.hash).err().expect("a fetch with no peer and no seed fails");
925 +
        assert!(e.to_string().contains("held only by runners"), "{e:#}");
926 +
        // With a peer holding it, it all comes from the peer.
927 +
        let (a_dir, b_dir) = (tempfile::tempdir().unwrap(), tempfile::tempdir().unwrap());
928 +
        std::fs::create_dir_all(a_dir.path().join("cas")).unwrap();
929 +
        std::fs::write(a_dir.path().join("cas").join(format!("{}.ext4", m.hash)), &bytes).unwrap();
930 +
        let (_a_home, a) = runner(a_dir.path(), &hub, 1, "ta", free_port(), "lab");
931 +
        let deadline = Instant::now() + Duration::from_secs(20);
932 +
        while !a.state.offers.lock().unwrap().contains_key(&m.hash) {
933 +
            assert!(Instant::now() < deadline, "runner A never offered its image");
934 +
            std::thread::sleep(Duration::from_millis(20));
935 +
        }
936 +
        let (b_home, b) = runner(b_dir.path(), &hub, 2, "tb", free_port(), "lab");
937 +
        let moved = b.fetch(&b_home, &m.hash).unwrap().unwrap();
938 +
        assert_eq!(std::fs::read(b_home.rootfs(&m.hash)).unwrap(), bytes);
939 +
        assert_eq!(moved.peer_bytes, moved.bytes, "every chunk came from the peer");
940 +
        assert_eq!(hub.chunks_served.load(Ordering::Relaxed), 0);
941 +
    }
942 +
943 +
    #[test]
944 +
    fn runners_starting_together_share_what_each_has_fetched() {
945 +
        let bytes = image();
946 +
        let m = jet::images::scan(&bytes[..], None, |_, _| Ok(())).unwrap();
947 +
        let needed = m.needed().len();
948 +
        // A slow hub, as a remote one is: the runners have time to swap.
949 +
        let hub = slow_hub(m.clone(), bytes.clone(), "lab", Duration::from_millis(400));
950 +
        let (b_dir, c_dir) = (tempfile::tempdir().unwrap(), tempfile::tempdir().unwrap());
951 +
        let (b_home, b) = runner(b_dir.path(), &hub, 2, "tb", free_port(), "lab");
952 +
        let (c_home, c) = runner(c_dir.path(), &hub, 3, "tc", free_port(), "lab");
953 +
        let (mb, mc) = std::thread::scope(|s| {
954 +
            let fb = s.spawn(|| b.fetch(&b_home, &m.hash).unwrap().unwrap());
955 +
            let fc = s.spawn(|| c.fetch(&c_home, &m.hash).unwrap().unwrap());
956 +
            (fb.join().unwrap(), fc.join().unwrap())
957 +
        });
958 +
        assert_eq!(std::fs::read(b_home.rootfs(&m.hash)).unwrap(), bytes);
959 +
        assert_eq!(std::fs::read(c_home.rootfs(&m.hash)).unwrap(), bytes);
960 +
        let served = hub.chunks_served.load(Ordering::Relaxed);
961 +
        assert!(served < 2 * needed, "the hub sent {served} chunks for two runners needing {needed} each");
962 +
        assert!(mb.peer_bytes + mc.peer_bytes > 0, "they shared");
963 +
    }
964 +
965 +
    #[test]
966 +
    fn without_peers_the_hub_sends_everything_and_a_ticket_from_another_site_is_refused() {
967 +
        let bytes = image();
968 +
        let m = jet::images::scan(&bytes[..], None, |_, _| Ok(())).unwrap();
969 +
        let hub = fake_hub(m.clone(), bytes.clone(), "lab");
970 +
        let dir = tempfile::tempdir().unwrap();
971 +
        let (home, s) = runner(dir.path(), &hub, 1, "t1", free_port(), "lab");
972 +
        let moved = s.fetch(&home, &m.hash).unwrap().unwrap();
973 +
        assert_eq!(moved.peer_bytes, 0);
974 +
        assert_eq!(hub.chunks_served.load(Ordering::Relaxed), m.needed().len());
975 +
        assert_eq!(std::fs::read(home.rootfs(&m.hash)).unwrap(), bytes);
976 +
977 +
        // A runner in another site shows a ticket for "lab": refused.
978 +
        let other = tempfile::tempdir().unwrap();
979 +
        let port = free_port();
980 +
        let (_h, o) = runner(other.path(), &hub, 3, "t3", port, "elsewhere");
981 +
        let ticket = s.state.client.announce(&m.hash, "fetching").unwrap().ticket;
982 +
        let err = hello(&format!("127.0.0.1:{port}"), &o.addr.cert, &ticket).unwrap_err();
983 +
        assert!(format!("{err:#}").contains("HTTP 403"), "{err:#}");
984 +
        // And no session, no chunks.
985 +
        let url = format!("https://127.0.0.1:{port}/swarm/v1/chunks/{}", m.chunks[0].hash());
986 +
        assert!(matches!(peer_agent(&o.addr.cert).unwrap().get(&url).call(), Err(ureq::Error::Status(401, _))));
987 +
        // Nor any answer at all without TLS.
988 +
        assert!(ureq::get(&url.replacen("https", "http", 1)).call().is_err());
989 +
    }
990 +
991 +
    #[test]
992 +
    fn a_peer_with_another_certificate_is_not_trusted() {
993 +
        let bytes = image();
994 +
        let m = jet::images::scan(&bytes[..], None, |_, _| Ok(())).unwrap();
995 +
        let hub = fake_hub(m.clone(), bytes.clone(), "lab");
996 +
        let (a_dir, b_dir) = (tempfile::tempdir().unwrap(), tempfile::tempdir().unwrap());
997 +
        std::fs::create_dir_all(a_dir.path().join("cas")).unwrap();
998 +
        std::fs::write(a_dir.path().join("cas").join(format!("{}.ext4", m.hash)), &bytes).unwrap();
999 +
        let (_a_home, a) = runner(a_dir.path(), &hub, 1, "ta", free_port(), "lab");
1000 +
        while !a.state.offers.lock().unwrap().contains_key(&m.hash) {
1001 +
            std::thread::sleep(Duration::from_millis(20));
1002 +
        }
1003 +
        let (b_home, b) = runner(b_dir.path(), &hub, 2, "tb", free_port(), "lab");
1004 +
        // The hub names A, but with B's certificate: whoever answers at A's
1005 +
        // address is not who the hub vouched for.
1006 +
        hub.runners.lock().unwrap()[0].1.cert = b.addr.cert.clone();
1007 +
        let moved = b.fetch(&b_home, &m.hash).unwrap().unwrap();
1008 +
        assert_eq!(moved.peer_bytes, 0, "nothing was taken from the stranger");
1009 +
        assert_eq!(hub.chunks_served.load(Ordering::Relaxed), m.needed().len());
1010 +
        assert_eq!(std::fs::read(b_home.rootfs(&m.hash)).unwrap(), bytes);
1011 +
        // The certificate is kept: the same runner, the same fingerprint.
1012 +
        let mut cfg = b_home.config.runner.clone().unwrap();
1013 +
        cfg.swarm.as_mut().unwrap().listen = "127.0.0.1:0".into();
1014 +
        let again = Swarm::start(&b_home, &cfg).unwrap().unwrap();
1015 +
        assert_eq!(again.addr.cert, b.addr.cert);
1016 +
    }
1017 +
1018 +
    #[test]
1019 +
    fn the_upload_rate_holds_back_a_burst() {
1020 +
        let dir = tempfile::tempdir().unwrap();
1021 +
        let (cert, key) = certificate(dir.path()).unwrap();
1022 +
        let state = State {
1023 +
            tls: server_tls(cert, key).unwrap(),
1024 +
            root: PathBuf::new(),
1025 +
            site: String::new(),
1026 +
            client: Client::new("http://127.0.0.1:1", "t", None).unwrap(),
1027 +
            hub: Mutex::new(None),
1028 +
            offers: Mutex::new(HashMap::new()),
1029 +
            located: Mutex::new(HashMap::new()),
1030 +
            sessions: Mutex::new(HashMap::new()),
1031 +
            conns: AtomicUsize::new(0),
1032 +
            bucket: Mutex::new((1_000_000.0, Instant::now())),
1033 +
            rate: 1_000_000.0,
1034 +
        };
1035 +
        let start = Instant::now();
1036 +
        throttle(&state, 1_000_000);
1037 +
        assert!(start.elapsed() < Duration::from_millis(100), "the first second's worth goes at once");
1038 +
        throttle(&state, 300_000);
1039 +
        assert!(start.elapsed() >= Duration::from_millis(250), "then it waits for the rate");
1040 +
    }
1041 +
}
modified crates/flow/src/syscall.rs +58 −46
…
2 unchanged lines
3 3
//! network, IPC and UTS namespaces, on the rootfs image's tree (unpacked
4 4
//! once per hash, read-only) with the workspace directory at `/src` and a
5 5
//! tmpfs at `/tmp`. Landlock limits writes to `/src`, `/tmp` and `/dev`; a
6 -
//! seccomp filter refuses sockets other than `AF_UNIX` and the calls that
6 +
//! seccomp filter refuses sockets other than `AF_UNIX` (and, with
7 +
//! `network "loopback"`, IPv4 and IPv6 on its own `lo`) and the calls that
7 8
//! reach past the sandbox (`ptrace`, `mount`, `bpf`, `perf_event_open`,
8 9
//! keys, modules, io_uring); and the sandbox's CPU and memory limits apply
9 10
//! through a cgroup v2 group of the runner's own delegated cgroup. Without
…
9 unchanged lines
19 20
use crate::guest::{dev_links, job_env, loopback_up, resolve_program, PATH};
20 21
use crate::home::Home;
21 22
use crate::microvm::{Outcome, Sink};
23 +
pub use crate::tier::disk_dir;
24 +
pub(crate) use crate::tier::{lines, self_hash, FAILED};
22 25
use anyhow::{bail, Context, Result};
23 26
use flow_core::api::{Record, SecretInput};
24 27
use flow_core::{Environment, Job, MountSource, Network, Sandbox};
25 28
use serde::{Deserialize, Serialize};
26 29
use std::ffi::CString;
27 -
use std::io::{BufRead, BufReader, Read};
28 30
use std::path::{Path, PathBuf};
29 31
use std::sync::{mpsc, OnceLock};
30 32
use std::time::{Duration, Instant};
…
23 unchanged lines
54 56
    }
55 57
}
56 58
57 -
fn self_hash() -> &'static str {
58 -
    static HASH: OnceLock<String> = OnceLock::new();
59 -
    HASH.get_or_init(|| crate::home::hash_file(Path::new("/proc/self/exe")).unwrap_or_else(|_| "unknown".into()))
60 -
}
61 -
62 59
fn kernel_release() -> String {
63 -
    std::fs::read_to_string("/proc/sys/kernel/osrelease").map(|s| s.trim().to_string()).unwrap_or_else(|_| "unknown".into())
60 +
    crate::facts::kernel().unwrap_or_else(|| "unknown".into())
64 61
}
65 62
66 63
/// Whether this machine can run the tier now: a cgroup this runner may
…
33 unchanged lines
100 97
            return Err("Landlock is not enabled in this kernel (it must be in the lsm= list)".into());
101 98
        }
102 99
        no_new_privs()?;
103 -
        seccomp(&[]).map_err(|e| format!("seccomp filters are not available: {e}"))
100 +
        seccomp(&[], families(false)).map_err(|e| format!("seccomp filters are not available: {e}"))
104 101
    })();
105 102
    match r {
106 103
        Ok(()) => 0,
…
179 unchanged lines
286 283
        timed_out,
287 284
        cancelled,
288 285
        peak_memory_mib,
286 +
        pruned: Vec::new(),
289 287
    })
290 -
}
291 -
292 -
/// Where the `n`th `ephemeral://disk` mount of a job keeps its files.
293 -
pub fn disk_dir(scratch: &Path, n: usize) -> PathBuf {
294 -
    scratch.join(format!("mount-{n}"))
295 -
}
296 -
297 -
/// How the helper says the job never started, on its stderr.
298 -
const FAILED: &str = "\u{1e}F ";
299 -
300 -
fn lines(r: impl Read, stdout: bool, tx: mpsc::Sender<(bool, Vec<u8>)>) {
301 -
    let mut r = BufReader::new(r);
302 -
    let mut buf = Vec::new();
303 -
    loop {
304 -
        buf.clear();
305 -
        // A long line is cut at 16 KiB, as flow-init cuts it.
306 -
        match (&mut r).take(16 * 1024).read_until(b'\n', &mut buf) {
307 -
            Ok(0) | Err(_) => break,
308 -
            Ok(_) => {
309 -
                if buf.last() == Some(&b'\n') {
310 -
                    buf.pop();
311 -
                }
312 -
                if tx.send((stdout, buf.clone())).is_err() {
313 -
                    break;
314 -
                }
315 -
            }
316 -
        }
317 -
    }
318 288
}
319 289
320 290
/// The image's tree, unpacked from its ext4 file once per hash with
…
325 unchanged lines
646 616
            }
647 617
            no_new_privs()?;
648 618
            landlock(&writable)?;
649 -
            seccomp(DENIED)
619 +
            seccomp(DENIED, families(spec.loopback))
650 620
        })();
651 621
        unsafe {
652 622
            if let Err(e) = confined {
…
175 unchanged lines
828 798
    libc::SYS_io_uring_register,
829 799
];
830 800
801 +
/// Classic BPF opcodes (linux/bpf_common.h), fixed by the kernel's ABI;
802 +
/// spelled out because libc does not export them for every target
803 +
/// (Android, where the runner is built for the WASM tier only).
804 +
const BPF_LD: u16 = 0x00;
805 +
const BPF_W: u16 = 0x00;
806 +
const BPF_ABS: u16 = 0x20;
807 +
const BPF_JMP: u16 = 0x05;
808 +
const BPF_JEQ: u16 = 0x10;
809 +
const BPF_JGE: u16 = 0x30;
810 +
const BPF_K: u16 = 0x00;
811 +
const BPF_RET: u16 = 0x06;
812 +
813 +
/// Socket families a job may open: `AF_UNIX` always; with `network
814 +
/// "loopback"` also IPv4 and IPv6 (oxygenflow#73), which in the job's own
815 +
/// network namespace reach only its `lo`.
816 +
fn families(loopback: bool) -> &'static [libc::c_int] {
817 +
    if loopback {
818 +
        &[libc::AF_UNIX, libc::AF_INET, libc::AF_INET6]
819 +
    } else {
820 +
        &[libc::AF_UNIX]
821 +
    }
822 +
}
823 +
831 824
/// Install a filter that fails each of `denied` with EPERM and `socket`
832 -
/// for any family but `AF_UNIX` with EAFNOSUPPORT (an empty `denied`
825 +
/// for a family not in `allowed` with EAFNOSUPPORT (an empty `denied`
833 826
/// still checks sockets). Another architecture's calls kill the process.
834 -
fn seccomp(denied: &[libc::c_long]) -> std::result::Result<(), String> {
835 -
    const LD: u16 = (libc::BPF_LD | libc::BPF_W | libc::BPF_ABS) as u16;
836 -
    const JEQ: u16 = (libc::BPF_JMP | libc::BPF_JEQ | libc::BPF_K) as u16;
837 -
    const RET: u16 = (libc::BPF_RET | libc::BPF_K) as u16;
827 +
fn seccomp(denied: &[libc::c_long], allowed: &[libc::c_int]) -> std::result::Result<(), String> {
828 +
    const LD: u16 = BPF_LD | BPF_W | BPF_ABS;
829 +
    const JEQ: u16 = BPF_JMP | BPF_JEQ | BPF_K;
830 +
    const RET: u16 = BPF_RET | BPF_K;
838 831
    let op = |code: u16, k: u32, jt: u8, jf: u8| libc::sock_filter { code, jt, jf, k };
839 832
    let errno = |e: libc::c_int| libc::SECCOMP_RET_ERRNO | (e as u32 & libc::SECCOMP_RET_DATA);
840 833
    // struct seccomp_data: nr at 0, arch at 4, args[0] at 16 (its low
…
2 unchanged lines
843 836
    #[cfg(target_arch = "x86_64")]
844 837
    {
845 838
        // x32 calls carry this bit; there are none in a 64-bit job.
846 -
        const JGE: u16 = (libc::BPF_JMP | libc::BPF_JGE | libc::BPF_K) as u16;
839 +
        const JGE: u16 = BPF_JMP | BPF_JGE | BPF_K;
847 840
        f.push(op(JGE, 0x4000_0000, 0, 1));
848 841
        f.push(op(RET, errno(libc::EPERM), 0, 0));
849 842
    }
…
1 unchanged lines
851 844
        f.push(op(JEQ, *nr as u32, 0, 1));
852 845
        f.push(op(RET, errno(libc::EPERM), 0, 0));
853 846
    }
854 -
    f.push(op(JEQ, libc::SYS_socket as u32, 0, 3));
847 +
    // socket(): past the family checks and the refusal to the allow when
848 +
    // it is another call; each allowed family jumps over the rest to it.
849 +
    let n = allowed.len() as u8;
850 +
    f.push(op(JEQ, libc::SYS_socket as u32, 0, n + 2));
855 851
    f.push(op(LD, 16, 0, 0));
856 -
    f.push(op(JEQ, libc::AF_UNIX as u32, 1, 0));
852 +
    for (i, fam) in allowed.iter().enumerate() {
853 +
        f.push(op(JEQ, *fam as u32, n - i as u8, 0));
854 +
    }
857 855
    f.push(op(RET, errno(libc::EAFNOSUPPORT), 0, 0));
858 856
    f.push(op(RET, libc::SECCOMP_RET_ALLOW, 0, 0));
859 857
    let prog = libc::sock_fprog { len: f.len() as u16, filter: f.as_mut_ptr() };
…
2 unchanged lines
862 860
    }
863 861
    Ok(())
864 862
}
863 +
864 +
#[cfg(all(test, not(target_os = "android")))]
865 +
mod tests {
866 +
    /// The opcodes written out above are the ABI's, as glibc's libc has them.
867 +
    #[test]
868 +
    fn bpf_opcodes_are_the_kernels() {
869 +
        use super::*;
870 +
        assert_eq!(
871 +
            [BPF_LD, BPF_W, BPF_ABS, BPF_JMP, BPF_JEQ, BPF_JGE, BPF_K, BPF_RET],
872 +
            [libc::BPF_LD, libc::BPF_W, libc::BPF_ABS, libc::BPF_JMP, libc::BPF_JEQ, libc::BPF_JGE, libc::BPF_K, libc::BPF_RET]
873 +
                .map(|v| v as u16)
874 +
        );
875 +
    }
876 +
}
added crates/flow/src/syscall_none.rs +46 −0
1 +
//! The syscall tier where it cannot be: it is built from Linux's user
2 +
//! namespaces, Landlock, seccomp and cgroups, so a runner elsewhere (a
3 +
//! Windows runner, oxygenflow#83; Haiku and FreeBSD, #91, #99) never
4 +
//! offers it, and these say why.
5 +
6 +
use crate::home::Home;
7 +
use crate::microvm::{Outcome, Sink};
8 +
use anyhow::{bail, Result};
9 +
use flow_core::api::SecretInput;
10 +
use flow_core::{Job, Sandbox};
11 +
use std::path::Path;
12 +
use std::time::Duration;
13 +
14 +
pub use crate::tier::disk_dir;
15 +
pub(crate) use crate::tier::{lines, self_hash, FAILED};
16 +
17 +
const WHY: &str = "the syscall tier runs only on Linux";
18 +
19 +
pub fn available() -> std::result::Result<(), String> {
20 +
    Err(WHY.into())
21 +
}
22 +
23 +
pub fn probe_main() -> i32 {
24 +
    eprintln!("{WHY}");
25 +
    1
26 +
}
27 +
28 +
pub fn sandbox_main(_spec_path: &Path) -> i32 {
29 +
    eprintln!("{WHY}");
30 +
    125
31 +
}
32 +
33 +
#[allow(clippy::too_many_arguments)]
34 +
pub fn run(
35 +
    _home: &Home,
36 +
    _sandbox: &Sandbox,
37 +
    _job: &Job,
38 +
    _secrets: &[SecretInput],
39 +
    _workspace: &Path,
40 +
    _scratch: &Path,
41 +
    _timeout: Duration,
42 +
    _sink: &mut dyn Sink,
43 +
    _name: &str,
44 +
) -> Result<Outcome> {
45 +
    bail!("{WHY}")
46 +
}
added crates/flow/src/tier.rs +52 −0
1 +
//! What the sandbox tiers share, on every system a runner runs on: where a
2 +
//! job's disks are kept, how a helper says a job never started, reading a
3 +
//! job's output by lines, and this binary's hash for receipts.
4 +
5 +
use std::io::{BufRead, BufReader, Read};
6 +
use std::path::{Path, PathBuf};
7 +
use std::sync::{mpsc, OnceLock};
8 +
9 +
/// Where the `n`th `ephemeral://disk` mount of a job keeps its files.
10 +
pub fn disk_dir(scratch: &Path, n: usize) -> PathBuf {
11 +
    scratch.join(format!("mount-{n}"))
12 +
}
13 +
14 +
/// How the helper says the job never started, on its stderr.
15 +
pub(crate) const FAILED: &str = "\u{1e}F ";
16 +
17 +
pub(crate) fn lines(r: impl Read, stdout: bool, tx: mpsc::Sender<(bool, Vec<u8>)>) {
18 +
    let mut r = BufReader::new(r);
19 +
    let mut buf = Vec::new();
20 +
    loop {
21 +
        buf.clear();
22 +
        // A long line is cut at 16 KiB, as flow-init cuts it.
23 +
        match (&mut r).take(16 * 1024).read_until(b'\n', &mut buf) {
24 +
            Ok(0) | Err(_) => break,
25 +
            Ok(_) => {
26 +
                if buf.last() == Some(&b'\n') {
27 +
                    buf.pop();
28 +
                }
29 +
                if tx.send((stdout, buf.clone())).is_err() {
30 +
                    break;
31 +
                }
32 +
            }
33 +
        }
34 +
    }
35 +
}
36 +
37 +
/// This binary, to start helpers from and hash: `/proc/self/exe` on Linux
38 +
/// and Android (the running file, even after an update replaced its name),
39 +
/// the path it was started from elsewhere (oxygenflow#83).
40 +
pub fn self_exe() -> PathBuf {
41 +
    if cfg!(target_os = "linux") || cfg!(target_os = "android") {
42 +
        PathBuf::from("/proc/self/exe")
43 +
    } else {
44 +
        std::env::current_exe().unwrap_or_default()
45 +
    }
46 +
}
47 +
48 +
/// This binary's hash, which receipts name as the sandbox.
49 +
pub(crate) fn self_hash() -> &'static str {
50 +
    static HASH: OnceLock<String> = OnceLock::new();
51 +
    HASH.get_or_init(|| crate::home::hash_file(&self_exe()).unwrap_or_else(|_| "unknown".into()))
52 +
}
added crates/flow/src/update.rs +593 −0
1 +
//! Runner updates, the runner's side (docs/jetstream.md §8.2, #85): check
2 +
//! a new `flow` build, stage it beside the running one, swap it in keeping
3 +
//! the old one as `flow.prev`, and have the running runner leave with
4 +
//! notice so its unit starts the new build. The new build proves itself by
5 +
//! reaching the hub; three failed starts, or ten minutes without the hub,
6 +
//! put `flow.prev` back.
7 +
8 +
use anyhow::{bail, Context, Result};
9 +
use serde::{Deserialize, Serialize};
10 +
use std::path::{Path, PathBuf};
11 +
use std::time::Duration;
12 +
13 +
/// The staged build, beside the installed one.
14 +
pub const NEW: &str = "flow.new";
15 +
/// The build an update replaced, kept to go back to.
16 +
pub const PREV: &str = "flow.prev";
17 +
/// In the runner's home: the update not yet proven.
18 +
const PENDING: &str = "update.json";
19 +
/// In the runner's home: every update and rollback, a JSON line each.
20 +
const LOG: &str = "updates.log";
21 +
/// In the runner's home: the running runner's process id.
22 +
const PID: &str = "runner.pid";
23 +
/// In the runner's home: a new build is in place, leave for it. How a
24 +
/// runner is told on Windows, which has no SIGUSR1.
25 +
const REQUEST: &str = "update.request";
26 +
/// Starts of an unproven build after which the previous one comes back.
27 +
pub const MAX_STARTS: u32 = 3;
28 +
/// How long a new build has to reach the hub.
29 +
pub const PROVE_WITHIN: Duration = Duration::from_secs(10 * 60);
30 +
31 +
/// An update swapped in and not yet proven.
32 +
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
33 +
pub struct Pending {
34 +
    pub from: String,
35 +
    pub to: String,
36 +
    /// The installed binary the update replaced.
37 +
    pub exe: PathBuf,
38 +
    pub at_ms: u64,
39 +
    /// How long the running runner had to finish what it held.
40 +
    pub notice_secs: u64,
41 +
    /// Starts of the new build so far.
42 +
    pub starts: u32,
43 +
}
44 +
45 +
/// What [`install`] did, or would do.
46 +
#[derive(Debug, PartialEq, Eq)]
47 +
pub struct Installed {
48 +
    pub from: String,
49 +
    pub to: String,
50 +
    pub exe: PathBuf,
51 +
    /// The running runner told to leave, by process id.
52 +
    pub told: Option<u32>,
53 +
}
54 +
55 +
/// The version a `flow --version` line names: `flow 0.3.1 (build)` → `0.3.1`.
56 +
pub fn parse_version(line: &str) -> Option<String> {
57 +
    let v = line.trim().strip_prefix("flow ")?.split_whitespace().next()?;
58 +
    v.split('.').all(|p| !p.is_empty() && p.bytes().all(|b| b.is_ascii_digit())).then(|| v.to_string())
59 +
}
60 +
61 +
/// Whether version `a` is older than `b`, part by part.
62 +
pub fn older(a: &str, b: &str) -> bool {
63 +
    let parts = |v: &str| v.split('.').map(|p| p.parse::<u64>().unwrap_or(0)).collect::<Vec<_>>();
64 +
    parts(a) < parts(b)
65 +
}
66 +
67 +
/// `exe`'s sibling named `stem`, with `exe`'s extension (`flow.prev.exe`
68 +
/// beside `flow.exe` on Windows, `flow.prev` beside `flow` elsewhere).
69 +
pub fn beside(exe: &Path, stem: &str) -> PathBuf {
70 +
    match exe.extension() {
71 +
        Some(e) => exe.with_file_name(format!("{stem}.{}", e.to_string_lossy())),
72 +
        None => exe.with_file_name(stem),
73 +
    }
74 +
}
75 +
76 +
/// The PE machine number of this build's architecture (Windows).
77 +
fn pe_machine_here() -> Option<u16> {
78 +
    Some(match std::env::consts::ARCH {
79 +
        "x86_64" => 0x8664,
80 +
        "aarch64" => 0xAA64,
81 +
        _ => return None,
82 +
    })
83 +
}
84 +
85 +
/// Refuse a file that is not a PE executable for this machine (Windows).
86 +
pub fn check_pe(bytes: &[u8]) -> Result<()> {
87 +
    if bytes.len() < 0x40 || &bytes[..2] != b"MZ" {
88 +
        bail!("not a Windows executable");
89 +
    }
90 +
    let at = u32::from_le_bytes([bytes[0x3c], bytes[0x3d], bytes[0x3e], bytes[0x3f]]) as usize;
91 +
    if bytes.len() < at + 6 || &bytes[at..at + 4] != b"PE\0\0" {
92 +
        bail!("not a Windows executable (no PE header in its first {} bytes)", bytes.len());
93 +
    }
94 +
    let machine = u16::from_le_bytes([bytes[at + 4], bytes[at + 5]]);
95 +
    match pe_machine_here() {
96 +
        Some(m) if m == machine => Ok(()),
97 +
        Some(_) => bail!("a build for another machine (PE machine {machine:#x}); this runner is {}", std::env::consts::ARCH),
98 +
        None => bail!("updates are not supported on {}", std::env::consts::ARCH),
99 +
    }
100 +
}
101 +
102 +
/// The ELF machine number of this build's architecture.
103 +
fn machine_here() -> Option<u16> {
104 +
    Some(match std::env::consts::ARCH {
105 +
        "x86_64" => 62,
106 +
        "aarch64" => 183,
107 +
        "riscv64" => 243,
108 +
        _ => return None,
109 +
    })
110 +
}
111 +
112 +
/// Refuse a file that is not a 64-bit little-endian ELF executable for this
113 +
/// machine: a build for another target would not start, and the rollback
114 +
/// would only notice after three tries.
115 +
pub fn check_target(bytes: &[u8]) -> Result<()> {
116 +
    if bytes.len() < 20 || &bytes[..4] != b"\x7fELF" {
117 +
        bail!("not an ELF executable");
118 +
    }
119 +
    if bytes[4] != 2 || bytes[5] != 1 {
120 +
        bail!("not a 64-bit little-endian build");
121 +
    }
122 +
    let machine = u16::from_le_bytes([bytes[18], bytes[19]]);
123 +
    match machine_here() {
124 +
        Some(m) if m == machine => Ok(()),
125 +
        Some(_) => bail!("a build for another machine (ELF machine {machine}); this runner is {}", std::env::consts::ARCH),
126 +
        None => bail!("updates are not supported on {}", std::env::consts::ARCH),
127 +
    }
128 +
}
129 +
130 +
/// Run `file args…` with a time limit, returning its standard output.
131 +
fn run(file: &Path, args: &[&str], limit: Duration) -> Result<String> {
132 +
    let mut child = std::process::Command::new(file)
133 +
        .args(args)
134 +
        .stdin(std::process::Stdio::null())
135 +
        .stdout(std::process::Stdio::piped())
136 +
        .stderr(std::process::Stdio::piped())
137 +
        .spawn()
138 +
        .with_context(|| format!("starting {}", file.display()))?;
139 +
    let start = std::time::Instant::now();
140 +
    loop {
141 +
        if child.try_wait()?.is_some() {
142 +
            break;
143 +
        }
144 +
        if start.elapsed() > limit {
145 +
            let _ = child.kill();
146 +
            let _ = child.wait();
147 +
            bail!("`{} {}` did not finish within {} s", file.display(), args.join(" "), limit.as_secs());
148 +
        }
149 +
        std::thread::sleep(Duration::from_millis(50));
150 +
    }
151 +
    let out = child.wait_with_output()?;
152 +
    if !out.status.success() {
153 +
        bail!("`{} {}` failed ({}): {}", file.display(), args.join(" "), out.status, String::from_utf8_lossy(&out.stderr).trim());
154 +
    }
155 +
    Ok(String::from_utf8_lossy(&out.stdout).into_owned())
156 +
}
157 +
158 +
/// Copy `from` to `to` if it is `size` bytes with BLAKE3 `blake3`, as the
159 +
/// hub attests a build: a part file renamed into place, so `to` is either
160 +
/// the whole checked build or absent.
161 +
pub fn copy_checked(from: &Path, to: &Path, size: u64, blake3: &str) -> Result<()> {
162 +
    let bytes = std::fs::read(from).with_context(|| format!("reading {}", from.display()))?;
163 +
    let got = blake3::hash(&bytes).to_hex().to_string();
164 +
    if bytes.len() as u64 != size || got != blake3 {
165 +
        bail!("{} is not the build the hub attests ({} bytes, BLAKE3 {}…)", from.display(), bytes.len(), &got[..12]);
166 +
    }
167 +
    let part = to.with_extension("partial");
168 +
    std::fs::write(&part, &bytes).with_context(|| format!("writing {}", part.display()))?;
169 +
    std::fs::rename(&part, to)?;
170 +
    Ok(())
171 +
}
172 +
173 +
/// Check a build as the runner would run it: this machine's target, it
174 +
/// starts and names its version, and it finds the tiers this home offers.
175 +
/// Returns its version.
176 +
pub fn check(file: &Path, home: &Path) -> Result<String> {
177 +
    let mut head = vec![0u8; 4096];
178 +
    let n = std::io::Read::read(&mut std::fs::File::open(file).with_context(|| format!("opening {}", file.display()))?, &mut head)?;
179 +
    if cfg!(windows) {
180 +
        check_pe(&head[..n])?;
181 +
    } else {
182 +
        check_target(&head[..n])?;
183 +
    }
184 +
    let limit = Duration::from_secs(30);
185 +
    let line = run(file, &["--version"], limit)?;
186 +
    let version = parse_version(&line).with_context(|| format!("{} does not say it is flow: {:?}", file.display(), line.trim()))?;
187 +
    let home_s = home.to_string_lossy();
188 +
    let tiers = run(file, &["--home", &home_s, "tiers"], limit)?;
189 +
    if !tiers.lines().any(|l| l.ends_with(": available")) {
190 +
        bail!("flow {version} finds no tier it can run here:\n{}", tiers.trim());
191 +
    }
192 +
    Ok(version)
193 +
}
194 +
195 +
fn now_ms() -> u64 {
196 +
    std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map(|d| d.as_millis() as u64).unwrap_or(0)
197 +
}
198 +
199 +
fn log(home: &Path, event: &str, from: &str, to: &str, why: &str) {
200 +
    use std::io::Write;
201 +
    let line = serde_json::json!({"at_ms": now_ms(), "event": event, "from": from, "to": to, "why": why});
202 +
    if let Ok(mut f) = std::fs::OpenOptions::new().create(true).append(true).open(home.join(LOG)) {
203 +
        let _ = writeln!(f, "{line}");
204 +
    }
205 +
}
206 +
207 +
/// The update not yet proven, if any.
208 +
pub fn pending(home: &Path) -> Option<Pending> {
209 +
    serde_json::from_slice(&std::fs::read(home.join(PENDING)).ok()?).ok()
210 +
}
211 +
212 +
fn save(home: &Path, p: &Pending) -> Result<()> {
213 +
    let tmp = home.join(format!("{PENDING}.tmp"));
214 +
    std::fs::write(&tmp, serde_json::to_vec_pretty(p)?)?;
215 +
    std::fs::rename(&tmp, home.join(PENDING))?;
216 +
    Ok(())
217 +
}
218 +
219 +
/// Whether `p` is the program running now.
220 +
fn is_self(p: &Path) -> bool {
221 +
    let canon = |p: &Path| std::fs::canonicalize(p).ok();
222 +
    std::env::current_exe().ok().and_then(|me| canon(&me)).is_some_and(|me| Some(me) == canon(p))
223 +
}
224 +
225 +
/// Rename program `from` to `to`. On Windows a running program cannot
226 +
/// rename its own file (other programs can), so a copy of it, run as
227 +
/// `flow __rename`, does it then.
228 +
fn rename_exe(from: &Path, to: &Path) -> Result<()> {
229 +
    if cfg!(windows) && is_self(from) {
230 +
        let helper = beside(from, "flow.rename");
231 +
        std::fs::copy(from, &helper).with_context(|| format!("copying {} to rename it", from.display()))?;
232 +
        let out = std::process::Command::new(&helper).arg("__rename").arg(from).arg(to).output();
233 +
        let _ = std::fs::remove_file(&helper);
234 +
        let out = out.with_context(|| format!("starting {}", helper.display()))?;
235 +
        if !out.status.success() {
236 +
            bail!("{}", String::from_utf8_lossy(&out.stderr).trim());
237 +
        }
238 +
        return Ok(());
239 +
    }
240 +
    Ok(crate::portable::rename(from, to)?)
241 +
}
242 +
243 +
/// Put `staged` in place of `exe`, keeping `exe` as [`PREV`] beside it.
244 +
/// `exe` is never missing: it is copied aside, then replaced in one rename.
245 +
/// On Windows, where a running program cannot be replaced but can be
246 +
/// renamed, `exe` is renamed aside and `staged` into its place.
247 +
pub fn swap(exe: &Path, staged: &Path) -> Result<()> {
248 +
    let prev = beside(exe, PREV);
249 +
    if cfg!(windows) {
250 +
        let _ = std::fs::remove_file(&prev);
251 +
        rename_exe(exe, &prev).with_context(|| format!("keeping {} as {}", exe.display(), prev.display()))?;
252 +
        if let Err(e) = crate::portable::rename(staged, exe) {
253 +
            let _ = crate::portable::rename(&prev, exe);
254 +
            return Err(e).with_context(|| format!("putting {} in place of {}", staged.display(), exe.display()));
255 +
        }
256 +
        return Ok(());
257 +
    }
258 +
    let _ = std::fs::remove_file(&prev);
259 +
    if std::fs::hard_link(exe, &prev).is_err() {
260 +
        std::fs::copy(exe, &prev).with_context(|| format!("keeping {} as {}", exe.display(), prev.display()))?;
261 +
    }
262 +
    std::fs::rename(staged, exe).with_context(|| format!("putting {} in place of {}", staged.display(), exe.display()))?;
263 +
    Ok(())
264 +
}
265 +
266 +
/// Put [`PREV`] back in place of `exe`.
267 +
pub fn restore(exe: &Path) -> Result<()> {
268 +
    let prev = beside(exe, PREV);
269 +
    if !prev.is_file() {
270 +
        bail!("{} is not there to go back to", prev.display());
271 +
    }
272 +
    if cfg!(windows) {
273 +
        // The failed build may be the one running: rename it aside first.
274 +
        let failed = beside(exe, "flow.failed");
275 +
        let _ = std::fs::remove_file(&failed);
276 +
        rename_exe(exe, &failed).with_context(|| format!("moving {} aside", exe.display()))?;
277 +
        return crate::portable::rename(&prev, exe).with_context(|| format!("putting {} back", prev.display()));
278 +
    }
279 +
    std::fs::rename(&prev, exe).with_context(|| format!("putting {} back", prev.display()))?;
280 +
    Ok(())
281 +
}
282 +
283 +
/// Check `file`, stage it beside `exe`, and unless `dry_run` swap it in and
284 +
/// tell the running runner (if any) to leave within `notice`, finishing
285 +
/// what fits and handing back the rest, so its unit starts the new build.
286 +
pub fn install(home: &crate::home::Home, exe: &Path, file: &Path, notice: Duration, dry_run: bool, allow_older: bool) -> Result<Installed> {
287 +
    if home.config.runner.as_ref().and_then(|r| r.update.as_ref()).is_some_and(|u| !u.allow) {
288 +
        bail!("flow.toml says [runner.update] allow = false: this runner takes no updates");
289 +
    }
290 +
    if let Some(p) = pending(&home.root) {
291 +
        bail!("the update from {} to {} is not proven yet; wait for it (or for its rollback)", p.from, p.to);
292 +
    }
293 +
    let from = env!("CARGO_PKG_VERSION").to_string();
294 +
    let staged = beside(exe, NEW);
295 +
    let _ = std::fs::remove_file(&staged);
296 +
    std::fs::copy(file, &staged).with_context(|| format!("staging {} as {}", file.display(), staged.display()))?;
297 +
    crate::portable::set_mode(&staged, 0o755)?;
298 +
    let checked = check(&staged, &home.root).and_then(|to| {
299 +
        if older(&to, &from) && !allow_older {
300 +
            bail!("flow {to} is older than this runner's {from} (--allow-older to go back)");
301 +
        }
302 +
        Ok(to)
303 +
    });
304 +
    let to = match checked {
305 +
        Ok(v) => v,
306 +
        Err(e) => {
307 +
            let _ = std::fs::remove_file(&staged);
308 +
            return Err(e);
309 +
        }
310 +
    };
311 +
    if dry_run {
312 +
        let _ = std::fs::remove_file(&staged);
313 +
        return Ok(Installed { from, to, exe: exe.to_path_buf(), told: None });
314 +
    }
315 +
    if let Err(e) = swap(exe, &staged) {
316 +
        let _ = std::fs::remove_file(&staged);
317 +
        return Err(e);
318 +
    }
319 +
    let p =
320 +
        Pending { from: from.clone(), to: to.clone(), exe: exe.to_path_buf(), at_ms: now_ms(), notice_secs: notice.as_secs(), starts: 0 };
321 +
    save(&home.root, &p)?;
322 +
    log(&home.root, "installed", &from, &to, "flow runner update --from");
323 +
    let told = running(&home.root).filter(|pid| cfg!(not(unix)) || tell(*pid));
324 +
    Ok(Installed { from, to, exe: exe.to_path_buf(), told })
325 +
}
326 +
327 +
/// Ask runner `pid` to leave for the new build (SIGUSR1).
328 +
#[cfg(unix)]
329 +
fn tell(pid: u32) -> bool {
330 +
    unsafe { libc::kill(pid as libc::pid_t, libc::SIGUSR1) == 0 }
331 +
}
332 +
333 +
/// On Windows: a request in the runner's home, which its loop reads.
334 +
#[cfg(not(unix))]
335 +
fn tell(_pid: u32) -> bool {
336 +
    false
337 +
}
338 +
339 +
/// In the runner's home: drain (claim nothing more, finish what it holds,
340 +
/// then stop), as SIGTERM asks on Linux: `flow runner drain`, for Windows.
341 +
const DRAIN: &str = "drain.request";
342 +
343 +
/// Ask the runner of `home` to drain; whether one is running to ask.
344 +
pub fn ask_to_drain(home: &Path) -> std::io::Result<bool> {
345 +
    std::fs::write(home.join(DRAIN), "")?;
346 +
    Ok(home.join(PID).is_file())
347 +
}
348 +
349 +
/// Whether the runner has been asked, by file, to drain (once).
350 +
pub fn drain_requested(home: &Path) -> bool {
351 +
    std::fs::remove_file(home.join(DRAIN)).is_ok()
352 +
}
353 +
354 +
/// Whether the runner has been asked, by file, to leave for a new build
355 +
/// (and forget the asking).
356 +
pub fn requested(home: &Path) -> bool {
357 +
    std::fs::remove_file(home.join(REQUEST)).is_ok()
358 +
}
359 +
360 +
/// The running runner's process id, from its pid file, if that process is
361 +
/// still a `flow runner`: a stale file's pid may be another program's now,
362 +
/// and SIGUSR1 would end it. Builds before #85 write no pid file (and do
363 +
/// not listen for SIGUSR1), so they are never told.
364 +
#[cfg(unix)]
365 +
fn running(home: &Path) -> Option<u32> {
366 +
    let pid: u32 = std::fs::read_to_string(home.join(PID)).ok()?.trim().parse().ok()?;
367 +
    let cmdline = std::fs::read(format!("/proc/{pid}/cmdline")).ok()?;
368 +
    is_runner(&cmdline).then_some(pid)
369 +
}
370 +
371 +
/// On Windows: a runner of this home has a pid file, and is asked by a
372 +
/// request file it reads (a stale one is cleared when a runner starts).
373 +
#[cfg(not(unix))]
374 +
fn running(home: &Path) -> Option<u32> {
375 +
    let pid: u32 = std::fs::read_to_string(home.join(PID)).ok()?.trim().parse().ok()?;
376 +
    std::fs::write(home.join(REQUEST), pid.to_string()).ok()?;
377 +
    Some(pid)
378 +
}
379 +
380 +
/// Whether a NUL-separated command line is `flow … runner` with no subcommand.
381 +
#[cfg_attr(not(unix), allow(dead_code))]
382 +
fn is_runner(cmdline: &[u8]) -> bool {
383 +
    let args: Vec<&[u8]> = cmdline.split(|b| *b == 0).filter(|a| !a.is_empty()).collect();
384 +
    let Some((argv0, rest)) = args.split_first() else { return false };
385 +
    let name = argv0.rsplit(|b| *b == b'/').next().unwrap_or(argv0);
386 +
    name.starts_with(b"flow") && rest.last() == Some(&&b"runner"[..])
387 +
}
388 +
389 +
/// Note this process as the home's runner, for `flow runner update` to tell.
390 +
pub fn note_runner(home: &Path) {
391 +
    let _ = std::fs::remove_file(home.join(REQUEST));
392 +
    let _ = std::fs::remove_file(home.join(DRAIN));
393 +
    let _ = std::fs::write(home.join(PID), std::process::id().to_string());
394 +
}
395 +
396 +
/// The runner stopped: forget its pid.
397 +
pub fn forget_runner(home: &Path) {
398 +
    if std::fs::read_to_string(home.join(PID)).is_ok_and(|p| p.trim() == std::process::id().to_string()) {
399 +
        let _ = std::fs::remove_file(home.join(PID));
400 +
    }
401 +
}
402 +
403 +
/// What a runner starting up should do about an update.
404 +
#[derive(Debug, PartialEq, Eq)]
405 +
pub enum Start {
406 +
    /// No update waits to be proven.
407 +
    Settled,
408 +
    /// The new build is on trial: prove it within [`PROVE_WITHIN`].
409 +
    Trial(Pending),
410 +
    /// It failed too often; the previous build is back and should be run.
411 +
    RolledBack(Pending),
412 +
}
413 +
414 +
/// Count this start of an unproven build; after [`MAX_STARTS`], put the
415 +
/// previous build back.
416 +
pub fn on_start(home: &Path) -> Result<Start> {
417 +
    let Some(mut p) = pending(home) else { return Ok(Start::Settled) };
418 +
    p.starts += 1;
419 +
    if p.starts > MAX_STARTS {
420 +
        roll_back(home, &p, &format!("{MAX_STARTS} starts without reaching the hub"))?;
421 +
        return Ok(Start::RolledBack(p));
422 +
    }
423 +
    save(home, &p)?;
424 +
    Ok(Start::Trial(p))
425 +
}
426 +
427 +
/// In the runner's home: what to tell the hub once it answers.
428 +
const NOTE: &str = "update.note";
429 +
430 +
/// Keep `note` to tell the hub (a proof, a rollback) once it answers.
431 +
fn note(home: &Path, note: &str) {
432 +
    let _ = std::fs::write(home.join(NOTE), note);
433 +
}
434 +
435 +
/// What there is to tell the hub, once.
436 +
pub fn take_note(home: &Path) -> Option<String> {
437 +
    let n = std::fs::read_to_string(home.join(NOTE)).ok()?;
438 +
    let _ = std::fs::remove_file(home.join(NOTE));
439 +
    Some(n).filter(|n| !n.trim().is_empty())
440 +
}
441 +
442 +
/// The new build reached the hub: it stays.
443 +
pub fn proven(home: &Path) {
444 +
    if let Some(p) = pending(home) {
445 +
        let _ = std::fs::remove_file(home.join(PENDING));
446 +
        log(home, "proven", &p.from, &p.to, "reached the hub");
447 +
        note(home, &format!("runs flow {} (was {}); proven", p.to, p.from));
448 +
        eprintln!("flow runner: flow {} proven (it reached the hub); flow {} stays as {}", p.to, p.from, beside(&p.exe, PREV).display());
449 +
    }
450 +
}
451 +
452 +
/// Put the previous build back and forget the update.
453 +
pub fn roll_back(home: &Path, p: &Pending, why: &str) -> Result<()> {
454 +
    restore(&p.exe)?;
455 +
    let _ = std::fs::remove_file(home.join(PENDING));
456 +
    log(home, "rolled back", &p.to, &p.from, why);
457 +
    note(home, &format!("rolled back from flow {} to {}: {why}", p.to, p.from));
458 +
    eprintln!("flow runner: rolled back from flow {} to {} ({why})", p.to, p.from);
459 +
    Ok(())
460 +
}
461 +
462 +
#[cfg(test)]
463 +
mod tests {
464 +
465 +
    #[test]
466 +
    fn a_build_from_the_swarm_is_copied_only_when_it_is_the_one_attested() {
467 +
        let dir = tempfile::tempdir().unwrap();
468 +
        let (from, to) = (dir.path().join("cas.ext4"), dir.path().join("flow-0.5.2"));
469 +
        std::fs::write(&from, b"flow build").unwrap();
470 +
        let hash = blake3::hash(b"flow build").to_hex().to_string();
471 +
        assert!(copy_checked(&from, &to, 9, &hash).is_err(), "the wrong size");
472 +
        assert!(copy_checked(&from, &to, 10, &"0".repeat(64)).is_err(), "the wrong hash");
473 +
        assert!(!to.exists() && !to.with_extension("partial").exists(), "nothing left behind");
474 +
        copy_checked(&from, &to, 10, &hash).unwrap();
475 +
        assert_eq!(std::fs::read(&to).unwrap(), b"flow build");
476 +
    }
477 +
    use super::*;
478 +
479 +
    #[test]
480 +
    fn versions_are_read_and_compared() {
481 +
        assert_eq!(parse_version("flow 0.3.1 (local build)\n").as_deref(), Some("0.3.1"));
482 +
        assert_eq!(parse_version("flow 0.10.0"), Some("0.10.0".into()));
483 +
        assert_eq!(parse_version("oxy 0.6.0"), None);
484 +
        assert_eq!(parse_version("flow banana"), None);
485 +
        assert!(older("0.3.1", "0.10.0"), "by number, not by text");
486 +
        assert!(!older("0.3.1", "0.3.1"));
487 +
        assert!(older("0.3", "0.3.1"));
488 +
    }
489 +
490 +
    #[test]
491 +
    fn only_a_build_for_this_machine_passes() {
492 +
        let me = std::fs::read(std::env::current_exe().unwrap()).unwrap();
493 +
        check_target(&me[..64]).unwrap();
494 +
        assert!(check_target(b"#!/bin/sh\necho flow 9.9.9\n").unwrap_err().to_string().contains("not an ELF"));
495 +
        let mut other = me[..64].to_vec();
496 +
        other[18..20].copy_from_slice(&(if std::env::consts::ARCH == "riscv64" { 62u16 } else { 243u16 }).to_le_bytes());
497 +
        assert!(check_target(&other).unwrap_err().to_string().contains("another machine"));
498 +
    }
499 +
500 +
    #[test]
501 +
    fn a_swap_keeps_the_old_build_and_three_failed_starts_bring_it_back() {
502 +
        let dir = tempfile::tempdir().unwrap();
503 +
        let (bin, home) = (dir.path().join("bin"), dir.path().join("home"));
504 +
        std::fs::create_dir_all(&bin).unwrap();
505 +
        std::fs::create_dir_all(&home).unwrap();
506 +
        let exe = bin.join("flow");
507 +
        std::fs::write(&exe, "old").unwrap();
508 +
        std::fs::write(bin.join(NEW), "new").unwrap();
509 +
        swap(&exe, &bin.join(NEW)).unwrap();
510 +
        assert_eq!(std::fs::read_to_string(&exe).unwrap(), "new");
511 +
        assert_eq!(std::fs::read_to_string(bin.join(PREV)).unwrap(), "old");
512 +
        assert!(!bin.join(NEW).exists());
513 +
514 +
        assert_eq!(on_start(&home).unwrap(), Start::Settled, "no update, nothing to do");
515 +
        let p = Pending { from: "0.3.1".into(), to: "0.3.2".into(), exe: exe.clone(), at_ms: 1, notice_secs: 600, starts: 0 };
516 +
        save(&home, &p).unwrap();
517 +
        for n in 1..=MAX_STARTS {
518 +
            assert_eq!(on_start(&home).unwrap(), Start::Trial(Pending { starts: n, ..p.clone() }));
519 +
        }
520 +
        assert!(matches!(on_start(&home).unwrap(), Start::RolledBack(_)));
521 +
        assert_eq!(std::fs::read_to_string(&exe).unwrap(), "old", "the previous build is back");
522 +
        assert!(pending(&home).is_none());
523 +
        let log = std::fs::read_to_string(home.join(LOG)).unwrap();
524 +
        assert!(log.contains("rolled back") && log.contains("0.3.2"), "{log}");
525 +
    }
526 +
527 +
    #[test]
528 +
    fn a_windows_build_is_checked_as_pe_and_named_beside_with_its_extension() {
529 +
        let mut pe = vec![0u8; 0x100];
530 +
        pe[..2].copy_from_slice(b"MZ");
531 +
        pe[0x3c..0x40].copy_from_slice(&0x80u32.to_le_bytes());
532 +
        pe[0x80..0x84].copy_from_slice(b"PE\0\0");
533 +
        pe[0x84..0x86].copy_from_slice(&0x8664u16.to_le_bytes());
534 +
        if std::env::consts::ARCH == "x86_64" {
535 +
            check_pe(&pe).unwrap();
536 +
        }
537 +
        pe[0x84..0x86].copy_from_slice(&0x01c4u16.to_le_bytes());
538 +
        assert!(check_pe(&pe).is_err(), "another machine");
539 +
        assert!(check_pe(b"\x7fELF and more bytes than enough for a header, really, quite a few more")
540 +
            .unwrap_err()
541 +
            .to_string()
542 +
            .contains("not a Windows"));
543 +
        assert_eq!(beside(Path::new("C:/flow/flow.exe"), PREV), Path::new("C:/flow/flow.prev.exe"));
544 +
        assert_eq!(beside(Path::new("/usr/local/bin/flow"), NEW), Path::new("/usr/local/bin/flow.new"));
545 +
    }
546 +
547 +
    #[test]
548 +
    fn a_drain_is_asked_by_file_once() {
549 +
        let dir = tempfile::tempdir().unwrap();
550 +
        assert!(!ask_to_drain(dir.path()).unwrap(), "no runner running");
551 +
        assert!(drain_requested(dir.path()) && !drain_requested(dir.path()));
552 +
        note_runner(dir.path());
553 +
        assert!(ask_to_drain(dir.path()).unwrap());
554 +
        note_runner(dir.path());
555 +
        assert!(!drain_requested(dir.path()), "a starting runner clears a stale request");
556 +
    }
557 +
558 +
    #[test]
559 +
    fn a_request_file_asks_once() {
560 +
        let dir = tempfile::tempdir().unwrap();
561 +
        assert!(!requested(dir.path()));
562 +
        std::fs::write(dir.path().join(REQUEST), "1").unwrap();
563 +
        assert!(requested(dir.path()) && !requested(dir.path()), "read once");
564 +
        std::fs::write(dir.path().join(REQUEST), "1").unwrap();
565 +
        note_runner(dir.path());
566 +
        assert!(!requested(dir.path()), "a starting runner clears a stale request");
567 +
    }
568 +
569 +
    #[test]
570 +
    fn only_a_running_runner_is_told() {
571 +
        assert!(is_runner(b"/usr/local/bin/flow\0runner\0"));
572 +
        assert!(is_runner(b"flow\0--home\0/srv/flow\0runner\0"));
573 +
        assert!(!is_runner(b"flow\0runner\0update\0--from\0x\0"), "not the update command itself");
574 +
        assert!(!is_runner(b"/usr/bin/vim\0runner\0"));
575 +
        assert!(!is_runner(b""));
576 +
        let dir = tempfile::tempdir().unwrap();
577 +
        std::fs::write(dir.path().join(PID), std::process::id().to_string()).unwrap();
578 +
        assert_eq!(running(dir.path()), None, "this test is not a flow runner");
579 +
    }
580 +
581 +
    #[test]
582 +
    fn a_proven_build_stays() {
583 +
        let dir = tempfile::tempdir().unwrap();
584 +
        let exe = dir.path().join("flow");
585 +
        std::fs::write(&exe, "new").unwrap();
586 +
        let p = Pending { from: "0.3.1".into(), to: "0.3.2".into(), exe: exe.clone(), at_ms: 1, notice_secs: 0, starts: 1 };
587 +
        save(dir.path(), &p).unwrap();
588 +
        proven(dir.path());
589 +
        assert!(pending(dir.path()).is_none());
590 +
        assert_eq!(on_start(dir.path()).unwrap(), Start::Settled);
591 +
        assert!(std::fs::read_to_string(dir.path().join(LOG)).unwrap().contains("proven"));
592 +
    }
593 +
}
added crates/flow/src/wasm.rs +598 −0
1 +
//! The WASM tier (spec §4.1, oxygenflow#75): a job's `exec` names a WASI
2 +
//! preview 1 module, or a glob of modules run one after another, which
3 +
//! wasmtime runs with the job's workspace at `/src` and nothing else: no
4 +
//! network, no other file of the runner. Every runner can offer it, and it
5 +
//! is the tier phones and browsers run (docs/handover.md).
6 +
//!
7 +
//! As in the syscall tier the module runs in a child of the runner
8 +
//! (`flow __wasm SPEC`), so the runner streams its output as it comes,
9 +
//! stops it at the time limit or when the run is cancelled, and measures
10 +
//! its memory the same way; wasmtime itself also holds it to the sandbox's
11 +
//! memory limit.
12 +
13 +
use crate::microvm::{Outcome, Sink};
14 +
use crate::syscall::{lines, self_hash, FAILED};
15 +
use anyhow::{bail, Context, Result};
16 +
use flow_core::api::Record;
17 +
use flow_core::{Environment, Job, MountSource, Sandbox};
18 +
use serde::{Deserialize, Serialize};
19 +
use std::path::{Path, PathBuf};
20 +
use std::sync::mpsc;
21 +
use std::time::{Duration, Instant};
22 +
23 +
/// The runtime, named in every receipt this tier writes.
24 +
pub const WASMTIME: &str = "49.0.4";
25 +
26 +
/// Whether this build has the tier: a build for a system wasmtime does not
27 +
/// run on (Haiku, oxygenflow#91) leaves it out with the `wasm` feature, and
28 +
/// its runner does not offer it.
29 +
pub const AVAILABLE: bool = cfg!(feature = "wasm");
30 +
31 +
/// What a build without the tier says when asked to run a job in it.
32 +
const WITHOUT: &str = "this flow was built without the WASM tier (the `wasm` feature): it runs on systems wasmtime supports";
33 +
34 +
/// A line the helper writes to say something about the job itself (which
35 +
/// module starts), shown as the runner's own note rather than the job's.
36 +
const NOTE: &str = "\u{1e}N ";
37 +
38 +
/// The exit code for a module that trapped (crashed) rather than exiting.
39 +
#[cfg(feature = "wasm")]
40 +
const TRAPPED: i32 = 134;
41 +
42 +
/// What the helper runs, written by the runner for `flow __wasm`.
43 +
#[derive(Serialize, Deserialize)]
44 +
struct Spec {
45 +
    /// The job's files, preopened as `/src` (and as the current directory).
46 +
    workspace: PathBuf,
47 +
    /// The modules to run in turn, relative to the workspace.
48 +
    modules: Vec<String>,
49 +
    /// The arguments after the module.
50 +
    args: Vec<String>,
51 +
    env: Vec<(String, String)>,
52 +
    memory_mib: u32,
53 +
    /// The job's mounts: a directory of the runner and where the module sees it.
54 +
    preopens: Vec<(PathBuf, String)>,
55 +
}
56 +
57 +
/// The receipt environment of a job in this tier: the runtime and its
58 +
/// version, this runner's binary, the memory allowed, and every module run
59 +
/// with its hash, so the receipt says exactly what code ran.
60 +
pub fn environment(sandbox: &Sandbox, job: &Job, modules: &[(String, String)]) -> Environment {
61 +
    Environment {
62 +
        runtime: "wasm".into(),
63 +
        parts: vec![
64 +
            ("wasmtime".into(), WASMTIME.into()),
65 +
            ("flow".into(), self_hash().to_string()),
66 +
            ("memory_mib".into(), sandbox.memory_mib.to_string()),
67 +
        ]
68 +
        .into_iter()
69 +
        .chain(modules.iter().map(|(path, hash)| (format!("module {path}"), hash.clone())))
70 +
        .collect(),
71 +
        rootfs: String::new(),
72 +
        argv: job.exec.clone(),
73 +
        env: job.env.iter().map(|(k, v)| (k.clone(), v.clone())).collect(),
74 +
    }
75 +
}
76 +
77 +
/// The modules `program` names in `workspace`: one file, or the files a
78 +
/// glob matches (`*` in its last part only), sorted. A path may start with
79 +
/// `/src/`, where the module sees the workspace.
80 +
pub fn modules(workspace: &Path, program: &str) -> Result<Vec<String>> {
81 +
    let rel = program.strip_prefix("/src/").unwrap_or(program).trim_start_matches("./");
82 +
    if rel.is_empty() || rel.starts_with('/') || rel.split('/').any(|p| p == "..") {
83 +
        bail!("module {program:?}: give a path inside the job's files, such as target/wasm32-wasip1/debug/app.wasm");
84 +
    }
85 +
    let (dir, last) = rel.rsplit_once('/').unwrap_or(("", rel));
86 +
    if dir.contains('*') {
87 +
        bail!("module {program:?}: only the file name may hold `*`");
88 +
    }
89 +
    if !last.contains('*') {
90 +
        if !workspace.join(rel).is_file() {
91 +
            bail!("module {rel} is not among the job's files (its scope or an earlier job's artifact)");
92 +
        }
93 +
        return Ok(vec![rel.to_string()]);
94 +
    }
95 +
    let mut found: Vec<String> = std::fs::read_dir(workspace.join(dir))
96 +
        .with_context(|| format!("module {program:?}: the directory {dir} is not among the job's files"))?
97 +
        .filter_map(|e| e.ok())
98 +
        .filter(|e| e.file_type().is_ok_and(|t| t.is_file()))
99 +
        .filter_map(|e| e.file_name().into_string().ok())
100 +
        .filter(|name| wildcard(last, name))
101 +
        .map(|name| if dir.is_empty() { name } else { format!("{dir}/{name}") })
102 +
        .collect();
103 +
    found.sort();
104 +
    if found.is_empty() {
105 +
        bail!("no module matches {program:?}");
106 +
    }
107 +
    Ok(found)
108 +
}
109 +
110 +
/// Whether `name` matches `pattern`, where `*` stands for any run of characters.
111 +
fn wildcard(pattern: &str, name: &str) -> bool {
112 +
    let parts: Vec<&str> = pattern.split('*').collect();
113 +
    let (first, last) = (parts[0], parts[parts.len() - 1]);
114 +
    if parts.len() == 1 {
115 +
        return pattern == name;
116 +
    }
117 +
    if !name.starts_with(first) || !name[first.len()..].ends_with(last) || name.len() < first.len() + last.len() {
118 +
        return false;
119 +
    }
120 +
    let mut rest = &name[first.len()..name.len() - last.len()];
121 +
    for middle in &parts[1..parts.len() - 1] {
122 +
        match rest.find(middle) {
123 +
            Some(i) => rest = &rest[i + middle.len()..],
124 +
            None => return false,
125 +
        }
126 +
    }
127 +
    true
128 +
}
129 +
130 +
/// Run `job` in this tier with its files in `workspace` (as the syscall
131 +
/// tier lays them out); `scratch` holds the helper's spec and the job's
132 +
/// mounts.
133 +
pub fn run(sandbox: &Sandbox, job: &Job, workspace: &Path, scratch: &Path, timeout: Duration, sink: &mut dyn Sink) -> Result<Outcome> {
134 +
    if !AVAILABLE {
135 +
        bail!("job {}: {WITHOUT}", job.name);
136 +
    }
137 +
    if !job.secrets.is_empty() || !job.identities.is_empty() {
138 +
        bail!("job {}: the WASM tier takes no secrets or identities yet; run it in a microvm or syscall sandbox", job.name);
139 +
    }
140 +
    let Some(program) = job.exec.first() else { bail!("job {}: exec names no module", job.name) };
141 +
    let modules = modules(workspace, program)?;
142 +
    let hashed: Vec<(String, String)> =
143 +
        modules.iter().map(|m| Ok((m.clone(), crate::home::hash_file(&workspace.join(m))?))).collect::<Result<_>>()?;
144 +
    let environment = environment(sandbox, job, &hashed);
145 +
    let mut preopens = Vec::new();
146 +
    let mut disks = 0;
147 +
    for (i, m) in job.mounts.iter().enumerate() {
148 +
        let dir = match m.source {
149 +
            MountSource::Disk => {
150 +
                disks += 1;
151 +
                crate::syscall::disk_dir(scratch, disks - 1)
152 +
            }
153 +
            MountSource::Tmpfs => scratch.join(format!("tmpfs-{i}")),
154 +
        };
155 +
        std::fs::create_dir_all(&dir)?;
156 +
        preopens.push((dir, m.target.clone()));
157 +
    }
158 +
    let spec = Spec {
159 +
        workspace: workspace.to_path_buf(),
160 +
        modules,
161 +
        args: job.exec[1..].to_vec(),
162 +
        env: job.env.iter().map(|(k, v)| (k.clone(), v.clone())).collect(),
163 +
        memory_mib: sandbox.memory_mib,
164 +
        preopens,
165 +
    };
166 +
    if in_process() {
167 +
        return run_in_thread(&spec, environment, timeout, sink);
168 +
    }
169 +
    let spec_path = scratch.join("wasm.json");
170 +
    std::fs::write(&spec_path, serde_json::to_vec(&spec)?)?;
171 +
172 +
    let started = Instant::now();
173 +
    let mut child = std::process::Command::new(crate::tier::self_exe())
174 +
        .arg("__wasm")
175 +
        .arg(&spec_path)
176 +
        .env_clear()
177 +
        .current_dir(scratch)
178 +
        .stdin(std::process::Stdio::null())
179 +
        .stdout(std::process::Stdio::piped())
180 +
        .stderr(std::process::Stdio::piped())
181 +
        .spawn()
182 +
        .context("start the WASM helper")?;
183 +
    let (tx, rx) = mpsc::channel::<(bool, Vec<u8>)>();
184 +
    let out = child.stdout.take().unwrap();
185 +
    let err = child.stderr.take().unwrap();
186 +
    let tx2 = tx.clone();
187 +
    let readers = [std::thread::spawn(move || lines(out, true, tx)), std::thread::spawn(move || lines(err, false, tx2))];
188 +
    let t = |s: &Instant| s.elapsed().as_millis() as u64;
189 +
    let (mut timed_out, mut cancelled, mut failed_to_start) = (false, false, false);
190 +
    let mut peak_memory_mib = None;
191 +
    let deadline = started + timeout;
192 +
    loop {
193 +
        if let Some(m) = crate::facts::peak_rss_mib(child.id()) {
194 +
            peak_memory_mib = Some(m);
195 +
        }
196 +
        match rx.recv_timeout(Duration::from_millis(200)) {
197 +
            Ok((stdout, line)) => {
198 +
                let text = String::from_utf8_lossy(&line).into_owned();
199 +
                let rec = |stream: &str, text: &str| Record { t: t(&started), stream: stream.into(), text: text.into() };
200 +
                if let Some(note) = text.strip_prefix(NOTE).filter(|_| !stdout) {
201 +
                    sink.record(rec("flow", note));
202 +
                } else if let Some(why) = text.strip_prefix(FAILED).filter(|_| !stdout) {
203 +
                    failed_to_start = true;
204 +
                    sink.record(rec("flow", &format!("the WASM tier could not run the job: {why}")));
205 +
                } else {
206 +
                    sink.record(rec(if stdout { "out" } else { "err" }, &text));
207 +
                }
208 +
            }
209 +
            Err(mpsc::RecvTimeoutError::Timeout) => {}
210 +
            Err(mpsc::RecvTimeoutError::Disconnected) => break,
211 +
        }
212 +
        if !timed_out && !cancelled && sink.cancelled() {
213 +
            cancelled = true;
214 +
            let _ = child.kill();
215 +
            sink.record(Record { t: t(&started), stream: "flow".into(), text: "stopped: the run was cancelled".into() });
216 +
        }
217 +
        if Instant::now() >= deadline && !timed_out && !cancelled {
218 +
            timed_out = true;
219 +
            let _ = child.kill();
220 +
            sink.record(Record {
221 +
                t: t(&started),
222 +
                stream: "flow".into(),
223 +
                text: format!("stopped at the time limit of {} s", timeout.as_secs()),
224 +
            });
225 +
        }
226 +
    }
227 +
    let status = child.wait()?;
228 +
    for r in readers {
229 +
        let _ = r.join();
230 +
    }
231 +
    let _ = std::fs::remove_file(&spec_path);
232 +
    let exit_code = status.code();
233 +
    if exit_code.is_none() && !timed_out && !cancelled && !failed_to_start {
234 +
        sink.record(Record {
235 +
            t: t(&started),
236 +
            stream: "flow".into(),
237 +
            text: format!("the WASM helper ended without an exit code ({status})"),
238 +
        });
239 +
    }
240 +
    Ok(Outcome {
241 +
        exit_code: if timed_out || cancelled { None } else { exit_code },
242 +
        duration_ms: t(&started),
243 +
        environment,
244 +
        timed_out,
245 +
        cancelled,
246 +
        peak_memory_mib,
247 +
        pruned: Vec::new(),
248 +
    })
249 +
}
250 +
251 +
/// Whether modules run in a thread of this process rather than in a child
252 +
/// `flow __wasm`: on Android, where the runner is a library inside an app
253 +
/// that cannot start itself as a command (oxygenhub-android#8), and
254 +
/// anywhere `FLOW_WASM_IN_PROCESS` is set (how the tests check this way).
255 +
pub fn in_process() -> bool {
256 +
    cfg!(target_os = "android") || std::env::var_os("FLOW_WASM_IN_PROCESS").is_some()
257 +
}
258 +
259 +
#[cfg(feature = "wasm")]
260 +
/// The most of a module's output kept before it is read, per stream.
261 +
const PIPE_BYTES: usize = 64 << 20;
262 +
263 +
#[cfg(feature = "wasm")]
264 +
/// Run the spec's modules in a thread of this process, streaming their
265 +
/// output as it comes: the pipes are read every 100 ms, and a time limit
266 +
/// or a cancelled run interrupts the module through wasmtime's epoch.
267 +
fn run_in_thread(spec: &Spec, environment: Environment, timeout: Duration, sink: &mut dyn Sink) -> Result<Outcome> {
268 +
    use wasmtime_wasi::p2::pipe::MemoryOutputPipe;
269 +
    let mut config = wasmtime::Config::new();
270 +
    config.epoch_interruption(true);
271 +
    let engine = ww(wasmtime::Engine::new(&config), || "start wasmtime".into())?;
272 +
    let (stdout, stderr) = (MemoryOutputPipe::new(PIPE_BYTES), MemoryOutputPipe::new(PIPE_BYTES));
273 +
    let (notes_tx, notes) = mpsc::channel::<String>();
274 +
    let io = Io::Pipes { stdout: stdout.clone(), stderr: stderr.clone(), notes: notes_tx };
275 +
    let started = Instant::now();
276 +
    let t = |s: &Instant| s.elapsed().as_millis() as u64;
277 +
    let deadline = started + timeout;
278 +
    let (mut timed_out, mut cancelled) = (false, false);
279 +
    // Bytes of each stream already passed on, and the part line not ended yet.
280 +
    let mut said = [(0usize, Vec::<u8>::new()), (0usize, Vec::<u8>::new())];
281 +
    let mut pass_on = |sink: &mut dyn Sink, last: bool| {
282 +
        for (i, pipe) in [&stdout, &stderr].into_iter().enumerate() {
283 +
            let all = pipe.contents();
284 +
            let (offset, partial) = &mut said[i];
285 +
            partial.extend_from_slice(&all[*offset..]);
286 +
            *offset = all.len();
287 +
            while let Some(n) = partial.iter().position(|b| *b == b'\n') {
288 +
                let line: Vec<u8> = partial.drain(..=n).collect();
289 +
                let text = String::from_utf8_lossy(&line[..line.len() - 1]).into_owned();
290 +
                sink.record(Record { t: t(&started), stream: if i == 0 { "out" } else { "err" }.into(), text });
291 +
            }
292 +
            if last && !partial.is_empty() {
293 +
                let text = String::from_utf8_lossy(partial).into_owned();
294 +
                partial.clear();
295 +
                sink.record(Record { t: t(&started), stream: if i == 0 { "out" } else { "err" }.into(), text });
296 +
            }
297 +
        }
298 +
        while let Ok(note) = notes.try_recv() {
299 +
            sink.record(Record { t: t(&started), stream: "flow".into(), text: note });
300 +
        }
301 +
    };
302 +
    let ran = std::thread::scope(|scope| {
303 +
        let worker = scope.spawn(|| run_modules(&engine, spec, &io));
304 +
        while !worker.is_finished() {
305 +
            std::thread::sleep(Duration::from_millis(100));
306 +
            pass_on(sink, false);
307 +
            // Once stopped, keep moving the epoch: a store made after the
308 +
            // first increment (its module still compiling when the time
309 +
            // limit or the cancel came) sets its deadline one epoch past
310 +
            // that, and a single increment would never reach it (#93).
311 +
            if timed_out || cancelled {
312 +
                engine.increment_epoch();
313 +
            }
314 +
            if !timed_out && !cancelled && sink.cancelled() {
315 +
                cancelled = true;
316 +
                engine.increment_epoch();
317 +
                sink.record(Record { t: t(&started), stream: "flow".into(), text: "stopped: the run was cancelled".into() });
318 +
            }
319 +
            if Instant::now() >= deadline && !timed_out && !cancelled {
320 +
                timed_out = true;
321 +
                engine.increment_epoch();
322 +
                sink.record(Record {
323 +
                    t: t(&started),
324 +
                    stream: "flow".into(),
325 +
                    text: format!("stopped at the time limit of {} s", timeout.as_secs()),
326 +
                });
327 +
            }
328 +
        }
329 +
        worker.join().unwrap_or_else(|_| Err(anyhow::anyhow!("the WASM worker panicked")))
330 +
    });
331 +
    pass_on(sink, true);
332 +
    let (exit_code, peak) = match ran {
333 +
        Ok((code, peak)) => (Some(code), Some(peak)),
334 +
        Err(e) => {
335 +
            sink.record(Record { t: t(&started), stream: "flow".into(), text: format!("the WASM tier could not run the job: {e:#}") });
336 +
            (Some(125), None)
337 +
        }
338 +
    };
339 +
    Ok(Outcome {
340 +
        exit_code: if timed_out || cancelled { None } else { exit_code },
341 +
        duration_ms: t(&started),
342 +
        environment,
343 +
        timed_out,
344 +
        cancelled,
345 +
        peak_memory_mib: peak.map(|b| (b as u64).div_ceil(1 << 20)),
346 +
        pruned: Vec::new(),
347 +
    })
348 +
}
349 +
350 +
#[cfg(feature = "wasm")]
351 +
/// `flow __wasm SPEC`: run the spec's modules in turn, each in a fresh
352 +
/// instance; the exit code is the first non-zero one, after all have run,
353 +
/// so one failing module does not hide another's results.
354 +
pub fn wasm_main(spec_path: &Path) -> i32 {
355 +
    let ran = (|| {
356 +
        let spec: Spec = serde_json::from_slice(&std::fs::read(spec_path)?).context("read the spec")?;
357 +
        let engine = ww(wasmtime::Engine::new(&wasmtime::Config::new()), || "start wasmtime".into())?;
358 +
        run_modules(&engine, &spec, &Io::Inherit)
359 +
    })();
360 +
    match ran {
361 +
        Ok((code, _)) => code,
362 +
        Err(e) => {
363 +
            eprintln!("{FAILED}{e:#}");
364 +
            125
365 +
        }
366 +
    }
367 +
}
368 +
369 +
/// `flow __wasm SPEC` in a build without the tier: says so.
370 +
#[cfg(not(feature = "wasm"))]
371 +
pub fn wasm_main(_spec_path: &Path) -> i32 {
372 +
    eprintln!("{FAILED}{WITHOUT}");
373 +
    125
374 +
}
375 +
376 +
#[cfg(not(feature = "wasm"))]
377 +
fn run_in_thread(_spec: &Spec, _environment: Environment, _timeout: Duration, _sink: &mut dyn Sink) -> Result<Outcome> {
378 +
    bail!(WITHOUT)
379 +
}
380 +
381 +
#[cfg(feature = "wasm")]
382 +
/// wasmtime's own error, as this crate's: with its whole chain of causes.
383 +
fn ww<T>(r: wasmtime::Result<T>, what: impl FnOnce() -> String) -> Result<T> {
384 +
    r.map_err(|e| anyhow::anyhow!("{}: {e:#}", what()))
385 +
}
386 +
387 +
#[cfg(feature = "wasm")]
388 +
/// Where a module's output and the tier's notes go: inherited by the
389 +
/// helper process (notes on stderr, marked), or pipes the runner reads.
390 +
enum Io {
391 +
    Inherit,
392 +
    Pipes {
393 +
        stdout: wasmtime_wasi::p2::pipe::MemoryOutputPipe,
394 +
        stderr: wasmtime_wasi::p2::pipe::MemoryOutputPipe,
395 +
        notes: mpsc::Sender<String>,
396 +
    },
397 +
}
398 +
399 +
#[cfg(feature = "wasm")]
400 +
impl Io {
401 +
    fn note(&self, text: String) {
402 +
        match self {
403 +
            Io::Inherit => eprintln!("{NOTE}{text}"),
404 +
            Io::Pipes { notes, .. } => {
405 +
                let _ = notes.send(text);
406 +
            }
407 +
        }
408 +
    }
409 +
}
410 +
411 +
#[cfg(feature = "wasm")]
412 +
/// The sandbox's memory limit, and the most a module asked for within it.
413 +
struct Limits {
414 +
    memory: usize,
415 +
    peak: usize,
416 +
}
417 +
418 +
#[cfg(feature = "wasm")]
419 +
impl wasmtime::ResourceLimiter for Limits {
420 +
    fn memory_growing(&mut self, _current: usize, desired: usize, maximum: Option<usize>) -> wasmtime::Result<bool> {
421 +
        let ok = desired <= self.memory && maximum.is_none_or(|m| desired <= m);
422 +
        if ok {
423 +
            self.peak = self.peak.max(desired);
424 +
        }
425 +
        Ok(ok)
426 +
    }
427 +
428 +
    fn table_growing(&mut self, _current: usize, desired: usize, maximum: Option<usize>) -> wasmtime::Result<bool> {
429 +
        Ok(maximum.is_none_or(|m| desired <= m))
430 +
    }
431 +
}
432 +
433 +
#[cfg(feature = "wasm")]
434 +
struct State {
435 +
    wasi: wasmtime_wasi::p1::WasiP1Ctx,
436 +
    limits: Limits,
437 +
}
438 +
439 +
#[cfg(feature = "wasm")]
440 +
/// Run each module in turn: the first non-zero exit code after all ran,
441 +
/// and the most memory any of them held, in bytes.
442 +
fn run_modules(engine: &wasmtime::Engine, spec: &Spec, io: &Io) -> Result<(i32, usize)> {
443 +
    let mut linker: wasmtime::Linker<State> = wasmtime::Linker::new(engine);
444 +
    ww(wasmtime_wasi::p1::add_to_linker_sync(&mut linker, |s: &mut State| &mut s.wasi), || "link WASI".into())?;
445 +
    let (mut first_failure, mut peak) = (0, 0);
446 +
    for path in &spec.modules {
447 +
        if spec.modules.len() > 1 {
448 +
            io.note(format!("running {path}"));
449 +
        }
450 +
        let (code, held) = run_module(engine, &linker, spec, path, io)?;
451 +
        peak = peak.max(held);
452 +
        if spec.modules.len() > 1 {
453 +
            io.note(format!("{path}: exit {code}"));
454 +
        }
455 +
        if first_failure == 0 && code != 0 {
456 +
            first_failure = code;
457 +
        }
458 +
    }
459 +
    Ok((first_failure, peak))
460 +
}
461 +
462 +
#[cfg(feature = "wasm")]
463 +
fn run_module(engine: &wasmtime::Engine, linker: &wasmtime::Linker<State>, spec: &Spec, path: &str, io: &Io) -> Result<(i32, usize)> {
464 +
    use wasmtime_wasi::{FsPerms, WasiCtxBuilder};
465 +
    let module = ww(wasmtime::Module::from_file(engine, spec.workspace.join(path)), || format!("module {path} does not load"))?;
466 +
    let name = path.rsplit('/').next().unwrap_or(path);
467 +
    let mut argv = vec![name.to_string()];
468 +
    argv.extend(spec.args.iter().cloned());
469 +
    let mut wasi = WasiCtxBuilder::new();
470 +
    match io {
471 +
        Io::Inherit => {
472 +
            wasi.inherit_stdout().inherit_stderr();
473 +
        }
474 +
        Io::Pipes { stdout, stderr, .. } => {
475 +
            wasi.stdout(stdout.clone()).stderr(stderr.clone());
476 +
        }
477 +
    }
478 +
    wasi.args(&argv).envs(&spec.env).env("PWD", "/src").allow_blocking_current_thread(true);
479 +
    let open = |wasi: &mut WasiCtxBuilder, dir: &Path, at: &str| {
480 +
        ww(wasi.preopened_dir(dir, at, FsPerms::ReadWrite).map(|_| ()), || format!("open {} as {at}", dir.display()))
481 +
    };
482 +
    open(&mut wasi, &spec.workspace, "/src")?;
483 +
    // Relative paths resolve against the workspace too.
484 +
    open(&mut wasi, &spec.workspace, ".")?;
485 +
    for (dir, target) in &spec.preopens {
486 +
        open(&mut wasi, dir, target)?;
487 +
    }
488 +
    let limits = Limits { memory: spec.memory_mib as usize * 1024 * 1024, peak: 0 };
489 +
    let mut store = wasmtime::Store::new(engine, State { wasi: wasi.build_p1(), limits });
490 +
    store.limiter(|s| &mut s.limits);
491 +
    // With epoch interruption on (in process), one tick from now traps:
492 +
    // the runner ticks only to stop the module.
493 +
    store.set_epoch_deadline(1);
494 +
    let instance = ww(linker.instantiate(&mut store, &module), || format!("module {path} cannot start here"))?;
495 +
    let start =
496 +
        ww(instance.get_typed_func::<(), ()>(&mut store, "_start"), || format!("module {path} has no _start: is it a WASI command?"))?;
497 +
    let code = match start.call(&mut store, ()) {
498 +
        Ok(()) => 0,
499 +
        Err(e) => match e.downcast_ref::<wasmtime_wasi::I32Exit>() {
500 +
            Some(exit) => exit.0,
501 +
            None => {
502 +
                io.note(format!("{path} trapped: {e:#}"));
503 +
                TRAPPED
504 +
            }
505 +
        },
506 +
    };
507 +
    Ok((code, store.data().limits.peak))
508 +
}
509 +
510 +
#[cfg(all(test, feature = "wasm"))]
511 +
mod tests {
512 +
    use super::*;
513 +
514 +
    /// A spec running `text` (WAT) as the only module, with 64 MiB.
515 +
    fn spec_of(dir: &Path, text: &str) -> Spec {
516 +
        std::fs::write(dir.join("m.wasm"), wat::parse_str(text).unwrap()).unwrap();
517 +
        Spec { workspace: dir.to_path_buf(), modules: vec!["m.wasm".into()], args: vec![], env: vec![], memory_mib: 64, preopens: vec![] }
518 +
    }
519 +
520 +
    fn env() -> Environment {
521 +
        Environment { runtime: "wasm".into(), parts: vec![], rootfs: String::new(), argv: vec![], env: vec![] }
522 +
    }
523 +
524 +
    /// Records, and whether to say the run was cancelled.
525 +
    struct Collect(Vec<Record>, bool);
526 +
    impl Sink for Collect {
527 +
        fn record(&mut self, r: Record) {
528 +
            self.0.push(r);
529 +
        }
530 +
        fn cancelled(&mut self) -> bool {
531 +
            self.1
532 +
        }
533 +
    }
534 +
535 +
    const PRINT: &str = r#"(module
536 +
      (import "wasi_snapshot_preview1" "fd_write" (func $w (param i32 i32 i32 i32) (result i32)))
537 +
      (memory (export "memory") 2)
538 +
      (data (i32.const 16) "line one\nline two")
539 +
      (func (export "_start")
540 +
        (i32.store (i32.const 0) (i32.const 16)) (i32.store (i32.const 4) (i32.const 17))
541 +
        (drop (call $w (i32.const 1) (i32.const 0) (i32.const 1) (i32.const 8)))))"#;
542 +
543 +
    const SPIN: &str = r#"(module (func (export "_start") (loop $l (br $l))))"#;
544 +
545 +
    #[test]
546 +
    fn in_process_output_streams_as_lines() {
547 +
        let dir = tempfile::tempdir().unwrap();
548 +
        let mut sink = Collect(vec![], false);
549 +
        let o = run_in_thread(&spec_of(dir.path(), PRINT), env(), Duration::from_secs(30), &mut sink).unwrap();
550 +
        assert_eq!(o.exit_code, Some(0));
551 +
        let out: Vec<&str> = sink.0.iter().filter(|r| r.stream == "out").map(|r| r.text.as_str()).collect();
552 +
        assert_eq!(out, ["line one", "line two"], "a last line without its newline still comes");
553 +
        assert_eq!(o.peak_memory_mib, Some(1), "two 64 KiB pages, in whole MiB");
554 +
    }
555 +
556 +
    #[test]
557 +
    fn in_process_stops_at_the_time_limit_and_when_cancelled() {
558 +
        let dir = tempfile::tempdir().unwrap();
559 +
        let mut sink = Collect(vec![], false);
560 +
        let started = Instant::now();
561 +
        let o = run_in_thread(&spec_of(dir.path(), SPIN), env(), Duration::from_millis(300), &mut sink).unwrap();
562 +
        assert!(o.timed_out && o.exit_code.is_none(), "{:?}", sink.0);
563 +
        assert!(started.elapsed() < Duration::from_secs(5), "the endless loop was interrupted");
564 +
        let mut sink = Collect(vec![], true);
565 +
        let o = run_in_thread(&spec_of(dir.path(), SPIN), env(), Duration::from_secs(30), &mut sink).unwrap();
566 +
        assert!(o.cancelled && o.exit_code.is_none());
567 +
        assert!(sink.0.iter().any(|r| r.text == "stopped: the run was cancelled"));
568 +
    }
569 +
570 +
    #[test]
571 +
    fn globs_match_names_with_stars() {
572 +
        assert!(wildcard("*.wasm", "oxy_core-1a2b.wasm"));
573 +
        assert!(wildcard("oxy_core-*.wasm", "oxy_core-1a2b.wasm"));
574 +
        assert!(!wildcard("oxy_core-*.wasm", "oxy_core-1a2b.d"));
575 +
        assert!(wildcard("a*b*c", "aXbYc") && !wildcard("a*b*c", "aXcYb"));
576 +
        assert!(wildcard("plain.wasm", "plain.wasm") && !wildcard("plain.wasm", "other.wasm"));
577 +
    }
578 +
579 +
    #[test]
580 +
    fn modules_are_files_of_the_job() {
581 +
        let dir = tempfile::tempdir().unwrap();
582 +
        let deps = dir.path().join("target/wasm32-wasip1/debug/deps");
583 +
        std::fs::create_dir_all(&deps).unwrap();
584 +
        for f in ["b-2.wasm", "a-1.wasm", "a-1.d"] {
585 +
            std::fs::write(deps.join(f), b"").unwrap();
586 +
        }
587 +
        let ws = dir.path();
588 +
        assert_eq!(
589 +
            modules(ws, "/src/target/wasm32-wasip1/debug/deps/*.wasm").unwrap(),
590 +
            ["target/wasm32-wasip1/debug/deps/a-1.wasm", "target/wasm32-wasip1/debug/deps/b-2.wasm"]
591 +
        );
592 +
        assert_eq!(modules(ws, "./target/wasm32-wasip1/debug/deps/b-2.wasm").unwrap(), ["target/wasm32-wasip1/debug/deps/b-2.wasm"]);
593 +
        assert!(modules(ws, "target/wasm32-wasip1/debug/deps/*.so").unwrap_err().to_string().contains("no module matches"));
594 +
        assert!(modules(ws, "../etc/passwd").is_err());
595 +
        assert!(modules(ws, "/etc/x.wasm").is_err());
596 +
        assert!(modules(ws, "target/*/debug/a.wasm").unwrap_err().to_string().contains("only the file name"));
597 +
    }
598 +
}
modified crates/flow/src/workspace.rs +171 −8
…
37 unchanged lines
38 38
                }
39 39
                if *kind == NodeKind::Symlink {
40 40
                    let to = String::from_utf8_lossy(&bytes).into_owned();
41 -
                    std::os::unix::fs::symlink(&to, &target).with_context(|| format!("symlink {path}"))?;
41 +
                    crate::portable::symlink(Path::new(&to), &target).with_context(|| format!("symlink {path}"))?;
42 42
                } else {
43 43
                    std::fs::write(&target, &bytes).with_context(|| format!("write {path}"))?;
44 44
                    if f.mode.exec {
45 -
                        use std::os::unix::fs::PermissionsExt;
46 -
                        std::fs::set_permissions(&target, std::fs::Permissions::from_mode(0o755))?;
45 +
                        crate::portable::set_mode(&target, 0o755)?;
47 46
                    }
48 47
                }
49 48
                files.push((path.clone(), *blake3::hash(&bytes).as_bytes()));
…
7 unchanged lines
57 56
/// `.gitignore` exclude) under `dir`, for `flow run`.
58 57
pub fn copy_tree(src: &Path, dir: &Path) -> Result<Files> {
59 58
    let mut files = Vec::new();
59 +
    // A file at the top would otherwise find no directory to go in.
60 +
    std::fs::create_dir_all(dir)?;
60 61
    let walk = ignore::WalkBuilder::new(src)
61 62
        .hidden(false)
62 63
        .git_ignore(true)
…
14 unchanged lines
77 78
            std::fs::create_dir_all(&target)?;
78 79
        } else if ft.is_symlink() {
79 80
            let to = std::fs::read_link(entry.path())?;
80 -
            std::os::unix::fs::symlink(&to, &target)?;
81 +
            crate::portable::symlink(&to, &target)?;
81 82
            files.push((rel_s, *blake3::hash(to.to_string_lossy().as_bytes()).as_bytes()));
82 83
        } else if ft.is_file() {
83 84
            std::fs::copy(entry.path(), &target)?;
…
16 unchanged lines
100 101
        .map(|m| m.len().max(4096))
101 102
        .sum();
102 103
    let mib = used * 12 / 10 / (1 << 20) + 512;
103 -
    let (uid, gid) = unsafe { (libc::getuid(), libc::getgid()) };
104 +
    let (uid, gid) = owner();
104 105
    let status = std::process::Command::new("mkfs.ext4")
105 106
        .args(["-q", "-F", "-L", "workspace", "-E"])
106 107
        .arg(format!("root_owner={uid}:{gid}"))
…
9 unchanged lines
116 117
    Ok(())
117 118
}
118 119
120 +
/// Give the files of `dir` whose content a cached disk was built from
121 +
/// the mtime they had then (oxygenflow#52), and describe this checkout's
122 +
/// sources for the disk to keep: unchanged files are fresh to a build
123 +
/// tool that judges by time, changed ones newer than anything it built.
124 +
/// Whole seconds, as the workspace image keeps them. Returns the sources
125 +
/// and how many files got their time back.
126 +
pub fn restore_times(dir: &Path, files: &Files, before: Option<&crate::cache::Sources>) -> (crate::cache::Sources, usize) {
127 +
    let mut out = crate::cache::Sources::new();
128 +
    let mut restored = 0;
129 +
    for (path, hash) in files {
130 +
        let p = dir.join(path);
131 +
        let Ok(md) = std::fs::symlink_metadata(&p) else { continue };
132 +
        if !md.is_file() {
133 +
            continue;
134 +
        }
135 +
        let hex = hex::encode(hash);
136 +
        let kept = before.and_then(|b| b.get(path)).filter(|(h, _)| *h == hex).map(|(_, t)| *t);
137 +
        let t = match kept {
138 +
            Some(t) => {
139 +
                let at = std::time::UNIX_EPOCH + std::time::Duration::from_secs(t.max(0) as u64);
140 +
                if std::fs::File::options().write(true).open(&p).and_then(|f| f.set_modified(at)).is_err() {
141 +
                    continue;
142 +
                }
143 +
                restored += 1;
144 +
                t
145 +
            }
146 +
            None => md.modified().ok().and_then(|m| m.duration_since(std::time::UNIX_EPOCH).ok()).map(|d| d.as_secs() as i64).unwrap_or(0),
147 +
        };
148 +
        out.insert(path.clone(), (hex, t));
149 +
    }
150 +
    // Directories too, after their files (#52): a build script's
151 +
    // `rerun-if-changed` on a directory looks at its time as well. A
152 +
    // directory whose listing is unchanged gets its old time back; one that
153 +
    // gained or lost an entry keeps its checkout's, so the change shows.
154 +
    restore_dir_times(dir, Path::new(""), before, &mut out, &mut restored);
155 +
    (out, restored)
156 +
}
157 +
158 +
/// The key a directory's entry has in the sources: its path and a slash.
159 +
fn dir_key(rel: &Path) -> String {
160 +
    format!("{}/", rel.display())
161 +
}
162 +
163 +
fn restore_dir_times(
164 +
    root: &Path,
165 +
    rel: &Path,
166 +
    before: Option<&crate::cache::Sources>,
167 +
    out: &mut crate::cache::Sources,
168 +
    restored: &mut usize,
169 +
) {
170 +
    let here = root.join(rel);
171 +
    let Ok(entries) = std::fs::read_dir(&here) else { return };
172 +
    let mut names: Vec<(String, bool)> = Vec::new();
173 +
    for e in entries.flatten() {
174 +
        let is_dir = e.file_type().is_ok_and(|t| t.is_dir());
175 +
        names.push((e.file_name().to_string_lossy().into_owned(), is_dir));
176 +
    }
177 +
    names.sort();
178 +
    for (name, is_dir) in &names {
179 +
        if *is_dir {
180 +
            restore_dir_times(root, &rel.join(name), before, out, restored);
181 +
        }
182 +
    }
183 +
    let mut h = blake3::Hasher::new();
184 +
    for (name, is_dir) in &names {
185 +
        h.update(name.as_bytes());
186 +
        h.update(if *is_dir { b"/\n".as_slice() } else { b"\n".as_slice() });
187 +
    }
188 +
    let hex = h.finalize().to_hex().to_string();
189 +
    let key = dir_key(rel);
190 +
    let kept = before.and_then(|b| b.get(&key)).filter(|(h, _)| *h == hex).map(|(_, t)| *t);
191 +
    let t = match kept {
192 +
        Some(t) => {
193 +
            let at = std::time::UNIX_EPOCH + std::time::Duration::from_secs(t.max(0) as u64);
194 +
            if std::fs::File::open(&here).and_then(|f| f.set_modified(at)).is_err() {
195 +
                return;
196 +
            }
197 +
            *restored += 1;
198 +
            t
199 +
        }
200 +
        None => std::fs::metadata(&here)
201 +
            .ok()
202 +
            .and_then(|m| m.modified().ok())
203 +
            .and_then(|m| m.duration_since(std::time::UNIX_EPOCH).ok())
204 +
            .map(|d| d.as_secs() as i64)
205 +
            .unwrap_or(0),
206 +
    };
207 +
    out.insert(key, (hex, t));
208 +
}
209 +
119 210
/// An empty ext4 filesystem of `mib` MiB at `out`, owned by this user like
120 211
/// the workspace. mkfs.ext4 creates the file sparse, so only what the job
121 212
/// writes takes space on the runner.
122 213
pub fn empty_ext4(out: &Path, mib: u32, label: &str) -> Result<()> {
123 -
    let (uid, gid) = unsafe { (libc::getuid(), libc::getgid()) };
214 +
    let (uid, gid) = owner();
124 215
    let status = std::process::Command::new("mkfs.ext4")
125 216
        .args(["-q", "-F", "-L", label, "-E"])
126 217
        .arg(format!("root_owner={uid}:{gid}"))
…
20 unchanged lines
147 238
    }
148 239
}
149 240
241 +
/// The runner's user and group, which own a workspace image's root.
242 +
fn owner() -> (u32, u32) {
243 +
    #[cfg(unix)]
244 +
    {
245 +
        unsafe { (libc::getuid(), libc::getgid()) }
246 +
    }
247 +
    #[cfg(not(unix))]
248 +
    {
249 +
        (0, 0)
250 +
    }
251 +
}
252 +
150 253
fn open_up(dir: &Path) {
151 -
    use std::os::unix::fs::PermissionsExt;
152 254
    let Ok(meta) = std::fs::symlink_metadata(dir) else { return };
153 255
    if !meta.is_dir() {
154 256
        return;
155 257
    }
156 -
    let _ = std::fs::set_permissions(dir, std::fs::Permissions::from_mode(0o700));
258 +
    let _ = crate::portable::set_mode(dir, 0o700);
157 259
    if let Ok(entries) = std::fs::read_dir(dir) {
158 260
        for e in entries.flatten() {
159 261
            open_up(&e.path());
160 262
        }
161 263
    }
162 264
}
265 +
266 +
#[cfg(test)]
267 +
mod tests {
268 +
    use super::*;
269 +
    use std::time::UNIX_EPOCH;
270 +
271 +
    fn secs(p: &Path) -> i64 {
272 +
        std::fs::metadata(p).unwrap().modified().unwrap().duration_since(UNIX_EPOCH).unwrap().as_secs() as i64
273 +
    }
274 +
275 +
    /// oxygenflow#52: an unchanged file gets the time it had when the cached
276 +
    /// disk was built; a changed or new one keeps its checkout's, newer.
277 +
    #[test]
278 +
    fn unchanged_sources_get_their_cached_times_back() {
279 +
        let dir = tempfile::tempdir().unwrap();
280 +
        for (name, text) in [("same.rs", "a"), ("changed.rs", "b2"), ("new.rs", "c")] {
281 +
            std::fs::write(dir.path().join(name), text).unwrap();
282 +
        }
283 +
        std::os::unix::fs::symlink("same.rs", dir.path().join("link.rs")).unwrap();
284 +
        let h = |t: &str| *blake3::hash(t.as_bytes()).as_bytes();
285 +
        let files: Files =
286 +
            vec![("same.rs".into(), h("a")), ("changed.rs".into(), h("b2")), ("new.rs".into(), h("c")), ("link.rs".into(), h("same.rs"))];
287 +
        let now = secs(&dir.path().join("same.rs"));
288 +
        let before: crate::cache::Sources =
289 +
            [("same.rs".to_string(), (hex::encode(h("a")), 1_000_000)), ("changed.rs".to_string(), (hex::encode(h("b1")), 1_000_000))]
290 +
                .into();
291 +
292 +
        let (sources, restored) = restore_times(dir.path(), &files, Some(&before));
293 +
        assert_eq!(restored, 1);
294 +
        assert_eq!(secs(&dir.path().join("same.rs")), 1_000_000, "unchanged: fresh to cargo");
295 +
        assert!(secs(&dir.path().join("changed.rs")) >= now, "changed: newer than the cached build");
296 +
        assert_eq!(sources["same.rs"], (hex::encode(h("a")), 1_000_000));
297 +
        assert_eq!(sources["changed.rs"].0, hex::encode(h("b2")), "kept with its new content");
298 +
        assert!(sources.contains_key("new.rs") && !sources.contains_key("link.rs"), "files only");
299 +
300 +
        // Without a cached build nothing moves, and the sources are listed:
301 +
        // three files and the directory holding them.
302 +
        let (again, restored) = restore_times(dir.path(), &files, None);
303 +
        assert_eq!((restored, again.len()), (0, 4));
304 +
        assert!(again.contains_key("/"));
305 +
306 +
        // A directory whose listing is unchanged gets its time back; one
307 +
        // that gained an entry keeps its new time, so a build script's
308 +
        // `rerun-if-changed` on it sees the change.
309 +
        std::fs::create_dir(dir.path().join("migrations")).unwrap();
310 +
        std::fs::write(dir.path().join("migrations/0001.sql"), "x").unwrap();
311 +
        let mut files = files;
312 +
        files.push(("migrations/0001.sql".into(), h("x")));
313 +
        let (mut kept, _) = restore_times(dir.path(), &files, None);
314 +
        kept.get_mut("migrations/").unwrap().1 = 2_000_000;
315 +
        kept.get_mut("migrations/0001.sql").unwrap().1 = 2_000_000;
316 +
        let (_, restored) = restore_times(dir.path(), &files, Some(&kept));
317 +
        assert_eq!(secs(&dir.path().join("migrations")), 2_000_000, "unchanged listing: its old time");
318 +
        assert!(restored >= 2);
319 +
        std::fs::write(dir.path().join("migrations/0002.sql"), "y").unwrap();
320 +
        files.push(("migrations/0002.sql".into(), h("y")));
321 +
        let (now_sources, _) = restore_times(dir.path(), &files, Some(&kept));
322 +
        assert!(secs(&dir.path().join("migrations")) > 2_000_000, "a new entry: it looks changed");
323 +
        assert_ne!(now_sources["migrations/"].0, kept["migrations/"].0);
324 +
    }
325 +
}
modified crates/flow/tests/syscall.rs +17 −1
…
9 unchanged lines
10 10
11 11
const APPLETS: &[&str] = &[
12 12
    "sh", "echo", "cat", "id", "touch", "wget", "tail", "cut", "tr", "true", "false", "ls", "grep", "wc", "mount", "hostname", "head",
13 -
    "sed",
13 +
    "sed", "httpd", "mkdir", "sleep",
14 14
];
15 15
16 16
/// `flow`, in a delegated cgroup scope of its own when systemd's user
…
52 unchanged lines
69 69
    rootfs "cas://blob/{rootfs}"
70 70
    memory-limit "256MiB"
71 71
}}
72 +
sandbox "bb-lo" {{
73 +
    runtime "syscall"
74 +
    rootfs "cas://blob/{rootfs}"
75 +
    memory-limit "256MiB"
76 +
    network "loopback"
77 +
}}
78 +
job "loopback" {{
79 +
    sandbox "bb-lo"
80 +
    exec "sh" "-c" "mkdir -p /tmp/www && echo SERVED > /tmp/www/x && httpd -p 127.0.0.1:8080 -h /tmp/www && sleep 1 && wget -q -O - http://127.0.0.1:8080/x; wget -q -T 2 -O /dev/null http://1.1.1.1 && echo LO-NETWORK || echo lo-only"
81 +
}}
72 82
job "literal" {{
73 83
    sandbox "bb"
74 84
    exec "echo" "a; touch /src/pwned && $(id) | `id`"
…
40 unchanged lines
115 125
    assert!(stdout.contains("   flow\n"), "its own host name: {all}");
116 126
    assert!(!stdout.contains("   MOUNTED\n") && stdout.contains("   done\n"), "no mounts: {all}");
117 127
    assert!(stderr.contains("Address family not supported"), "sockets other than AF_UNIX are refused: {all}");
128 +
    // oxygenflow#73: with network "loopback", a server on 127.0.0.1 answers
129 +
    // its own job, and nothing past lo is reached.
130 +
    assert!(
131 +
        stdout.contains("   SERVED\n") && stdout.contains("   lo-only\n") && !stdout.contains("   LO-NETWORK\n"),
132 +
        "loopback TCP: {all}"
133 +
    );
118 134
    assert!(!work.join("w").exists(), "jobs write to their own copy: {all}");
119 135
    let procs: u32 = stdout.lines().find_map(|l| l.trim().strip_prefix("procs ")).and_then(|n| n.parse().ok()).expect(&all);
120 136
    assert!(procs < 10, "only the sandbox's processes are visible: {all}");
…
7 unchanged lines
added crates/flow/tests/wasm.rs +150 −0
1 +
//! The WASM tier (oxygenflow#75): modules written as text, run by `flow
2 +
//! run` through wasmtime. Every machine offers the tier, so nothing is
3 +
//! skipped.
4 +
5 +
use std::path::Path;
6 +
use std::process::Command;
7 +
8 +
const IMPORTS: &str = r#"
9 +
  (import "wasi_snapshot_preview1" "fd_write" (func $fd_write (param i32 i32 i32 i32) (result i32)))
10 +
  (import "wasi_snapshot_preview1" "proc_exit" (func $proc_exit (param i32)))
11 +
  (import "wasi_snapshot_preview1" "path_open"
12 +
    (func $path_open (param i32 i32 i32 i32 i32 i64 i64 i32 i32) (result i32)))
13 +
  (import "wasi_snapshot_preview1" "args_sizes_get" (func $args_sizes_get (param i32 i32) (result i32)))
14 +
  (memory (export "memory") 1)
15 +
"#;
16 +
17 +
/// A module whose `_start` is `body`, with the WASI calls above.
18 +
fn module(dir: &Path, path: &str, data: &str, body: &str) {
19 +
    let text = format!("(module {IMPORTS} {data} (func (export \"_start\") (local $e i32) {body}))");
20 +
    let bytes = wat::parse_str(&text).unwrap_or_else(|e| panic!("{path}: {e}"));
21 +
    let file = dir.join(path);
22 +
    std::fs::create_dir_all(file.parent().unwrap()).unwrap();
23 +
    std::fs::write(file, bytes).unwrap();
24 +
}
25 +
26 +
/// Print the `len` bytes at 16 to fd `fd`.
27 +
fn print(fd: i32, len: i32) -> String {
28 +
    format!("(i32.store (i32.const 0) (i32.const 16)) (i32.store (i32.const 4) (i32.const {len})) (drop (call $fd_write (i32.const {fd}) (i32.const 0) (i32.const 1) (i32.const 8)))")
29 +
}
30 +
31 +
#[test]
32 +
fn wasi_modules_run_confined_in_the_wasm_tier() {
33 +
    confined(false);
34 +
}
35 +
36 +
/// The same in a thread of the runner, as on Android, where the runner is
37 +
/// a library in an app that cannot start itself as a command.
38 +
#[test]
39 +
fn wasi_modules_run_confined_in_process() {
40 +
    confined(true);
41 +
}
42 +
43 +
fn confined(in_process: bool) {
44 +
    let dir = tempfile::tempdir().unwrap();
45 +
    let home = dir.path().join("home");
46 +
    let work = dir.path().join("work");
47 +
    let tiers = Command::new(env!("CARGO_BIN_EXE_flow")).arg("--home").arg(&home).arg("tiers").output().unwrap();
48 +
    assert!(String::from_utf8_lossy(&tiers.stdout).contains("wasm"), "{}", String::from_utf8_lossy(&tiers.stdout));
49 +
50 +
    // Prints, and says how many arguments it had (2: its name and one).
51 +
    module(
52 +
        &work,
53 +
        "mods/hello.wasm",
54 +
        r#"(data (i32.const 16) "hello from wasm\n")"#,
55 +
        &format!(
56 +
            "{} (drop (call $args_sizes_get (i32.const 100) (i32.const 104))) (if (i32.ne (i32.load (i32.const 100)) (i32.const 2)) (then (call $proc_exit (i32.const 7))))",
57 +
            print(1, 16)
58 +
        ),
59 +
    );
60 +
    // Two modules under one glob: the first passes, the second exits 3.
61 +
    module(&work, "mods/each/a.wasm", r#"(data (i32.const 16) "module a\n")"#, &print(1, 9));
62 +
    module(&work, "mods/each/b.wasm", r#"(data (i32.const 16) "module b\n")"#, &format!("{} (call $proc_exit (i32.const 3))", print(2, 9)));
63 +
    // Creates a file in the workspace (preopen 3 is /src).
64 +
    module(
65 +
        &work,
66 +
        "mods/writer.wasm",
67 +
        r#"(data (i32.const 64) "made.txt") (data (i32.const 16) "made in wasm\n")"#,
68 +
        "(local.set $e (call $path_open (i32.const 3) (i32.const 0) (i32.const 64) (i32.const 8) (i32.const 9) (i64.const 0x1fffffff) (i64.const 0x1fffffff) (i32.const 0) (i32.const 200)))
69 +
         (if (local.get $e) (then (call $proc_exit (i32.const 10))))
70 +
         (i32.store (i32.const 0) (i32.const 16)) (i32.store (i32.const 4) (i32.const 13))
71 +
         (drop (call $fd_write (i32.load (i32.const 200)) (i32.const 0) (i32.const 1) (i32.const 8)))",
72 +
    );
73 +
    // Tries to create a file outside the workspace: refused, so it exits 0.
74 +
    module(
75 +
        &work,
76 +
        "mods/escape.wasm",
77 +
        r#"(data (i32.const 64) "../escaped")"#,
78 +
        "(local.set $e (call $path_open (i32.const 3) (i32.const 0) (i32.const 64) (i32.const 10) (i32.const 9) (i64.const 0x1fffffff) (i64.const 0x1fffffff) (i32.const 0) (i32.const 200)))
79 +
         (if (i32.eqz (local.get $e)) (then (call $proc_exit (i32.const 9))))",
80 +
    );
81 +
    // Asks for 512 MiB in a 256 MiB sandbox: refused, so it exits 0.
82 +
    module(
83 +
        &work,
84 +
        "mods/greedy.wasm",
85 +
        "",
86 +
        "(if (i32.ne (memory.grow (i32.const 8192)) (i32.const -1)) (then (call $proc_exit (i32.const 9))))",
87 +
    );
88 +
    module(&work, "mods/trap.wasm", "", "unreachable");
89 +
90 +
    // A file at the top of the workspace, beside the directories.
91 +
    std::fs::write(work.join("README"), "top\n").unwrap();
92 +
    std::fs::create_dir_all(work.join(".flow")).unwrap();
93 +
    std::fs::write(
94 +
        work.join(".flow/pipeline.kdl"),
95 +
        r#"schema-version 1
96 +
sandbox "w" {
97 +
    runtime "wasm"
98 +
    memory-limit "256MiB"
99 +
}
100 +
job "hello" {
101 +
    sandbox "w"
102 +
    exec "/src/mods/hello.wasm" "one"
103 +
}
104 +
job "each" {
105 +
    sandbox "w"
106 +
    exec "mods/each/*.wasm"
107 +
}
108 +
job "writer" {
109 +
    sandbox "w"
110 +
    exec "mods/writer.wasm"
111 +
}
112 +
job "escape" {
113 +
    sandbox "w"
114 +
    exec "mods/escape.wasm"
115 +
}
116 +
job "greedy" {
117 +
    sandbox "w"
118 +
    exec "mods/greedy.wasm"
119 +
}
120 +
job "trap" {
121 +
    sandbox "w"
122 +
    exec "mods/trap.wasm"
123 +
}
124 +
"#,
125 +
    )
126 +
    .unwrap();
127 +
    let mut flow = Command::new(env!("CARGO_BIN_EXE_flow"));
128 +
    if in_process {
129 +
        flow.env("FLOW_WASM_IN_PROCESS", "1");
130 +
    }
131 +
    let run = flow.arg("--home").arg(&home).arg("run").current_dir(&work).output().unwrap();
132 +
    let out = String::from_utf8_lossy(&run.stdout).to_string() + &String::from_utf8_lossy(&run.stderr);
133 +
    for want in [
134 +
        "hello from wasm",
135 +
        "── hello passed",
136 +
        "running mods/each/a.wasm",
137 +
        "module a",
138 +
        "mods/each/b.wasm: exit 3",
139 +
        "── each failed (exit 3)",
140 +
        "── writer passed",
141 +
        "── escape passed",
142 +
        "── greedy passed",
143 +
        "mods/trap.wasm trapped",
144 +
        "── trap failed (exit 134)",
145 +
    ] {
146 +
        assert!(out.contains(want), "{want:?} not in:\n{out}");
147 +
    }
148 +
    assert!(!dir.path().join("escaped").exists() && !work.join("escaped").exists(), "a module wrote outside the workspace");
149 +
    assert_eq!(run.status.code(), Some(1), "two jobs failed:\n{out}");
150 +
}
modified crates/jet/src/lib.rs +4 −1
…
1 unchanged lines
2 2
//! its own. Flow says what to run; Jet carries it to where it should run.
3 3
//!
4 4
//! It starts with images by hash (oxygenflow#39): a rootfs image is kept by
5 -
//! a hub as content-defined chunks, sent and fetched by what is missing.
5 +
//! a hub as content-defined chunks, sent and fetched by what is missing,
6 +
//! and shared between runners in a swarm (oxygenflow#47).
6 7
7 8
pub mod images;
9 +
pub mod rootfs;
10 +
pub mod swarm;
added crates/jet/src/rootfs.rs +383 −0
1 +
//! What a rootfs image is for, read from its own files (OxygenFlow spec
2 +
//! §11.2): the ELF header of its shell. An image is an ext4 filesystem,
3 +
//! as `flow rootfs import` makes it; this reads just enough of one to
4 +
//! follow a path, symlinks included, to a file's first bytes. It needs no
5 +
//! mount and no privileges, so a runner can check an image it fetched and
6 +
//! a hub an image it keeps as chunks.
7 +
//!
8 +
//! Read only, and defensive: a malformed image answers `None`, never a
9 +
//! panic or an unbounded read.
10 +
11 +
use std::io::{Read, Seek, SeekFrom};
12 +
use std::path::Path;
13 +
14 +
/// The files whose architecture is the image's, in order: the shell's,
15 +
/// else the first program found.
16 +
const PROGRAMS: [&str; 5] = ["bin/sh", "bin/busybox", "usr/bin/env", "bin/bash", "usr/bin/sh"];
17 +
18 +
const MAGIC: u16 = 0xEF53;
19 +
const ROOT_INODE: u32 = 2;
20 +
const INCOMPAT_64BIT: u32 = 0x80;
21 +
const EXTENTS_FL: u32 = 0x80000;
22 +
const INLINE_DATA_FL: u32 = 0x1000_0000;
23 +
const EXTENT_MAGIC: u16 = 0xF30A;
24 +
/// More symlinks than this on one path is a loop.
25 +
const MAX_LINKS: u32 = 40;
26 +
/// The largest directory read, in bytes.
27 +
const MAX_DIR: u64 = 64 << 20;
28 +
29 +
/// The architecture an ELF header names (`x86_64`, `aarch64`, `riscv64`,
30 +
/// …), as `std::env::consts::ARCH` spells it; `None` for anything else.
31 +
pub fn elf_arch(header: &[u8]) -> Option<&'static str> {
32 +
    if header.len() < 20 || &header[..4] != b"\x7fELF" {
33 +
        return None;
34 +
    }
35 +
    let wide = header[4] == 2;
36 +
    let machine = if header[5] == 1 { u16::from_le_bytes([header[18], header[19]]) } else { u16::from_be_bytes([header[18], header[19]]) };
37 +
    Some(match machine {
38 +
        0x3E => "x86_64",
39 +
        0xB7 => "aarch64",
40 +
        0xF3 if wide => "riscv64",
41 +
        0xF3 => "riscv32",
42 +
        0x03 => "x86",
43 +
        0x28 => "arm",
44 +
        0x15 => "powerpc64",
45 +
        0x16 => "s390x",
46 +
        _ => return None,
47 +
    })
48 +
}
49 +
50 +
/// The architecture of the ext4 image read through `read` (fill the
51 +
/// buffer from that byte offset): its shell's, else the first program of
52 +
/// [`PROGRAMS`] it has. `None` when it is not ext4 or has none of them.
53 +
pub fn arch(read: &mut dyn FnMut(u64, &mut [u8]) -> std::io::Result<()>) -> Option<&'static str> {
54 +
    let fs = Fs::open(read)?;
55 +
    PROGRAMS.iter().find_map(|p| {
56 +
        let inode = fs.resolve(read, p)?;
57 +
        if inode.mode & 0xF000 != 0x8000 {
58 +
            return None;
59 +
        }
60 +
        let mut head = [0u8; 20];
61 +
        fs.read_file(read, &inode, &mut head)?;
62 +
        elf_arch(&head)
63 +
    })
64 +
}
65 +
66 +
/// The architecture of the ext4 image at `path` ([`arch`]).
67 +
pub fn arch_of_file(path: &Path) -> Option<&'static str> {
68 +
    let mut f = std::fs::File::open(path).ok()?;
69 +
    arch(&mut |at: u64, buf: &mut [u8]| {
70 +
        f.seek(SeekFrom::Start(at))?;
71 +
        f.read_exact(buf)
72 +
    })
73 +
}
74 +
75 +
struct Fs {
76 +
    block: u64,
77 +
    first_data_block: u64,
78 +
    inodes_per_group: u32,
79 +
    inode_size: u64,
80 +
    desc_size: u64,
81 +
    wide: bool,
82 +
    groups: u32,
83 +
}
84 +
85 +
struct Inode {
86 +
    mode: u16,
87 +
    size: u64,
88 +
    flags: u32,
89 +
    block: [u8; 60],
90 +
}
91 +
92 +
fn u16_at(b: &[u8], at: usize) -> u16 {
93 +
    u16::from_le_bytes([b[at], b[at + 1]])
94 +
}
95 +
96 +
fn u32_at(b: &[u8], at: usize) -> u32 {
97 +
    u32::from_le_bytes([b[at], b[at + 1], b[at + 2], b[at + 3]])
98 +
}
99 +
100 +
type ReadFn<'a> = dyn FnMut(u64, &mut [u8]) -> std::io::Result<()> + 'a;
101 +
102 +
impl Fs {
103 +
    fn open(read: &mut ReadFn) -> Option<Fs> {
104 +
        let mut sb = [0u8; 1024];
105 +
        read(1024, &mut sb).ok()?;
106 +
        if u16_at(&sb, 0x38) != MAGIC {
107 +
            return None;
108 +
        }
109 +
        let log = u32_at(&sb, 0x18);
110 +
        if log > 6 {
111 +
            return None;
112 +
        }
113 +
        let block = 1024u64 << log;
114 +
        let inode_size = if u32_at(&sb, 0x4C) == 0 { 128 } else { u16_at(&sb, 0x58) as u64 };
115 +
        let wide = u32_at(&sb, 0x60) & INCOMPAT_64BIT != 0;
116 +
        let desc_size = if wide { (u16_at(&sb, 0xFE) as u64).max(64) } else { 32 };
117 +
        let inodes_per_group = u32_at(&sb, 0x28);
118 +
        let inodes = u32_at(&sb, 0);
119 +
        if !(128..=1024).contains(&inode_size) || desc_size > 1024 || inodes_per_group == 0 {
120 +
            return None;
121 +
        }
122 +
        Some(Fs {
123 +
            block,
124 +
            first_data_block: u32_at(&sb, 0x14) as u64,
125 +
            inodes_per_group,
126 +
            inode_size,
127 +
            desc_size,
128 +
            wide,
129 +
            groups: inodes.div_ceil(inodes_per_group),
130 +
        })
131 +
    }
132 +
133 +
    fn inode(&self, read: &mut ReadFn, n: u32) -> Option<Inode> {
134 +
        let (group, index) = ((n.checked_sub(1)?) / self.inodes_per_group, (n - 1) % self.inodes_per_group);
135 +
        if group >= self.groups {
136 +
            return None;
137 +
        }
138 +
        let mut desc = [0u8; 64];
139 +
        let at = (self.first_data_block + 1) * self.block + group as u64 * self.desc_size;
140 +
        let len = if self.wide { 64 } else { 32 };
141 +
        read(at, &mut desc[..len]).ok()?;
142 +
        let hi = if self.wide { u32_at(&desc, 0x28) as u64 } else { 0 };
143 +
        let table = (hi << 32) | u32_at(&desc, 8) as u64;
144 +
        let mut raw = [0u8; 160];
145 +
        read(table * self.block + index as u64 * self.inode_size, &mut raw[..128]).ok()?;
146 +
        let mut block = [0u8; 60];
147 +
        block.copy_from_slice(&raw[0x28..0x28 + 60]);
148 +
        Some(Inode {
149 +
            mode: u16_at(&raw, 0),
150 +
            size: ((u32_at(&raw, 0x6C) as u64) << 32) | u32_at(&raw, 4) as u64,
151 +
            flags: u32_at(&raw, 0x20),
152 +
            block,
153 +
        })
154 +
    }
155 +
156 +
    /// The physical block holding logical block `n` of `inode`; `None` for
157 +
    /// a hole or an extent not yet written.
158 +
    fn map(&self, read: &mut ReadFn, inode: &Inode, n: u64) -> Option<u64> {
159 +
        if inode.flags & EXTENTS_FL != 0 {
160 +
            let mut node = inode.block.to_vec();
161 +
            for _ in 0..6 {
162 +
                if u16_at(&node, 0) != EXTENT_MAGIC {
163 +
                    return None;
164 +
                }
165 +
                let entries = (u16_at(&node, 2) as usize).min((node.len() - 12) / 12);
166 +
                let depth = u16_at(&node, 6);
167 +
                let entry = |i: usize| &node[12 + 12 * i..24 + 12 * i];
168 +
                if depth == 0 {
169 +
                    return (0..entries).map(entry).find_map(|e| {
170 +
                        let (first, len) = (u32_at(e, 0) as u64, u16_at(e, 4) as u64);
171 +
                        let start = ((u16_at(e, 6) as u64) << 32) | u32_at(e, 8) as u64;
172 +
                        (len <= 32768 && n >= first && n < first + len).then_some(start + (n - first))
173 +
                    });
174 +
                }
175 +
                let next = (0..entries).map(entry).take_while(|e| u32_at(e, 0) as u64 <= n).last()?;
176 +
                let child = ((u16_at(next, 8) as u64) << 32) | u32_at(next, 4) as u64;
177 +
                let mut buf = vec![0u8; self.block as usize];
178 +
                read(child * self.block, &mut buf).ok()?;
179 +
                node = buf;
180 +
            }
181 +
            return None;
182 +
        }
183 +
        // The old block map: twelve direct blocks, then one and two levels
184 +
        // of indirection.
185 +
        let per = self.block / 4;
186 +
        let ptr = |i: usize| u32_at(&inode.block, i * 4) as u64;
187 +
        let mut indirect = |table: u64, i: u64| -> Option<u64> {
188 +
            let mut b = [0u8; 4];
189 +
            read(table * self.block + i * 4, &mut b).ok()?;
190 +
            Some(u32::from_le_bytes(b) as u64).filter(|&p| p != 0)
191 +
        };
192 +
        let p = if n < 12 {
193 +
            ptr(n as usize)
194 +
        } else if n < 12 + per {
195 +
            indirect(ptr(12), n - 12)?
196 +
        } else if n < 12 + per + per * per {
197 +
            let k = n - 12 - per;
198 +
            let mid = indirect(ptr(13), k / per)?;
199 +
            indirect(mid, k % per)?
200 +
        } else {
201 +
            return None;
202 +
        };
203 +
        (p != 0).then_some(p)
204 +
    }
205 +
206 +
    /// Fill `buf` from the start of `inode`'s data.
207 +
    fn read_file(&self, read: &mut ReadFn, inode: &Inode, buf: &mut [u8]) -> Option<()> {
208 +
        if inode.flags & INLINE_DATA_FL != 0 || (buf.len() as u64) > inode.size.min(self.block) {
209 +
            return None;
210 +
        }
211 +
        let at = self.map(read, inode, 0)?;
212 +
        read(at * self.block, buf).ok()
213 +
    }
214 +
215 +
    /// Where a symlink points.
216 +
    fn link_target(&self, read: &mut ReadFn, inode: &Inode) -> Option<String> {
217 +
        if inode.size == 0 || inode.size > 4096 {
218 +
            return None;
219 +
        }
220 +
        let bytes = if inode.flags & (EXTENTS_FL | INLINE_DATA_FL) == 0 && inode.size < 60 {
221 +
            inode.block[..inode.size as usize].to_vec()
222 +
        } else {
223 +
            let mut b = vec![0u8; inode.size as usize];
224 +
            self.read_file(read, inode, &mut b)?;
225 +
            b
226 +
        };
227 +
        String::from_utf8(bytes).ok()
228 +
    }
229 +
230 +
    /// The inode named `name` in directory `dir`.
231 +
    fn lookup(&self, read: &mut ReadFn, dir: &Inode, name: &str) -> Option<u32> {
232 +
        if dir.mode & 0xF000 != 0x4000 || dir.flags & INLINE_DATA_FL != 0 || dir.size > MAX_DIR {
233 +
            return None;
234 +
        }
235 +
        let mut buf = vec![0u8; self.block as usize];
236 +
        for n in 0..dir.size.div_ceil(self.block) {
237 +
            let Some(at) = self.map(read, dir, n) else { continue };
238 +
            read(at * self.block, &mut buf).ok()?;
239 +
            let mut off = 0usize;
240 +
            while off + 8 <= buf.len() {
241 +
                let (ino, rec, len) = (u32_at(&buf, off), u16_at(&buf, off + 4) as usize, buf[off + 6] as usize);
242 +
                if rec < 8 || off + rec > buf.len() {
243 +
                    break;
244 +
                }
245 +
                if ino != 0 && off + 8 + len <= buf.len() && &buf[off + 8..off + 8 + len] == name.as_bytes() {
246 +
                    return Some(ino);
247 +
                }
248 +
                off += rec;
249 +
            }
250 +
        }
251 +
        None
252 +
    }
253 +
254 +
    /// The inode at `path` from the root, following every symlink.
255 +
    fn resolve(&self, read: &mut ReadFn, path: &str) -> Option<Inode> {
256 +
        let mut todo: Vec<String> = path.split('/').filter(|p| !p.is_empty()).rev().map(String::from).collect();
257 +
        let mut dirs = vec![ROOT_INODE];
258 +
        let mut links = 0;
259 +
        while let Some(name) = todo.pop() {
260 +
            match name.as_str() {
261 +
                "." => continue,
262 +
                ".." => {
263 +
                    if dirs.len() > 1 {
264 +
                        dirs.pop();
265 +
                    }
266 +
                    continue;
267 +
                }
268 +
                _ => {}
269 +
            }
270 +
            let dir = self.inode(read, *dirs.last()?)?;
271 +
            let n = self.lookup(read, &dir, &name)?;
272 +
            let inode = self.inode(read, n)?;
273 +
            if inode.mode & 0xF000 == 0xA000 {
274 +
                links += 1;
275 +
                if links > MAX_LINKS {
276 +
                    return None;
277 +
                }
278 +
                let target = self.link_target(read, &inode)?;
279 +
                if target.starts_with('/') {
280 +
                    dirs.truncate(1);
281 +
                }
282 +
                todo.extend(target.split('/').filter(|p| !p.is_empty()).rev().map(String::from));
283 +
                continue;
284 +
            }
285 +
            dirs.push(n);
286 +
        }
287 +
        self.inode(read, *dirs.last()?)
288 +
    }
289 +
}
290 +
291 +
#[cfg(test)]
292 +
mod tests {
293 +
    use super::*;
294 +
    use std::process::Command;
295 +
296 +
    /// The first 64 bytes of an ELF program for `machine`, 64-bit little
297 +
    /// endian.
298 +
    fn elf(machine: u16) -> Vec<u8> {
299 +
        let mut h = vec![0u8; 64];
300 +
        h[..4].copy_from_slice(b"\x7fELF");
301 +
        h[4] = 2;
302 +
        h[5] = 1;
303 +
        h[6] = 1;
304 +
        h[16] = 3;
305 +
        h[18..20].copy_from_slice(&machine.to_le_bytes());
306 +
        h
307 +
    }
308 +
309 +
    /// An ext4 image of `tree`, as `flow rootfs import` makes one; `None`
310 +
    /// where mkfs.ext4 is missing.
311 +
    fn image(tree: &Path, out: &Path, size: &str, extra: &[&str]) -> Option<()> {
312 +
        let status = Command::new("mkfs.ext4")
313 +
            .args(["-q", "-F", "-L", "rootfs", "-E", "root_owner=0:0"])
314 +
            .args(extra)
315 +
            .arg("-d")
316 +
            .arg(tree)
317 +
            .arg(out)
318 +
            .arg(size)
319 +
            .status()
320 +
            .ok()?;
321 +
        assert!(status.success(), "mkfs.ext4 failed");
322 +
        Some(())
323 +
    }
324 +
325 +
    #[test]
326 +
    fn elf_headers_name_their_architecture() {
327 +
        assert_eq!(elf_arch(&elf(0x3E)), Some("x86_64"));
328 +
        assert_eq!(elf_arch(&elf(0xF3)), Some("riscv64"));
329 +
        assert_eq!(elf_arch(&elf(0xB7)), Some("aarch64"));
330 +
        assert_eq!(elf_arch(b"#!/bin/sh\necho hello\n"), None);
331 +
        assert_eq!(elf_arch(&elf(0x9999)), None);
332 +
    }
333 +
334 +
    #[test]
335 +
    fn an_images_shell_names_its_architecture_through_symlinks() {
336 +
        let dir = tempfile::tempdir().unwrap();
337 +
        // A merged /usr, as Debian and Ubuntu have it: bin -> usr/bin, and
338 +
        // sh -> dash; a riscv64 shell.
339 +
        let tree = dir.path().join("tree");
340 +
        std::fs::create_dir_all(tree.join("usr/bin")).unwrap();
341 +
        std::fs::write(tree.join("usr/bin/dash"), elf(0xF3)).unwrap();
342 +
        std::os::unix::fs::symlink("dash", tree.join("usr/bin/sh")).unwrap();
343 +
        std::os::unix::fs::symlink("usr/bin", tree.join("bin")).unwrap();
344 +
        // A full directory, so lookups cross blocks.
345 +
        for i in 0..400 {
346 +
            std::fs::write(tree.join(format!("usr/bin/tool-with-a-long-name-{i:04}")), b"#!/bin/sh\n").unwrap();
347 +
        }
348 +
        let img = dir.path().join("rootfs.ext4");
349 +
        if image(&tree, &img, "8M", &[]).is_none() {
350 +
            eprintln!("mkfs.ext4 not found: skipped");
351 +
            return;
352 +
        }
353 +
        assert_eq!(arch_of_file(&img), Some("riscv64"));
354 +
        // 1 KiB blocks, and an absolute link through a deeper directory.
355 +
        let tree2 = dir.path().join("tree2");
356 +
        std::fs::create_dir_all(tree2.join("bin")).unwrap();
357 +
        std::fs::create_dir_all(tree2.join("opt/busybox/bin")).unwrap();
358 +
        std::fs::write(tree2.join("opt/busybox/bin/busybox"), elf(0x3E)).unwrap();
359 +
        std::os::unix::fs::symlink("/opt/busybox/bin/../bin/busybox", tree2.join("bin/busybox")).unwrap();
360 +
        let img2 = dir.path().join("small.ext4");
361 +
        image(&tree2, &img2, "8M", &["-b", "1024"]).unwrap();
362 +
        assert_eq!(arch_of_file(&img2), Some("x86_64"), "no sh: busybox decides");
363 +
    }
364 +
365 +
    #[test]
366 +
    fn what_is_not_an_image_or_has_no_shell_answers_nothing() {
367 +
        let dir = tempfile::tempdir().unwrap();
368 +
        let junk = dir.path().join("junk");
369 +
        std::fs::write(&junk, vec![0x5Au8; 64 << 10]).unwrap();
370 +
        assert_eq!(arch_of_file(&junk), None);
371 +
        assert_eq!(arch_of_file(&dir.path().join("missing")), None);
372 +
        let tree = dir.path().join("tree");
373 +
        std::fs::create_dir_all(tree.join("bin")).unwrap();
374 +
        std::fs::write(tree.join("bin/sh"), b"#!/not/an/elf\n").unwrap();
375 +
        std::os::unix::fs::symlink("loop", tree.join("bin/loop")).unwrap();
376 +
        std::os::unix::fs::symlink("loop", tree.join("bin/busybox")).unwrap();
377 +
        let img = dir.path().join("rootfs.ext4");
378 +
        if image(&tree, &img, "8M", &[]).is_none() {
379 +
            return;
380 +
        }
381 +
        assert_eq!(arch_of_file(&img), None, "a script shell and a symlink loop name nothing");
382 +
    }
383 +
}
added crates/jet/src/swarm.rs +500 −0
1 +
//! The swarm (oxygenflow#47, docs/jetstream.md §8.1): runners fetch image
2 +
//! chunks from each other as well as from the hub.
3 +
//!
4 +
//! This module holds what both ends agree on and what can be tested
5 +
//! without a network: the have-map a peer offers ([`Bits`]), the ticket the
6 +
//! hub signs to admit a runner to its peers ([`Ticket`]), and the order in
7 +
//! which a fetcher asks its sources for chunks ([`Picker`]): rarest first,
8 +
//! the hub only for what no peer has, and an endgame at the close.
9 +
10 +
use oxy_core::hash::KeyId;
11 +
use oxy_core::identity::{verify_challenge, Identity, SecretKey, Signature};
12 +
use serde::{Deserialize, Serialize};
13 +
use std::collections::HashMap;
14 +
use std::time::{Duration, Instant};
15 +
16 +
/// How long a ticket admits a runner to its peers.
17 +
pub const TICKET_LIFE_MS: u64 = 15 * 60 * 1000;
18 +
/// Peers make no progress for this long and the hub is asked for anything.
19 +
pub const STALL: Duration = Duration::from_secs(2);
20 +
/// The most neighbouring chunks a fetcher takes in order where rarity does
21 +
/// not decide (about 5.6 MB of an image's 88 KB chunks); a small image's
22 +
/// runs are shorter, so it still has a few hundred to spread runners over.
23 +
const RUN: usize = 64;
24 +
/// Most sources asked for the same chunk in the endgame.
25 +
const ENDGAME_COPIES: u8 = 2;
26 +
27 +
/// One bit per chunk of a manifest's needed list, in its order.
28 +
#[derive(Clone, Debug, Default, PartialEq, Eq)]
29 +
pub struct Bits {
30 +
    words: Vec<u64>,
31 +
    len: usize,
32 +
}
33 +
34 +
impl Bits {
35 +
    pub fn new(len: usize) -> Bits {
36 +
        Bits { words: vec![0; len.div_ceil(64)], len }
37 +
    }
38 +
39 +
    /// Every bit set: a peer that holds the whole image.
40 +
    pub fn full(len: usize) -> Bits {
41 +
        let mut b = Bits::new(len);
42 +
        (0..len).for_each(|i| b.set(i));
43 +
        b
44 +
    }
45 +
46 +
    pub fn len(&self) -> usize {
47 +
        self.len
48 +
    }
49 +
50 +
    pub fn is_empty(&self) -> bool {
51 +
        self.len == 0
52 +
    }
53 +
54 +
    pub fn get(&self, i: usize) -> bool {
55 +
        i < self.len && self.words[i / 64] & (1 << (i % 64)) != 0
56 +
    }
57 +
58 +
    pub fn set(&mut self, i: usize) {
59 +
        if i < self.len {
60 +
            self.words[i / 64] |= 1 << (i % 64);
61 +
        }
62 +
    }
63 +
64 +
    pub fn clear(&mut self, i: usize) {
65 +
        if i < self.len {
66 +
            self.words[i / 64] &= !(1 << (i % 64));
67 +
        }
68 +
    }
69 +
70 +
    pub fn count(&self) -> usize {
71 +
        self.words.iter().map(|w| w.count_ones() as usize).sum()
72 +
    }
73 +
74 +
    /// Little-endian bytes, as `GET …/have` sends them.
75 +
    pub fn to_bytes(&self) -> Vec<u8> {
76 +
        let mut out: Vec<u8> = self.words.iter().flat_map(|w| w.to_le_bytes()).collect();
77 +
        out.truncate(self.len.div_ceil(8));
78 +
        out
79 +
    }
80 +
81 +
    /// The bits of a `have` answer for a list of `len` chunks; bits past
82 +
    /// `len` are ignored, and a short answer means the rest are unset.
83 +
    pub fn from_bytes(len: usize, bytes: &[u8]) -> Bits {
84 +
        let mut b = Bits::new(len);
85 +
        for (i, byte) in bytes.iter().enumerate().take(len.div_ceil(8)) {
86 +
            b.words[i / 8] |= (*byte as u64) << (8 * (i % 8));
87 +
        }
88 +
        if let Some(last) = b.words.last_mut() {
89 +
            if !len.is_multiple_of(64) {
90 +
                *last &= (1u64 << (len % 64)) - 1;
91 +
            }
92 +
        }
93 +
        b
94 +
    }
95 +
}
96 +
97 +
/// What the hub signs to admit a runner to its peers.
98 +
#[derive(Clone, Debug, PartialEq, Eq, Serialize, Deserialize)]
99 +
pub struct Ticket {
100 +
    /// The hub's key, hex: a peer accepts only its own hub's tickets.
101 +
    pub hub: String,
102 +
    pub runner: i64,
103 +
    pub name: String,
104 +
    pub site: String,
105 +
    pub expires_ms: u64,
106 +
}
107 +
108 +
impl Ticket {
109 +
    /// The ticket as text: its JSON and the hub's signature over it, hex,
110 +
    /// joined by a dot.
111 +
    pub fn seal(&self, key: &KeyId, secret: &SecretKey) -> String {
112 +
        let body = serde_json::to_vec(self).expect("a ticket serialises");
113 +
        let sig = secret.sign_challenge(key, &body).expect("a ticket is longer than 16 bytes");
114 +
        format!("{}.{}", hex::encode(&body), hex::encode(Signature::encode_list(&[sig])))
115 +
    }
116 +
117 +
    /// Read and check a ticket: signed by `hub`, not expired at `now_ms`.
118 +
    pub fn open(text: &str, hub: &Identity, now_ms: u64) -> Result<Ticket, String> {
119 +
        let (body, sig) = text.trim().split_once('.').ok_or("not a ticket")?;
120 +
        let body = hex::decode(body).map_err(|_| "not a ticket")?;
121 +
        let sigs = hex::decode(sig).ok().and_then(|b| Signature::decode_list(&b).ok()).ok_or("not a ticket's signature")?;
122 +
        let sig = sigs.first().ok_or("an unsigned ticket")?;
123 +
        verify_challenge(hub, sig, &body).map_err(|_| "the ticket is not signed by this runner's hub")?;
124 +
        let t: Ticket = serde_json::from_slice(&body).map_err(|_| "not a ticket")?;
125 +
        if t.hub != hub.hash().to_hex() {
126 +
            return Err("the ticket names another hub".into());
127 +
        }
128 +
        if t.expires_ms <= now_ms {
129 +
            return Err("the ticket has expired".into());
130 +
        }
131 +
        Ok(t)
132 +
    }
133 +
}
134 +
135 +
/// Where a chunk comes from: the hub, or a peer by its index.
136 +
#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]
137 +
pub enum Source {
138 +
    Hub,
139 +
    Peer(usize),
140 +
}
141 +
142 +
/// What a source should do next.
143 +
#[derive(Debug, PartialEq, Eq)]
144 +
pub enum Pick {
145 +
    /// Ask for this chunk (an index into the needed list).
146 +
    Chunk(usize),
147 +
    /// Nothing for this source now; ask again shortly.
148 +
    Wait,
149 +
    /// Every chunk is here.
150 +
    Done,
151 +
}
152 +
153 +
/// The order a fetcher asks its sources for chunks (docs/jetstream.md
154 +
/// §8.1): the rarest chunk a peer holds first; the hub for chunks no peer
155 +
/// holds, or for anything once the peers stall; and in the endgame, a
156 +
/// second source for a chunk already asked for.
157 +
pub struct Picker {
158 +
    have: Bits,
159 +
    asked: Vec<u8>,
160 +
    /// The source first asked for each chunk in flight: the endgame asks
161 +
    /// a different one.
162 +
    first: Vec<Option<Source>>,
163 +
    /// Each peer's map: what it has, and what it is fetching now.
164 +
    peers: HashMap<usize, (Bits, Bits)>,
165 +
    /// Missing chunks, rarest first; rebuilt when the peers' maps change.
166 +
    order: Vec<usize>,
167 +
    /// Where each chunk sits in `order`.
168 +
    pos: Vec<usize>,
169 +
    /// No chunk before this place in `order` can be picked fresh: each is
170 +
    /// here or asked for. Moved back when one fails.
171 +
    skip: usize,
172 +
    /// How many peers hold each chunk, as of the last rebuild.
173 +
    holders: Vec<u16>,
174 +
    /// Whether some peer is fetching each chunk now: the hub is not asked
175 +
    /// for it, the peer will have it shortly.
176 +
    coming: Vec<bool>,
177 +
    progress: Instant,
178 +
    seed: u64,
179 +
}
180 +
181 +
impl Picker {
182 +
    /// A picker for `len` chunks, of which `have` are already here.
183 +
    pub fn new(len: usize, have: Bits, seed: u64) -> Picker {
184 +
        let mut p = Picker {
185 +
            have,
186 +
            asked: vec![0; len],
187 +
            first: vec![None; len],
188 +
            peers: HashMap::new(),
189 +
            order: Vec::new(),
190 +
            pos: vec![0; len],
191 +
            skip: 0,
192 +
            holders: vec![0; len],
193 +
            coming: vec![false; len],
194 +
            progress: Instant::now(),
195 +
            seed: seed | 1,
196 +
        };
197 +
        p.reorder();
198 +
        p
199 +
    }
200 +
201 +
    pub fn len(&self) -> usize {
202 +
        self.asked.len()
203 +
    }
204 +
205 +
    pub fn is_empty(&self) -> bool {
206 +
        self.asked.is_empty()
207 +
    }
208 +
209 +
    pub fn missing(&self) -> usize {
210 +
        self.len() - self.have.count()
211 +
    }
212 +
213 +
    pub fn have(&self) -> &Bits {
214 +
        &self.have
215 +
    }
216 +
217 +
    /// A peer's have-map, new or refreshed.
218 +
    pub fn peer(&mut self, peer: usize, bits: Bits) {
219 +
        let none = Bits::new(bits.len());
220 +
        self.peer_fetching(peer, bits, none);
221 +
    }
222 +
223 +
    /// A peer's have-map and the chunks it is fetching now.
224 +
    pub fn peer_fetching(&mut self, peer: usize, have: Bits, fetching: Bits) {
225 +
        self.peers.insert(peer, (have, fetching));
226 +
        self.reorder();
227 +
    }
228 +
229 +
    /// A peer that failed: it is asked for nothing more.
230 +
    pub fn drop_peer(&mut self, peer: usize) {
231 +
        self.peers.remove(&peer);
232 +
        self.reorder();
233 +
    }
234 +
235 +
    fn next_rand(&mut self) -> u64 {
236 +
        // xorshift64: enough to spread runners that start together.
237 +
        self.seed ^= self.seed << 13;
238 +
        self.seed ^= self.seed >> 7;
239 +
        self.seed ^= self.seed << 17;
240 +
        self.seed
241 +
    }
242 +
243 +
    /// Rarest first; among chunks as rare as each other, runs of up to
244 +
    /// [`RUN`] neighbours in order, the runs shuffled. A peer reads them out of a
245 +
    /// whole image, and on a spinning disk scattered reads cost a seek each
246 +
    /// (Earth sent 2.3 GiB in 20 minutes, picked one chunk at a time at
247 +
    /// random); runners that start together still spread over the runs.
248 +
    fn reorder(&mut self) {
249 +
        let run = (self.len() / 256).clamp(1, RUN);
250 +
        let shuffle: Vec<u64> = (0..self.len().div_ceil(run)).map(|_| self.next_rand()).collect();
251 +
        let mut keyed: Vec<(u16, u64, usize)> = Vec::new();
252 +
        for i in 0..self.len() {
253 +
            self.holders[i] = self.peers.values().filter(|(b, _)| b.get(i)).count().min(u16::MAX as usize) as u16;
254 +
            self.coming[i] = self.peers.values().any(|(_, f)| f.get(i));
255 +
            if !self.have.get(i) {
256 +
                keyed.push((self.holders[i], shuffle[i / run], i));
257 +
            }
258 +
        }
259 +
        keyed.sort_unstable();
260 +
        self.order = keyed.into_iter().map(|(_, _, i)| i).collect();
261 +
        for (at, &i) in self.order.iter().enumerate() {
262 +
            self.pos[i] = at;
263 +
        }
264 +
        self.skip = 0;
265 +
    }
266 +
267 +
    /// The next chunk for `src` to fetch.
268 +
    pub fn pick(&mut self, src: Source) -> Pick {
269 +
        if self.missing() == 0 {
270 +
            return Pick::Done;
271 +
        }
272 +
        let holds = |i: usize| match src {
273 +
            Source::Hub => true,
274 +
            Source::Peer(p) => self.peers.get(&p).is_some_and(|(b, _)| b.get(i)),
275 +
        };
276 +
        let stalled = self.progress.elapsed() >= STALL;
277 +
        // Not yet asked for: a peer takes the rarest it holds; the hub
278 +
        // takes what no peer holds, or anything once the peers stall.
279 +
        while self.skip < self.order.len() && (self.have.get(self.order[self.skip]) || self.asked[self.order[self.skip]] > 0) {
280 +
            self.skip += 1;
281 +
        }
282 +
        let fresh = self.order[self.skip..].iter().copied().find(|&i| {
283 +
            !self.have.get(i)
284 +
                && self.asked[i] == 0
285 +
                && holds(i)
286 +
                && (src != Source::Hub || stalled || self.peers.is_empty() || (self.holders[i] == 0 && !self.coming[i]))
287 +
        });
288 +
        if let Some(i) = fresh {
289 +
            self.asked[i] += 1;
290 +
            self.first[i] = Some(src);
291 +
            return Pick::Chunk(i);
292 +
        }
293 +
        // The endgame: everything missing is asked for already. The hub
294 +
        // joins it only once the peers stall.
295 +
        if src == Source::Hub && !stalled && !self.peers.is_empty() {
296 +
            return Pick::Wait;
297 +
        }
298 +
        let mut left = self.order.iter().copied().filter(|&i| !self.have.get(i));
299 +
        if left.clone().all(|i| self.asked[i] > 0) {
300 +
            if let Some(i) = left.find(|&i| self.asked[i] < ENDGAME_COPIES && self.first[i] != Some(src) && holds(i)) {
301 +
                self.asked[i] += 1;
302 +
                return Pick::Chunk(i);
303 +
            }
304 +
        }
305 +
        Pick::Wait
306 +
    }
307 +
308 +
    /// Chunk `i` arrived and was checked.
309 +
    pub fn got(&mut self, i: usize) {
310 +
        self.have.set(i);
311 +
        self.asked[i] = self.asked[i].saturating_sub(1);
312 +
        self.first[i] = None;
313 +
        self.progress = Instant::now();
314 +
    }
315 +
316 +
    /// Chunk `i` did not come: it can be asked for again.
317 +
    pub fn failed(&mut self, i: usize) {
318 +
        self.asked[i] = self.asked[i].saturating_sub(1);
319 +
        if self.asked[i] == 0 {
320 +
            self.first[i] = None;
321 +
            if !self.have.get(i) {
322 +
                self.skip = self.skip.min(self.pos[i]);
323 +
            }
324 +
        }
325 +
    }
326 +
}
327 +
328 +
#[cfg(test)]
329 +
mod tests {
330 +
    #[test]
331 +
    fn from_one_peer_neighbouring_chunks_come_in_order() {
332 +
        let n = RUN * 256;
333 +
        let mut p = Picker::new(n, Bits::new(n), 3);
334 +
        p.peer(0, Bits::full(n));
335 +
        let first: Vec<usize> = (0..RUN)
336 +
            .map(|_| match p.pick(Source::Peer(0)) {
337 +
                Pick::Chunk(i) => i,
338 +
                other => panic!("{other:?}"),
339 +
            })
340 +
            .collect();
341 +
        assert_eq!(first[0] % RUN, 0, "a run starts at its first chunk: {first:?}");
342 +
        assert!(first.windows(2).all(|w| w[1] == w[0] + 1), "one run, in order: {first:?}");
343 +
        // A failed chunk is asked for again before going further.
344 +
        p.failed(first[3]);
345 +
        assert_eq!(p.pick(Source::Peer(0)), Pick::Chunk(first[3]));
346 +
    }
347 +
348 +
    /// A fetch the size of the 14 GB Rust image (142,244 chunks) from one
349 +
    /// peer that holds them all: how long picking takes.
350 +
    #[test]
351 +
    #[ignore]
352 +
    fn picking_a_big_image_from_one_peer_is_quick() {
353 +
        let n = 142_244;
354 +
        let mut p = Picker::new(n, Bits::new(n), 7);
355 +
        p.peer(0, Bits::full(n));
356 +
        let started = std::time::Instant::now();
357 +
        let mut in_flight = std::collections::VecDeque::new();
358 +
        loop {
359 +
            match p.pick(Source::Peer(0)) {
360 +
                Pick::Chunk(i) => {
361 +
                    in_flight.push_back(i);
362 +
                    if in_flight.len() >= 4 {
363 +
                        p.got(in_flight.pop_front().unwrap());
364 +
                    }
365 +
                }
366 +
                Pick::Wait => p.got(in_flight.pop_front().unwrap()),
367 +
                Pick::Done => break,
368 +
            }
369 +
        }
370 +
        eprintln!("picked {n} chunks in {:?}", started.elapsed());
371 +
    }
372 +
373 +
    use super::*;
374 +
375 +
    #[test]
376 +
    fn bits_travel_as_bytes() {
377 +
        let mut b = Bits::new(70);
378 +
        [0, 9, 63, 64, 69].iter().for_each(|&i| b.set(i));
379 +
        let back = Bits::from_bytes(70, &b.to_bytes());
380 +
        assert_eq!(back, b);
381 +
        assert_eq!(b.to_bytes().len(), 9);
382 +
        assert_eq!(Bits::from_bytes(10, &[0xff, 0xff, 0xff]).count(), 10, "bits past the list are ignored");
383 +
        assert_eq!(Bits::from_bytes(10, &[]).count(), 0, "a short answer leaves the rest unset");
384 +
        assert_eq!(Bits::full(130).count(), 130);
385 +
    }
386 +
387 +
    #[test]
388 +
    fn a_ticket_opens_only_with_its_hub_and_in_time() {
389 +
        let sk = SecretKey::generate();
390 +
        let hub = sk.identity("hub", "hub@example", 1);
391 +
        let t = Ticket { hub: hub.hash().to_hex(), runner: 7, name: "earth-1".into(), site: "home".into(), expires_ms: 10_000 };
392 +
        let text = t.seal(&hub.hash(), &sk);
393 +
        assert_eq!(Ticket::open(&text, &hub, 9_999).unwrap(), t);
394 +
        assert!(Ticket::open(&text, &hub, 10_000).unwrap_err().contains("expired"));
395 +
        let other_sk = SecretKey::generate();
396 +
        let other = other_sk.identity("other", "o@example", 1);
397 +
        assert!(Ticket::open(&text, &other, 1).is_err(), "another hub's key");
398 +
        let mut forged = text.clone();
399 +
        forged.replace_range(2..4, "ff");
400 +
        assert!(Ticket::open(&forged, &hub, 1).is_err(), "a changed body");
401 +
        assert!(Ticket::open("nonsense", &hub, 1).is_err());
402 +
    }
403 +
404 +
    #[test]
405 +
    fn peers_are_asked_for_the_rarest_chunk_they_hold() {
406 +
        let mut p = Picker::new(4, Bits::new(4), 42);
407 +
        let mut a = Bits::new(4);
408 +
        [0, 1, 2].iter().for_each(|&i| a.set(i));
409 +
        let mut b = Bits::new(4);
410 +
        [0, 1].iter().for_each(|&i| b.set(i));
411 +
        p.peer(0, a);
412 +
        p.peer(1, b);
413 +
        // Chunk 2 only peer 0 holds; chunk 3 nobody does.
414 +
        assert_eq!(p.pick(Source::Peer(0)), Pick::Chunk(2));
415 +
        assert_eq!(p.pick(Source::Hub), Pick::Chunk(3), "the hub seeds what no peer holds");
416 +
        assert_eq!(p.pick(Source::Hub), Pick::Wait, "and leaves the rest to the peers");
417 +
        let first = p.pick(Source::Peer(1));
418 +
        assert!(matches!(first, Pick::Chunk(0) | Pick::Chunk(1)));
419 +
    }
420 +
421 +
    #[test]
422 +
    fn the_hub_is_not_asked_for_what_a_peer_is_fetching() {
423 +
        let mut p = Picker::new(2, Bits::new(2), 4);
424 +
        let mut fetching = Bits::new(2);
425 +
        fetching.set(0);
426 +
        p.peer_fetching(0, Bits::new(2), fetching);
427 +
        assert_eq!(p.pick(Source::Hub), Pick::Chunk(1));
428 +
        assert_eq!(p.pick(Source::Hub), Pick::Wait, "chunk 0 is coming from the peer");
429 +
        let mut have = Bits::new(2);
430 +
        have.set(0);
431 +
        p.peer(0, have);
432 +
        assert_eq!(p.pick(Source::Peer(0)), Pick::Chunk(0));
433 +
    }
434 +
435 +
    #[test]
436 +
    fn without_peers_the_hub_serves_everything() {
437 +
        let mut p = Picker::new(3, Bits::new(3), 1);
438 +
        let mut seen = Vec::new();
439 +
        while let Pick::Chunk(i) = p.pick(Source::Hub) {
440 +
            seen.push(i);
441 +
            p.got(i);
442 +
        }
443 +
        seen.sort();
444 +
        assert_eq!(seen, vec![0, 1, 2]);
445 +
        assert_eq!(p.pick(Source::Hub), Pick::Done);
446 +
    }
447 +
448 +
    #[test]
449 +
    fn the_endgame_never_asks_the_same_source_twice() {
450 +
        let mut p = Picker::new(1, Bits::new(1), 9);
451 +
        assert_eq!(p.pick(Source::Hub), Pick::Chunk(0));
452 +
        assert_eq!(p.pick(Source::Hub), Pick::Wait);
453 +
        p.peer(0, Bits::full(1));
454 +
        assert_eq!(p.pick(Source::Peer(0)), Pick::Chunk(0), "a peer may race the hub for it");
455 +
        assert_eq!(p.pick(Source::Hub), Pick::Wait, "but not the hub a peer, while the peers make progress");
456 +
    }
457 +
458 +
    #[test]
459 +
    fn the_endgame_asks_a_second_source_and_a_failure_frees_the_chunk() {
460 +
        let mut p = Picker::new(1, Bits::new(1), 3);
461 +
        p.peer(0, Bits::full(1));
462 +
        p.peer(1, Bits::full(1));
463 +
        assert_eq!(p.pick(Source::Peer(0)), Pick::Chunk(0));
464 +
        assert_eq!(p.pick(Source::Peer(1)), Pick::Chunk(0), "the endgame");
465 +
        assert_eq!(p.pick(Source::Peer(1)), Pick::Wait, "at most two copies");
466 +
        p.failed(0);
467 +
        p.failed(0);
468 +
        p.drop_peer(0);
469 +
        assert_eq!(p.pick(Source::Peer(1)), Pick::Chunk(0));
470 +
        p.got(0);
471 +
        assert_eq!(p.pick(Source::Peer(1)), Pick::Done);
472 +
    }
473 +
474 +
    #[test]
475 +
    fn chunks_already_here_are_never_asked_for() {
476 +
        let mut have = Bits::new(3);
477 +
        have.set(1);
478 +
        let mut p = Picker::new(3, have, 5);
479 +
        assert_eq!(p.missing(), 2);
480 +
        let mut seen = Vec::new();
481 +
        while let Pick::Chunk(i) = p.pick(Source::Hub) {
482 +
            seen.push(i);
483 +
            p.got(i);
484 +
        }
485 +
        assert!(!seen.contains(&1));
486 +
    }
487 +
488 +
    #[test]
489 +
    fn runners_starting_together_ask_the_hub_for_different_chunks() {
490 +
        let a: Vec<_> = {
491 +
            let mut p = Picker::new(64, Bits::new(64), 11);
492 +
            (0..8).filter_map(|_| if let Pick::Chunk(i) = p.pick(Source::Hub) { Some(i) } else { None }).collect()
493 +
        };
494 +
        let b: Vec<_> = {
495 +
            let mut p = Picker::new(64, Bits::new(64), 12);
496 +
            (0..8).filter_map(|_| if let Pick::Chunk(i) = p.pick(Source::Hub) { Some(i) } else { None }).collect()
497 +
        };
498 +
        assert_ne!(a, b);
499 +
    }
500 +
}
added docs/handover.md +271 −0
1 +
# Handover: jobs that outlive their runner
2 +
3 +
Proposed 2026-10-25, from a design conversation with Andy. Issues:
4 +
oxygenflow#75 (WASM tier), #76 (leases and drain), #77 (checkpoints),
5 +
#78 (hedging), #79 (trust classes), #80 (browser runners); the phone
6 +
runner is oxygenhub-android#8. Builds on Jetstream (docs/jetstream.md),
7 +
whose pools (hosts, Kubernetes, cloud) and requeue of a lost runner's job
8 +
this extends.
9 +
10 +
## 1. Why
11 +
12 +
Three situations, one problem: a job is tied to the runner that started
13 +
it.
14 +
15 +
- **The runner goes away.** A phone leaves the office Wi-Fi or comes off
16 +
  charge; a laptop sleeps; a spot VM is reclaimed with 30 seconds' notice;
17 +
  a Kubernetes node is drained. Today the job starts again from nothing
18 +
  elsewhere, or fails.
19 +
- **A better runner frees up.** A job went to the small runner B because
20 +
  the big runner A was busy. A finishes its own job when B is 10% through,
21 +
  then sits idle while B grinds through the other 90%.
22 +
- **The same work runs twice.** A requeued job redoes everything; two
23 +
  pipelines that build the same thing both build it.
24 +
25 +
Solving the first makes unreliable runners usable (phones, browsers,
26 +
spare machines, spot instances). Solving the second keeps a mixed fleet
27 +
busy. Together they are the case a company makes for running CI on the
28 +
hardware it already has: more use from it, and less waiting for
29 +
developers and agents.
30 +
31 +
## 2. What Flow already has
32 +
33 +
Most of the parts exist:
34 +
35 +
- **Jobs are the packets.** A job runs one `exec`; jobs form a graph and
36 +
  pass artifacts (§5 of the spec, oxygenflow#58). A run is already spread
37 +
  over runners: oxygenhub's run #439 built on earth-1, ran clippy on
38 +
  plex-1, dist on plex-1 and riscv64 on narvi-1.
39 +
- **Receipts already chain.** Each job's receipt (spec §6.1) is signed by
40 +
  its runner and names its inputs, which are other jobs' outputs, whose
41 +
  receipts name *their* runners. A run's provenance is already a graph of
42 +
  signed receipts from several runners.
43 +
- **A lost runner costs only its job.** Jetstream requeues it after a
44 +
  grace, and a spot reclaim notice requeues it at once (jetstream.md §11).
45 +
- **Placement can wait for a better runner** ("held for a better
46 +
  runner"), and the hub learns each job's duration per speed class
47 +
  (spec §11).
48 +
49 +
What is missing: a job that is *partly done* is lost when it moves; there
50 +
is no graceful way to leave; a better runner cannot take over; nothing
51 +
says how far to trust a runner; and phones and browsers cannot run jobs
52 +
at all (no WASM tier).
53 +
54 +
## 3. Decision: move progress, not processes
55 +
56 +
**No live migration of running VMs** (oxygenflow#33 suggested Firecracker
57 +
snapshots). A snapshot fixes the VM's shape: a job started with 2 vCPUs
58 +
on a small runner is restored with 2 vCPUs on the big one, since
59 +
Firecracker cannot add vCPUs to a running VM, so it gets no faster. It
60 +
moves gigabytes of memory, works only between hosts of the same
61 +
architecture and CPU features, and cannot help when the runner simply
62 +
disappears, because the snapshot is on the runner that is gone.
63 +
64 +
Instead a job's **progress** moves: the outputs of finished jobs (as
65 +
today) and, within a job, the directories it declares as checkpoints
66 +
(§7). Everything is content-addressed, so the new runner fetches only what
67 +
it lacks, and every piece comes with a receipt saying who made it.
68 +
69 +
## 4. Runners
70 +
71 +
| Kind | Tiers | Typical trust (§9) | Leaves by |
72 +
|---|---|---|---|
73 +
| Hosts (earth-1, plex-1, mars-1, narvi-1) | microvm, syscall, wasm | owned | restart or update (drain) |
74 +
| Kubernetes pods | microvm (KVM nodes), syscall, wasm | owned or attested | node drain, eviction, scale-down (SIGTERM) |
75 +
| Cloud VMs, spot or on demand | microvm, syscall, wasm | attested | reclaim notice, scale-down |
76 +
| Phone, the easy way (oxygenhub-android#8) | wasm | device | off charge, off the allowed network, too hot, owner stops it |
77 +
| Phone, full Linux (oxygenhub-android#9, later) | syscall in Android's Linux VM | device | as above |
78 +
| Browser tab (#80) | wasm | anonymous, or device when a member signs in | tab closed or hidden |
79 +
80 +
Cloud and Kubernetes are where handover pays most: spot capacity costs
81 +
60 to 90% less, and handover is what makes it safe to run CI on it.
82 +
Phones and browsers are where it shows best in a demonstration.
83 +
84 +
**Architecture still matters.** Native outputs and checkpoints only move
85 +
between runners of the same image and target (x86_64 to x86_64, arm64 to
86 +
arm64). WASM work moves anywhere.
87 +
88 +
## 5. The WASM tier (oxygenflow#75)
89 +
90 +
Spec §4.1 names it; built 2026-10-25 (wasmtime 49, WASI preview 1).
91 +
92 +
- wasmtime, WASI preview 1 first (preview 2 components later).
93 +
- A virtual file system holding the job's scope and inputs; outputs
94 +
  collected from it as for the other tiers.
95 +
- No network. Memory from the sandbox's limit; CPU time bounded by fuel
96 +
  and the job's time limit.
97 +
- The receipt's environment names the runtime, its version and the
98 +
  module's hash.
99 +
- Every Linux runner offers it (`tiers` gains `wasm`); phones and
100 +
  browsers offer only it.
101 +
102 +
What runs there is what compiles to WASM: linters, formatters, policy
103 +
checks, generated docs, and **test suites built for `wasm32-wasip1`**.
104 +
That last one makes the first demonstration (§11): a native job builds
105 +
the test modules, a phone runs them.
106 +
107 +
## 6. Leases and drain (oxygenflow#76)
108 +
109 +
**Lease.** A runner holding a job holds a lease, renewed by its logs and
110 +
a heartbeat. When the lease lapses the job is requeued, as today.
111 +
112 +
**Drain** is one message in the runner contract (jetstream.md §4) for
113 +
every *planned* departure:
114 +
115 +
| Trigger | Notice |
116 +
|---|---|
117 +
| SIGTERM, `flow` restart or update | the runner's grace (75 min on Earth) |
118 +
| Kubernetes node drain or eviction | the pod's grace period |
119 +
| Spot reclaim | 30 s (GCP), 2 min (AWS) |
120 +
| Phone off charge, off the allowed network, too hot | the owner's policy (a minute or so) |
121 +
| Browser tab hidden or closing | a few seconds |
122 +
123 +
On drain the runner decides from the job's learned duration (spec §11):
124 +
125 +
1. the rest is shorter than the notice: finish the job, then leave;
126 +
2. otherwise: save its checkpoints (§7), upload them, release the lease
127 +
   with `drained`, and stop the job.
128 +
129 +
A drained job is requeued at once, **ahead of new work**, and resumes
130 +
from its checkpoints. A runner that drains a job is not penalised; one
131 +
that drops leases without draining loses placement priority.
132 +
133 +
## 7. Checkpoints (oxygenflow#77)
134 +
135 +
A job may declare directories that hold its progress:
136 +
137 +
```kdl
138 +
job "build" {
139 +
    sandbox "rust"
140 +
    exec "cargo" "build" "--release"
141 +
    checkpoint "/src/target"
142 +
}
143 +
```
144 +
145 +
- **Saved** by content (FastCDC chunks, deduplicated, kept for the run's
146 +
  life) on drain and every few minutes while it runs (a sandbox setting,
147 +
  default 5 minutes, only while the job has run at least that long).
148 +
- **Restored** before a resumed job runs the **same exec again**. Tools
149 +
  that skip finished work (cargo's fingerprints, gradle, make, ninja)
150 +
  then carry on where they stopped; a tool that does not just starts
151 +
  again, so it is never wrong, only slower.
152 +
- **Receipts.** The partial run ends in a receipt with `exit` "drained",
153 +
  naming the checkpoint's hash as an output and signed by its runner. The
154 +
  resumed job's receipt names that checkpoint among its inputs. So
155 +
  provenance says that plex built the first part and mars finished it,
156 +
  and both are signed.
157 +
- **Cheap to move inside a cloud:** checkpoints live in the run's store
158 +
  (a bucket under Strata for cloud pools), so a new pod in the same region
159 +
  restores them quickly. From a phone, the policy (charger, office
160 +
  network) keeps uploads cheap.
161 +
- **Checkpoints are code.** A tampered `.rlib` in `target/` ends up in
162 +
  the binary. So a job accepts a checkpoint only from a runner of at
163 +
  least its trust class (§9), and its receipt says that it used one.
164 +
165 +
## 8. Hedging (oxygenflow#78)
166 +
167 +
When a markedly faster runner frees up while a job runs on a slow one,
168 +
and the learned durations say it would finish sooner (counting the time
169 +
to restore the latest checkpoint), start the job there too. The first to
170 +
finish wins; the other is cancelled.
171 +
172 +
- **Bounded:** one hedge per job, and only where a policy allows it (the
173 +
  critical path of a run, or always on owned runners). Never for jobs
174 +
  with side effects (deployments, publishing).
175 +
- **No tricky handover:** the slow runner is never interrupted. If the
176 +
  fast one fails, nothing is lost.
177 +
- **A free determinism check:** when both finish (a race), their outputs
178 +
  are compared; a difference is reported as a non-hermetic job.
179 +
- **Cost-aware in clouds:** starting a bigger instance costs money, so a
180 +
  pool may forbid hedging, or allow it only for jobs a person is waiting
181 +
  on.
182 +
183 +
Hedging plus holding briefly for a better runner (which exists) gets most
184 +
of the benefit of a full handover, for a fraction of the work. Build it
185 +
before anything finer (§12).
186 +
187 +
## 9. Trust classes (oxygenflow#79)
188 +
189 +
How far to believe a receipt depends on who signed it:
190 +
191 +
| Class | Who | Key |
192 +
|---|---|---|
193 +
| `attested` | cloud confidential VMs, vTPM-measured hosts | keyless workload identity (oxygenhub#122) |
194 +
| `owned` | the organisation's registered hosts | the runner's identity, or keyless |
195 +
| `device` | members' phones and laptops | the device's Keystore key, under the member's grant |
196 +
| `anonymous` | browsers not signed in | a key held non-extractable by WebCrypto |
197 +
198 +
- A sandbox or job may require a minimum (`trust "owned"`). Release and
199 +
  publish jobs default to `owned`.
200 +
- A job resumes only from checkpoints made at its class or above.
201 +
- Receipts and provenance record the class of every runner that
202 +
  contributed. The trust page shows a run's weakest link.
203 +
- **Spot checks:** a sample of jobs run by `device` and `anonymous`
204 +
  runners is run again elsewhere and the outputs compared (with
205 +
  oxygenhub#298's independent rebuilds). A runner whose results disagree
206 +
  is held for its owner to look at.
207 +
208 +
## 10. Phones and browsers
209 +
210 +
**Phones** (oxygenhub-android#8): one tap registers the phone as a
211 +
WASM-tier runner, as a foreground service, scoped like any runner. The
212 +
owner's policy says when it may run: on charge, on these networks (the
213 +
office Wi-Fi), under a temperature, within a storage budget. Leaving
214 +
those conditions drains it (§6). The runner itself is Flow's runner code
215 +
built for Android through the app's Rust core (rust-core, uniffi), with
216 +
wasmtime for the tier. The full Linux runner in Android's Linux terminal
217 +
(#9) comes later.
218 +
219 +
**Browsers** (#80): a page on the hub turns a tab into a WASM-tier runner
220 +
in a Web Worker: WASI through a shim, OPFS (the browser's private file
221 +
storage) for files, uploads as for any runner. Limits: about 4 GB of
222 +
memory, threads only on a cross-origin-isolated page, throttled or
223 +
stopped in a background tab (a drain). This is mostly a demonstration of
224 +
handover and spot checks on the least reliable runner there is, not
225 +
capacity.
226 +
227 +
## 11. The first demonstration
228 +
229 +
A pipeline over oxy-core:
230 +
231 +
1. `build-wasm-tests` (microvm, any x86_64 host): `cargo test
232 +
   --target wasm32-wasip1 --no-run`, outputs the test modules.
233 +
2. `test-wasm` (wasm, `runs-on "android"`, depends on 1): runs each
234 +
   module under the WASM tier, on the phone.
235 +
236 +
The script: start the run; the phone takes `test-wasm`; switch the
237 +
phone's Wi-Fi off halfway (a drain under its policy); the job goes back
238 +
to the queue and plex-1 takes it in its WASM tier; the run page shows
239 +
which runner did what, with the receipts and their trust classes. Then
240 +
the same with a pipeline that has a checkpoint, to show a job resuming
241 +
rather than restarting.
242 +
243 +
## 12. Order of work
244 +
245 +
1. **WASM tier** (#75), in `flow` on Linux runners.
246 +
2. **Phone runner** (oxygenhub-android#8) on that tier, with its policy.
247 +
3. **Leases and drain** (#76), with requeue ahead of new work.
248 +
4. The demonstration (§11), job-level: drain and requeue.
249 +
5. **Checkpoints** (#77): resume within a job; receipts that say so.
250 +
6. **Trust classes** (#79), before anything other than our own runners
251 +
   joins.
252 +
7. **Hedging** (#78), then measure on our own CI: queue and run times
253 +
   with and without it.
254 +
8. **Browser runners** (#80).
255 +
9. **Finer pieces,** only if the measurements call for them: test
256 +
   sharding and per-crate builds (#29), so that a long job is many short
257 +
   ones a better runner can take over between.
258 +
259 +
Cloud and Kubernetes pools come through Jetstream (jetstream.md §9,
260 +
oxygenflow#33), and get drain and checkpoints as they arrive.
261 +
262 +
## 13. Open questions
263 +
264 +
- Checkpoint retention: the run's life, or kept as a cache for the next
265 +
  run of the same job (a remote cache, which would then need the same
266 +
  trust rules)?
267 +
- How long may a drained job wait for a runner of its trust class before
268 +
  it falls back to restarting from nothing on any runner?
269 +
- Hedging costs compute: a per-pool budget, or a per-organisation one?
270 +
- A phone's battery and heat: are the owner's policy limits enough, or
271 +
  should the runner also cap its own CPU use?
modified docs/jetstream.md +178 −0
…
238 unchanged lines
239 239
- **Architecture recorded at build**, so an x86 image never reaches a
240 240
  RISC-V runner.
241 241
242 +
### 8.1 The swarm: runners share images
243 +
244 +
Runners that are configured for it fetch image chunks from each other as
245 +
well as from the hub (oxygenflow #47, oxygenhub #144). Fifty fresh
246 +
runners asking for the same 2 GiB image then cost the hub about one copy,
247 +
not fifty, and a site behind a thin link pulls each chunk across it once.
248 +
249 +
- **Trust comes from the hashes.** The manifest comes from the hub, and
250 +
  every chunk is checked against its name before it is kept. A peer can
251 +
  fail to help; it can never change what a job runs.
252 +
- **The hub is the tracker.** A runner reports its swarm address, its site
253 +
  and its certificate's fingerprint with every claim. `POST /api/v1/flow/swarm/announce {image, state}`
254 +
  (`fetching`, `done` or `stopped`) returns the peers in the same site that
255 +
  hold the image or are fetching it, and a **ticket**.
256 +
- **Tickets admit peers.** A ticket names the runner, its site and an
257 +
  expiry (15 minutes), signed with the hub key that `/.well-known/oxyhub-key`
258 +
  publishes. A runner shows its ticket once to each peer
259 +
  (`POST /swarm/v1/hello`) and gets a session for the ticket's life; no
260 +
  other runner and no account can take chunks from a peer.
261 +
- **Encrypted, and pinned by the hub.** Each runner makes its own TLS
262 +
  certificate on first use (`swarm/cert.der`, its key readable by the
263 +
  runner alone) and keeps it, so its fingerprint (BLAKE3 of the
264 +
  certificate) holds across restarts. The tracker hands each peer out with
265 +
  that fingerprint, and a fetcher talks only to a peer showing that
266 +
  certificate and proving its key in the handshake; no CA is involved. A
267 +
  peer at another address, or a stranger at a peer's, gets nothing and
268 +
  sees nothing: chunks, tickets and sessions all travel inside TLS.
269 +
- **The peer protocol** is HTTPS on the runner's swarm address:
270 +
  - `POST /swarm/v1/hello` with the ticket: a session token.
271 +
  - `GET /swarm/v1/images/{image}/have`: `complete`, or two bit maps over
272 +
    the manifest's needed list (`Manifest::needed`), in its order: the
273 +
    chunks it has, then the chunks it is fetching now.
274 +
  - `GET /swarm/v1/chunks/{chunk}`: the chunk's bytes, from the chunk
275 +
    cache or read out of a whole image; `503` when the peer is at its
276 +
    connection limit, and the fetcher asks it again shortly.
277 +
- **Rarest first.** The fetcher refreshes every peer's maps twice a
278 +
  second and asks each peer, four requests at a time, for the chunk the
279 +
  fewest peers hold. The hub is a seed that holds everything: it is asked
280 +
  only for chunks no peer has or is fetching, or any chunk once the peers
281 +
  have made no progress for two seconds. Runners that start together so
282 +
  split the hub's chunks between them rather than each taking them all. When every missing chunk is already asked
283 +
  for, idle workers ask a second source for it (the endgame; the hub joins
284 +
  it only once the peers stall). A peer that
285 +
  fails is dropped and its chunks go back to the pool.
286 +
- **Serving while fetching.** A chunk is offered to others as soon as it
287 +
  is checked and kept, so runners starting together share the load.
288 +
- **Bounded.** A runner serves at most 16 connections and `upload_mib_s`
289 +
  (default 50 MiB/s), so serving never starves its own jobs.
290 +
- **Configured, never assumed.** Without a `[runner.swarm]` section a
291 +
  runner neither listens nor asks peers. Peers are not offered across
292 +
  sites, so a site is the network its runners can reach each other on.
293 +
294 +
In flow.toml (or as `[swarm]` in runner.toml, which stands in for the
295 +
`[runner]` section):
296 +
297 +
```toml
298 +
[runner.swarm]
299 +
listen = "0.0.0.0:7401"        # where peers reach this runner
300 +
advertise = "10.0.0.5:7401"    # the address peers are given (default: listen)
301 +
site = "home"                  # peers are only ever in the same site
302 +
upload_mib_s = 50
303 +
```
304 +
305 +
#### 8.1.1 Images only runners hold (oxygenflow#94, oxygenhub#342)
306 +
307 +
Until now an image had to be sent to the hub (`flow rootfs push`) before any
308 +
runner but the one that imported it could take its jobs: the hub hands out
309 +
the manifest, and placement offers a job to a runner that fetches only when
310 +
the hub holds its image. On 2026-10-25 the Rust image was re-pinned and not
311 +
pushed, and for a day every Rust job queued on the one runner that had
312 +
imported it while the others sat idle.
313 +
314 +
- **Runners say what they hold.** At start, every ten minutes and after an
315 +
  import, a runner sends the hashes of the images it holds
316 +
  (`POST /flow/images/held {images}`); the hub answers with those it keeps
317 +
  no manifest for (`{wanted}`), and the runner sends each manifest alone
318 +
  (`PUT /flow/images/{hash}/held`, a few MB even for a 14 GB image): no
319 +
  chunks. (Not in the claim: an idle runner's claim gets no answer at all.)
320 +
- **The hub keeps it as peer-held.** It checks the manifest is well formed
321 +
  and names the hash it is filed under, and records the image as held by
322 +
  runners, not by itself: it cannot read the image to check it. A pushed
323 +
  image (`flow rootfs push`, verified whole) replaces a peer-held record.
324 +
- **Trust still comes from the hashes.** The pipeline pins the image's hash;
325 +
  a fetcher asks for the manifest by that hash, checks every chunk against
326 +
  its name, and checks the assembled image against the pin before it keeps
327 +
  it (`jet::images::assemble`). A wrong manifest can make a fetch fail; it
328 +
  cannot change what a job runs. Only runner tokens may send one.
329 +
- **Placement counts peers.** A job whose image the hub holds only as
330 +
  peer-held goes to a runner that fetches only when a runner of the same
331 +
  swarm site holds the image, and says so when none does ("held only by
332 +
  earth-1, in another site").
333 +
- **The hub is no seed for it.** The announce answer says whether the hub
334 +
  holds the chunks (`seed`); when it does not, the fetcher asks peers only,
335 +
  and a fetch with no peer left fails with that reason rather than after
336 +
  the hub refused chunk after chunk.
337 +
338 +
### 8.2 Runners update themselves
339 +
340 +
Proposed 2026-10-25 from Andy's experience: keeping a large fleet's
341 +
runners up to date by hand is one of the biggest costs of running CI.
342 +
Runners fetch new runner builds the way they fetch images, from each other
343 +
through the swarm, check them as strictly as a release, and swap
344 +
themselves in without losing work, under a policy the organisation sets,
345 +
testing on a few runners before the rest.
346 +
347 +
**A runner build** is a `flow` binary from an OxygenFlow release, one per
348 +
target (linux-x86_64, linux-arm64, linux-riscv64; Windows and macOS when
349 +
those runners exist), built by Flow from the tagged state with a signed
350 +
receipt, as oxy's and OxygenHub's releases are. The hub serves it as a
351 +
blob with a chunk manifest, so the swarm (§8.1) carries it unchanged: one
352 +
runner of a site fetches each chunk from the hub and the rest from it.
353 +
354 +
**Off unless chosen, nearest wins.** The policy is set for the site, an
355 +
organisation, a test group and a runner; the nearest setting wins:
356 +
357 +
- `updates`: `off` (the default) or `on`;
358 +
- `channel`: `stable`, `canary`, or a pinned version (`0.3.4`);
359 +
- `window`: hours in which a runner may update (outside them it waits);
360 +
- `dry_run`: fetch and check, say what would happen, do not swap.
361 +
362 +
A runner's own `flow.toml` has the last word: `[runner.update] allow =
363 +
false` refuses every update whatever the hub says, so a site can hold a
364 +
machine back while it tests.
365 +
366 +
**Test groups and waves.** A test group is a named set of runners: chosen
367 +
by name, by label or by pool. A rollout is a release and its waves, in
368 +
order; each wave is a group and its share of it, **a percentage or chosen
369 +
runners**. A percentage is taken by a stable hash of each runner's id, so
370 +
"10% of build-farm" is the same machines every rollout, the ones the
371 +
organisation watches. A wave starts when the one before has been healthy
372 +
for its soak time (say 24 hours) or when an admin promotes it. For
373 +
example: canary (two chosen runners), then build-farm at 10%, then
374 +
build-farm at 100%, then everyone.
375 +
376 +
**Healthy** means the updated runners came back on the new version within
377 +
ten minutes, claim jobs, and their jobs' infrastructure errors (`lost`,
378 +
`error`, declines) over the soak are no worse than the same runners' on
379 +
the version before, over enough jobs to tell. A wave that is not healthy
380 +
halts the rollout, and its runners roll back. Job failures from the
381 +
pipelines themselves do not count.
382 +
383 +
**On the runner,** in order:
384 +
385 +
1. The hub's answer to a claim or heartbeat names an update when the
386 +
   policy says so: the version, its blob and manifest, its receipt.
387 +
2. Fetch the blob through the swarm (hub as seed); every chunk checked
388 +
   against the manifest from the hub.
389 +
3. Check it as a release: its BLAKE3 matches the release asset the hub
390 +
   attests; the receipt is signed by an `owned` build runner (trust
391 +
   classes, #79) and names the tagged state; the target is this machine's.
392 +
   A peer can only serve bytes; it cannot change what is installed.
393 +
4. Stage it (`bin/flow.new`) and run its self-check (`flow.new tiers`,
394 +
   `--version`).
395 +
5. Drain with notice (`leave_within`, §11.5.2 of the spec): finish what
396 +
   fits, hand back the rest with its checkpoints.
397 +
6. Swap: `flow` → `flow.prev`, `flow.new` → `flow`, and restart.
398 +
7. Prove it: the first claim reports the new version. A runner that will
399 +
   not start, or that the hub does not hear from within ten minutes, goes
400 +
   back to `flow.prev` (a start counter in the runner's home: three
401 +
   failed starts after an update restore the previous build).
402 +
8. The runner's page shows its version, channel, group, last update and
403 +
   why; the audit log records every update and rollback.
404 +
405 +
**Phones** take runner updates with the app's releases (Android updates an
406 +
app only through its store or an install), so they follow the app.
407 +
408 +
**Order of work:** (1) the runner side, with a local source (`flow runner
409 +
update --from FILE`): check, stage, drain, swap, prove, roll back (done,
410 +
#85: spec §11.5.3, Linux and Windows); runners asking the hub and taking
411 +
its offer (done); (2) the
412 +
policy on the hub (opt in and out, channels, windows, dry run) and the
413 +
update a runner is offered (done: oxygenhub#322, for the site and for each
414 +
organisation's own runners, `GET /api/v1/flow/runner/update`); (3) test groups (done: chosen or a percentage), waves and health (a
415 +
build that rolls back on one runner is held for every other, oxygenhub#326;
416 +
waves that advance by themselves after a soak are not built yet); (4) the
417 +
swarm as transport; (5) Admin › Runners › Updates (done) and `oh runner
418 +
update`.
419 +
242 420
## 9. Pools
243 421
244 422
A **pool** is one place runners come from. All pools report the same
…
165 unchanged lines
modified docs/m0.md +14 −6
…
146 unchanged lines
147 147
   variable the pipeline sets keeps its value. They are part of the job's
148 148
   environment, so of its receipt.
149 149
5. `flow-init` frames the child's output on the serial console as
150 -
   `\x1eO <line>` (stdout) and `\x1eE <line>` (stderr), then
151 -
   `\x1eX <code>`, and powers off. The kernel runs `quiet` so the host sees
150 +
   `\x1eO <line>` (stdout) and `\x1eE <line>` (stderr). After a passing
151 +
   job it prunes each build cache disk (mounted `strictatime`) of the
152 +
   files the job did not use when it is over the runner's
153 +
   `cache_prune_percent` of its size, and reports each as
154 +
   `\x1eC <json>` (spec §5.4, oxygenflow#72); then `\x1eX <code>`, and
155 +
   it powers off. The kernel runs `quiet` so the host sees
152 156
   only frames; anything else is kept as `console` records.
153 157
6. The host reads frames, streams them as log records, and kills the VM
154 158
   at the timeout (the guest cannot extend it). Scratch files are deleted
…
9 unchanged lines
164 168
UTS namespaces (no interfaces but loopback), the rootfs image's tree
165 169
(unpacked once per hash, read-only) as `/`, the workspace directory at
166 170
`/src`, tmpfs at `/tmp`; Landlock limits writes to `/src` and `/tmp`; a
167 -
seccomp filter refuses `socket` for anything but `AF_UNIX`, plus
171 +
seccomp filter refuses `socket` for anything but `AF_UNIX` (and, with
172 +
`network "loopback"`, `AF_INET` and `AF_INET6`, which reach only the
173 +
job's own `lo`: oxygenflow#73), plus
168 174
`ptrace`, `mount`, `bpf`, `perf_event_open`, `keyctl` and module calls;
169 175
`cpu-limit` and `memory-limit` apply through a cgroup v2 scope when the
170 176
runner has a delegated cgroup, and the job fails to start otherwise
…
1 unchanged lines
172 178
173 179
**How it runs.**
174 180
- **Helper:** for each job the runner starts `flow __sandbox <spec>` (its own binary, through `/proc/self/exe`). The helper joins the job's cgroup, makes the namespaces with the job's user (uid and gid 1000) mapped to the runner's, and forks the sandbox's PID 1.
175 -
- **Root:** PID 1 binds the unpacked tree read-only as the new root, the workspace at `/src`, tmpfs at `/tmp`, `/run` and `/run/secrets`, a fresh `/proc`, and a `/dev` of `null`, `zero`, `full`, `random` and `urandom` with `/dev/shm`. It brings `lo` up for `network "loopback"`, pivots into the root and detaches the old one, so the host's files, the spec and the scratch directory are out of reach.
181 +
- **Root:** PID 1 binds the unpacked tree read-only as the new root, the workspace at `/src`, tmpfs at `/tmp`, `/run` and `/run/secrets`, a fresh `/proc`, and a `/dev` of `null`, `zero`, `full`, `random` and `urandom` with `/dev/shm`. It brings `lo` up for `network "loopback"` and `"egress"` (with the vsock relay of spec §4.3.1 for the latter), pivots into the root and detaches the old one, so the host's files, the spec and the scratch directory are out of reach.
176 182
- **Job:** the job gets no new privileges, then Landlock (read and execute everywhere, write beneath `/src`, `/tmp`, `/dev` and its mounts), then the seccomp filter, then `execve` with exactly the job's environment, found the way `flow-init` finds it (the two share that code). The filter also refuses io_uring (which could open sockets past it), the new mount calls, kexec, swap and reboot.
177 183
- **Unpacking:** `debugfs rdump` unpacks the image into `trees/<hash>` in the home once, without mount rights. Set-id bits and device nodes are dropped.
178 -
- **Ephemeral mounts:** a `tmpfs` mount is a tmpfs of its size. An `ephemeral://disk` mount is a scratch directory, so its size is not enforced in this tier.
184 +
- **Ephemeral mounts:** a `tmpfs` mount is a tmpfs of its size. An `ephemeral://disk` mount is a scratch directory, so its size is not enforced in this tier. One that names a `cache` starts from a directory a passing job of the slot kept, and goes back into the runner's cache pool when the job passes (spec §5.4, oxygenflow#70), pruned first by the runner of the files the job did not use when over the cap (oxygenflow#72).
179 185
- **Artifacts:** they are read from the workspace and mount directories without following links.
180 186
- **Receipts:** the environment's runtime is `syscall`. Its parts are the `flow` binary's BLAKE3, the host kernel's release, memory, CPUs and mounts.
181 187
- **Limits:** the job's cgroup gets `memory.max`, `memory.swap.max 0` and `cpu.max`. A job killed for memory is told so in its log. At the end the group is killed and removed.
…
61 unchanged lines
243 249
`kernel/vmlinux` (pinned), `cas/` (rootfs images), `work/` (scratch), and
244 250
`runner.toml`. `flow runner` runs as the `flow-runner` systemd service
245 251
under the `claude` user (member of `kvm`), polling the hub at
246 -
`https://earth.andymccall.xyz:7343`.
252 +
`https://earth.andymccall.xyz:7343`. The unit has `KillMode=mixed` and
253 +
`TimeoutStopSec=75min`, so a restart drains the runner instead of killing
254 +
its job (spec §11.5.1).
247 255
248 256
## 7. Not in M0
249 257
…
4 unchanged lines
modified docs/spec.md +382 −7
…
90 unchanged lines
91 91
    rootfs "cas://blob/9b5c2a1e8f3d4e7a6b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a"
92 92
    memory-limit "4GiB"
93 93
    cpu-limit 4
94 -
    network "isolated"      // Options: "isolated" | "loopback" | "egress"
94 +
    network "isolated"      // Options: "isolated" | "loopback" | "egress" { allow "host"... } (§4.3.1)
95 95
}
96 96
97 97
// Job evaluated based on Oxygen change closure
…
81 unchanged lines
179 179
180 180
**Time limits.** A job may say how long it may run with `timeout "90m"` (hours, minutes and seconds, as `"2h"`, `"1h30m"` or `"45s"`, from a minute to a day); a sandbox's `timeout` is the default for its jobs. Without one, the hub's default applies, and the hub caps every job at the site's maximum (both set by an administrator on Admin → Runners). The runner stops a job at its limit, and the hub ends one its runner stopped reporting a few minutes after it. A timeout is part of the compiled job, so a job without one compiles as before.
181 181
182 +
**Priority.** A pipeline may lower its jobs' place in the hub's queue, never raise it (oxygenhub#244): `priority "low"` (or `"normal"`) on a job, or at the top of the file for every job that does not say. A job's level is then the lower of its own and its repository's (the repository's, organisation's or site's, set on the hub), so nightly or bulk jobs can stand back for everyday ones; `high` and `urgent` are refused here, being the hub's to give, and a run a person expedites stays urgent whatever its jobs say. A priority is part of the compiled job; a job without one compiles as before.
183 +
182 184
**Runner labels.** A sandbox or a job may say `runs-on "LABEL" ...`: lower-case letters, digits, `-`, `_` and `.`, at most 64 bytes each, given on one or more lines. A job needs a runner that has every label its sandbox and it name. A runner reports its labels and architecture when it registers and on each claim (`flow runner register --label`). The hub hands a job only to a runner with all of them, within the runner's scope (OxygenHub spec 13.2), and a job no runner can take waits, saying so on the queue. Labels are part of the compiled job, so a job that asks for none compiles as before.
183 185
184 186
### 3.2 Canonical Compilation
…
6 unchanged lines
191 193
OxygenFlow implements three distinct isolation tiers depending on the workload:
192 194
193 195
### 4.1 Level 0: WASM Component Sandbox
194 -
- Intended for static linters, formatters, doc generators, and policy checks.
195 -
- Runs on a standard WebAssembly System Interface (WASI) runtime.
196 -
- Startup latency: < 5 ms.
197 -
- Zero host I/O; virtualized file system presented in-memory.
196 +
- Intended for static linters, formatters, doc generators, policy checks, and
197 +
  test suites built for `wasm32-wasip1`; the tier phones and browsers run
198 +
  (docs/handover.md).
199 +
- Built (oxygenflow#75): wasmtime 49 runs WASI preview 1 modules, each in a
200 +
  fresh instance, compiled by Cranelift (tens of milliseconds a module).
201 +
  Preview 2 components later.
202 +
- `runtime "wasm"`; no `rootfs` and no network (`network` absent or
203 +
  `"isolated"`); `memory-limit` holds the module's linear memory.
204 +
- `exec` names the module, a path among the job's files (its scope or an
205 +
  earlier job's artifact), optionally under `/src/`, or a glob with `*` in
206 +
  its file name: each match runs in turn, and the job's exit code is the
207 +
  first non-zero one after all have run. The rest of `exec` are the
208 +
  module's arguments. A module that traps exits 134.
209 +
- The job's files are the only host files it sees, preopened as `/src` (and
210 +
  as the current directory, for relative paths); each `mount` is preopened at
211 +
  its target. Secrets and identities are not offered yet.
212 +
- The receipt's environment names `wasmtime` and its version, the runner
213 +
  binary, the memory, and every module run with its BLAKE3.
198 214
199 215
### 4.2 Level 1: Intercepted Syscall Isolation (gVisor / Landlock + seccomp-bpf)
200 216
- Intended for standard compilation and hermetic test suites.
…
9 unchanged lines
210 226
  directly to FastCDC chunks stored in Oxygen's CAS.
211 227
- Guest VM memory is cleared upon termination.
212 228
229 +
#### 4.3.1 Allow-listed egress (oxygenflow#55)
230 +
231 +
A sandbox reaches the network only by naming its hosts:
232 +
233 +
```kdl
234 +
sandbox "deploy" {
235 +
    runtime "microvm"
236 +
    network "egress" { allow "*.googleapis.com" "registry.opentofu.org" }
237 +
}
238 +
```
239 +
240 +
The VM still has no network device. `flow-init` serves a proxy on the
241 +
guest's `127.0.0.1:3128` (the job gets `HTTPS_PROXY`, `HTTP_PROXY` and
242 +
their lower-case forms, unless it sets them) and relays each connection,
243 +
unread, over Firecracker's vsock to the runner. The runner accepts only
244 +
`CONNECT host:443` for a host the list allows (a name exactly, or any
245 +
name under a `*.` domain, never the domain itself; never an address),
246 +
resolves it itself (the guest does not resolve), refuses any name that
247 +
resolves to an address that is not public, and relays the bytes without
248 +
opening the TLS. Every decision, allowed or refused, is a `flow` line in
249 +
the job's log. Patterns are lower-case host names of two or more labels,
250 +
at most 32 per sandbox, and part of the sandbox's hash.
251 +
252 +
Egress is off unless a runner turns it on (`egress = true` under
253 +
`[runner]` in `flow.toml`); such a runner offers the `egress` label, and
254 +
an egress sandbox needs it, so the hub hands its jobs only to runners
255 +
that allow it, and a runner without it refuses one anyway. The syscall
256 +
runtime has no egress yet.
257 +
258 +
**Keyless receipts** (oxygenhub#241). With `keyless = true` under
259 +
`[runner]` (or in `runner.toml`), a runner signs each job's receipt with a
260 +
fresh key instead of its own: just before signing, it asks the hub to
261 +
certify the key (`POST /api/v1/keyless/certify` with the job's token and
262 +
the key's proof of possession, scope `seal`), checks the certificate is
263 +
for that key and signed by the key the hub publishes at
264 +
`/.well-known/oxyhub-key`, signs, attaches the certificate and the hub's
265 +
identity to the receipt (Oxygen spec §14.6), and drops the key. When the
266 +
hub cannot certify, the runner's own key signs and the job's log says
267 +
why. The runner's key then only authenticates it to the hub.
268 +
269 +
**Metrics** (oxygenflow#35). With `metrics = "127.0.0.1:9464"` under
270 +
`[runner]` (off unless set), a runner serves Prometheus text at
271 +
`/metrics` on that address, from a thread of its own: its version and
272 +
uptime, jobs claimed and declined, finished jobs by result (`passed`,
273 +
`failed`, `error`, `cancelled`), the seconds they ran together, and the
274 +
jobs running now, each labelled with the runner's name. With
275 +
`metrics_token`, a scrape must send it as a bearer token; without one,
276 +
bind it to an address only the scraper reaches. The hub's own `/metrics`
277 +
has the queue and run times of every runner's jobs.
278 +
213 279
## 5. Content-Addressable Storage (CAS) & Zero-Copy Mounting
214 280
215 281
OxygenFlow runners bypass traditional clone operations. Repositories are accessed
…
70 unchanged lines
286 352
}
287 353
```
288 354
355 +
### 5.3.1 Artifacts between jobs (oxygenflow#58)
356 +
357 +
A job takes an artifact of a job it depends on:
358 +
359 +
```kdl
360 +
job "apply" {
361 +
    depends-on "plan"
362 +
    input {
363 +
        artifact "tfplan" from "plan"
364 +
        path "/src/tfplan"
365 +
    }
366 +
}
367 +
```
368 +
369 +
The producer must be named in `depends-on` and declare that name as an
370 +
artifact (not a report); paths are clean absolute paths under `/src`, at
371 +
most 8 inputs, and they are part of the job's hash (`[artifact, from,
372 +
path]`, field 10). Since the producer has passed when the job is handed
373 +
out, the hub puts each input in the claim with the blob the producer's
374 +
receipt names (a passing job's receipt names exactly the artifacts it
375 +
uploaded). Before the job starts, the runner fetches each from
376 +
`GET /api/v1/flow/jobs/{job}/inputs/{index}`, checks its Oxygen blob
377 +
reference against that blob, refuses the job if they differ, and writes it
378 +
at its path in the workspace (not part of the scope hash: it belongs to the
379 +
run, not the repository). The job's receipt lists the blobs as its
380 +
`inputs`, so an apply job's receipt proves which plan it applied.
381 +
382 +
### 5.3.2 Approval gates (oxygenflow#57)
383 +
384 +
A job can wait for a person:
385 +
386 +
```kdl
387 +
job "apply" {
388 +
    depends-on "plan"
389 +
    approval {
390 +
        by "maintain"        // the least role: "write", "maintain" (default) or "admin"
391 +
        others-only #true    // not the person whose push started the run
392 +
        expires "24h"        // from when the jobs it depends on are done; up to 720h
393 +
        sealed #true         // signed with the approver's key; default: on with an environment
394 +
    }
395 +
}
396 +
```
397 +
398 +
The hub hands such a job to no runner until someone with that role on the
399 +
repository approves it, on the run page or through the API; others-only
400 +
leaves out whoever pushed. Approving or rejecting is recorded on the job
401 +
(who, when) and in the audit log. A rejected or expired job ends as
402 +
cancelled and the jobs after it are skipped; a newer push to the
403 +
bookmark cancels the run as before, so a stale plan is never applied. The
404 +
approval is part of the job's hash (field 11).
405 +
406 +
**Sealed approvals (oxygenhub#235).** With `sealed` (the default for a job
407 +
that names an environment, §5.3.3), pressing Approve is not enough: the
408 +
approver signs an Oxygen `approval` attestation with their own key
409 +
(`oxy seal --flow URL --job NAME`; Oxygen spec §14.4) covering the run, the
410 +
job's hash, the state, the input blobs (the reviewed plan) and the
411 +
environment. The hub starts the job only once it verifies, the key is
412 +
linked to an account with the role, and it is not a delegated key; it keeps
413 +
the approval in the repository, and the job's identity tokens name the
414 +
approver and the approval's hash. Rejecting needs no seal.
415 +
416 +
**Agents (oxygenhub#229, #258).** A gate is there so a person decides. A
417 +
decision made through an agent's token (an agent-marked token, or a keyless
418 +
agent's grant) is refused unless the repository lets agents decide its
419 +
gates (its Settings › OxygenFlow, for maintainers; off by default, audited
420 +
when changed). Where allowed, it says so on the job, the run and the audit
421 +
log ("approved by carol via claude-code"). A sealed approval is a person's
422 +
own key, whatever carried it, and is not affected.
423 +
424 +
### 5.3.3 Environments (oxygenflow#59)
425 +
426 +
`environment "prod"` on a job says where it deploys (1 to 32 of `a-z`,
427 +
`0-9`, `-`; part of the job's hash, field 12). The hub keeps what each
428 +
environment of a repository runs, from the jobs that passed for it (state,
429 +
run, when, who approved), on the repository's Deployments page. The job's
430 +
identity tokens carry an `environment` claim only when the job was
431 +
approved (§5.3.2): a job with no approval block, or one not approved, gets
432 +
tokens without it. So a cloud's trust condition can require
433 +
`environment == "prod"`, and nothing unapproved deploys there, even from
434 +
the deploy bookmark. The approval block is the environment's protection
435 +
rule for now.
436 +
437 +
### 5.4 The Build Cache (oxygenflow#28)
438 +
439 +
An `ephemeral://disk` mount may name a `cache`, and the files that key it:
440 +
441 +
```kdl
442 +
mount {
443 +
    source "ephemeral://disk"
444 +
    target "/src/target"
445 +
    size "24GiB"
446 +
    cache "target"
447 +
    key "Cargo.lock" "rust-toolchain.toml"
448 +
}
449 +
```
450 +
451 +
Such a disk is not always empty at the start. Each runner keeps a pool of
452 +
the disks of passing jobs, by repository, job, cache name and bookmark,
453 +
named by the **key**: the BLAKE3 of the sandbox image, the job's name, the
454 +
cache name, and each key file's content in the job's state. A job's disk
455 +
starts as, in order: its bookmark's disk with the same key (moved), its
456 +
bookmark's newest disk (moved), another bookmark's with the same key and
457 +
then another's newest (copied, so a proposal never takes the default
458 +
bookmark's), and otherwise an empty disk. A microVM's disk must be the
459 +
size the mount asks for.
460 +
461 +
- **Only passing jobs keep a disk.** When the job and its outputs pass,
462 +
  its cached disks go back into the pool under their key; otherwise they
463 +
  are discarded with the scratch directory, so nothing a failing job wrote
464 +
  reaches a later one.
465 +
- **The receipt says so.** The job's environment gains `cache TARGET` with
466 +
  the key and what the disk started from, so the receipt's environment
467 +
  hash differs from a cold build's.
468 +
- **Bounded.** The pool keeps under `cache_gb` in the runner's config
469 +
  (default 64 GiB) by dropping the disks used longest ago.
470 +
- **Pruned before it is kept** (oxygenflow#72). A kept disk would
471 +
  otherwise carry every build's files (each crate under every hash a
472 +
  lockfile or feature set gave it) until the job's disk is full. When a
473 +
  passing job's cache holds more than `cache_prune_percent` of its
474 +
  mount's `size` (runner config, default 60), the files the job did not
475 +
  use are removed, used longest ago first, until it is under that cap; one
476 +
  still over it is not kept, so the next run starts empty. A file was
477 +
  used if it was read, written or changed since the job started, or it
478 +
  shares a unit hash with one that was (a 16-digit hex word after a `-`
479 +
  in its path, as cargo names a unit's fingerprint, rlibs, binaries and
480 +
  build-script output: a crate cargo found fresh is only looked at, but
481 +
  its fingerprint is read), or is another link of one, or lies under one
482 +
  of the job's outputs. A microVM's guest (`flow-init`) prunes its disks
483 +
  before it powers off, having mounted them `strictatime` so every read
484 +
  is seen, and tells the runner what it kept (`\x1eC` frames); without
485 +
  word from it the runner only measures the image (`dumpe2fs`) and keeps
486 +
  it if it is under the cap. The syscall tier's runner prunes a directory
487 +
  itself, having set each file's atime to the epoch when the job took it,
488 +
  so even a `relatime` file system records the job's first read of each;
489 +
  on a `noatime` one nothing is pruned and only the size decides. The
490 +
  job's log says what happened: `cache target: kept 12.3 GiB of 23.1 GiB
491 +
  (pruned 10.8 GiB in 4113 files this run did not use; the cap is 60% of
492 +
  its 24.0 GiB disk)`, or `not kept:` and why.
493 +
- **Unchanged sources keep their times** (oxygenflow#52). Beside each kept
494 +
  disk the pool keeps its sources: each workspace file's content hash and
495 +
  the mtime it had, in whole seconds (what the workspace image keeps). A
496 +
  job that takes the disk gives each file whose content is unchanged that
497 +
  mtime back before the workspace is imaged; a changed or new file keeps
498 +
  the time of its checkout, newer than anything on the disk. A build tool
499 +
  that judges freshness by time, such as cargo, then rebuilds exactly the
500 +
  crates whose sources changed, where a fresh checkout would make every
501 +
  path crate look changed. Directories are kept the same way, by their
502 +
  listing (the names in them): an unchanged one gets its old time back,
503 +
  one that gained or lost an entry keeps its new time, so a build
504 +
  script's `rerun-if-changed` on a directory sees exactly what changed.
505 +
  The log says how many files and directories kept their times. A build
506 +
  script that reads a per-state value (the Oxygen crates' `FLOW_HEADS`)
507 +
  reruns on every state; those crates read it only when `OXY_BUILD` is
508 +
  unset, so a pipeline that sets `OXY_BUILD` keeps them fresh.
509 +
- **Jobs go where their cache is** (oxygenflow#51). Each claim lists the
510 +
  caches the runner holds a disk of (`caches`, each
511 +
  `flow_core::api::cache_slot(repo, job)`), and the hub ranks a runner
512 +
  holding a job's cache above one holding its image (§11.4): a job another
513 +
  free runner holds the cache for is held for it for up to 15 seconds, and
514 +
  `placed` says "holds its build cache". When several hold it, the
515 +
  freshest copy wins (oxygenflow#54): claims also say when each slot last
516 +
  kept a disk (`cache_times`, Unix ms), and a copy kept at least a minute
517 +
  later ranks above an older one, before the image and speed are
518 +
  weighed; `placed` says how old the copy is.
519 +
- **The syscall tier keeps directories** (oxygenflow#70). Its
520 +
  `ephemeral://disk` mount is a host directory bound at the target, so
521 +
  its pool holds directories (`<key>.tree`) beside the microVM's images
522 +
  (`<key>.ext4`), taken and kept the same way: moved out of the pool for a
523 +
  job of the same bookmark (so two jobs of one slot running at once never
524 +
  take the same one), copied with their modes and times from another
525 +
  bookmark's, kept only when the job and its outputs pass and otherwise
526 +
  thrown away, and counted by the blocks of everything in them toward
527 +
  `cache_gb`. A directory has no size, so `size` does not choose between
528 +
  them. Each tier takes only its own form: a syscall job never starts from
529 +
  a microVM's image, nor a microVM from a directory. The job runs as the
530 +
  runner's user, so what it wrote is the runner's; before a directory goes
531 +
  into the pool nothing in it is left writable by group or others, or
532 +
  set-id, and the pool is readable by the runner alone. Claims report the
533 +
  slots of both forms in `caches` and `cache_times`, so jobs go where
534 +
  their cache is whichever tier they run in.
535 +
- A cache is a speed-up, never a source of truth: jobs must build
536 +
  correctly from an empty disk, which a key change or an eviction gives
537 +
  them at any time.
538 +
289 539
## 6. Commutative Execution & Change Caching
290 540
291 541
In Oxygen, changes that do not share scopes (referenced atoms, placements, or
…
222 unchanged lines
514 764
`flow rootfs import` reads the ELF header of the image's `/bin/sh` and
515 765
records its architecture in the CAS manifest beside the hash. A runner
516 766
refuses to import an image for another architecture (there is no
517 -
emulation tier). An image imported before this section has no recorded
518 -
architecture and is treated as the importing runner's.
767 +
emulation tier).
768 +
769 +
An image whose manifest records no architecture (imported before this
770 +
section, or fetched from a hub that did not know it) is read, not
771 +
guessed: `jet::rootfs` follows `/bin/sh` (else busybox, env or bash)
772 +
through the image's ext4, symlinks included, to its ELF header, with no
773 +
mount and no privileges. The runner records the answer in the manifest,
774 +
so each image is read once, reports it with its claims, and declines a
775 +
job whose image is for another architecture instead of running it and
776 +
failing (§11.5). The hub reads the images it keeps the same way
777 +
(OxygenHub, oxygenhub#173), and an image nobody can read is offered only
778 +
to the architecture its jobs have passed on before.
519 779
520 780
### 11.3 Job Requirements
521 781
…
45 unchanged lines
567 827
works out why each ready job waits every 15 seconds, and the run page and
568 828
the API (`waiting`) show it.
569 829
830 +
### 11.5.1 Restarts (oxygenflow#49, #50)
831 +
832 +
Neither a hub restart nor a runner restart costs a job anything.
833 +
834 +
- **The hub away.** A job's uploads (artifacts, report files) and declines
835 +
  are tried again through refused connections and server errors (5xx,
836 +
  as a proxy answers while the hub restarts) for three minutes, waiting
837 +
  1, 2, 4, 8 and then 15 seconds; a refusal (4xx) is final. Log lines
838 +
  the hub could not take are kept and sent with the next batch, 200 lines
839 +
  at a time, up to 50,000 (beyond that the oldest go, and the log says how
840 +
  many). A job's result is kept on disk and offered for two hours, as
841 +
  before. Claims are not retried through a lost connection, so a job is
842 +
  never claimed twice; the runner simply asks again.
843 +
- **The runner restarted.** The first SIGTERM or SIGINT drains the runner:
844 +
  it claims nothing more and stops once every job it holds has ended and
845 +
  been reported. A second signal stops it at once. Its systemd unit says
846 +
  `KillMode=mixed`, so only the runner is signalled and its microVMs run
847 +
  on, and `TimeoutStopSec` longer than the longest job, after which
848 +
  systemd kills what is left. A new binary is installed by replacing it
849 +
  and restarting the unit (`systemctl restart --no-block`): the old runner
850 +
  finishes its job, and `Restart=always` starts the new one.
851 +
  `flow runner drain` asks the same by a `drain.request` file in the
852 +
  runner's home, which its loop reads: the way on Windows, which has no
853 +
  SIGTERM (oxygenflow#83).
854 +
855 +
### 11.5.2 Handing back (oxygenflow#76)
856 +
857 +
A runner that leaves while it holds a job, with no time to finish it,
858 +
hands the job back: `POST /api/v1/flow/jobs/{id}/release` with
859 +
`{"reason": …}` (a phone off its allowed network, docs/handover.md §6).
860 +
The hub queues the job again **ahead of new work**, whatever its
861 +
priority, adds a line to its log saying who handed it back and why, and
862 +
lets the next attempt's log follow on. It is not a decline: the runner is
863 +
not held from the job and it does not count towards unplaceable.
864 +
865 +
**Leases.** Holding a job is a lease the runner keeps by being heard from:
866 +
claims, a heartbeat every 30 seconds while it claims nothing, its jobs'
867 +
logs. When the hub has not heard from a runner for 3 minutes, its jobs
868 +
are handed back the same way, so a runner that vanishes (out of every
869 +
network, crashed) costs a restart of its job, not the job. A job handed
870 +
back more than 5 times ends as an error saying so.
871 +
872 +
**Notice.** A departure may come with notice (a phone taken off charge
873 +
has a minute; a spot reclaim 30 seconds to 2 minutes). A claim carries
874 +
`expected_ms`, how long the job is expected to take, when the hub has
875 +
learned it: the median of the job's last five passing runs in that
876 +
repository (reused passes left out). A leaving runner finishes and
877 +
reports a job whose expected rest fits within the notice, and hands back
878 +
the others at once, and any job still running when the notice is up. A
879 +
job with no expected time is handed back at once.
880 +
881 +
**Checkpoints (#77).** A job may name directories that hold its progress:
882 +
`checkpoint "/src/target"` (one or more, under /src, not /src itself).
883 +
When its runner hands it back, the runner packs them (files with their
884 +
modes and modification times, which cargo's fingerprints read; at most
885 +
512 MiB) and keeps them with `POST /api/v1/flow/jobs/{id}/checkpoint`
886 +
before the release; the claim of the next attempt names the blob
887 +
(`checkpoint`), and that runner fetches it (`GET` on the same path),
888 +
checks it, and restores it over the job's cache before running the same
889 +
exec, so a tool that skips finished work carries on. The receipt lists
890 +
the checkpoint among the job's inputs. As built: the WASM and syscall
891 +
tiers save and restore checkpoints, under a disk mount or in the tree;
892 +
the MicroVM tier, whose files are in an image, starts again for now.
893 +
894 +
As built (flow 0.3.2, OxygenHub 0.9.4): the runner as a library has
895 +
`runner::leave(reason)` (no notice) and `runner::leave_within(reason,
896 +
notice)` beside `runner::stop()`: each claims nothing more and stops the
897 +
jobs it does not finish (their logs sent first), releasing them; if the
898 +
release cannot be sent, the lease does it. A host's SIGTERM still drains
899 +
by finishing what it holds. Checkpoints (#77), so the next attempt
900 +
resumes rather than restarts, come next.
901 +
902 +
### 11.5.3 Updating a runner (oxygenflow#85)
903 +
904 +
`flow runner update --from FILE` installs a new `flow` build in place of
905 +
the one it runs as (`--exe` names another), the runner's side of
906 +
self-updating runners (docs/jetstream.md §8.2):
907 +
908 +
1. **Refused** when the home's `flow.toml` says `[runner.update] allow =
909 +
   false`, when an earlier update is not yet proven, or when the build is
910 +
   older than the installed one (unless `--allow-older`).
911 +
2. **Checked:** a 64-bit little-endian ELF for this machine; it names
912 +
   itself (`--version`, `flow X.Y.Z …`) and finds a tier it can run here
913 +
   (`tiers` against the runner's home), each within 30 seconds.
914 +
3. **Staged and swapped:** copied beside the installed binary as
915 +
   `flow.new` and checked there; the installed one is kept as
916 +
   `flow.prev` (a hard link, else a copy), then `flow.new` replaces it in
917 +
   one rename, so `flow` is never missing. `--dry-run` stops before the
918 +
   swap and changes nothing.
919 +
4. **The running runner leaves:** a runner writes its pid to
920 +
   `runner.pid` in its home, and the update sends it SIGUSR1 (after
921 +
   checking that pid is still a `flow … runner`). It then leaves with
922 +
   `--notice` (default 600 s, §11.5.2): jobs expected to finish in time
923 +
   finish, the rest are handed back with their checkpoints, and it exits
924 +
   0, so its unit's `Restart=always` starts the new build. A runner from
925 +
   before #85 is not told; restarting its unit drains it as before.
926 +
5. **Proven or rolled back:** the update waits in `update.json` until the
927 +
   new build's first claim gets an answer from the hub. Each start counts;
928 +
   on the fourth start without that answer, or after ten minutes running
929 +
   without it, `flow.prev` is put back (and on a start, run in its place).
930 +
   `updates.log` in the home records every install, proof and rollback.
931 +
   The runner tells the hub at its next answer (`POST /api/v1/flow/runner/update`):
932 +
   `runs flow X (was Y); proven`, or `rolled back from flow X to Y: why`,
933 +
   on which the hub holds X back from every other runner until an
934 +
   administrator lifts the hold (oxygenhub#326).
935 +
6. **On Windows** (oxygenflow#83) the same steps run with Windows' means:
936 +
   the build is checked as a PE executable for this machine; `flow.exe`
937 +
   is renamed to `flow.prev.exe` and `flow.new.exe` into its place (a
938 +
   running program can be renamed but not replaced, and cannot rename
939 +
   its own file, so a copy of it, `flow __rename`, does that); the runner
940 +
   is told by an `update.request` file in its home, which it reads in its
941 +
   loop, there being no SIGUSR1; and having left, it exits 75, so a Task
942 +
   Scheduler task set to restart on failure starts the new build. A
943 +
   rollback runs the restored build in its place and ends with it.
944 +
570 945
### 11.6 Conformance
571 946
572 947
1. Architecture Safety: a job whose image is `x86_64` MUST never be
…
10 unchanged lines
added docs/workspaces.md +73 −0
1 +
# Workspaces: VS Code in a microVM, the runner side
2 +
3 +
**Status: paused** (Andy, 2026-10-17): too large to take on, and to keep up,
4 +
before the work already promised is delivered. Nothing below is built; the
5 +
plan stands for when it is picked up again.
6 +
7 +
oxygenhub#69, hub spec §13.3. Written 2026-10-17 with Andy's decisions:
8 +
build it, **runner side first**; files **lazy from the start**; editor
9 +
traffic over a **WebSocket with multiplexing**. This is the plan for the
10 +
runner; the hub's part (the workspaces table, the button, the proxy,
11 +
commits) follows it.
12 +
13 +
## What is there today
14 +
15 +
- Jobs run to their exit under a hard time limit, with a job token that
16 +
  lasts at most two hours (hub `JOB_TOKEN_MAX_AGE_MS`).
17 +
- A microVM has no network device. Its only traffic is guest-initiated:
18 +
  the egress proxy over vsock (`flow-init/src/egress.rs`).
19 +
- The runner speaks to the hub in short HTTPS calls; nothing long-lived.
20 +
- flow-casfs mounts a state lazily with FUSE, on the host. Firecracker has
21 +
  no virtio-fs, so a FUSE mount on the host cannot be handed to a guest.
22 +
- The pinned guest kernel (6.1.155) has overlayfs and vsock, but no FUSE.
23 +
24 +
## The shape
25 +
26 +
```
27 +
browser ──HTTPS/WS──► hub /{owner}/{name}/workspace/{id}/ ──┐
28 +
                                                            │ one WebSocket per workspace,
29 +
runner ◄────────────────────────────────────────────────────┘ opened by the runner (yamux streams)
30 +
  │ vsock (host connects in)
31 +
  ▼
32 +
microVM: flow-init ─► openvscode-server on 127.0.0.1:3000
33 +
         /workspace = overlay(lower: casfs over vsock, upper: scratch disk)
34 +
```
35 +
36 +
## Steps
37 +
38 +
Each is an issue of its own, and each lands working and tested.
39 +
40 +
1. **A guest kernel with FUSE.** The same 6.1 kernel and Firecracker's
41 +
   guest config, with `CONFIG_FUSE_FS=y`; pinned by BLAKE3 in `flow.toml`
42 +
   on earth-1, plex-1 and mars-1. Pipeline jobs are unaffected.
43 +
2. **casfs over vsock.** flow-casfs gains a server half: on the host it
44 +
   answers lookups, listings and reads for one state over a vsock port of
45 +
   the workspace's VM, from the same graph-key cache it uses today. The
46 +
   guest half is a small FUSE client in the workspace rootfs. `/workspace`
47 +
   is an overlay: the state read-only below, the VM's scratch disk above,
48 +
   so edits are plain files. Opening a workspace costs a listing, not an
49 +
   export; the target is under a second on the Linux tree.
50 +
3. **The workspace rootfs.** openvscode-server (pinned release, its
51 +
   checksum checked), `oxy`, the Oxygen VCS extension, a non-root user;
52 +
   built and imported like the rust image (`images/workspace`).
53 +
4. **The `workspace` job kind.** `Claim` says `kind: "workspace"`. The
54 +
   runner keeps the VM until it is told to stop, or the hub's idle limit
55 +
   passes, instead of waiting for an exit; it reports liveness and open
56 +
   connections every 30 s, and the hub renews the job token while the
57 +
   workspace lives. A runner advertises whether it takes workspaces.
58 +
5. **The tunnel.** flow-init accepts host connections on a vsock port and
59 +
   forwards each to openvscode-server. The runner opens one WebSocket to
60 +
   the hub per workspace (`/api/v1/flow/workspaces/{id}/tunnel`, its
61 +
   runner token) and carries each browser connection as a yamux stream to
62 +
   that port. Works for runners behind NAT and through Cloudflare.
63 +
64 +
Then the hub (hub spec §13.3): the `workspaces` table and limits, "Open in
65 +
workspace" behind a site setting (off), the owner-only reverse proxy for
66 +
HTTP and WebSocket into the tunnel, and commits pushed to `ws/{account}/{id}`
67 +
and signed by the hub as the account's delegate.
68 +
69 +
## Downloads this needs
70 +
71 +
Linux 6.1 sources (kernel.org) and Firecracker's guest config for step 1,
72 +
and an openvscode-server release (GitHub, Gitpod) for step 3: each pinned
73 +
and checked against its published checksum, as cosign was.
added images/android/Dockerfile +58 −0
1 +
# The OxygenFlow Android image (oxygenflow#19): a JDK, the Android SDK and
2 +
# NDK, Gradle, cargo-ndk, the crates and Gradle dependencies of the
3 +
# OxygenHub app, so its build and tests run in a job with no network.
4 +
# Built by ./build.sh, which fetches and checks the downloads first.
5 +
FROM rust:1.97.1-slim-bookworm
6 +
7 +
RUN apt-get update \
8 +
 && apt-get install -y --no-install-recommends ca-certificates openjdk-17-jdk-headless unzip make git pkg-config \
9 +
 && rm -rf /var/lib/apt/lists/*
10 +
11 +
# The guest's PATH is fixed and has no /usr/local/cargo/bin, and a rustup
12 +
# proxy would need RUSTUP_HOME: link the toolchain's own binaries instead.
13 +
RUN rustup target add aarch64-linux-android x86_64-linux-android \
14 +
 && cargo install --locked --version 4.1.2 --root /opt/cargo-ndk cargo-ndk \
15 +
 && rm -rf /usr/local/cargo/registry /usr/local/cargo/git \
16 +
 && tc=$(ls -d /usr/local/rustup/toolchains/*/ | head -1) \
17 +
 && for b in cargo rustc rustdoc; do ln -sf "$tc/bin/$b" /usr/local/bin/$b; done \
18 +
 && ln -sf /opt/cargo-ndk/bin/cargo-ndk /usr/local/bin/cargo-ndk
19 +
20 +
# The SDK: Google's command-line tools (checked by build.sh), then the
21 +
# platform, build tools and NDK the app names, licences accepted here.
22 +
ENV ANDROID_HOME=/opt/android-sdk JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64
23 +
COPY dl/cmdline-tools.zip /tmp/cmdline-tools.zip
24 +
RUN mkdir -p "$ANDROID_HOME/cmdline-tools" \
25 +
 && unzip -q /tmp/cmdline-tools.zip -d /tmp/ct \
26 +
 && mv /tmp/ct/cmdline-tools "$ANDROID_HOME/cmdline-tools/latest" \
27 +
 && rm -rf /tmp/ct /tmp/cmdline-tools.zip \
28 +
 && yes | "$ANDROID_HOME/cmdline-tools/latest/bin/sdkmanager" --licenses >/dev/null \
29 +
 && "$ANDROID_HOME/cmdline-tools/latest/bin/sdkmanager" --install \
30 +
      "platforms;android-37.0" "build-tools;36.0.0" "ndk;30.0.16248370" >/dev/null
31 +
32 +
# Gradle itself (checked by build.sh): jobs run `gradle`, not the wrapper,
33 +
# which would download its distribution.
34 +
ARG GRADLE_VERSION
35 +
COPY dl/gradle.zip /tmp/gradle.zip
36 +
RUN unzip -q /tmp/gradle.zip -d /opt && rm /tmp/gradle.zip \
37 +
 && ln -sf /opt/gradle-${GRADLE_VERSION}/bin/gradle /usr/local/bin/gradle
38 +
39 +
# The app's crates, vendored: cargo reads /.cargo/config.toml for any
40 +
# directory under /, so rust-core builds offline.
41 +
COPY vendor /opt/vendor
42 +
COPY cargo-config.toml /.cargo/config.toml
43 +
44 +
# Gradle's dependencies: build the app once here, with the network, and
45 +
# keep what Gradle fetched as a read-only cache (GRADLE_RO_DEP_CACHE) that
46 +
# a job's own, writable GRADLE_USER_HOME reads first. A failing test does
47 +
# not matter here; only the downloads do.
48 +
COPY app /tmp/app
49 +
RUN cd /tmp/app \
50 +
 && printf 'sdk.dir=%s\n' "$ANDROID_HOME" > local.properties \
51 +
 && (GRADLE_USER_HOME=/opt/gradle-home gradle --no-daemon -q --continue \
52 +
      :app:assembleDebug :app:createDebugUnitTestCoverageReport :app:lintDebug :app:validateDebugScreenshotTest || true) \
53 +
 && test -d /opt/gradle-home/caches/modules-2 \
54 +
 && cd / && rm -rf /tmp/app /opt/gradle-home/daemon /opt/gradle-home/native /opt/gradle-home/wrapper \
55 +
      /opt/gradle-home/caches/[0-9]* /opt/gradle-home/caches/transforms-* /opt/gradle-home/caches/build-cache-* \
56 +
      /opt/gradle-home/caches/journal-* /opt/gradle-home/caches/jars-* \
57 +
 && rm -f /opt/gradle-home/caches/modules-2/*.lock /opt/gradle-home/caches/modules-2/gc.properties \
58 +
 && chmod -R a+rX /opt/gradle-home "$ANDROID_HOME" /opt/vendor
added images/android/README.md +49 −0
1 +
# The OxygenFlow Android image
2 +
3 +
A root file system for building and testing the OxygenHub Android app (`oxygenhub-android`) in a job with no network (oxygenflow#19):
4 +
5 +
- OpenJDK 17 (`JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64`).
6 +
- The Android SDK in `/opt/android-sdk` (`ANDROID_HOME`): command-line tools 23.0 (checked against Google's published SHA-1), `platforms;android-37.0`, `build-tools;36.0.0` and `ndk;30.0.16248370`, licences accepted.
7 +
- Gradle 9.6.0 as `gradle` (checked against its published SHA-256). Jobs run `gradle --offline`, not `./gradlew`, which would download its distribution.
8 +
- Rust 1.97.1 with the `aarch64-linux-android` and `x86_64-linux-android` targets and cargo-ndk 4.1.2, for the app's `rust-core`.
9 +
- `rust-core`'s crates, vendored in `/opt/vendor`; `/.cargo/config.toml` points cargo at them, as in the rust image.
10 +
- The app's Gradle dependencies in `/opt/gradle-home/caches`, fetched by building the app once while the image is built.
11 +
12 +
The Gradle cache is read-only in the guest. A job points `GRADLE_RO_DEP_CACHE` at it and gives Gradle its own writable `GRADLE_USER_HOME`, so Gradle reads dependencies from the image and writes everything else under `/tmp`.
13 +
14 +
## Building and importing
15 +
16 +
```
17 +
images/android/build.sh /path/to/oxygenflow-android.tar
18 +
scp /path/to/oxygenflow-android.tar claude@earth:/home/claude/flow/dl/
19 +
ssh claude@earth 'cd ~/flow && FLOW_HOME=~/flow bin/flow rootfs import dl/oxygenflow-android.tar'
20 +
```
21 +
22 +
`build.sh` expects the app next to this repository (`../oxygenhub-android`, subrepos checked out). It downloads and checks the command-line tools and Gradle into `images/android/dl/`, copies the app without its build output, vendors `rust-core`'s crates, then builds the image with Docker and exports it as a plain tarball. The tarball is about 6 GB, so build it where the disk has room (Earth) and delete it once imported.
23 +
24 +
Rebuild and re-import when the app changes its SDK, NDK, Gradle or dependency versions, or `rust-core/Cargo.lock` changes, then pin the new hash in the app's pipeline. A dependency missing from the image fails the job with Gradle's "No cached version … available for offline mode", or cargo's "no matching package".
25 +
26 +
## Using it
27 +
28 +
```kdl
29 +
sandbox "android" {
30 +
    runtime "microvm"
31 +
    rootfs "cas://blob/<hash printed by flow rootfs import>"
32 +
    memory-limit "8GiB"
33 +
    cpu-limit 4
34 +
    network "loopback"
35 +
}
36 +
37 +
job "test" {
38 +
    sandbox "android"
39 +
    env {
40 +
        ANDROID_HOME "/opt/android-sdk"
41 +
        JAVA_HOME "/usr/lib/jvm/java-17-openjdk-amd64"
42 +
        GRADLE_RO_DEP_CACHE "/opt/gradle-home/caches"
43 +
        GRADLE_USER_HOME "/tmp/gradle"
44 +
    }
45 +
    exec "gradle" "--offline" "--no-daemon" ":app:testDebugUnitTest"
46 +
}
47 +
```
48 +
49 +
Use `network "loopback"`, not `"isolated"`: with no interface up, Gradle stops with "Could not determine a usable wildcard IP for this machine". Loopback keeps the job offline.
added images/android/build.sh +41 −0
1 +
#!/bin/sh
2 +
# Build the OxygenFlow Android image and write it as a root file system
3 +
# tarball for `flow rootfs import` (oxygenflow#19). Run from any directory;
4 +
# the app is expected next to this repository (../oxygenhub-android), with
5 +
# its subrepos checked out. Rebuild when the app's SDK, NDK, Gradle or
6 +
# dependency versions change, or rust-core's Cargo.lock does.
7 +
set -eu
8 +
here=$(cd "$(dirname "$0")" && pwd)
9 +
dev=$(cd "$here/../../.." && pwd)
10 +
app="$dev/oxygenhub-android"
11 +
out=${1:-$here/oxygenflow-android.tar}
12 +
# Command-line tools 23.0, as Google's repository2-3.xml lists them.
13 +
CMDLINE_TOOLS=commandlinetools-linux-16111833_latest.zip
14 +
CMDLINE_TOOLS_SHA1=e025545c62a8e64c7559119566a569fb1dec5f60
15 +
GRADLE_VERSION=9.6.0
16 +
17 +
mkdir -p "$here/dl"
18 +
if [ ! -f "$here/dl/cmdline-tools.zip" ]; then
19 +
    curl -fsSL "https://dl.google.com/android/repository/$CMDLINE_TOOLS" -o "$here/dl/cmdline-tools.zip"
20 +
fi
21 +
echo "$CMDLINE_TOOLS_SHA1  $here/dl/cmdline-tools.zip" | sha1sum -c - >/dev/null
22 +
if [ ! -f "$here/dl/gradle.zip" ]; then
23 +
    curl -fsSL "https://services.gradle.org/distributions/gradle-$GRADLE_VERSION-bin.zip" -o "$here/dl/gradle.zip"
24 +
fi
25 +
# Gradle publishes each distribution's SHA-256 beside it.
26 +
want=$(curl -fsSL "https://services.gradle.org/distributions/gradle-$GRADLE_VERSION-bin.zip.sha256")
27 +
echo "$want  $here/dl/gradle.zip" | sha256sum -c - >/dev/null
28 +
29 +
# The app, without build output, and rust-core's crates vendored.
30 +
rm -rf "$here/app" "$here/vendor"
31 +
mkdir -p "$here/app"
32 +
tar -C "$app" --exclude='./build' --exclude='./app/build' --exclude='./rust-core/target' \
33 +
    --exclude='./.gradle' --exclude='./.oxy' --exclude='./local.properties' -cf - . | tar -C "$here/app" -xf -
34 +
cargo vendor --locked --versioned-dirs --manifest-path "$app/rust-core/Cargo.toml" "$here/vendor" >/dev/null
35 +
chmod -R a+rX "$here/vendor" "$here/app"
36 +
docker build -q -t oxygenflow-android --build-arg GRADLE_VERSION=$GRADLE_VERSION "$here"
37 +
id=$(docker create oxygenflow-android)
38 +
docker export "$id" -o "$out"
39 +
docker rm "$id" >/dev/null
40 +
rm -rf "${here:?}/app" "${here:?}/vendor"
41 +
echo "$out"
added images/android/cargo-config.toml +9 −0
1 +
# Crates come from the image, never the network (OxygenFlow jobs are isolated).
2 +
[source.crates-io]
3 +
replace-with = "vendored"
4 +
5 +
[source.vendored]
6 +
directory = "/opt/vendor"
7 +
8 +
[net]
9 +
offline = true
modified images/rust/Dockerfile +23 −2
…
16 unchanged lines
17 17
# Release builds (oxygen#59, oxygenhub#85): each project's scripts/dist.sh
18 18
# cross-builds Linux x86_64, arm64 and riscv64, macOS and Windows, with
19 19
# cargo-zigbuild and zig (no Apple SDK needed) and MinGW, and packs .deb,
20 -
# .tar.gz and .zip. The rustup targets' standard libraries, zig checked
20 +
# .tar.gz and .zip; wasm32-wasip1 rebuilds the cheap checks' WASI module
21 +
# (scripts/checks-wasm.sh --check, #96). The rustup targets' standard libraries, zig checked
21 22
# against its published SHA-256, and cargo-zigbuild, all linked onto the
22 23
# guest's fixed PATH.
23 24
ARG ZIG_VERSION
24 25
ARG ZIG_SHA256
25 -
RUN rustup target add x86_64-pc-windows-gnu aarch64-unknown-linux-gnu riscv64gc-unknown-linux-gnu x86_64-apple-darwin aarch64-apple-darwin \
26 +
RUN rustup target add x86_64-pc-windows-gnu aarch64-unknown-linux-gnu riscv64gc-unknown-linux-gnu x86_64-apple-darwin aarch64-apple-darwin wasm32-wasip1 \
26 27
 && apt-get update \
27 28
 && apt-get install -y --no-install-recommends gcc-mingw-w64-x86-64 zip \
28 29
 && rm -rf /var/lib/apt/lists/*
…
14 unchanged lines
43 44
 && ln -sf /opt/cargo-llvm-cov/bin/cargo-llvm-cov /usr/local/bin/cargo-llvm-cov \
44 45
 && rm -rf /usr/local/cargo/registry /usr/local/cargo/git
45 46
47 +
# Test reports (oxygenflow#81): cargo-nextest writes JUnit XML, which the
48 +
# hub reads as a job's `junit` artifact; `cargo llvm-cov nextest` gives the
49 +
# LCOV coverage and the JUnit report from one run of the tests.
50 +
RUN cargo install --locked --version 0.9.148 --root /opt/cargo-nextest cargo-nextest \
51 +
 && ln -sf /opt/cargo-nextest/bin/cargo-nextest /usr/local/bin/cargo-nextest \
52 +
 && rm -rf /usr/local/cargo/registry /usr/local/cargo/git
53 +
46 54
ARG NODE_VERSION
47 55
ARG NODE_SHA256
48 56
ADD https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-x64.tar.xz /tmp/node.tar.xz
…
1 unchanged lines
50 58
 && tar -xJf /tmp/node.tar.xz -C /usr/local --strip-components=1 --exclude='*/include' --exclude='*/share/doc' \
51 59
 && rm /tmp/node.tar.xz
52 60
61 +
# PostgreSQL (oxygenhub#211): OxygenHub's tests on Postgres start a server
62 +
# of their own on loopback inside the job (.flow/test-postgres.sh). Debian's
63 +
# server, its programs linked onto the guest's fixed PATH, and nss_wrapper:
64 +
# a job runs as the runner's builder, a user this image's /etc/passwd does
65 +
# not name, and initdb must look its user up.
66 +
RUN apt-get update \
67 +
 && apt-get install -y --no-install-recommends postgresql libnss-wrapper \
68 +
 && rm -rf /var/lib/apt/lists/* \
69 +
 && for b in initdb pg_ctl postgres; do ln -sf /usr/lib/postgresql/*/bin/$b /usr/local/bin/$b; done \
70 +
 && initdb --version
71 +
53 72
# npm's cache for the Oxygen extensions' lock files: `npm ci --offline
54 73
# --cache /opt/npm-cache` installs from it, read-only as it is.
55 74
COPY npm-cache /opt/npm-cache
…
1 unchanged lines
57 76
# Cargo reads /.cargo/config.toml for anything under /, so /src uses the
58 77
# vendored crates; CARGO_HOME stays the default $HOME/.cargo (/tmp).
59 78
COPY vendor /opt/vendor
79 +
# What it vendors, `NAME VERSION` a line: `flow deps check --list` (#101).
80 +
COPY crates.txt /opt/crates.txt
60 81
COPY cargo-config.toml /.cargo/config.toml
61 82
62 83
# Dependencies prebuilt (backlog F6b2): each workspace's recipe (its
…
21 unchanged lines
modified images/rust/README.md +2 −0
…
5 unchanged lines
6 6
- Node 24.21.0, checked against its published SHA-256.
7 7
- git, make and pkg-config.
8 8
- Coverage: llvm-tools and cargo-llvm-cov 0.9.1, so a job can write an LCOV report for the hub's Coverage tab.
9 +
- Test reports: cargo-nextest 0.9.148, so a job can write JUnit XML for the hub (with `cargo llvm-cov nextest`, both from one run).
9 10
- Release cross-building: the Windows, Linux arm64 and riscv64, and macOS targets, zig 0.16.0 (checked against its SHA-256), cargo-zigbuild and MinGW, with zip, so `scripts/dist.sh` runs in a job.
11 +
- WASI: the wasm32-wasip1 target, so the fmt job can check that `.flow/checks.wasm` is what `crates/flow-checks` builds (#96).
10 12
- Every crate that oxygen, oxygenhub and oxygenflow use, vendored in `/opt/vendor`.
11 13
12 14
OxygenFlow jobs have no network (`network "isolated"`). So cargo is pointed at the vendored crates by `/.cargo/config.toml`, which cargo reads for any directory under `/`, including `/src`. `CARGO_HOME` stays at its default, `$HOME/.cargo`, which is under `/tmp` and writable. The toolchain's binaries are linked into `/usr/local/bin`, because the guest's `PATH` is fixed and nothing at run time needs rustup.
…
41 unchanged lines
modified images/rust/build.sh +8 −2
…
17 unchanged lines
18 18
    --sync "$dev/oxygenhub/Cargo.toml" \
19 19
    --sync "$dev/oxygenflow/Cargo.toml" \
20 20
    "$here/vendor" >/dev/null
21 +
# The image's crates as `NAME VERSION` lines (#101): at /opt/crates.txt in
22 +
# the image and beside the tarball, for `flow deps check --list` to read.
23 +
for m in "$here"/vendor/*/Cargo.toml; do
24 +
    awk -F'"' '/^\[package\]/{p=1;next} /^\[/{p=0} p&&/^name = /{n=$2} p&&/^version = /{v=$2} END{print n" "v}' "$m"
25 +
done | sort > "$here/crates.txt"
21 26
# npm packages of the two extensions, into a cache `npm ci --offline` reads.
22 27
rm -rf "${here:?}/npm-cache"
23 28
for e in oxygen-vscode oxygenhub-vscode; do
…
12 unchanged lines
36 41
# Readable by any user: a job runs as its workspace's owner on the runner
37 42
# host, which differs between runners (oxygen#59: 0640 files from Earth's
38 43
# umask failed plex's dist job).
39 -
chmod -R a+rX "$here/vendor" "$here/npm-cache" "$here/warm"
44 +
chmod -R a+rX "$here/vendor" "$here/npm-cache" "$here/warm" "$here/crates.txt"
40 45
docker build -q -t oxygenflow-build \
41 46
    --build-arg NODE_VERSION=$NODE_VERSION --build-arg NODE_SHA256=$NODE_SHA256 \
42 47
    --build-arg ZIG_VERSION=$ZIG_VERSION --build-arg ZIG_SHA256=$ZIG_SHA256 "$here"
43 48
id=$(docker create oxygenflow-build)
44 49
docker export "$id" -o "$out"
45 50
docker rm "$id" >/dev/null
46 -
rm -rf "${here:?}/vendor" "${here:?}/npm-cache" "${here:?}/warm"
51 +
cp "$here/crates.txt" "${out%.tar}.crates.txt"
52 +
rm -rf "${here:?}/vendor" "${here:?}/npm-cache" "${here:?}/warm" "${here:?}/crates.txt"
47 53
echo "$out"
added images/tofu/Dockerfile +56 −0
1 +
# The OxygenFlow OpenTofu image (oxygenflow#56): OpenTofu and the providers
2 +
# the deploy pipelines use, mirrored inside, so `tofu init` needs no
3 +
# registry. A job reaches a cloud's API only through allow-listed egress
4 +
# (#55); formatting and validation need no network at all.
5 +
6 +
# Download and check OpenTofu, and mirror the providers, with the tools
7 +
# only this stage needs.
8 +
FROM debian:bookworm-slim AS fetch
9 +
ARG TOFU_VERSION
10 +
ARG TOFU_SHA256
11 +
RUN apt-get update \
12 +
 && apt-get install -y --no-install-recommends ca-certificates curl unzip \
13 +
 && rm -rf /var/lib/apt/lists/*
14 +
RUN curl -fsSL -o /tmp/tofu.zip \
15 +
      "https://github.com/opentofu/opentofu/releases/download/v${TOFU_VERSION}/tofu_${TOFU_VERSION}_linux_amd64.zip" \
16 +
 && echo "${TOFU_SHA256}  /tmp/tofu.zip" | sha256sum -c - \
17 +
 && unzip -q /tmp/tofu.zip tofu -d /usr/local/bin \
18 +
 && rm /tmp/tofu.zip
19 +
# The exact provider versions of the pipelines' lock files; `tofu
20 +
# providers mirror` checks each package against the registry's signed
21 +
# checksums as it downloads it.
22 +
COPY mirror.tf /tmp/mirror/
23 +
RUN cd /tmp/mirror && tofu providers mirror -platform=linux_amd64 /opt/tofu/mirror
24 +
# TFLint and its Google ruleset (HCL linting), each checked against its
25 +
# release's SHA-256; the ruleset where TFLint looks for plugins, so
26 +
# linting needs no `tflint --init` and no network.
27 +
ARG TFLINT_VERSION
28 +
ARG TFLINT_SHA256
29 +
ARG TFLINT_GOOGLE_VERSION
30 +
ARG TFLINT_GOOGLE_SHA256
31 +
RUN curl -fsSL -o /tmp/tflint.zip \
32 +
      "https://github.com/terraform-linters/tflint/releases/download/v${TFLINT_VERSION}/tflint_linux_amd64.zip" \
33 +
 && echo "${TFLINT_SHA256}  /tmp/tflint.zip" | sha256sum -c - \
34 +
 && unzip -q /tmp/tflint.zip tflint -d /usr/local/bin \
35 +
 && curl -fsSL -o /tmp/google.zip \
36 +
      "https://github.com/terraform-linters/tflint-ruleset-google/releases/download/v${TFLINT_GOOGLE_VERSION}/tflint-ruleset-google_linux_amd64.zip" \
37 +
 && echo "${TFLINT_GOOGLE_SHA256}  /tmp/google.zip" | sha256sum -c - \
38 +
 && d=/opt/tflint/plugins/github.com/terraform-linters/tflint-ruleset-google/${TFLINT_GOOGLE_VERSION} \
39 +
 && mkdir -p "$d" && unzip -q /tmp/google.zip -d "$d" \
40 +
 && rm /tmp/tflint.zip /tmp/google.zip
41 +
42 +
FROM debian:bookworm-slim
43 +
# curl, jq, xz, sfdisk and e2fsprogs' debugfs: oxygenhub-gcp's bake job
44 +
# builds the hub's VM image by writing files into Debian's cloud disk image,
45 +
# no root needed, and publishes it through Google's APIs (oxygenhub-gcp #7).
46 +
RUN apt-get update \
47 +
 && apt-get install -y --no-install-recommends ca-certificates curl jq xz-utils fdisk e2fsprogs \
48 +
 && rm -rf /var/lib/apt/lists/*
49 +
COPY --from=fetch /usr/local/bin/tofu /usr/local/bin/tofu
50 +
COPY --from=fetch /opt/tofu/mirror /opt/tofu/mirror
51 +
COPY --from=fetch /usr/local/bin/tflint /usr/local/bin/tflint
52 +
COPY --from=fetch /opt/tflint/plugins /opt/tflint/plugins
53 +
# Providers come only from the mirror: a job never reaches a registry.
54 +
COPY cli.tfrc /etc/tofu/cli.tfrc
55 +
ENV TF_CLI_CONFIG_FILE=/etc/tofu/cli.tfrc \
56 +
    TFLINT_PLUGIN_DIR=/opt/tflint/plugins
added images/tofu/README.md +54 −0
1 +
# The OxygenFlow OpenTofu image
2 +
3 +
A root file system for deploy jobs (oxygenflow#56):
4 +
5 +
- Debian bookworm, slim, with CA certificates.
6 +
- OpenTofu 1.13.1, checked against its published SHA-256.
7 +
- A provider mirror in `/opt/tofu/mirror`: `hashicorp/google` 8.7.0,
8 +
  `hashicorp/google-beta` 8.7.0 (Firebase) and `cloudflare/cloudflare`
9 +
  5.28.0, checked by `tofu providers mirror` against the registry's signed
10 +
  checksums when the image is built.
11 +
- `/etc/tofu/cli.tfrc` (set by `TF_CLI_CONFIG_FILE`) installs providers
12 +
  from the mirror only.
13 +
- TFLint 0.64.0 and its Google ruleset 0.40.0, each checked against its
14 +
  release's SHA-256, the ruleset in `/opt/tflint/plugins`
15 +
  (`TFLINT_PLUGIN_DIR`): a repository's `.tflint.hcl` enables it with
16 +
  `plugin "google" { enabled = true  version = "0.40.0"  source =
17 +
  "github.com/terraform-linters/tflint-ruleset-google" }`, and linting
18 +
  needs no `tflint --init`.
19 +
20 +
So `tofu init`, `fmt` and `validate` need no network; `plan` and `apply`
21 +
reach the cloud's APIs through allow-listed egress (#55). The guest's
22 +
environment is the pipeline's, so a job sets `TF_CLI_CONFIG_FILE` itself
23 +
(below).
24 +
25 +
## Building and importing
26 +
27 +
```
28 +
images/tofu/build.sh /path/to/oxygenflow-tofu.tar
29 +
scp /path/to/oxygenflow-tofu.tar claude@earth:/home/claude/flow/dl/
30 +
ssh claude@earth 'cd ~/flow && FLOW_HOME=~/flow bin/flow rootfs import dl/oxygenflow-tofu.tar'
31 +
```
32 +
33 +
Pin the printed hash in the pipelines that use it, and push it to the hub
34 +
(`flow rootfs push HASH`) so every runner can fetch it.
35 +
36 +
A provider version in a pipeline's lock file must be in the mirror: bump
37 +
`mirror.tf`, rebuild and re-pin when a lock file changes.
38 +
39 +
## Using it
40 +
41 +
```kdl
42 +
sandbox "tofu" {
43 +
    runtime "microvm"
44 +
    rootfs "cas://blob/<hash printed by flow rootfs import>"
45 +
    memory-limit "2GiB"
46 +
    cpu-limit 2
47 +
    network "isolated"
48 +
    env {
49 +
        TF_CLI_CONFIG_FILE "/etc/tofu/cli.tfrc"
50 +
        TF_IN_AUTOMATION "1"
51 +
        TFLINT_PLUGIN_DIR "/opt/tflint/plugins"
52 +
    }
53 +
}
54 +
```
added images/tofu/build.sh +23 −0
1 +
#!/bin/sh
2 +
# Build the OxygenFlow OpenTofu image and write it as a root file system
3 +
# tarball for `flow rootfs import`. Rebuild when OpenTofu or a pinned
4 +
# provider version, TFLint or its Google ruleset changes.
5 +
set -eu
6 +
here=$(cd "$(dirname "$0")" && pwd)
7 +
out=${1:-$here/oxygenflow-tofu.tar}
8 +
TOFU_VERSION=1.13.1
9 +
TOFU_SHA256=8ccbc6f8ee21d2827715f3c6e08a9b3e0209b1e62057c05067ef117e047c1a80
10 +
TFLINT_VERSION=0.64.0
11 +
TFLINT_SHA256=cca9d13e2e1d7a2c627af60ff899a3c9b74212899416aeb96ec764d2ef954537
12 +
TFLINT_GOOGLE_VERSION=0.40.0
13 +
TFLINT_GOOGLE_SHA256=a5c02b3de937456de2521155831eaf1cc876014a21445bf7dd1621654511fe28
14 +
15 +
docker build -q -t oxygenflow-tofu \
16 +
    --build-arg TOFU_VERSION=$TOFU_VERSION --build-arg TOFU_SHA256=$TOFU_SHA256 \
17 +
    --build-arg TFLINT_VERSION=$TFLINT_VERSION --build-arg TFLINT_SHA256=$TFLINT_SHA256 \
18 +
    --build-arg TFLINT_GOOGLE_VERSION=$TFLINT_GOOGLE_VERSION --build-arg TFLINT_GOOGLE_SHA256=$TFLINT_GOOGLE_SHA256 \
19 +
    "$here"
20 +
id=$(docker create oxygenflow-tofu)
21 +
docker export "$id" -o "$out"
22 +
docker rm "$id" >/dev/null
23 +
echo "$out"
added images/tofu/cli.tfrc +8 −0
1 +
provider_installation {
2 +
  filesystem_mirror {
3 +
    path    = "/opt/tofu/mirror"
4 +
    include = ["registry.opentofu.org/*/*"]
5 +
  }
6 +
  # Nothing is fetched from a registry: a provider missing from the
7 +
  # mirror fails `tofu init` instead of reaching out.
8 +
}
added images/tofu/mirror.tf +20 −0
1 +
# The providers the image carries, at the versions the deploy pipelines
2 +
# lock. Bump here, rebuild, re-pin (README.md).
3 +
terraform {
4 +
  required_providers {
5 +
    google = {
6 +
      source  = "hashicorp/google"
7 +
      version = "8.7.0"
8 +
    }
9 +
    # Firebase (oxygenhub-gcp decision 0008): its project and apps are
10 +
    # google-beta resources.
11 +
    google-beta = {
12 +
      source  = "hashicorp/google-beta"
13 +
      version = "8.7.0"
14 +
    }
15 +
    cloudflare = {
16 +
      source  = "cloudflare/cloudflare"
17 +
      version = "5.28.0"
18 +
    }
19 +
  }
20 +
}
added scripts/changelog-check.sh +27 −0
1 +
#!/bin/sh
2 +
# The changelog check (2026-10-19): a commit that changes what users get
3 +
# adds a line under "## Unreleased" in CHANGELOG.md, so a release's notes
4 +
# are written as the work is done and versions say what changed.
5 +
#
6 +
#   scripts/changelog-check.sh [PATH...]
7 +
#
8 +
# PATHs are what the commit will take (as given to `oxy commit`); without
9 +
# any, the working copy's changes. Code is anything under src/, crates/,
10 +
# templates/, assets/, packaging/, images/ or scripts/, except tests.
11 +
# Docs, pipelines, pins (deps/) and lock files need no line. For a code
12 +
# change users won't notice (a refactor, a test helper), set NO_CHANGELOG=1.
13 +
set -eu
14 +
[ "${NO_CHANGELOG:-}" = 1 ] && exit 0
15 +
cd "$(dirname "$0")/.."
16 +
if [ "$#" -gt 0 ]; then
17 +
    paths=$(printf '%s\n' "$@")
18 +
else
19 +
    paths=$(oxy --plain status --json | python3 -c 'import json,sys; print("\n".join(p["path"] for p in json.load(sys.stdin)["paths"]))')
20 +
fi
21 +
code=$(printf '%s\n' "$paths" | grep -E '^(src|crates|templates|assets|packaging|images|scripts)/' | grep -vE '(^|/)tests?/|_test\.rs$' || true)
22 +
[ -z "$code" ] && exit 0
23 +
printf '%s\n' "$paths" | grep -qx 'CHANGELOG.md' && exit 0
24 +
echo "changelog: these changes touch code but CHANGELOG.md has no new line under ## Unreleased:" >&2
25 +
printf '%s\n' "$code" | sed 's/^/  /' >&2
26 +
echo "add one (Added / Changed / Fixed / Security, with the issue number), or set NO_CHANGELOG=1 if users won't notice" >&2
27 +
exit 1
added scripts/checks-wasm.sh +36 −0
1 +
#!/bin/sh
2 +
# Build the cheap checks (crates/flow-checks, #96) as a WASI module and keep
3 +
# it at .flow/checks.wasm, where the pipeline's WASM jobs run it from.
4 +
#
5 +
#   scripts/checks-wasm.sh          rebuild .flow/checks.wasm
6 +
#   scripts/checks-wasm.sh --check  fail unless .flow/checks.wasm is what
7 +
#                                   the source builds (the fmt job runs this)
8 +
#
9 +
# The build is the same on any machine with the same toolchain (the build
10 +
# image's, images/rust): the crate is std only, the workspace path is
11 +
# remapped to /src and rust-src's to /rustc/COMMIT, and the module carries
12 +
# no debug info.
13 +
# Needs: rustup target wasm32-wasip1.
14 +
set -eu
15 +
cd "$(dirname "$0")/.."
16 +
target=$(mktemp -d)
17 +
trap 'rm -rf "$target"' EXIT
18 +
# Where rust-src is installed, the standard library's panic locations name
19 +
# it; map it to /rustc/COMMIT, as a toolchain without rust-src has them.
20 +
src="$(rustc --print sysroot)/lib/rustlib/src/rust"
21 +
commit=$(rustc -vV | sed -n 's/^commit-hash: //p')
22 +
RUSTFLAGS="--remap-path-prefix=$(pwd)=/src --remap-path-prefix=$src=/rustc/$commit" \
23 +
CARGO_PROFILE_RELEASE_DEBUG=0 CARGO_PROFILE_RELEASE_STRIP=true CARGO_INCREMENTAL=0 \
24 +
    cargo build -q --release --locked --target wasm32-wasip1 -p flow-checks --target-dir "$target"
25 +
built="$target/wasm32-wasip1/release/flow-checks.wasm"
26 +
if [ "${1:-}" = --check ]; then
27 +
    if cmp -s "$built" .flow/checks.wasm; then
28 +
        echo ".flow/checks.wasm is what crates/flow-checks builds ($(rustc --version))"
29 +
    else
30 +
        echo ".flow/checks.wasm differs from what crates/flow-checks builds: run scripts/checks-wasm.sh and commit it" >&2
31 +
        exit 1
32 +
    fi
33 +
else
34 +
    cp "$built" .flow/checks.wasm
35 +
    echo "built .flow/checks.wasm ($(wc -c < .flow/checks.wasm) bytes, $(rustc --version))"
36 +
fi
added scripts/dist.sh +41 −0
1 +
#!/usr/bin/env bash
2 +
# Build the runner, `flow`, for every Linux machine a runner runs on
3 +
# (glibc 2.28 and later), as release assets runners update themselves from
4 +
# (docs/jetstream.md §8.2, oxygenhub#322): target/dist-flat/flow-linux-ARCH,
5 +
# named by the architecture a runner reports (x86_64, aarch64, riscv64),
6 +
# plain stripped binaries; flow-windows-x86_64.exe (the WASM tier only);
7 +
# and SHA256SUMS of them.
8 +
#
9 +
#   scripts/dist.sh [VERSION]    (default: the package version)
10 +
#   ONLY=x86_64 scripts/dist.sh  builds one architecture (a glob)
11 +
#
12 +
# Needs: rustup targets aarch64-unknown-linux-gnu and
13 +
# riscv64gc-unknown-linux-gnu and x86_64-pc-windows-gnu; cargo-zigbuild
14 +
# and zig on PATH; the MinGW-w64 gcc.
15 +
set -euo pipefail
16 +
17 +
cd "$(dirname "$0")/.."
18 +
VERSION="${1:-$(cargo pkgid -p flow | sed 's/.*[#@]//')}"
19 +
VERSION="${VERSION#v}"
20 +
OUT="${DIST_FLAT:-target/dist-flat}"
21 +
rm -rf "$OUT" && mkdir -p "$OUT"
22 +
export PATH="$HOME/.local/opt/zig:$PATH"
23 +
# Stripped, with no debug info: about a tenth of the size.
24 +
export CARGO_PROFILE_RELEASE_DEBUG=0 CARGO_PROFILE_RELEASE_STRIP=true CARGO_INCREMENTAL=0
25 +
# The version the binary reports, which the hub compares with its releases.
26 +
export OXY_BUILD="v$VERSION"
27 +
28 +
for p in "x86_64-unknown-linux-gnu|x86_64" "aarch64-unknown-linux-gnu|aarch64" "riscv64gc-unknown-linux-gnu|riscv64"; do
29 +
  IFS='|' read -r triple arch <<<"$p"
30 +
  if [ -n "${ONLY:-}" ] && [[ "$arch" != $ONLY ]]; then continue; fi
31 +
  echo "== linux-$arch ($triple)"
32 +
  cargo zigbuild -q --release --locked --target "$triple.2.28" -p flow --bin flow
33 +
  install -m 0755 "target/$triple/release/flow" "$OUT/flow-linux-$arch"
34 +
done
35 +
# Windows (oxygenflow#83): the WASM tier only, with the MinGW-w64 toolchain.
36 +
if [ -z "${ONLY:-}" ] || [[ windows-x86_64 == $ONLY ]]; then
37 +
  echo "== windows-x86_64 (x86_64-pc-windows-gnu)"
38 +
  cargo build -q --release --locked --target x86_64-pc-windows-gnu -p flow --bin flow
39 +
  install -m 0755 target/x86_64-pc-windows-gnu/release/flow.exe "$OUT/flow-windows-x86_64.exe"
40 +
fi
41 +
(cd "$OUT" && sha256sum flow-* > SHA256SUMS && cat SHA256SUMS)