Build image for Rust and Node jobs (images/rust: toolchain, rustfmt, clippy, Node 24, vendored crates, offline cargo); flow-init finds the program in the job's PATH (glibc's execvpe used PID 1's own); network "loopback" in M0 eb5c4db673e5
closes #3
- Author
- Andy McCall @andymccall <[email protected]>
- When
- · 2026-09-28 17:39 UTC
- Signature
- ✔ signed by Andy McCall
- Hash
eb5c4db673e5480f4d5069e905fc1bb1f8cbd722af230f826fdb52bc3418c520- Depends on
- e0918b887850 fc10d9e80383
- Depended on by
- f3b5b86e23ef 5544c8998aa7 c8917cf1f931 28cc603f737d 077a65242359 60d848f2474b ea81796bb757 e3fd8257ce05 fad6152112b1 ea56c56eaf90 e2c125610f43 22007df200f3 85ad9087a566 2b40cb9b87e1 eba589f87e4f a72abeeedd78 c4285535b03a 50354f576ea6
- In bookmarks
- main newer-pipeline push-uses-login
- In tags
- v0.2.0 v0.3.0 v0.3.1 v0.4.0 v0.5.0 v0.5.1 v0.5.2 v0.5.3
- Edits
- 21 atoms, 6 tree operations, 10 files
Browse the tree at this change
added images/rust/Dockerfile +26 −0
| 1 | + | # The OxygenFlow build image (backlog F1b): Rust with rustfmt and clippy, |
|
| 2 | + | # Node, and every crate the Oxygen projects use, vendored, because jobs |
|
| 3 | + | # have no network. Built by ./build.sh, which vendors the crates first. |
|
| 4 | + | FROM rust:1.97.1-slim-bookworm |
|
| 5 | + | ||
| 6 | + | RUN rustup component add rustfmt clippy \ |
|
| 7 | + | && apt-get update \ |
|
| 8 | + | && apt-get install -y --no-install-recommends ca-certificates xz-utils make git pkg-config \ |
|
| 9 | + | && rm -rf /var/lib/apt/lists/* |
|
| 10 | + | ||
| 11 | + | # The guest's PATH is fixed and has no /usr/local/cargo/bin, and a rustup |
|
| 12 | + | # proxy would need RUSTUP_HOME: link the toolchain's own binaries instead. |
|
| 13 | + | RUN tc=$(ls -d /usr/local/rustup/toolchains/*/ | head -1) \ |
|
| 14 | + | && for b in cargo rustc rustdoc cargo-fmt rustfmt cargo-clippy clippy-driver; do ln -sf "$tc/bin/$b" /usr/local/bin/$b; done |
|
| 15 | + | ||
| 16 | + | ARG NODE_VERSION |
|
| 17 | + | ARG NODE_SHA256 |
|
| 18 | + | ADD https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-x64.tar.xz /tmp/node.tar.xz |
|
| 19 | + | RUN echo "${NODE_SHA256} /tmp/node.tar.xz" | sha256sum -c - \ |
|
| 20 | + | && tar -xJf /tmp/node.tar.xz -C /usr/local --strip-components=1 --exclude='*/include' --exclude='*/share/doc' \ |
|
| 21 | + | && rm /tmp/node.tar.xz |
|
| 22 | + | ||
| 23 | + | # Cargo reads /.cargo/config.toml for anything under /, so /src uses the |
|
| 24 | + | # vendored crates; CARGO_HOME stays the default $HOME/.cargo (/tmp). |
|
| 25 | + | COPY vendor /opt/vendor |
|
| 26 | + | COPY cargo-config.toml /.cargo/config.toml |
added images/rust/README.md +51 −0
| 1 | + | # The OxygenFlow build image |
|
| 2 | + | ||
| 3 | + | A root file system for Rust and Node jobs: |
|
| 4 | + | ||
| 5 | + | - Rust 1.97.1 with rustfmt and clippy. |
|
| 6 | + | - Node 24.21.0, checked against its published SHA-256. |
|
| 7 | + | - git, make and pkg-config. |
|
| 8 | + | - Every crate that oxygen, oxygenhub and oxygenflow use, vendored in `/opt/vendor`. |
|
| 9 | + | ||
| 10 | + | OxygenFlow jobs have no network (`network "isolated"`). So cargo is pointed at the vendored crates by `/.cargo/config.toml`, which cargo reads for any directory under `/`, including `/src`. `CARGO_HOME` stays at its default, `$HOME/.cargo`, which is under `/tmp` and writable. The toolchain's binaries are linked into `/usr/local/bin`, because the guest's `PATH` is fixed and nothing at run time needs rustup. |
|
| 11 | + | ||
| 12 | + | ## Building and importing |
|
| 13 | + | ||
| 14 | + | ``` |
|
| 15 | + | images/rust/build.sh /path/to/oxygenflow-build.tar |
|
| 16 | + | scp /path/to/oxygenflow-build.tar claude@earth:/home/claude/flow/dl/ |
|
| 17 | + | ssh claude@earth 'cd ~/flow && FLOW_HOME=~/flow bin/flow rootfs import dl/oxygenflow-build.tar' |
|
| 18 | + | ``` |
|
| 19 | + | ||
| 20 | + | `build.sh` vendors the crates of the three workspaces next to this repository with `cargo vendor --locked`, then builds the image. Docker is used only on the machine that builds it, as a way to assemble a file system: the image is exported as a plain tarball. On the runner, `flow rootfs import` makes an ext4 image pinned by BLAKE3, which Firecracker boots read-only. |
|
| 21 | + | ||
| 22 | + | Rebuild and re-import whenever a `Cargo.lock` of those projects changes, then pin the new hash in the pipelines that use the image. A crate missing from the image fails the job with cargo's "no matching package" error, not with a network error. |
|
| 23 | + | ||
| 24 | + | ## Using it |
|
| 25 | + | ||
| 26 | + | ```kdl |
|
| 27 | + | sandbox "rust" { |
|
| 28 | + | runtime "microvm" |
|
| 29 | + | rootfs "cas://blob/<hash printed by flow rootfs import>" |
|
| 30 | + | memory-limit "8GiB" |
|
| 31 | + | cpu-limit 4 |
|
| 32 | + | network "loopback" // tests that serve on 127.0.0.1 need it; nothing leaves the guest |
|
| 33 | + | } |
|
| 34 | + | ||
| 35 | + | job "test" { |
|
| 36 | + | sandbox "rust" |
|
| 37 | + | mount { |
|
| 38 | + | source "ephemeral://disk" |
|
| 39 | + | target "/src/target" |
|
| 40 | + | size "24GiB" |
|
| 41 | + | } |
|
| 42 | + | env { |
|
| 43 | + | CARGO_INCREMENTAL "0" |
|
| 44 | + | } |
|
| 45 | + | exec "cargo" "test" "--workspace" |
|
| 46 | + | } |
|
| 47 | + | ``` |
|
| 48 | + | ||
| 49 | + | Measured on Earth (4 cores, the job given all of them and 8 GiB): oxygen's `cargo fmt --check`, `clippy -D warnings` and `cargo test --workspace` from a cold build take 135 s. |
|
| 50 | + | ||
| 51 | + | The build tree goes on an `ephemeral://disk` mount. The workspace itself has only 512 MiB to spare, and a `tmpfs` would take the build tree out of the job's memory. |
added images/rust/build.sh +25 −0
| 1 | + | #!/bin/sh |
|
| 2 | + | # Build the OxygenFlow build image and write it as a root file system |
|
| 3 | + | # tarball for `flow rootfs import`. Run from any directory; the Oxygen |
|
| 4 | + | # projects are expected next to this repository (../oxygen, ../oxygenhub). |
|
| 5 | + | # Rebuild whenever a Cargo.lock of those projects changes. |
|
| 6 | + | set -eu |
|
| 7 | + | here=$(cd "$(dirname "$0")" && pwd) |
|
| 8 | + | dev=$(cd "$here/../../.." && pwd) |
|
| 9 | + | out=${1:-$here/oxygenflow-build.tar} |
|
| 10 | + | NODE_VERSION=24.21.0 |
|
| 11 | + | NODE_SHA256=fd8e59d5a511510f6a298afb548f18c7d2b1be404d8b4a27d94fbe49f56cb2d6 |
|
| 12 | + | ||
| 13 | + | rm -rf "$here/vendor" |
|
| 14 | + | cargo vendor --locked --versioned-dirs \ |
|
| 15 | + | --manifest-path "$dev/oxygen/Cargo.toml" \ |
|
| 16 | + | --sync "$dev/oxygenhub/Cargo.toml" \ |
|
| 17 | + | --sync "$dev/oxygenflow/Cargo.toml" \ |
|
| 18 | + | "$here/vendor" >/dev/null |
|
| 19 | + | docker build -q -t oxygenflow-build \ |
|
| 20 | + | --build-arg NODE_VERSION=$NODE_VERSION --build-arg NODE_SHA256=$NODE_SHA256 "$here" |
|
| 21 | + | id=$(docker create oxygenflow-build) |
|
| 22 | + | docker export "$id" -o "$out" |
|
| 23 | + | docker rm "$id" >/dev/null |
|
| 24 | + | rm -rf "$here/vendor" |
|
| 25 | + | echo "$out" |
added images/rust/cargo-config.toml +9 −0
| 1 | + | # Crates come from the image, never the network (OxygenFlow jobs are isolated). |
|
| 2 | + | [source.crates-io] |
|
| 3 | + | replace-with = "vendored" |
|
| 4 | + | ||
| 5 | + | [source.vendored] |
|
| 6 | + | directory = "/opt/vendor" |
|
| 7 | + | ||
| 8 | + | [net] |
|
| 9 | + | offline = true |
modified .oxyignore +2 −0
| 1 | 1 | target/ |
|
| 2 | 2 | *.vsix |
|
| 3 | + | images/rust/vendor/ |
|
| 4 | + | images/rust/*.tar |
modified crates/flow-core/src/pipeline.rs +5 −2
| … | 497 unchanged lines |
||
| 498 | 498 | "network" => { |
|
| 499 | 499 | network = match ctx.one_string(c).as_deref() { |
|
| 500 | 500 | Some("isolated") => Some(Network::Isolated), |
|
| 501 | - | Some("loopback") | Some("egress") => { |
|
| 502 | - | ctx.err(c.span(), "only network \"isolated\" is in M0"); |
|
| 501 | + | // Loopback only: the guest's `lo` comes up, and it still has |
|
| 502 | + | // no device that reaches anything outside it. |
|
| 503 | + | Some("loopback") => Some(Network::Loopback), |
|
| 504 | + | Some("egress") => { |
|
| 505 | + | ctx.err(c.span(), "network \"egress\" is not in M0; M0 has \"isolated\" and \"loopback\""); |
|
| 503 | 506 | None |
|
| 504 | 507 | } |
|
| 505 | 508 | Some(other) => { |
|
| … | 186 unchanged lines |
modified crates/flow-core/src/tests.rs +11 −0
| … | 245 unchanged lines |
||
| 246 | 246 | (0..5).map(|i| format!(" mount {{ source \"ephemeral://tmpfs\"; target \"/tmp/m{i}\"; size \"16MiB\"; }}\n")).collect(); |
|
| 247 | 247 | has(doc(&format!("job \"b\" {{\n sandbox \"alpine\"\n{five} exec \"true\"\n}}\n")), "at most 4 mounts"); |
|
| 248 | 248 | } |
|
| 249 | + | ||
| 250 | + | #[test] |
|
| 251 | + | fn loopback_is_in_m0_and_egress_is_not() { |
|
| 252 | + | let src = |net: &str| { |
|
| 253 | + | format!("schema-version 1\nsandbox \"s\" {{\n rootfs \"cas://blob/{ROOTFS}\"\n network \"{net}\"\n}}\njob \"a\" {{ sandbox \"s\"; exec \"true\"; }}\n") |
|
| 254 | + | }; |
|
| 255 | + | let p = Pipeline::parse(&src("loopback")).unwrap(); |
|
| 256 | + | assert_eq!(p.sandboxes["s"].network, Network::Loopback); |
|
| 257 | + | assert_ne!(p.hash(), Pipeline::parse(&src("isolated")).unwrap().hash(), "the network is part of the sandbox"); |
|
| 258 | + | assert!(problems(&src("egress")).iter().any(|(l, m)| *l == 4 && m.contains("not in M0"))); |
|
| 259 | + | } |
modified crates/flow-init/src/main.rs +75 −1
| … | 20 unchanged lines |
||
| 21 | 21 | env: Vec<(String, String)>, |
|
| 22 | 22 | #[serde(default)] |
|
| 23 | 23 | mounts: Vec<MountSpec>, |
|
| 24 | + | /// `network "loopback"`: bring up `lo` (the guest has no other device). |
|
| 25 | + | #[serde(default)] |
|
| 26 | + | loopback: bool, |
|
| 24 | 27 | } |
|
| 25 | 28 | ||
| 26 | 29 | /// An ephemeral mount from the host: a tmpfs of `size_mib`, or the disk at |
|
| … | 62 unchanged lines |
||
| 89 | 92 | for d in missing.iter().rev() { |
|
| 90 | 93 | std::fs::create_dir(d).map_err(|e| format!("mkdir {}: {e}", d.display()))?; |
|
| 91 | 94 | chown(&d.to_string_lossy(), uid, gid)?; |
|
| 95 | + | } |
|
| 96 | + | Ok(()) |
|
| 97 | + | } |
|
| 98 | + | ||
| 99 | + | /// The file `program` names: itself when it holds a `/`, otherwise the first |
|
| 100 | + | /// executable regular file of that name in `path`'s directories. |
|
| 101 | + | fn resolve_program(program: &str, path: &str) -> Option<String> { |
|
| 102 | + | use std::os::unix::fs::PermissionsExt; |
|
| 103 | + | if program.is_empty() { |
|
| 104 | + | return None; |
|
| 105 | + | } |
|
| 106 | + | if program.contains('/') { |
|
| 107 | + | return Some(program.to_string()); |
|
| 108 | + | } |
|
| 109 | + | path.split(':') |
|
| 110 | + | .filter(|d| !d.is_empty()) |
|
| 111 | + | .map(|d| format!("{d}/{program}")) |
|
| 112 | + | .find(|p| std::fs::metadata(p).is_ok_and(|m| m.is_file() && m.permissions().mode() & 0o111 != 0)) |
|
| 113 | + | } |
|
| 114 | + | ||
| 115 | + | /// Set `lo` up; the kernel gives it 127.0.0.1 and ::1 itself. |
|
| 116 | + | fn loopback_up() -> Result<(), String> { |
|
| 117 | + | let fd = unsafe { libc::socket(libc::AF_INET, libc::SOCK_DGRAM | libc::SOCK_CLOEXEC, 0) }; |
|
| 118 | + | if fd < 0 { |
|
| 119 | + | return Err(format!("socket for lo: {}", std::io::Error::last_os_error())); |
|
| 120 | + | } |
|
| 121 | + | let mut req: libc::ifreq = unsafe { std::mem::zeroed() }; |
|
| 122 | + | for (i, b) in b"lo".iter().enumerate() { |
|
| 123 | + | req.ifr_name[i] = *b as libc::c_char; |
|
| 124 | + | } |
|
| 125 | + | let ok = unsafe { |
|
| 126 | + | libc::ioctl(fd, libc::SIOCGIFFLAGS, &mut req) == 0 && { |
|
| 127 | + | req.ifr_ifru.ifru_flags |= libc::IFF_UP as libc::c_short; |
|
| 128 | + | libc::ioctl(fd, libc::SIOCSIFFLAGS, &req) == 0 |
|
| 129 | + | } |
|
| 130 | + | }; |
|
| 131 | + | let err = std::io::Error::last_os_error(); |
|
| 132 | + | unsafe { libc::close(fd) }; |
|
| 133 | + | if !ok { |
|
| 134 | + | return Err(format!("bring up lo: {err}")); |
|
| 92 | 135 | } |
|
| 93 | 136 | Ok(()) |
|
| 94 | 137 | } |
|
| … | 52 unchanged lines |
||
| 147 | 190 | (kind, _) => return Err(format!("unknown mount kind {kind}")), |
|
| 148 | 191 | } |
|
| 149 | 192 | } |
|
| 193 | + | if job.loopback { |
|
| 194 | + | loopback_up()?; |
|
| 195 | + | } |
|
| 150 | 196 | let root = cstr("/newroot")?; |
|
| 151 | 197 | if unsafe { libc::chroot(root.as_ptr()) } != 0 || std::env::set_current_dir("/src").is_err() { |
|
| 152 | 198 | return Err(format!("enter the rootfs: {}", std::io::Error::last_os_error())); |
|
| 153 | 199 | } |
|
| 154 | 200 | ||
| 155 | 201 | let argv: Vec<CString> = job.argv.iter().map(|a| cstr(a)).collect::<Result<_, _>>()?; |
|
| 202 | + | // glibc's execvpe searches the caller's PATH, and PID 1 has none of its |
|
| 203 | + | // own, so the program is found here, in the job's PATH, before forking. |
|
| 204 | + | let program = cstr(&resolve_program(&job.argv[0], PATH).ok_or_else(|| format!("cannot run {:?}: not found in {PATH}", job.argv[0]))?)?; |
|
| 156 | 205 | let mut env: Vec<CString> = vec![cstr(&format!("PATH={PATH}"))?, cstr("HOME=/tmp")?]; |
|
| 157 | 206 | for (k, v) in &job.env { |
|
| 158 | 207 | env.push(cstr(&format!("{k}={v}"))?); |
|
| … | 22 unchanged lines |
||
| 181 | 230 | a.push(std::ptr::null()); |
|
| 182 | 231 | let mut e: Vec<*const libc::c_char> = env.iter().map(|c| c.as_ptr()).collect(); |
|
| 183 | 232 | e.push(std::ptr::null()); |
|
| 184 | - | libc::execvpe(a[0], a.as_ptr(), e.as_ptr()); |
|
| 233 | + | libc::execve(program.as_ptr(), a.as_ptr(), e.as_ptr()); |
|
| 185 | 234 | let msg = format!("flow: cannot run {:?}: {}\n", job.argv[0], std::io::Error::last_os_error()); |
|
| 186 | 235 | libc::write(2, msg.as_ptr() as *const libc::c_void, msg.len()); |
|
| 187 | 236 | libc::_exit(127); |
|
| … | 46 unchanged lines |
||
| 234 | 283 | } |
|
| 235 | 284 | if !line.is_empty() { |
|
| 236 | 285 | frame(console, kind, &line); |
|
| 286 | + | } |
|
| 287 | + | } |
|
| 288 | + | ||
| 289 | + | #[cfg(test)] |
|
| 290 | + | mod tests { |
|
| 291 | + | use super::resolve_program; |
|
| 292 | + | use std::os::unix::fs::PermissionsExt; |
|
| 293 | + | ||
| 294 | + | #[test] |
|
| 295 | + | fn programs_are_found_in_the_jobs_path() { |
|
| 296 | + | let dir = std::env::temp_dir().join(format!("flow-init-test-{}", std::process::id())); |
|
| 297 | + | let (a, b) = (dir.join("a"), dir.join("b")); |
|
| 298 | + | std::fs::create_dir_all(&a).unwrap(); |
|
| 299 | + | std::fs::create_dir_all(&b).unwrap(); |
|
| 300 | + | std::fs::write(a.join("tool"), "not executable").unwrap(); |
|
| 301 | + | std::fs::write(b.join("tool"), "#!/bin/sh\n").unwrap(); |
|
| 302 | + | std::fs::set_permissions(b.join("tool"), std::fs::Permissions::from_mode(0o755)).unwrap(); |
|
| 303 | + | std::fs::create_dir_all(a.join("dir")).unwrap(); |
|
| 304 | + | let path = format!("{}::{}", a.display(), b.display()); |
|
| 305 | + | assert_eq!(resolve_program("tool", &path), Some(format!("{}/tool", b.display())), "skips a file that is not executable"); |
|
| 306 | + | assert_eq!(resolve_program("dir", &path), None, "and directories"); |
|
| 307 | + | assert_eq!(resolve_program("missing", &path), None); |
|
| 308 | + | assert_eq!(resolve_program("./x", &path).as_deref(), Some("./x"), "a path is used as given"); |
|
| 309 | + | assert_eq!(resolve_program("", &path), None); |
|
| 310 | + | std::fs::remove_dir_all(&dir).unwrap(); |
|
| 237 | 311 | } |
|
| 238 | 312 | } |
modified crates/flow/src/microvm.rs +2 −1
| … | 61 unchanged lines |
||
| 62 | 62 | let rootfs = home.verified(&rootfs, &sandbox.rootfs)?; |
|
| 63 | 63 | let environment = environment(home, sandbox, job)?; |
|
| 64 | 64 | ||
| 65 | - | let spec = guest_spec(job); |
|
| 65 | + | let mut spec = guest_spec(job); |
|
| 66 | + | spec["loopback"] = serde_json::Value::Bool(sandbox.network == flow_core::Network::Loopback); |
|
| 66 | 67 | // Each `ephemeral://disk` mount is a fresh sparse ext4 disk in the |
|
| 67 | 68 | // scratch directory, attached after the workspace (vdc, vdd, ...). |
|
| 68 | 69 | let mut disks = Vec::new(); |
|
| … | 157 unchanged lines |
modified docs/m0.md +5 −3
| … | 23 unchanged lines |
||
| 24 | 24 | rootfs "cas://blob/<64 hex>" // BLAKE3 of an ext4 image the runner holds |
|
| 25 | 25 | memory-limit "1GiB" // KiB, MiB, GiB |
|
| 26 | 26 | cpu-limit 2 |
|
| 27 | - | network "isolated" // M0 accepts only "isolated" |
|
| 27 | + | network "isolated" // "isolated" or "loopback" in M0 |
|
| 28 | 28 | } |
|
| 29 | 29 | ||
| 30 | 30 | job "lint" { |
|
| … | 52 unchanged lines |
||
| 83 | 83 | directory (a clone at the state), removes `.oxy`, and builds a |
|
| 84 | 84 | workspace image with `mkfs.ext4 -d` sized to the tree plus 512 MiB. |
|
| 85 | 85 | 3. Boots Firecracker with no network device (so `isolated` holds by |
|
| 86 | - | construction), `vcpu_count` and `mem_size_mib` from the sandbox, the |
|
| 86 | + | construction; `loopback` only brings up the guest's `lo`, so tests can |
|
| 87 | + | serve and connect on 127.0.0.1 while nothing leaves the guest), `vcpu_count` and `mem_size_mib` from the sandbox, the |
|
| 87 | 88 | pinned guest kernel, the rootfs as a **read-only** `vda`, the |
|
| 88 | 89 | workspace as a read-write `vdb`, and an initramfs holding `flow-init`. |
|
| 89 | 90 | 4. `flow-init` (a static binary, PID 1 in the guest) mounts `/proc`, |
|
| … | 1 unchanged lines |
||
| 91 | 92 | `/newroot/src`, and tmpfs at `/newroot/tmp` and `/newroot/run`, then the |
|
| 92 | 93 | job's scratch mounts (creating their mount points, owned by the job's |
|
| 93 | 94 | user); it |
|
| 94 | - | switches root, then runs `argv` with exactly the job's `env` (plus |
|
| 95 | + | switches root, then runs `argv` (its program found in the guest's `PATH` by |
|
| 96 | + | `flow-init` itself) with exactly the job's `env` (plus |
|
| 95 | 97 | `PATH`, `HOME=/tmp`) in `/src` as uid 1000, with a job timeout. |
|
| 96 | 98 | 5. `flow-init` frames the child's output on the serial console as |
|
| 97 | 99 | `\x1eO <line>` (stdout) and `\x1eE <line>` (stderr), then |
|
| … | 74 unchanged lines |