OXYGENHUB

Build image for Rust and Node jobs (images/rust: toolchain, rustfmt, clippy, Node 24, vendored crates, offline cargo); flow-init finds the program in the job's PATH (glibc's execvpe used PID 1's own); network "loopback" in M0 eb5c4db673e5

closes #3
Author
Andy McCall @andymccall <[email protected]>
When
· 2026-09-28 17:39 UTC
Signature
✔ signed by Andy McCall
Hash
eb5c4db673e5480f4d5069e905fc1bb1f8cbd722af230f826fdb52bc3418c520
Depends on
e0918b887850 fc10d9e80383
Depended on by
f3b5b86e23ef 5544c8998aa7 c8917cf1f931 28cc603f737d 077a65242359 60d848f2474b ea81796bb757 e3fd8257ce05 fad6152112b1 ea56c56eaf90 e2c125610f43 22007df200f3 85ad9087a566 2b40cb9b87e1 eba589f87e4f a72abeeedd78 c4285535b03a 50354f576ea6
In bookmarks
main newer-pipeline push-uses-login
In tags
v0.2.0 v0.3.0 v0.3.1 v0.4.0 v0.5.0 v0.5.1 v0.5.2 v0.5.3
Edits
21 atoms, 6 tree operations, 10 files

Browse the tree at this change

added images/rust/Dockerfile +26 −0
1 +
# The OxygenFlow build image (backlog F1b): Rust with rustfmt and clippy,
2 +
# Node, and every crate the Oxygen projects use, vendored, because jobs
3 +
# have no network. Built by ./build.sh, which vendors the crates first.
4 +
FROM rust:1.97.1-slim-bookworm
5 +
6 +
RUN rustup component add rustfmt clippy \
7 +
 && apt-get update \
8 +
 && apt-get install -y --no-install-recommends ca-certificates xz-utils make git pkg-config \
9 +
 && rm -rf /var/lib/apt/lists/*
10 +
11 +
# The guest's PATH is fixed and has no /usr/local/cargo/bin, and a rustup
12 +
# proxy would need RUSTUP_HOME: link the toolchain's own binaries instead.
13 +
RUN tc=$(ls -d /usr/local/rustup/toolchains/*/ | head -1) \
14 +
 && for b in cargo rustc rustdoc cargo-fmt rustfmt cargo-clippy clippy-driver; do ln -sf "$tc/bin/$b" /usr/local/bin/$b; done
15 +
16 +
ARG NODE_VERSION
17 +
ARG NODE_SHA256
18 +
ADD https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-x64.tar.xz /tmp/node.tar.xz
19 +
RUN echo "${NODE_SHA256}  /tmp/node.tar.xz" | sha256sum -c - \
20 +
 && tar -xJf /tmp/node.tar.xz -C /usr/local --strip-components=1 --exclude='*/include' --exclude='*/share/doc' \
21 +
 && rm /tmp/node.tar.xz
22 +
23 +
# Cargo reads /.cargo/config.toml for anything under /, so /src uses the
24 +
# vendored crates; CARGO_HOME stays the default $HOME/.cargo (/tmp).
25 +
COPY vendor /opt/vendor
26 +
COPY cargo-config.toml /.cargo/config.toml
added images/rust/README.md +51 −0
1 +
# The OxygenFlow build image
2 +
3 +
A root file system for Rust and Node jobs:
4 +
5 +
- Rust 1.97.1 with rustfmt and clippy.
6 +
- Node 24.21.0, checked against its published SHA-256.
7 +
- git, make and pkg-config.
8 +
- Every crate that oxygen, oxygenhub and oxygenflow use, vendored in `/opt/vendor`.
9 +
10 +
OxygenFlow jobs have no network (`network "isolated"`). So cargo is pointed at the vendored crates by `/.cargo/config.toml`, which cargo reads for any directory under `/`, including `/src`. `CARGO_HOME` stays at its default, `$HOME/.cargo`, which is under `/tmp` and writable. The toolchain's binaries are linked into `/usr/local/bin`, because the guest's `PATH` is fixed and nothing at run time needs rustup.
11 +
12 +
## Building and importing
13 +
14 +
```
15 +
images/rust/build.sh /path/to/oxygenflow-build.tar
16 +
scp /path/to/oxygenflow-build.tar claude@earth:/home/claude/flow/dl/
17 +
ssh claude@earth 'cd ~/flow && FLOW_HOME=~/flow bin/flow rootfs import dl/oxygenflow-build.tar'
18 +
```
19 +
20 +
`build.sh` vendors the crates of the three workspaces next to this repository with `cargo vendor --locked`, then builds the image. Docker is used only on the machine that builds it, as a way to assemble a file system: the image is exported as a plain tarball. On the runner, `flow rootfs import` makes an ext4 image pinned by BLAKE3, which Firecracker boots read-only.
21 +
22 +
Rebuild and re-import whenever a `Cargo.lock` of those projects changes, then pin the new hash in the pipelines that use the image. A crate missing from the image fails the job with cargo's "no matching package" error, not with a network error.
23 +
24 +
## Using it
25 +
26 +
```kdl
27 +
sandbox "rust" {
28 +
    runtime "microvm"
29 +
    rootfs "cas://blob/<hash printed by flow rootfs import>"
30 +
    memory-limit "8GiB"
31 +
    cpu-limit 4
32 +
    network "loopback"   // tests that serve on 127.0.0.1 need it; nothing leaves the guest
33 +
}
34 +
35 +
job "test" {
36 +
    sandbox "rust"
37 +
    mount {
38 +
        source "ephemeral://disk"
39 +
        target "/src/target"
40 +
        size "24GiB"
41 +
    }
42 +
    env {
43 +
        CARGO_INCREMENTAL "0"
44 +
    }
45 +
    exec "cargo" "test" "--workspace"
46 +
}
47 +
```
48 +
49 +
Measured on Earth (4 cores, the job given all of them and 8 GiB): oxygen's `cargo fmt --check`, `clippy -D warnings` and `cargo test --workspace` from a cold build take 135 s.
50 +
51 +
The build tree goes on an `ephemeral://disk` mount. The workspace itself has only 512 MiB to spare, and a `tmpfs` would take the build tree out of the job's memory.
added images/rust/build.sh +25 −0
1 +
#!/bin/sh
2 +
# Build the OxygenFlow build image and write it as a root file system
3 +
# tarball for `flow rootfs import`. Run from any directory; the Oxygen
4 +
# projects are expected next to this repository (../oxygen, ../oxygenhub).
5 +
# Rebuild whenever a Cargo.lock of those projects changes.
6 +
set -eu
7 +
here=$(cd "$(dirname "$0")" && pwd)
8 +
dev=$(cd "$here/../../.." && pwd)
9 +
out=${1:-$here/oxygenflow-build.tar}
10 +
NODE_VERSION=24.21.0
11 +
NODE_SHA256=fd8e59d5a511510f6a298afb548f18c7d2b1be404d8b4a27d94fbe49f56cb2d6
12 +
13 +
rm -rf "$here/vendor"
14 +
cargo vendor --locked --versioned-dirs \
15 +
    --manifest-path "$dev/oxygen/Cargo.toml" \
16 +
    --sync "$dev/oxygenhub/Cargo.toml" \
17 +
    --sync "$dev/oxygenflow/Cargo.toml" \
18 +
    "$here/vendor" >/dev/null
19 +
docker build -q -t oxygenflow-build \
20 +
    --build-arg NODE_VERSION=$NODE_VERSION --build-arg NODE_SHA256=$NODE_SHA256 "$here"
21 +
id=$(docker create oxygenflow-build)
22 +
docker export "$id" -o "$out"
23 +
docker rm "$id" >/dev/null
24 +
rm -rf "$here/vendor"
25 +
echo "$out"
added images/rust/cargo-config.toml +9 −0
1 +
# Crates come from the image, never the network (OxygenFlow jobs are isolated).
2 +
[source.crates-io]
3 +
replace-with = "vendored"
4 +
5 +
[source.vendored]
6 +
directory = "/opt/vendor"
7 +
8 +
[net]
9 +
offline = true
modified .oxyignore +2 −0
1 1
target/
2 2
*.vsix
3 +
images/rust/vendor/
4 +
images/rust/*.tar
modified crates/flow-core/src/pipeline.rs +5 −2
…
497 unchanged lines
498 498
            "network" => {
499 499
                network = match ctx.one_string(c).as_deref() {
500 500
                    Some("isolated") => Some(Network::Isolated),
501 -
                    Some("loopback") | Some("egress") => {
502 -
                        ctx.err(c.span(), "only network \"isolated\" is in M0");
501 +
                    // Loopback only: the guest's `lo` comes up, and it still has
502 +
                    // no device that reaches anything outside it.
503 +
                    Some("loopback") => Some(Network::Loopback),
504 +
                    Some("egress") => {
505 +
                        ctx.err(c.span(), "network \"egress\" is not in M0; M0 has \"isolated\" and \"loopback\"");
503 506
                        None
504 507
                    }
505 508
                    Some(other) => {
…
186 unchanged lines
modified crates/flow-core/src/tests.rs +11 −0
…
245 unchanged lines
246 246
        (0..5).map(|i| format!("    mount {{ source \"ephemeral://tmpfs\"; target \"/tmp/m{i}\"; size \"16MiB\"; }}\n")).collect();
247 247
    has(doc(&format!("job \"b\" {{\n    sandbox \"alpine\"\n{five}    exec \"true\"\n}}\n")), "at most 4 mounts");
248 248
}
249 +
250 +
#[test]
251 +
fn loopback_is_in_m0_and_egress_is_not() {
252 +
    let src = |net: &str| {
253 +
        format!("schema-version 1\nsandbox \"s\" {{\n  rootfs \"cas://blob/{ROOTFS}\"\n  network \"{net}\"\n}}\njob \"a\" {{ sandbox \"s\"; exec \"true\"; }}\n")
254 +
    };
255 +
    let p = Pipeline::parse(&src("loopback")).unwrap();
256 +
    assert_eq!(p.sandboxes["s"].network, Network::Loopback);
257 +
    assert_ne!(p.hash(), Pipeline::parse(&src("isolated")).unwrap().hash(), "the network is part of the sandbox");
258 +
    assert!(problems(&src("egress")).iter().any(|(l, m)| *l == 4 && m.contains("not in M0")));
259 +
}
modified crates/flow-init/src/main.rs +75 −1
…
20 unchanged lines
21 21
    env: Vec<(String, String)>,
22 22
    #[serde(default)]
23 23
    mounts: Vec<MountSpec>,
24 +
    /// `network "loopback"`: bring up `lo` (the guest has no other device).
25 +
    #[serde(default)]
26 +
    loopback: bool,
24 27
}
25 28
26 29
/// An ephemeral mount from the host: a tmpfs of `size_mib`, or the disk at
…
62 unchanged lines
89 92
    for d in missing.iter().rev() {
90 93
        std::fs::create_dir(d).map_err(|e| format!("mkdir {}: {e}", d.display()))?;
91 94
        chown(&d.to_string_lossy(), uid, gid)?;
95 +
    }
96 +
    Ok(())
97 +
}
98 +
99 +
/// The file `program` names: itself when it holds a `/`, otherwise the first
100 +
/// executable regular file of that name in `path`'s directories.
101 +
fn resolve_program(program: &str, path: &str) -> Option<String> {
102 +
    use std::os::unix::fs::PermissionsExt;
103 +
    if program.is_empty() {
104 +
        return None;
105 +
    }
106 +
    if program.contains('/') {
107 +
        return Some(program.to_string());
108 +
    }
109 +
    path.split(':')
110 +
        .filter(|d| !d.is_empty())
111 +
        .map(|d| format!("{d}/{program}"))
112 +
        .find(|p| std::fs::metadata(p).is_ok_and(|m| m.is_file() && m.permissions().mode() & 0o111 != 0))
113 +
}
114 +
115 +
/// Set `lo` up; the kernel gives it 127.0.0.1 and ::1 itself.
116 +
fn loopback_up() -> Result<(), String> {
117 +
    let fd = unsafe { libc::socket(libc::AF_INET, libc::SOCK_DGRAM | libc::SOCK_CLOEXEC, 0) };
118 +
    if fd < 0 {
119 +
        return Err(format!("socket for lo: {}", std::io::Error::last_os_error()));
120 +
    }
121 +
    let mut req: libc::ifreq = unsafe { std::mem::zeroed() };
122 +
    for (i, b) in b"lo".iter().enumerate() {
123 +
        req.ifr_name[i] = *b as libc::c_char;
124 +
    }
125 +
    let ok = unsafe {
126 +
        libc::ioctl(fd, libc::SIOCGIFFLAGS, &mut req) == 0 && {
127 +
            req.ifr_ifru.ifru_flags |= libc::IFF_UP as libc::c_short;
128 +
            libc::ioctl(fd, libc::SIOCSIFFLAGS, &req) == 0
129 +
        }
130 +
    };
131 +
    let err = std::io::Error::last_os_error();
132 +
    unsafe { libc::close(fd) };
133 +
    if !ok {
134 +
        return Err(format!("bring up lo: {err}"));
92 135
    }
93 136
    Ok(())
94 137
}
…
52 unchanged lines
147 190
            (kind, _) => return Err(format!("unknown mount kind {kind}")),
148 191
        }
149 192
    }
193 +
    if job.loopback {
194 +
        loopback_up()?;
195 +
    }
150 196
    let root = cstr("/newroot")?;
151 197
    if unsafe { libc::chroot(root.as_ptr()) } != 0 || std::env::set_current_dir("/src").is_err() {
152 198
        return Err(format!("enter the rootfs: {}", std::io::Error::last_os_error()));
153 199
    }
154 200
155 201
    let argv: Vec<CString> = job.argv.iter().map(|a| cstr(a)).collect::<Result<_, _>>()?;
202 +
    // glibc's execvpe searches the caller's PATH, and PID 1 has none of its
203 +
    // own, so the program is found here, in the job's PATH, before forking.
204 +
    let program = cstr(&resolve_program(&job.argv[0], PATH).ok_or_else(|| format!("cannot run {:?}: not found in {PATH}", job.argv[0]))?)?;
156 205
    let mut env: Vec<CString> = vec![cstr(&format!("PATH={PATH}"))?, cstr("HOME=/tmp")?];
157 206
    for (k, v) in &job.env {
158 207
        env.push(cstr(&format!("{k}={v}"))?);
…
22 unchanged lines
181 230
            a.push(std::ptr::null());
182 231
            let mut e: Vec<*const libc::c_char> = env.iter().map(|c| c.as_ptr()).collect();
183 232
            e.push(std::ptr::null());
184 -
            libc::execvpe(a[0], a.as_ptr(), e.as_ptr());
233 +
            libc::execve(program.as_ptr(), a.as_ptr(), e.as_ptr());
185 234
            let msg = format!("flow: cannot run {:?}: {}\n", job.argv[0], std::io::Error::last_os_error());
186 235
            libc::write(2, msg.as_ptr() as *const libc::c_void, msg.len());
187 236
            libc::_exit(127);
…
46 unchanged lines
234 283
    }
235 284
    if !line.is_empty() {
236 285
        frame(console, kind, &line);
286 +
    }
287 +
}
288 +
289 +
#[cfg(test)]
290 +
mod tests {
291 +
    use super::resolve_program;
292 +
    use std::os::unix::fs::PermissionsExt;
293 +
294 +
    #[test]
295 +
    fn programs_are_found_in_the_jobs_path() {
296 +
        let dir = std::env::temp_dir().join(format!("flow-init-test-{}", std::process::id()));
297 +
        let (a, b) = (dir.join("a"), dir.join("b"));
298 +
        std::fs::create_dir_all(&a).unwrap();
299 +
        std::fs::create_dir_all(&b).unwrap();
300 +
        std::fs::write(a.join("tool"), "not executable").unwrap();
301 +
        std::fs::write(b.join("tool"), "#!/bin/sh\n").unwrap();
302 +
        std::fs::set_permissions(b.join("tool"), std::fs::Permissions::from_mode(0o755)).unwrap();
303 +
        std::fs::create_dir_all(a.join("dir")).unwrap();
304 +
        let path = format!("{}::{}", a.display(), b.display());
305 +
        assert_eq!(resolve_program("tool", &path), Some(format!("{}/tool", b.display())), "skips a file that is not executable");
306 +
        assert_eq!(resolve_program("dir", &path), None, "and directories");
307 +
        assert_eq!(resolve_program("missing", &path), None);
308 +
        assert_eq!(resolve_program("./x", &path).as_deref(), Some("./x"), "a path is used as given");
309 +
        assert_eq!(resolve_program("", &path), None);
310 +
        std::fs::remove_dir_all(&dir).unwrap();
237 311
    }
238 312
}
modified crates/flow/src/microvm.rs +2 −1
…
61 unchanged lines
62 62
    let rootfs = home.verified(&rootfs, &sandbox.rootfs)?;
63 63
    let environment = environment(home, sandbox, job)?;
64 64
65 -
    let spec = guest_spec(job);
65 +
    let mut spec = guest_spec(job);
66 +
    spec["loopback"] = serde_json::Value::Bool(sandbox.network == flow_core::Network::Loopback);
66 67
    // Each `ephemeral://disk` mount is a fresh sparse ext4 disk in the
67 68
    // scratch directory, attached after the workspace (vdc, vdd, ...).
68 69
    let mut disks = Vec::new();
…
157 unchanged lines
modified docs/m0.md +5 −3
…
23 unchanged lines
24 24
    rootfs "cas://blob/<64 hex>"  // BLAKE3 of an ext4 image the runner holds
25 25
    memory-limit "1GiB"          // KiB, MiB, GiB
26 26
    cpu-limit 2
27 -
    network "isolated"           // M0 accepts only "isolated"
27 +
    network "isolated"           // "isolated" or "loopback" in M0
28 28
}
29 29
30 30
job "lint" {
…
52 unchanged lines
83 83
   directory (a clone at the state), removes `.oxy`, and builds a
84 84
   workspace image with `mkfs.ext4 -d` sized to the tree plus 512 MiB.
85 85
3. Boots Firecracker with no network device (so `isolated` holds by
86 -
   construction), `vcpu_count` and `mem_size_mib` from the sandbox, the
86 +
   construction; `loopback` only brings up the guest's `lo`, so tests can
87 +
   serve and connect on 127.0.0.1 while nothing leaves the guest), `vcpu_count` and `mem_size_mib` from the sandbox, the
87 88
   pinned guest kernel, the rootfs as a **read-only** `vda`, the
88 89
   workspace as a read-write `vdb`, and an initramfs holding `flow-init`.
89 90
4. `flow-init` (a static binary, PID 1 in the guest) mounts `/proc`,
…
1 unchanged lines
91 92
   `/newroot/src`, and tmpfs at `/newroot/tmp` and `/newroot/run`, then the
92 93
   job's scratch mounts (creating their mount points, owned by the job's
93 94
   user); it
94 -
   switches root, then runs `argv` with exactly the job's `env` (plus
95 +
   switches root, then runs `argv` (its program found in the guest's `PATH` by
96 +
   `flow-init` itself) with exactly the job's `env` (plus
95 97
   `PATH`, `HOME=/tmp`) in `/src` as uid 1000, with a job timeout.
96 98
5. `flow-init` frames the child's output on the serial console as
97 99
   `\x1eO <line>` (stdout) and `\x1eE <line>` (stderr), then
…
74 unchanged lines