OXYGENHUB
# 1408324d5d55 oxygenflow / CHANGELOG.md
14.4 KiB · Markdown · last changed OxygenFlow 0.3.1: pins oxy 0.6.0 11 days ago renderedsource blame history raw

Changelog

0.2.0 is OxygenFlow's first tagged release; until then the runners ran main. This file starts on 2026-10-01; earlier history is in oxy log.

Unreleased

0.3.1 (2026-10-25)

  • Runners read repositories with oxy 0.6.0: a tag withdrawn on the hub (oxygenhub#301) is dropped when a runner fetches, a keyless agent's changes verify in a repository that pins its owner's newest key (oxygen#135) or has never seen it (oxygen#134), and shallow clones of big repositories work (oxygen#137). The runner itself is unchanged.

0.3.0 (2026-10-21)

  • A runner's own metrics (#35): metrics = "127.0.0.1:9464" under [runner] serves Prometheus text at /metrics: claims, declines, finished jobs by result, the seconds they ran, the jobs running now, version and uptime, optionally behind metrics_token. Off unless set.
  • Built with oxy 0.5.1. A runner reads repositories whose release tags a keyless agent made (oxygen#129), and declares certified-tags to the hub (oxygen#131).
  • Build caches are pruned before they are kept (#72, spec §5.4). A kept cache no longer grows until the job's disk is full. When a passing job's cache holds more than cache_prune_percent of its mount's size (runner config, default 60), the files the job did not read, write or share a cargo unit hash with are removed, oldest first, until it is under; one still over is not kept, and the next run starts empty. A microVM's guest init prunes its disk before it powers off; the runner prunes the syscall tier's directories. The job's log says how much was kept and pruned, or why the cache was not kept.

0.2.0 (2026-10-20)

  • Approval gates and environments (#57, #59, oxygenhub#235, spec §5.3.2, §5.3.3). approval { by "maintain"; others-only; expires "24h" } on a job makes the hub hold it until a person with that role (write, maintain by default, or admin) approves it, and with others-only not the person who pushed. approval { sealed #true } asks for an approval signed with the approver's own key (oxy seal --flow), the default for a job that names an environment "prod"; the hub names the environment in the job's identity tokens only once it is approved. Both are part of the job's hash.

  • Artifacts between jobs (#58, spec §5.3.1). input { artifact "x" from "job"; path "/src/x" } takes an artifact from a job named in depends-on, up to 8 per job. The runner fetches each from the hub, refuses the job when the blob is not the one the producer's receipt names, writes it at its path and lists it in the receipt's inputs.

  • Allow-listed egress (#55, spec §4.3.1). network "egress" { allow "host" "*.example.com" } lets a microVM job reach those names on port 443 and nothing else. The VM still has no network device: a proxy on the runner takes only CONNECT host:443 for an allowed name, refuses non-public addresses and logs every decision in the job's log; the job gets HTTPS_PROXY. Off unless a runner sets egress = true under [runner]; such jobs go only to runners that do.

  • Receipts signed with a key that dies with the job (oxygenhub#241). With keyless = true under [runner], the runner makes a fresh key just before signing a job's receipt, has the hub certify it with the job's token, signs, attaches the certificate and drops the key. If the hub cannot certify, the runner's own key signs and the job's log says why.

  • A pipeline may lower its jobs' priority (oxygenhub#244). priority "low" (or "normal") on a job, or at the top of the file for every job, puts it behind other work in the hub's queue. high and urgent are refused with the reason: they are the hub's to give.

  • Triggers (#24). on { push; tag; manual; schedule } for the pipeline and each job: push and tag take globs, manual is Run pipeline, and schedule takes five-field cron lines. A job's own on wins over the pipeline's; without one a job runs on every push and by hand, as before.

  • Runner labels (oxygenhub#41). A sandbox or a job may say runs-on "LABEL"; a job goes only to a runner that has every label it and its sandbox name (flow runner register --label).

  • Images by hash (#39, #44). flow rootfs push HASH sends an image to a hub, only the chunks it lacks, and flow rootfs pull HASH fetches one. A runner whose job's image is missing fetches it before starting, saying so in the job's log, and keeps the chunks, so a rebuilt image costs only its changed chunks. Both commands use the login oh auth login saved when OXY_TOKEN is not set.

  • Runners say what they have and what jobs used (#40, #41). Each claim carries the runner's free disk, load, kernel and a speed class from a short benchmark, taken at start and every 15 minutes while idle (flow runner bench prints them). A job's finish carries its peak disk and memory, for the hub to learn what its next run needs.

  • Reports (oxygenhub#127). report "NAME" { path "…"; when "…" } in a job's output block uploads a directory holding report.toml, file by file with a manifest, within 2,000 files and 512 MiB; the receipt names the manifest.

  • Artifacts from a failing job (#37). An artifact with when "always" is uploaded when the job fails or times out too, so a test report survives the failure it describes. It never goes in the receipt of a job that did not pass.

  • A cancelled run stops its jobs (#32). A job whose run is cancelled is stopped: its microVM ends, or never boots if the job was still being prepared, and nothing more is sent.

  • The OpenTofu image (#56). images/tofu: OpenTofu 1.13.1 with a provider mirror (hashicorp/google, cloudflare/cloudflare), TFLint with its Google ruleset, and curl, jq, xz, sfdisk and debugfs, for deploy jobs. Providers come only from the mirror, so tofu init and linting need no network.

  • Coverage in the Rust build image (oxygenhub#113). cargo-llvm-cov and the toolchain's llvm-tools are on the guest's PATH, so a job can write an LCOV report for the hub's Coverage tab.

  • The syscall tier keeps its build caches too (#70, spec §5.4). A cached ephemeral://disk mount in a runtime "syscall" job now starts from the directory a passing job of the same slot kept, and goes back into the runner's pool when the job passes, with the same order, bookmark rules, cache_gb eviction and claim reporting as the microVM tier's disks, so jobs go where their cache is on boards without KVM as well. Nothing kept is writable by others or set-id, and the pool is the runner's alone.

  • The build cache pays off (#51, #52, spec §5.4). A cached disk keeps the content hashes and times of the sources it was built from, and the next job gives every unchanged file its old time back, so cargo rebuilds only the crates that changed instead of every path crate. Claims list the caches a runner holds, and the hub sends a job to the runner holding its cache, as it does for images.

  • Restarts cost no job (#49, #50, spec §11.5.1). A job's uploads and declines wait out a hub that is away for up to three minutes (a deploy restarting it), through refused connections as well as server errors, and log lines it could not send go with the next batch instead of being dropped. A runner told to stop drains: it claims nothing more, finishes and reports the jobs it holds, then exits; a second signal stops it at once. With KillMode=mixed and a TimeoutStopSec longer than the longest job in its unit, deploying a runner no longer has to wait for it to be idle.

  • A build cache (#28, spec §5.4). A disk mount may name a cache and the key files it depends on, such as Cargo.lock. The runner keeps passing jobs' disks and starts the next job of the same repository, job and bookmark from one, so builds carry their compiled dependencies forward; another bookmark copies it rather than taking it. Failing jobs' disks are never kept, the receipt names what a disk started from, and the pool stays under cache_gb (default 64 GiB).

  • Runners share images (#47, docs/jetstream.md §8.1). A runner with a [runner.swarm] section serves the image chunks it holds to other runners in its site, and fetches an image it lacks from them and the hub together, rarest chunks first. The hub is asked only for chunks no peer has or is fetching, so runners starting together split its work. Peers talk over TLS: each runner keeps its own certificate, the hub hands its fingerprint on with the peer, and a fetcher trusts no other. Peers admit each other with a ticket the hub signs, and every chunk is checked against its hash. Uploads are capped (upload_mib_s, default 50) at 16 connections. Without the section a runner fetches from the hub alone, as before.

  • A runner declines a job it cannot run (#36, spec §11.5). Before fetching anything it checks the tier, the architecture, that the image is there and verifies, and that the sandbox's memory is free. If any check fails it hands the job back to the hub with the reason instead of running and failing it.

  • The syscall tier (#36, docs/m0.md §3). runtime "syscall" jobs run without KVM, so boards such as the OrangePi RV2 can take them. The sandbox uses user, mount, PID, network, IPC and UTS namespaces on the image's unpacked tree, read-only. Landlock limits writes to /src and /tmp, and a seccomp filter refuses network sockets, tracing, mounts, BPF, keys, modules and io_uring. A delegated cgroup v2 group enforces the CPU and memory limits, and a job killed for memory is told so. A runner offers the tier only where all of this works. flow tiers shows which tiers a machine can run and why the others cannot, and flow run runs syscall jobs too.

  • A runner offers only the tiers it can run (#36). It checks that /dev/kvm opens before offering the MicroVM tier, says which tiers it serves when it starts, and refuses to start with none, naming why, rather than claiming jobs it would fail.

  • A runner tries a fetch again after a network hiccup (oxygenhub#111). Fetching or cloning the job's repository or a subrepo is retried on DNS, connection, timeout and 502/503/504 errors, waiting 2, 4, 8, 16 and 30 seconds; any other error still ends the job at once.

  • Architecture awareness (#36, spec §11.2). flow rootfs import records an image's architecture from its shell's ELF header and refuses an image for another architecture; a claim reports each image's architecture, and a sandbox may say arch "riscv64".

  • A job's identity (oxygenhub#103). identity "AUDIENCE" { mount-path "/run/secrets/…" } asks the hub for an OIDC token for that audience, mounted as a file like a secret.

  • Secrets as files (oxygenhub#107). A job's secret "name" { mount-path "/run/secrets/…" } names a secret the hub holds; it comes with the claim and flow-init writes it at that path, 0400, on a tmpfs, before the job starts. Never in the environment, the receipt or an artifact (spec §9). flow run --secret NAME=VALUE locally.

  • Variables from the hub (oxygenhub#102). A claim carries env, the variables the hub holds for the repository; they join the job's environment under FLOW_* and the pipeline's own env. flow run --var NAME=VALUE does the same locally.

  • A job says how long it may run (#23). timeout "90m" on a job, or on a sandbox for its jobs (hours, minutes, seconds; a minute to a day). The hub caps it at the site's maximum.

  • A busy runner still says it is there (oxygenhub#243). A runner whose slots are full, or that is draining, claims nothing, so the hub showed it offline through a quiet job. It now sends a heartbeat every 30 s while it cannot claim.

  • One slow fetch no longer holds every job on a runner (#63). Workspaces were prepared under one runner-wide lock, so a job fetching a large repository from a bucket held all the others (one waited 819 s, and another's token expired before its fetch began). Fetches and exports now take only their repository's lock, and the job's log says how long it waited and how long fetching took.

  • An image with no recorded architecture is read, not trusted (#53). Such an image (imported before architectures were recorded, or fetched from a hub that did not know it) matched every runner, so a riscv64 runner could be handed an x86_64 image and fail. The runner now reads the architecture from the image's shell once, records it, reports it and declines a job for another architecture.

  • flow check says when a pipeline needs a newer flow (#43). A pipeline that names a node, output or event this flow does not know was called "not valid". flow check and flow run now say it uses something this version of flow does not know, and that a pipeline written for a newer OxygenFlow needs a newer flow.

  • Pushing a large image works against any hub (oxygenhub#147). flow rootfs push asked about 50,000 chunks at once, over a hub's 2 MB body limit, so an 11 GB image failed with 413. It now asks 20,000 at a time.

  • A pinned file is re-read from the disk before its hash is called wrong (#22).

  • A runner says which jobs it holds (oxygenhub#71). Each claim lists the jobs the runner is running or still has a result for, so the hub ends at once any job it lost by restarting, instead of an hour later at the time limit.

  • A runner says which images it holds (oxygenhub#73). Each claim lists the rootfs images in the runner's cas/ (read each time, so an image imported while it runs counts), and the hub offers only jobs whose image is among them. plex-1 had failed metalgear's jobs for want of one.

  • A job's result is kept until the hub has it (oxygenhub#71). The runner offered a finished job's result three times over seven seconds and then dropped it, so a hub that was briefly down (Earth's reset) left the job running forever. The result is now kept on disk (pending/ in the Flow home) and offered again, every 5 s doubling to a minute, for up to two hours, through any server error or lost connection; a runner that restarts sends what it kept. A refusal (the hub ended the job already) is final.

  • The Rust build image vendors mimalloc (#20). Oxygen's new allocator was missing from the image, so oxygen's runs #150 to #153 failed. The image was rebuilt with images/rust/build.sh, imported on Earth and pinned in the oxygen, oxygenhub and oxygenflow pipelines. Rebuild it whenever a Cargo.lock of those projects changes.

  • A hint for shell escapes in a pipeline. KDL 2.0 strings have no \033, \e or \x escapes; a pipeline that uses one (for ANSI colours, say) is still invalid, but flow check and the hub now name the line and say to write ESC as \u{1b}.