Image push: the manifest step times out on a big image and the image is never offered issue #341 closed: completed andymccall opened this 9 days ago · 0 comments

flow rootfs push of a large image fails at the last step on a busy hub, and the image never becomes fetchable.

Seen 2026-10-26: pushing the Rust image 836461b6… (14.5 GB, 25 October pin) from Earth. The chunks went up (about 600 MB new, 30 min), then the manifest PUT /flow/images/{hash} failed with "timed out reading response". No manifest was kept and no jet_images row was written. So placement still saw plex-1 and mars-1 as lacking the image, and every Rust job in oxygen, oxygenhub and oxygenflow queued on earth-1 alone.

Cause:

• put_manifest verifies by reading and hashing every chunk (the whole image) before it answers. On a busy disk that takes longer than the client's fixed 120 s timeout (flow/src/images.rs, the ureq agent).
• The jet_images row is written after the verification returns, in the request's own future. If the client disconnects, the future is dropped: a manifest the blocking task did keep has no row, and the hub never offers the image.
• A retry verifies the whole image again, so it times out again.

Fix:

• Hub: record the image in a detached task once verification passes, so a client that gives up doesn't lose it.
• Hub: answer a re-sent manifest the hub already keeps without verifying again (write the row if it is missing).
• Flow: give the manifest PUT a long timeout, since the hub reads the whole image before it answers.

Workaround used: the same manifest sent with curl --max-time 3600.