OXYGENHUB

Image push: a big image's manifest is recorded even when the sender gives up, and a kept one is not read again (closes #341) d9d0d314fdcb

Author
Andy McCall @andymccall <[email protected]>
via
agent agent:claude@andymccall <[email protected]> · keyless, under your grant for changes,refs,seal · key a669da278f48
When
· 2026-10-26 21:27 UTC
Signature
◈ sealed by agent:claude@andymccall
Hash
d9d0d314fdcbc309903206d93879124522323789dd8f8d2d28c7c92acb4bb52f
Depends on
42fa95540468 92a797b20912 93501da2a162 a7e6e8e6094c b5668108487f d7e2c2754807
Depended on by
7a91f1fd71b0
In bookmarks
diffusion-pin-bookmark main
In tags
v0.9.11 v0.9.15 v0.9.16 v0.9.19 v0.9.20 v0.9.21 v0.9.22 v0.9.23 v0.9.24 v0.9.7 v0.9.8 v0.9.9
Edits
5 atoms, 0 tree operations, 3 files

Browse the tree at this change

modified CHANGELOG.md +1 −0
…
3 unchanged lines
4 4
5 5
## Unreleased
6 6
7 +
- **A big image's push no longer fails at the last step (#341).** The hub reads a whole image before it accepts the manifest, which on a busy disk outlasted `flow rootfs push`'s wait. The image could then be kept but never offered, and a retry read it all again. The hub now finishes and records the image even if the sender gives up, and a manifest it already keeps is recorded without reading the image again.
7 8
- **Listing landed proposals is quick, and pages (#331).** `oh proposal list --state landed` on a repository with a few hundred proposals timed out: the API worked out every proposal's projected state again, landed and closed ones too. Those now come from what was recorded, and the list takes `limit` (`oh proposal list --limit N`, 30 by default), newest first.
8 9
- **The API refuses a field an edit does not take (#330).** `PATCH` on an issue, proposal, project, milestone, release or drop answered 200 to a field it does not know and changed nothing (`{"ready": true}` on a draft proposal, say), which looked like success. It now answers 422 naming the field and the ones it takes.
9 10
- **Mail reads `[OxygenHub]`, in a wider message (#334).** A notification's subject was tagged with the site's name (`[Oxygenhub on Earth]`); mail now goes by the From address's display name (`OxygenHub <noreply@…>`) in the subject and the footer, falling back to the site's name, so the site keeps its own name (which its apt repository is labelled with). The message card is 720px wide rather than 560.
…
350 unchanged lines
modified crates/hub/src/api/images.rs +20 −8
…
93 unchanged lines
94 94
    if m.hash != hash {
95 95
        return Err(ApiError(StatusCode::BAD_REQUEST, format!("the manifest is for {}, not {}", m.hash, hash)));
96 96
    }
97 -
    let store = app.images();
98 -
    // Kept with the architecture its own files name (oxygenhub#173).
99 -
    let m = tokio::task::spawn_blocking(move || store.keep_reading_arch(&m))
100 -
        .await
101 -
        .map_err(|e| ApiError(StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?
102 -
        .map_err(|e| ApiError(StatusCode::CONFLICT, e))?;
103 -
    crate::images::record(&app.db, &m, &who).await?;
104 -
    crate::db::accounts::audit(&app.db, None, "flow.image", &who, &format!("{} ({} MiB)", &m.hash[..12], m.size >> 20)).await.ok();
97 +
    // In a task of its own (#341): reading a big image can outlast the
98 +
    // sender's wait, and a sender that gives up must not leave an image
99 +
    // kept but never offered.
100 +
    let m = tokio::spawn(async move {
101 +
        let store = app.images();
102 +
        // A manifest kept before was verified then; only its row may be missing.
103 +
        let m = match store.manifest(&m.hash).filter(|k| k.chunks == m.chunks) {
104 +
            Some(kept) => kept,
105 +
            // Kept with the architecture its own files name (oxygenhub#173).
106 +
            None => tokio::task::spawn_blocking(move || store.keep_reading_arch(&m))
107 +
                .await
108 +
                .map_err(|e| ApiError(StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?
109 +
                .map_err(|e| ApiError(StatusCode::CONFLICT, e))?,
110 +
        };
111 +
        crate::images::record(&app.db, &m, &who).await?;
112 +
        crate::db::accounts::audit(&app.db, None, "flow.image", &who, &format!("{} ({} MiB)", &m.hash[..12], m.size >> 20)).await.ok();
113 +
        Ok::<_, ApiError>(m)
114 +
    })
115 +
    .await
116 +
    .map_err(|e| ApiError(StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))??;
105 117
    Ok((StatusCode::CREATED, Json(serde_json::json!({ "hash": m.hash, "size": m.size, "chunks": m.chunks.len(), "arch": m.arch }))))
106 118
}
107 119
…
17 unchanged lines
modified crates/hub/tests/flow.rs +11 −0
…
3186 unchanged lines
3187 3187
        call(&h.app, "PUT", &format!("/api/v1/flow/images/{}", wrong.hash), Some("site"), Some(serde_json::to_string(&wrong).unwrap()))
3188 3188
            .await;
3189 3189
    assert_eq!(st, StatusCode::CONFLICT, "chunks that make another image");
3190 +
    let (st, body) = call(&h.app, "PUT", &format!("/api/v1/flow/images/{hash}"), Some("site"), Some(manifest.clone())).await;
3191 +
    assert_eq!(st, StatusCode::CREATED, "{body}");
3192 +
    // #341: a sender that gave up during the check left the manifest kept
3193 +
    // but no row; sending it again records it without reading the image
3194 +
    // again (a chunk gone from disk would fail a second check).
3195 +
    sqlx::query("DELETE FROM jet_images").execute(&h.app.db).await.unwrap();
3196 +
    let gone = h.app.config.data.join("images/chunks").join(&missing[0][..2]).join(&missing[0]);
3197 +
    let kept = std::fs::read(&gone).unwrap();
3198 +
    std::fs::remove_file(&gone).unwrap();
3190 3199
    let (st, body) = call(&h.app, "PUT", &format!("/api/v1/flow/images/{hash}"), Some("site"), Some(manifest)).await;
3191 3200
    assert_eq!(st, StatusCode::CREATED, "{body}");
3201 +
    assert!(oxygenhub::images::held(&h.app.db).await.unwrap().contains(&hash));
3202 +
    std::fs::write(&gone, kept).unwrap();
3192 3203
3193 3204
    // A runner fetches the manifest and the chunks, and the image comes back whole.
3194 3205
    let (st, body) = call(&h.app, "GET", &format!("/api/v1/flow/images/{hash}"), Some("runner-secret"), None).await;
…
3717 unchanged lines