Image push: a big image's manifest is recorded even when the sender gives up, and a kept one is not read again (closes #341) d9d0d314fdcb
- Author
- Andy McCall @andymccall <[email protected]>
- via
- agent
agent:claude@andymccall <[email protected]> · keyless, under your grant for changes,refs,seal · key a669da278f48
- When
- · 2026-10-26 21:27 UTC
- Signature
- ◈ sealed by agent:claude@andymccall
- Hash
d9d0d314fdcbc309903206d93879124522323789dd8f8d2d28c7c92acb4bb52f- Depends on
- 42fa95540468 92a797b20912 93501da2a162 a7e6e8e6094c b5668108487f d7e2c2754807
- Depended on by
- 7a91f1fd71b0
- In bookmarks
- diffusion-pin-bookmark main
- In tags
- v0.9.11 v0.9.15 v0.9.16 v0.9.19 v0.9.20 v0.9.21 v0.9.22 v0.9.23 v0.9.24 v0.9.7 v0.9.8 v0.9.9
- Edits
- 5 atoms, 0 tree operations, 3 files
Browse the tree at this change
modified CHANGELOG.md +1 −0
| … | 3 unchanged lines |
||
| 4 | 4 | ||
| 5 | 5 | ## Unreleased |
|
| 6 | 6 | ||
| 7 | + | - **A big image's push no longer fails at the last step (#341).** The hub reads a whole image before it accepts the manifest, which on a busy disk outlasted `flow rootfs push`'s wait. The image could then be kept but never offered, and a retry read it all again. The hub now finishes and records the image even if the sender gives up, and a manifest it already keeps is recorded without reading the image again. |
|
| 7 | 8 | - **Listing landed proposals is quick, and pages (#331).** `oh proposal list --state landed` on a repository with a few hundred proposals timed out: the API worked out every proposal's projected state again, landed and closed ones too. Those now come from what was recorded, and the list takes `limit` (`oh proposal list --limit N`, 30 by default), newest first. |
|
| 8 | 9 | - **The API refuses a field an edit does not take (#330).** `PATCH` on an issue, proposal, project, milestone, release or drop answered 200 to a field it does not know and changed nothing (`{"ready": true}` on a draft proposal, say), which looked like success. It now answers 422 naming the field and the ones it takes. |
|
| 9 | 10 | - **Mail reads `[OxygenHub]`, in a wider message (#334).** A notification's subject was tagged with the site's name (`[Oxygenhub on Earth]`); mail now goes by the From address's display name (`OxygenHub <noreply@…>`) in the subject and the footer, falling back to the site's name, so the site keeps its own name (which its apt repository is labelled with). The message card is 720px wide rather than 560. |
|
| … | 350 unchanged lines |
modified crates/hub/src/api/images.rs +20 −8
| … | 93 unchanged lines |
||
| 94 | 94 | if m.hash != hash { |
|
| 95 | 95 | return Err(ApiError(StatusCode::BAD_REQUEST, format!("the manifest is for {}, not {}", m.hash, hash))); |
|
| 96 | 96 | } |
|
| 97 | - | let store = app.images(); |
|
| 98 | - | // Kept with the architecture its own files name (oxygenhub#173). |
|
| 99 | - | let m = tokio::task::spawn_blocking(move || store.keep_reading_arch(&m)) |
|
| 100 | - | .await |
|
| 101 | - | .map_err(|e| ApiError(StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))? |
|
| 102 | - | .map_err(|e| ApiError(StatusCode::CONFLICT, e))?; |
|
| 103 | - | crate::images::record(&app.db, &m, &who).await?; |
|
| 104 | - | crate::db::accounts::audit(&app.db, None, "flow.image", &who, &format!("{} ({} MiB)", &m.hash[..12], m.size >> 20)).await.ok(); |
|
| 97 | + | // In a task of its own (#341): reading a big image can outlast the |
|
| 98 | + | // sender's wait, and a sender that gives up must not leave an image |
|
| 99 | + | // kept but never offered. |
|
| 100 | + | let m = tokio::spawn(async move { |
|
| 101 | + | let store = app.images(); |
|
| 102 | + | // A manifest kept before was verified then; only its row may be missing. |
|
| 103 | + | let m = match store.manifest(&m.hash).filter(|k| k.chunks == m.chunks) { |
|
| 104 | + | Some(kept) => kept, |
|
| 105 | + | // Kept with the architecture its own files name (oxygenhub#173). |
|
| 106 | + | None => tokio::task::spawn_blocking(move || store.keep_reading_arch(&m)) |
|
| 107 | + | .await |
|
| 108 | + | .map_err(|e| ApiError(StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))? |
|
| 109 | + | .map_err(|e| ApiError(StatusCode::CONFLICT, e))?, |
|
| 110 | + | }; |
|
| 111 | + | crate::images::record(&app.db, &m, &who).await?; |
|
| 112 | + | crate::db::accounts::audit(&app.db, None, "flow.image", &who, &format!("{} ({} MiB)", &m.hash[..12], m.size >> 20)).await.ok(); |
|
| 113 | + | Ok::<_, ApiError>(m) |
|
| 114 | + | }) |
|
| 115 | + | .await |
|
| 116 | + | .map_err(|e| ApiError(StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))??; |
|
| 105 | 117 | Ok((StatusCode::CREATED, Json(serde_json::json!({ "hash": m.hash, "size": m.size, "chunks": m.chunks.len(), "arch": m.arch })))) |
|
| 106 | 118 | } |
|
| 107 | 119 | ||
| … | 17 unchanged lines |
modified crates/hub/tests/flow.rs +11 −0
| … | 3186 unchanged lines |
||
| 3187 | 3187 | call(&h.app, "PUT", &format!("/api/v1/flow/images/{}", wrong.hash), Some("site"), Some(serde_json::to_string(&wrong).unwrap())) |
|
| 3188 | 3188 | .await; |
|
| 3189 | 3189 | assert_eq!(st, StatusCode::CONFLICT, "chunks that make another image"); |
|
| 3190 | + | let (st, body) = call(&h.app, "PUT", &format!("/api/v1/flow/images/{hash}"), Some("site"), Some(manifest.clone())).await; |
|
| 3191 | + | assert_eq!(st, StatusCode::CREATED, "{body}"); |
|
| 3192 | + | // #341: a sender that gave up during the check left the manifest kept |
|
| 3193 | + | // but no row; sending it again records it without reading the image |
|
| 3194 | + | // again (a chunk gone from disk would fail a second check). |
|
| 3195 | + | sqlx::query("DELETE FROM jet_images").execute(&h.app.db).await.unwrap(); |
|
| 3196 | + | let gone = h.app.config.data.join("images/chunks").join(&missing[0][..2]).join(&missing[0]); |
|
| 3197 | + | let kept = std::fs::read(&gone).unwrap(); |
|
| 3198 | + | std::fs::remove_file(&gone).unwrap(); |
|
| 3190 | 3199 | let (st, body) = call(&h.app, "PUT", &format!("/api/v1/flow/images/{hash}"), Some("site"), Some(manifest)).await; |
|
| 3191 | 3200 | assert_eq!(st, StatusCode::CREATED, "{body}"); |
|
| 3201 | + | assert!(oxygenhub::images::held(&h.app.db).await.unwrap().contains(&hash)); |
|
| 3202 | + | std::fs::write(&gone, kept).unwrap(); |
|
| 3192 | 3203 | ||
| 3193 | 3204 | // A runner fetches the manifest and the chunks, and the image comes back whole. |
|
| 3194 | 3205 | let (st, body) = call(&h.app, "GET", &format!("/api/v1/flow/images/{hash}"), Some("runner-secret"), None).await; |
|
| … | 3717 unchanged lines |