Runner: claim only the memory the host has free (closes #92); WASM: a job stopped while its module compiles stops (closes #93) 77c8c24066ef
- Author
- Andy McCall @andymccall <[email protected]>
- via
- agent
agent:claude@andymccall <[email protected]> · keyless, under your grant for changes,refs,seal · key fa9b904673f7
- When
- · 2026-10-27 00:57 UTC
- Signature
- ◈ sealed by agent:claude@andymccall
- Hash
77c8c24066ef11f737fb0dcd1ff02aab93c1b3ead2e65d7e86a548cacdf79833- Depends on
- 28cc603f737d 3007d8f011fa 5cf1e8aefbb6 8a4f5174526a b11e30f5031d bb643c252515 c3db750ed96e c5b702e59c1f e0918b887850
- Depended on by
- 32f54b393396
- In bookmarks
- main
- In tags
- v0.5.0 v0.5.1 v0.5.2 v0.5.3
- Edits
- 6 atoms, 0 tree operations, 3 files
Browse the tree at this change
modified CHANGELOG.md +2 −0
| … | 3 unchanged lines |
||
| 4 | 4 | ||
| 5 | 5 | ## Unreleased |
|
| 6 | 6 | ||
| 7 | + | - **A WASM job stopped while its module compiles stops (#93).** The time limit or a cancel moved the engine's epoch once; a store made after that set its deadline past it, so the job ran on forever (seen as a test spinning 35 minutes under load). Once stopped, the epoch now moves every tick until the job ends. |
|
| 8 | + | - **A runner claims only what its host can start (#92).** An idle runner told the hub it had no memory limit, so it was offered sandboxes bigger than its machine, fetched their images, and declined them one by one (mars-1, 7 GB, and the 8 GiB jobs). A claim now asks for the budget's room capped at the host's free memory. |
|
| 7 | 9 | - **`flow rootfs push` waits for the hub to check a big image (oxygenhub#341).** The hub reads the whole image before it accepts the manifest; the manifest request now waits up to an hour, not the 120 s every other request gets. |
|
| 8 | 10 | - **The Android image (#19).** `images/android`: OpenJDK 17, the Android SDK (command-line tools 23.0, platform 37.0, build-tools 36.0.0, NDK 30.0), Gradle 9.6.0, Rust with the Android targets and cargo-ndk, and the OxygenHub app's crates and Gradle dependencies, so the app builds and tests offline. Each download is checked against its published hash. |
|
| 9 | 11 | - **`flow runner bench` says when load is not measured (#83):** a Windows runner printed `load 0.00`; it now says `load not measured here`. Its kernel line reads `Windows 10.0.26300` (checked with 0.4.0 in the Windows VM). |
|
| … | 109 unchanged lines |
modified crates/flow/src/runner.rs +23 −1
| … | 650 unchanged lines |
||
| 651 | 651 | None |
|
| 652 | 652 | } |
|
| 653 | 653 | ||
| 654 | + | /// The memory to claim with: the budget's room, never more than the host |
|
| 655 | + | /// has free now. An idle runner's budget is unlimited (a lone job may use |
|
| 656 | + | /// it all), so without this it was offered sandboxes it then declined for |
|
| 657 | + | /// want of memory, one after another, having fetched their images first |
|
| 658 | + | /// (an 8 GiB job on a 7 GB machine). |
|
| 659 | + | fn claim_memory(room: Option<u32>, host_free_mib: Option<u64>) -> Option<u32> { |
|
| 660 | + | match (room, host_free_mib) { |
|
| 661 | + | (Some(r), Some(h)) => Some(r.min(u32::try_from(h).unwrap_or(u32::MAX))), |
|
| 662 | + | (None, Some(h)) => Some(u32::try_from(h).unwrap_or(u32::MAX)), |
|
| 663 | + | (room, None) => room, |
|
| 664 | + | } |
|
| 665 | + | } |
|
| 666 | + | ||
| 654 | 667 | /// `MemAvailable` from /proc/meminfo, in MiB. |
|
| 655 | 668 | fn available_memory_mib() -> Option<u64> { |
|
| 656 | 669 | let info = std::fs::read_to_string("/proc/meminfo").ok()?; |
|
| … | 211 unchanged lines |
||
| 868 | 881 | let req = ClaimRequest { |
|
| 869 | 882 | runner: name.clone(), |
|
| 870 | 883 | runtimes: runtimes.clone(), |
|
| 871 | - | free_memory_mib, |
|
| 884 | + | free_memory_mib: claim_memory(free_memory_mib, available_memory_mib()), |
|
| 872 | 885 | free_cpus, |
|
| 873 | 886 | info: Some(RunnerInfo { |
|
| 874 | 887 | version: env!("OXY_VERSION_LONG").into(), |
|
| … | 1382 unchanged lines |
||
| 2257 | 2270 | assert_eq!(one.room(used(1, 0, 0)), None, "the default runs one job at a time"); |
|
| 2258 | 2271 | let open = Budget { max_jobs: 2, memory_mib: None, cpus: None }; |
|
| 2259 | 2272 | assert_eq!(open.room(used(1, 8192, 4)), Some((None, None)), "no limits configured"); |
|
| 2273 | + | } |
|
| 2274 | + | ||
| 2275 | + | #[test] |
|
| 2276 | + | fn a_claim_never_asks_for_more_memory_than_the_host_has_free() { |
|
| 2277 | + | assert_eq!(claim_memory(None, Some(6541)), Some(6541), "idle: the host's free memory, not unlimited"); |
|
| 2278 | + | assert_eq!(claim_memory(Some(16_384), Some(6541)), Some(6541)); |
|
| 2279 | + | assert_eq!(claim_memory(Some(4096), Some(6541)), Some(4096), "the budget when it is smaller"); |
|
| 2280 | + | assert_eq!(claim_memory(None, None), None, "nothing known: no limit, as before"); |
|
| 2281 | + | assert_eq!(claim_memory(Some(4096), None), Some(4096)); |
|
| 2260 | 2282 | } |
|
| 2261 | 2283 | } |
modified crates/flow/src/wasm.rs +7 −0
| … | 289 unchanged lines |
||
| 290 | 290 | while !worker.is_finished() { |
|
| 291 | 291 | std::thread::sleep(Duration::from_millis(100)); |
|
| 292 | 292 | pass_on(sink, false); |
|
| 293 | + | // Once stopped, keep moving the epoch: a store made after the |
|
| 294 | + | // first increment (its module still compiling when the time |
|
| 295 | + | // limit or the cancel came) sets its deadline one epoch past |
|
| 296 | + | // that, and a single increment would never reach it (#93). |
|
| 297 | + | if timed_out || cancelled { |
|
| 298 | + | engine.increment_epoch(); |
|
| 299 | + | } |
|
| 293 | 300 | if !timed_out && !cancelled && sink.cancelled() { |
|
| 294 | 301 | cancelled = true; |
|
| 295 | 302 | engine.increment_epoch(); |
|
| … | 261 unchanged lines |