A rootfs whose cached pages go bad fails every job until the cache drops them issue #22
closed: completed andymccall opened this · 0 comments
andymccall
2026-10-02: earth-1 failed every job on the rust image for ~3 hours: 'a3b49f… has BLAKE3 aa095cfe…, but a3b49f… is pinned'. The file on disk was intact (identical to plex's copy; hashes right after the page cache was flushed by an 8 GB copy); one cmp showed a single differing byte, a second none: a corrupted page in Linux's page cache, most likely a RAM fault on Earth (no ECC). The runner re-hashed through the same bad cached page each time. Fix: on a mismatch, drop the file's cached pages (posix_fadvise DONTNEED) and re-read it once from disk before failing, logging which it was. Earth's memory should be tested (memtest86+).
2026-10-02: earth-1 failed every job on the rust image for ~3 hours: 'a3b49f… has BLAKE3 aa095cfe…, but a3b49f… is pinned'. The file on disk was intact (identical to plex's copy; hashes right after the page cache was flushed by an 8 GB copy); one cmp showed a single differing byte, a second none: a corrupted page in Linux's page cache, most likely a RAM fault on Earth (no ECC). The runner re-hashed through the same bad cached page each time. Fix: on a mismatch, drop the file's cached pages (posix_fadvise DONTNEED) and re-read it once from disk before failing, logging which it was. Earth's memory should be tested (memtest86+).