A repository kept in a bucket lists its packs on every open issue #116 closed: completed andymccall opened this 17 days ago · 4 comments

Progress (2026-10-18, live on oxygenhub.io):

• a359c806: pack listings kept 30 s;
• 051f6efc: the pack size comes from the listing, and a bucket store's stat is a HEAD;
• 45b7327c: the counters split into list_packs / list_kept / list.

Pack listings are fixed: in 265 s, 6 pack LISTs were sent and 51 answered from the cache.

What remains is list (other prefixes): 210 in those 265 s. From the code, these are the per-object fanout listings: Store::signatures(r) (sig/…) and index_get (seal, cert, accepted, receipt indexes), one LIST per object looked at, mostly empty (unsigned changes).

Options, for Andy:

1. Keep those listings for a short window too (the pack listing's mechanism). Cheap, but a repository that clients write directly in the bucket (Strata remotes, e.g. doomsdayone/tetris) would have the hub miss a signature for up to the window, and a policy-checked fetch could fail meanwhile. Pushes through the hub stay exact (its own writes forget the listing).
2. Store signatures and index entries in the packs' own index when objects are packed (one GET per pack instead of a LIST per object), so only loose objects need listing. More work, no staleness. My recommendation for the lasting fix.
3. Both: 2, plus 1 with a short window only for repositories the hub alone writes.

Not changed tonight.

Decided by Andy (2026-10-18): keep signatures and index entries in the packs' index, read with the cached pack listing, so bucket stores make no per-object LISTs. A format change: rebuild all three runners before any repository's data uses it.

Overnight analysis (2026-10-18 20:45), nothing implemented: the decided design can't be done without bending one of three constraints, so it's back to Andy.

Constraints: (1) never miss a signature/index entry, including ones added after packing and ones written by other processes or clients straight into a bucket (tetris); (2) older readers (runners, oxy 0.4.6) keep working; (3) signed formats unchanged.

Why: released oxy 0.4.6 writes each signature and index entry only as a per-object key (sig/<hh>/<rest>/<signer>, seal|receipt|approval|cert|accepted/<hh>/<rest>/<value>), sometimes without moving any operation head (add_receipt, add_approval, sign_change). The value is part of the key, so only a LIST covering it, taken at lookup time, finds it. A pack sidecar can hold what existed at packing, not what arrives after.

Options:

• A, as decided (entries in a pack sidecar, read with the cached pack listing): entries added after packing are invisible until the next repack; new-code writers elsewhere seen after 30 s. Breaks (1).
• B, A plus a journal (pack-entries, appended by compare-and-swap like pack-retired, read fresh on every open: one read instead of N LISTs; folded into the pack sidecar on repack, kept an hour): exact for every writer running the new oxy; misses entries written by 0.4.6 clients until a repack. Bends (1) for old writers only. My recommendation, given no compat before 1.0: bucket-repo writers (tetris clients, runners) move to the new oxy first.
• C, one full listing per prefix per request: exact, but 6-12 paginated LISTs over all history per request: no saving.
• D, cache those listings 30 s: a time window for other writers. Breaks (1).
• E, send the per-object LISTs concurrently: keeps (1)-(3), cuts latency, same LIST count.

Separate saving found: every signatures()/index_get() also GETs a legacy one-list key (sigs/…, seals/…, certs/…, accepted-lists/…) that no current version writes, almost always a miss, so bucket requests are about twice the LIST figure. Looked at separately.

Andy chose B (2026-10-19): entries in a pack sidecar plus a pack-entries journal (compare-and-swap appends like pack-retired), read once per open; folded into the sidecar on repack. Entries written by 0.4.6 clients are missed until a repack, so bucket-repository writers (runners, tetris clients) move to the new oxy first. Also drop the legacy one-list key GETs (sigs/…, seals/…, certs/…, accepted-lists/…) with a fold-on-gc migration and a marker. Spec §13 first; rebuild all three runners before any repository's data uses it; measure /health cloud_stores before and after.