Cylinder: artifacts, the Oxygen way

Status: Draft 0.3 (proposal revision 3), 2026-10-12. Draft 0.2 added the object model, deployment leases, inverted vulnerability indexing, multi-party consensus, agent scopes and revset queries; 0.3 corrects the MicroVM mount path, the reach of the file index, revocation, encryption, leases and upstream quarantine, and keeps §7 open. C0 is built; the rest is not. "Cylinder" is a working name (the artifact-repository idea parked on 2026-10-02); nothing here is decided until Andy says so.

Cylinder keeps and moves everything a build produces: runner images, container images, packages, binaries, SBOMs and reports. It would be a service of its own, embedded in OxygenHub by default and shown through the hub's UI, the same pattern as Jetstream. It is meant to be better than what exists, not compatible with it; compatibility is an edge we add for adoption, never the core.

1. What exists today

And the bolt-ons each of them needs:

• Distribution: Dragonfly, Kraken or Spegel, run beside the registry.
• Lazy pulling: eStargz, SOCI, Nydus or zstd:chunked, each a format plus a snapshotter. It matters: pulling took 76% of container start time while only 6.4% of the pulled data was read (Slacker, FAST '16).
• Signing and provenance: cosign and Rekor, SLSA generators, in-toto, Notation, attached afterwards through the OCI referrers API.
• Retention: cleanup policies plus a separate garbage collection.

2. The pattern behind the weaknesses

1. The unit is the tarball layer. One changed file re-pushes and re-pulls a whole layer; layers share nothing below their own size; and a layer has two digests (compressed and not), so tools disagree on what it is called.
2. Names are mutable pointers. A tag moves silently and leaves no history. Pinning by digest is safe but unreadable, so most people don't.
3. Trust is bolted on afterwards. The registry stores bytes; a separate tool signs them, a third says how they were built, and each brings its own keys and identities. Nothing stops an unsigned image from being pushed, and most pulls never verify anything.
4. Distribution is someone else's problem. Every registry is a central server; peer-to-peer and lazy loading are extra products.
5. The forge and the registry are strangers. Permissions, identity, retention and "which commit is this?" are kept twice and drift apart.

3. What Oxygen already has

Every piece Cylinder needs exists in some form:

• Content-defined chunks with BLAKE3 (FastCDC, 32 to 256 KiB), shared by repositories, release assets, and runner images: deduplication below file level, across everything.
• Hybrid post-quantum signatures (Ed25519 with ML-DSA-65), identities with delegation chains, and rotation recorded in the repository.
• Signed Flow receipts: a runner signs what it built, from which state, with which pipeline and environment, producing which outputs. That is provenance by construction, not a statement added later.
• flow-casfs: a content-addressed tree mounted lazily over FUSE or virtio-fs. Lazy pulling is native, not a format bolted on.
• The Jetstream swarm: peer-to-peer chunks, rarest first, TLS pinned by the hub, tickets that only the hub signs.
• Strata: storage on disk, S3, GCS or Azure, with no server in between, sovereign keys and client-side hash verification.
• OxygenHub: one identity and permission model, checks, proposals and reports.
• Scoped delegation: key scopes (changes, refs, seal) allowing principals and AI agents to act within strict cryptographic boundaries.

4. The design

4.1 One kind of thing: a tree of chunks

An artifact is an immutable, content-addressed object in the Oxygen store naming a file tree:

Artifact {
  version     [0]: 1
  kind        [1]: "container" | "runner-image" | "package" | "archive" | "report"
  arch        [2]: text | null              // "x86_64", "riscv64", "aarch64"
  os          [3]: text | null              // "linux", "darwin", "none"
  tree        [4]: ObjectRef                // DirTree blob: paths, modes, blobs, sizes
  entrypoint  [5]: [text] | null            // default exec argv
  env         [6]: map<text, text> | null   // environment variables
  labels      [7]: map<text, text> | null   // metadata / OCI labels
  total_size  [8]: u64                      // uncompressed bytes
  receipt     [9]: ObjectRef | null         // Receipt object that produced it
}

Files in the DirTree are ordinary Oxygen blob objects composed of FastCDC BLAKE3 chunks. So:

• An image rebuilt with one changed file moves only that file's changed chunks: no layers.
• A release tarball, an npm package, and the container image containing the same binary share its chunks verbatim. If a 100 MB model or library is committed in the repo and then packaged, zero new chunk bytes are written to Strata.
• A single file can be fetched independently: a container or job starts after fetching only the files it reads.
• Every artifact has exactly one name for its content: the BLAKE3 of its canonical CBOR encoding.
• MicroVMs: Flow's MicroVM tier runs on Firecracker, which has no virtio-fs, so a microVM gets the artifact as a block device: an ext4 image served on the fly from the chunk tree (a userspace block device, ublk, or an NBD server), fetching blocks as the guest reads them and duplicating no bytes. That removes copying whole images onto each runner's disk without changing hypervisor. The syscall tier and containers mount the tree directly over flow-casfs; mounting it with virtio-fs would need Cloud Hypervisor or QEMU instead of Firecracker.

4.2 Names are signed histories, not pointers

A channel (app:stable, runner-image:rust, pkg:cargo/my-crate) is an append-only log of signed moves, recorded as an attestation of kind channel:

Attest (kind = "channel") {
  channel   [0]: text             // e.g. "app:stable"
  old_art   [1]: ObjectRef | null // previous artifact hash (CAS check)
  new_art   [2]: ObjectRef        // target artifact hash
  serial    [3]: u64              // monotonically increasing
  receipt   [4]: ObjectRef | null // Receipt that built new_art
  reason    [5]: text             // commit message or promotion note
}

A move needs policy: signed by an authorized key under the repository's trust policy (spec.md §14.5), from an approved bookmark, after required checks or human approvals pass. Clients follow a channel and verify each step:

• A tag cannot move silently. The tj-actions attack needed one stolen token; here it needs a move signed by an authorized key, and the move is logged in the append-only history.
• Rolling back is a move to an earlier artifact, recorded and attested like any other move.
• Pinning stays readable and tamper-evident: app:stable@42 names the 42nd entry of an audited log.
• On Strata stores, updates use compare-and-swap on the channel record (old_art and serial), matching how bookmark refs operate.

4.3 Provenance by construction and multi-party consensus

An artifact is published by a Flow job, and its record is the unbroken chain:

source state → pipeline → runner (identity, tier, image) → signed receipt → artifact

Each link is a hash or signature verifiable offline. Beyond what attestations give today:

• Assurance levels: a MicroVM job on a registered runner is worth more than a syscall-tier job, which is worth more than a signed local build. Trust policies can enforce minimum assurance floors for sensitive channels.
• Multi-party consensus (reproduced): when multiple independent runners on disjoint machines or architectures build the same artifact hash from the same state, each signs a receipt. A policy rule can require threshold consensus: require: "reproduced(2)" demands two independent runner receipts before an artifact can be promoted to a production channel.
• Upstream lineage: an artifact built from other artifacts (a base image, a package) names their hashes in its receipt inputs, so the chain reaches back transitively to their sources.
• An SBOM with every artifact: the job that publishes an artifact also records what it contains by package and version (an SBOM, itself an artifact named in the receipt's outputs), so "which artifacts hold a vulnerable version of this package" is a lookup, not a scan (§4.6).

4.4 Distribution is built in

• The store is the seed; the swarm delivers. Jetstream's swarm, generalized: any runner, cluster node or developer machine in a site shares the chunks it has, over TLS pinned by the hub, admitted by hub-signed tickets.
• Lazy by default: artifacts are mounted via flow-casfs and fetched on demand.
• Learned prefetch: each run records which chunks were accessed during its initial execution window; subsequent pulls request those chunks first.
• Sites and air gaps: a site syncs chunks and channel logs from another hub as a signed bundle, so a disconnected network gets verified artifacts with no live connection.

4.5 Sovereign storage, and our own transparency

• Bytes live on any Strata store: local disk, S3, GCS or Azure, in the customer's own account and region, encrypted at rest by the store with customer-managed keys (for example S3 with a KMS key). Chunks stay named by the hash of their plain bytes: encrypting them on the client with separate keys would stop them deduplicating and stop peers checking them by hash.
• Channel moves and signatures are recorded in the hub's append-only Merkle log. Auditing requires no dependence on public ledgers like Rekor, though Cylinder can bridge to them when desired.

4.6 Policy at point of use, and the inverted vulnerability index

Verification happens where the artifact is used, not only where it is stored:

• A runner checks the job's image against policy before execution.
• A Kubernetes admission hook checks pod images against channel policies and assurance levels.
• oxy and flow verify what they fetch.

The inverted file index: Because artifacts are trees of content-addressed files rather than opaque tarballs, Cylinder keeps an index from each file's hash to the artifacts that contain it:

$$\text{FileHash} \longrightarrow [\text{ArtifactHash}]$$

• Exact files, at once. Given a known-bad file (a compromised binary, a leaked key, the exact libcrypto.so.3 an advisory names), one lookup lists every container image, runner image and package across the fleet holding it, with no layers to unpack.
• Not versions. A patched or rebuilt library is a different file with a different hash, so this index does not answer "which images have a vulnerable version of OpenSSL". That comes from package identity: the SBOM each artifact carries (§4.3), indexed by package and version. The two together answer both questions; the file index alone answers the first.
• Revocation without an outage. When an artifact, base image or file is revoked in policy, admission hooks refuse new pods that use it at once. Pods already running are not touched by admission; a controller marks them, then replaces or evicts them within a grace period the policy sets, so one policy change cannot take production down.

4.7 Retention by reachability and deployment leases

What a channel, a release, a pinned pipeline or an active deployment reaches is kept; unreferenced chunks are collected by oxy gc.

Deployment leases: Container registries often suffer from garbage collection deleting images still running on long-lived cluster pods. To prevent this, nodes or Kubernetes operators register periodic deployment leases:

Attest (kind = "lease") {
  artifact [0]: ObjectRef  // artifact hash currently running
  scope    [1]: text       // e.g. "k8s:cluster-lon/ns/workload"
  expires  [2]: u64        // timestamp
}

Active leases act as garbage collection roots. If a workload terminates or a cluster is decommissioned, the lease expires naturally, enabling safe automated retention without operational risk. A lease is signed by a key scoped to leases and to its scope, and has a maximum length (a day, renewed while the workload runs), so a compromised node cannot hold storage forever or pin artifacts outside its own scope.

4.8 Compatibility at the edge

• An OCI registry façade: docker pull and Kubernetes work unchanged. Manifests and layers are assembled on the fly from chunks. docker push is accepted and chunked via FastCDC, applying deduplication to legacy images.
• A containerd snapshotter speaking the native protocol, mounting file trees via flow-casfs for instant sub-second container starts and P2P delivery on Kubernetes.
• Package façades (Cargo, npm, PyPI): Served from the same chunk store. Façade endpoints synthesize package metadata (Cargo sparse indexes, npm packuments, PyPI simple indexes) dynamically from artifact manifests.
• Upstream dependency quarantine: When Flow builds fetch third-party packages from crates.io or npm, Cylinder proxies them into the organization's Strata store, locking the hashes into internal mirror channels. Upstream outages or deletions cannot break builds. Packages arrive compressed (.crate and .tgz are gzip), which chunks poorly, so the original archive is kept for clients that check its hash and its unpacked files are chunked as an artifact tree, where they deduplicate with everything else.

4.9 AI-native agent integration

Extending Oxygen's scoped delegation model (ai_native_agent_workflows.md §2), agent keys can hold artifact-specific scopes:

delegates [8]: [{ key: KeyId, scope: ["changes" | "refs" | "seal" | "artifacts" | "channels"], expires: u64? }]

• An AI agent (e.g. Aider, Claude Code) working autonomously can be granted --scope artifacts and restricted channel rights:
  • Permitted: building artifacts and moving agents/NAME/** channels.
  • Forbidden: moving production channels (app:stable, release/*).
• Agents can test containerized workloads end-to-end in isolated channels, while production promotion remains guarded by human landing rules or multi-party consensus.

4.10 Revset query integration

Oxygen's revset language (revset_query_language.md) bridges source changes and deployed artifacts:

Running oxy log -r 'main ~ artifact(app:stable)' immediately shows all changes pending deployment.

4.11 The UI (inside the hub)

• Per repository, an Artifacts tab: published artifacts, channel histories, and the provenance graph from commit to run, artifact, and deployment lease.
• Site-wide: chunk distribution across hubs and runners, active swarm transfers, deduplication savings, and the global CVE inverted index.
• Runner images are listed as standard artifacts with their build receipts and hardware architecture badges.

flowchart LR
  subgraph Build
    S[Source state] --> P[Pipeline]
    P --> R1[Runner 1: microvm]
    P --> R2[Runner 2: disjoint host]
    R1 -->|receipt 1| A[Artifact: tree of chunks]
    R2 -->|receipt 2| A
  end
  A -->|consensus move| C[Channel log: app:stable@42]
  A --> ST[(Strata store)]
  ST -->|seed| SW{{Swarm}}
  SW --> N1[Runner]
  SW --> N2[K8s node: containerd snapshotter]
  SW --> N3[Developer]
  N2 -->|deployment lease| DL[Active Leases: GC Roots]
  C -->|policy check| N2
  OCI[OCI façade] --- ST
  PKG[Package façades: cargo/npm/pypi] --- ST
  H[OxygenHub UI & CVE Index] --- C
  H --- A

4.12 Where it stands among the services

• OxygenHub owns repositories, proposals, checks, user identity, and the UI.
• Jetstream owns the fleet: queue, placement, and runner swarms. It fetches runner images from Cylinder by artifact hash.
• Flow executes pipelines and produces signed execution receipts.
• Cylinder owns artifacts: manifests, channel logs, provenance chains, distribution, package façades, and deployment leases.
• Strata provides the underlying content-addressed chunk storage.

Each service runs embedded in OxygenHub by default, or standalone for distributed scale.

5. Better how, in one table

6. How we could build it

• C0. Images in the hub (built 2026-10-12, oxygenhub #146). Admin › Runners › Images: every runner image, its size and architecture, where it is (the hub, each runner, swarm fetches in progress), the repositories whose jobs used it, and an image held nowhere marked.
• C1. The service boundary and artifact model. Implement the Artifact CBOR object type and FastCDC chunk engine in cylinder; unify runner images with artifacts.
• C2. Channels, provenance and consensus. Flow publishes artifacts; signed channel move attestations; multi-party reproduced(N) consensus; revset predicates.
• C3. The OCI façade and containerd snapshotter. OCI pull/push translation; containerd snapshotter over flow-casfs.
• C4. Leases and the indexes. Kubernetes deployment lease operator for safe GC; the inverted file index and the SBOM package index; the controller that replaces revoked running workloads within their grace period.
• C5. Reach and package façades. Package façades for Cargo, npm, and PyPI; upstream dependency quarantine mirroring.

7. Open questions

These are Andy's decisions. Each carries the position proposed in review, not an answer.

• The name. Proposed: keep Cylinder. It fits the elemental progression: Oxygen (VCS) is carried in Cylinders (artifacts) by Jetstream (the fleet), driven by Flow (pipelines).
• Where it lives. Proposed: first in oxygenflow as crates/oxygen-cylinder, beside jet and flow, to share flow-casfs and the chunking code; its own repository from C3.
• Local builds. Proposed: allowed, at the lowest assurance level (local); production channels require more (tier-microvm and reproduced(2)).
• A public catalogue and federation. Proposed: public hubs federate by publishing signed Strata bundles of channels and artifacts, checked offline with no central intermediary.
• Its place in the queue. Proposed: C1 alongside Jetstream's service boundary, so Jetstream's image store is built on Cylinder from the start rather than on a temporary mechanism. Against: the core work of 2026-10-03 (Strata's remaining stores, agents, hub polish) waits longer.

Sources

• Docker Hub pull limits: docs.docker.com, pull usage and limits; heise, "only 10 pulls per hour"
• GHCR billing: GitHub community discussion #183054
• GitLab cleanup policies and garbage collection: GitLab docs, reduce container registry storage
• Bitbucket Packages: Atlassian blog, Bitbucket Packages; GA announcement
• Pull time versus data read: SOCI paper (Amazon), citing Slacker; Grab engineering, lazy loading
• zstd:chunked: Red Hat, faster container image pulls
• Peer-to-peer meshes: Spegel vs Kraken vs Dragonfly
• Sigstore in registries: Sigstore in image registries: adoption and reality; cosign and the referrers API
• tj-actions tag attack: CVE-2025-30066; Legit Security, immutable tags
• Artifactory and Nexus: Codefresh, Artifactory alternatives; PeerSpot, Artifactory improvements