Cylinder: artifacts, the Oxygen way

Status: Draft 0.2 (proposal revision 2), 2026-10-12. Incorporates review of object model, deployment leases, inverted vulnerability indexing, multi-party consensus, agent scopes, and revset queries. Not built. "Cylinder" is a working name (the artifact-repository idea parked on 2026-10-02); nothing here is decided until Andy says so.

Cylinder keeps and moves everything a build produces: runner images, container images, packages, binaries, SBOMs and reports. It would be a service of its own, embedded in OxygenHub by default and shown through the hub's UI, the same pattern as Jetstream. It is meant to be better than what exists, not compatible with it; compatibility is an edge we add for adoption, never the core.

1. What exists today

And the bolt-ons each of them needs:

• Distribution: Dragonfly, Kraken or Spegel, run beside the registry.
• Lazy pulling: eStargz, SOCI, Nydus or zstd:chunked, each a format plus a snapshotter. It matters: pulling took 76% of container start time while only 6.4% of the pulled data was read (Slacker, FAST '16).
• Signing and provenance: cosign and Rekor, SLSA generators, in-toto, Notation, attached afterwards through the OCI referrers API.
• Retention: cleanup policies plus a separate garbage collection.

2. The pattern behind the weaknesses

1. The unit is the tarball layer. One changed file re-pushes and re-pulls a whole layer; layers share nothing below their own size; and a layer has two digests (compressed and not), so tools disagree on what it is called.
2. Names are mutable pointers. A tag moves silently and leaves no history. Pinning by digest is safe but unreadable, so most people don't.
3. Trust is bolted on afterwards. The registry stores bytes; a separate tool signs them, a third says how they were built, and each brings its own keys and identities. Nothing stops an unsigned image from being pushed, and most pulls never verify anything.
4. Distribution is someone else's problem. Every registry is a central server; peer-to-peer and lazy loading are extra products.
5. The forge and the registry are strangers. Permissions, identity, retention and "which commit is this?" are kept twice and drift apart.

3. What Oxygen already has

Every piece Cylinder needs exists in some form:

• Content-defined chunks with BLAKE3 (FastCDC, 32 to 256 KiB), shared by repositories, release assets, and runner images: deduplication below file level, across everything.
• Hybrid post-quantum signatures (Ed25519 with ML-DSA-65), identities with delegation chains, and rotation recorded in the repository.
• Signed Flow receipts: a runner signs what it built, from which state, with which pipeline and environment, producing which outputs. That is provenance by construction, not a statement added later.
• flow-casfs: a content-addressed tree mounted lazily over FUSE or virtio-fs. Lazy pulling is native, not a format bolted on.
• The Jetstream swarm: peer-to-peer chunks, rarest first, TLS pinned by the hub, tickets that only the hub signs.
• Strata: storage on disk, S3, GCS or Azure, with no server in between, sovereign keys and client-side hash verification.
• OxygenHub: one identity and permission model, checks, proposals and reports.
• Scoped delegation: key scopes (changes, refs, seal) allowing principals and AI agents to act within strict cryptographic boundaries.

4. The design

4.1 One kind of thing: a tree of chunks

An artifact is an immutable, content-addressed object in the Oxygen store naming a file tree:

Artifact {
  version     [0]: 1
  kind        [1]: "container" | "runner-image" | "package" | "archive" | "report"
  arch        [2]: text | null              // "x86_64", "riscv64", "aarch64"
  os          [3]: text | null              // "linux", "darwin", "none"
  tree        [4]: ObjectRef                // DirTree blob: paths, modes, blobs, sizes
  entrypoint  [5]: [text] | null            // default exec argv
  env         [6]: map<text, text> | null   // environment variables
  labels      [7]: map<text, text> | null   // metadata / OCI labels
  total_size  [8]: u64                      // uncompressed bytes
  receipt     [9]: ObjectRef | null         // Receipt object that produced it
}

Files in the DirTree are ordinary Oxygen blob objects composed of FastCDC BLAKE3 chunks. So:

• An image rebuilt with one changed file moves only that file's changed chunks: no layers.
• A release tarball, an npm package, and the container image containing the same binary share its chunks verbatim. If a 100 MB model or library is committed in the repo and then packaged, zero new chunk bytes are written to Strata.
• A single file can be fetched independently: a container or job starts after fetching only the files it reads.
• Every artifact has exactly one name for its content: the BLAKE3 of its canonical CBOR encoding.
• MicroVMs and virtio-fs: Jetstream microVM runners mount an artifact's tree directly via virtio-fs over flow-casfs, eliminating ext4 block images, loop devices, and cold disk allocation. Where a raw block device is required by legacy hardware or external hypervisors, an ext4 block map is synthesized on the fly from the chunk tree without duplicating bytes.

4.2 Names are signed histories, not pointers

A channel (app:stable, runner-image:rust, pkg:cargo/my-crate) is an append-only log of signed moves, recorded as an attestation of kind channel:

Attest (kind = "channel") {
  channel   [0]: text             // e.g. "app:stable"
  old_art   [1]: ObjectRef | null // previous artifact hash (CAS check)
  new_art   [2]: ObjectRef        // target artifact hash
  serial    [3]: u64              // monotonically increasing
  receipt   [4]: ObjectRef | null // Receipt that built new_art
  reason    [5]: text             // commit message or promotion note
}

A move needs policy: signed by an authorized key under the repository's trust policy (spec.md §14.5), from an approved bookmark, after required checks or human approvals pass. Clients follow a channel and verify each step:

• A tag cannot move silently. The tj-actions attack needed one stolen token; here it needs a move signed by an authorized key, and the move is logged in the append-only history.
• Rolling back is a move to an earlier artifact, recorded and attested like any other move.
• Pinning stays readable and tamper-evident: app:stable@42 names the 42nd entry of an audited log.
• On Strata stores, updates use compare-and-swap on the channel record (old_art and serial), matching how bookmark refs operate.

4.3 Provenance by construction and multi-party consensus

An artifact is published by a Flow job, and its record is the unbroken chain:

source state → pipeline → runner (identity, tier, image) → signed receipt → artifact

Each link is a hash or signature verifiable offline. Beyond what attestations give today:

• Assurance levels: a MicroVM job on a registered runner is worth more than a syscall-tier job, which is worth more than a signed local build. Trust policies can enforce minimum assurance floors for sensitive channels.
• Multi-party consensus (reproduced): when multiple independent runners on disjoint machines or architectures build the same artifact hash from the same state, each signs a receipt. A policy rule can require threshold consensus: require: "reproduced(2)" demands two independent runner receipts before an artifact can be promoted to a production channel.
• Upstream lineage: an artifact built from other artifacts (a base image, a package) names their hashes in its receipt inputs, so the chain reaches back transitively to their sources.

4.4 Distribution is built in

• The store is the seed; the swarm delivers. Jetstream's swarm, generalized: any runner, cluster node or developer machine in a site shares the chunks it has, over TLS pinned by the hub, admitted by hub-signed tickets.
• Lazy by default: artifacts are mounted via flow-casfs and fetched on demand.
• Learned prefetch: each run records which chunks were accessed during its initial execution window; subsequent pulls request those chunks first.
• Sites and air gaps: a site syncs chunks and channel logs from another hub as a signed bundle, so a disconnected network gets verified artifacts with no live connection.

4.5 Sovereign storage, and our own transparency

• Bytes live on any Strata store: local disk, S3, GCS or Azure, in the customer's own account and region, encrypted with customer-managed KMS.
• Channel moves and signatures are recorded in the hub's append-only Merkle log. Auditing requires no dependence on public ledgers like Rekor, though Cylinder can bridge to them when desired.

4.6 Policy at point of use, and the inverted vulnerability index

Verification happens where the artifact is used, not only where it is stored:

• A runner checks the job's image against policy before execution.
• A Kubernetes admission hook checks pod images against channel policies and assurance levels.
• oxy and flow verify what they fetch.

The global inverted vulnerability index: Because artifacts are trees of content-addressed files and chunks rather than opaque tarballs, Cylinder maintains an inverted index from chunk/file hash to artifacts:

$$\text{FileHash} \longrightarrow [\text{ArtifactHash}]$$

When a security advisory or zero-day is disclosed for a binary or library (e.g., a vulnerable libcrypto.so.3 or Go binary hash):

• Cylinder performs an immediate $O(1)$ lookup to identify every container image, runner image, and package across the fleet containing that file.
• There is no need to unpack gigabytes of tarball layers across thousands of images to discover exposure.
• Continuous revocation: If an upstream library or base image is revoked in policy, point-of-use admission hooks immediately block downstream pods from running any artifact containing that chunk, eliminating the gap between vulnerability discovery and redeployment.

4.7 Retention by reachability and deployment leases

What a channel, a release, a pinned pipeline or an active deployment reaches is kept; unreferenced chunks are collected by oxy gc.

Deployment leases: Container registries often suffer from garbage collection deleting images still running on long-lived cluster pods. To prevent this, nodes or Kubernetes operators register periodic deployment leases:

Attest (kind = "lease") {
  artifact [0]: ObjectRef  // artifact hash currently running
  scope    [1]: text       // e.g. "k8s:cluster-lon/ns/workload"
  expires  [2]: u64        // timestamp
}

Active leases act as garbage collection roots. If a workload terminates or a cluster is decommissioned, the lease expires naturally, enabling safe automated retention without operational risk.

4.8 Compatibility at the edge

• An OCI registry façade: docker pull and Kubernetes work unchanged. Manifests and layers are assembled on the fly from chunks. docker push is accepted and chunked via FastCDC, applying deduplication to legacy images.
• A containerd snapshotter speaking the native protocol, mounting file trees via flow-casfs for instant sub-second container starts and P2P delivery on Kubernetes.
• Package façades (Cargo, npm, PyPI): Served from the same chunk store. Façade endpoints synthesize package metadata (Cargo sparse indexes, npm packuments, PyPI simple indexes) dynamically from artifact manifests.
• Upstream dependency quarantine: When Flow builds fetch third-party packages from crates.io or npm, Cylinder proxies and chunks them into the organization's Strata store, locking the hashes into internal mirror channels. Upstream outages or deletions cannot break builds.

4.9 AI-native agent integration

Extending Oxygen's scoped delegation model (ai_native_agent_workflows.md §2), agent keys can hold artifact-specific scopes:

delegates [8]: [{ key: KeyId, scope: ["changes" | "refs" | "seal" | "artifacts" | "channels"], expires: u64? }]

• An AI agent (e.g. Aider, Claude Code) working autonomously can be granted --scope artifacts and restricted channel rights:
  • Permitted: building artifacts and moving agents/NAME/** channels.
  • Forbidden: moving production channels (app:stable, release/*).
• Agents can test containerized workloads end-to-end in isolated channels, while production promotion remains guarded by human landing rules or multi-party consensus.

4.10 Revset query integration

Oxygen's revset language (revset_query_language.md) bridges source changes and deployed artifacts:

Running oxy log -r 'main ~ artifact(app:stable)' immediately shows all changes pending deployment.

4.11 The UI (inside the hub)

• Per repository, an Artifacts tab: published artifacts, channel histories, and the provenance graph from commit to run, artifact, and deployment lease.
• Site-wide: chunk distribution across hubs and runners, active swarm transfers, deduplication savings, and the global CVE inverted index.
• Runner images are listed as standard artifacts with their build receipts and hardware architecture badges.

flowchart LR
  subgraph Build
    S[Source state] --> P[Pipeline]
    P --> R1[Runner 1: microvm]
    P --> R2[Runner 2: disjoint host]
    R1 -->|receipt 1| A[Artifact: tree of chunks]
    R2 -->|receipt 2| A
  end
  A -->|consensus move| C[Channel log: app:stable@42]
  A --> ST[(Strata store)]
  ST -->|seed| SW{{Swarm}}
  SW --> N1[Runner]
  SW --> N2[K8s node: containerd snapshotter]
  SW --> N3[Developer]
  N2 -->|deployment lease| DL[Active Leases: GC Roots]
  C -->|policy check| N2
  OCI[OCI façade] --- ST
  PKG[Package façades: cargo/npm/pypi] --- ST
  H[OxygenHub UI & CVE Index] --- C
  H --- A

4.12 Where it stands among the services

• OxygenHub owns repositories, proposals, checks, user identity, and the UI.
• Jetstream owns the fleet: queue, placement, and runner swarms. It fetches runner images from Cylinder by artifact hash.
• Flow executes pipelines and produces signed execution receipts.
• Cylinder owns artifacts: manifests, channel logs, provenance chains, distribution, package façades, and deployment leases.
• Strata provides the underlying content-addressed chunk storage.

Each service runs embedded in OxygenHub by default, or standalone for distributed scale.

5. Better how, in one table

6. How we could build it

• C0. Images in the hub, now. A Flow › Images page: each image, where it is (hub, runners, swarm activity) and what pins it.
• C1. The service boundary and artifact model. Implement the Artifact CBOR object type and FastCDC chunk engine in cylinder; unify runner images with artifacts. Started 2026-10-28 (oxygen#108): crates/cylinder has the Artifact and DirTree encodings of §4.1 (canonical CBOR; a name is the BLAKE3 of the encoding; env and labels as sorted pairs, since the profile's maps take integer keys), packing a directory into the repository's own blobs and chunks, verifying and unpacking, and the roots garbage collection keeps; oxy artifact pack | list | show | unpack | verify. Until the artifact is an object type of its own, its encoding and tree are blobs and .oxy/artifacts/NAME records it. Next: the object type, then runner images as artifacts.
• C2. Channels, provenance and consensus. Flow publishes artifacts; signed channel move attestations; multi-party reproduced(N) consensus; revset predicates.
• C3. The OCI façade and containerd snapshotter. OCI pull/push translation; containerd snapshotter over flow-casfs.
• C4. Leases and the inverted index. Kubernetes deployment lease operator for safe GC; global inverted chunk index for instantaneous CVE discovery.
• C5. Reach and package façades. Package façades for Cargo, npm, and PyPI; upstream dependency quarantine mirroring.

7. Open questions

• The name: Keep Cylinder. It fits the elemental progression: Oxygen (VCS) is transported in Cylinders (artifacts) via Jetstream (fleet transport) driven by Flow (pipelines).
• Where it lives: Initially in oxygenflow / crates/oxygen-cylinder alongside jet and flow to share flow-casfs and chunking primitives, splitting into an independent repository at C3.
• Local builds: Allowed, but marked with lower assurance level (local). Production channel rules require higher assurance levels (tier-microvm + reproduced(2)).
• A public catalogue / Federation: Public hubs federate by publishing signed Strata bundles of channels and artifacts, verified offline without centralized intermediaries.
• Place in the queue: C0/C1 should proceed alongside Jetstream's service boundary work, ensuring Jetstream's image store is built directly on Cylinder rather than creating a temporary mechanism.

Sources

• Docker Hub pull limits: docs.docker.com, pull usage and limits; heise, "only 10 pulls per hour"
• GHCR billing: GitHub community discussion #183054
• GitLab cleanup policies and garbage collection: GitLab docs, reduce container registry storage
• Bitbucket Packages: Atlassian blog, Bitbucket Packages; GA announcement
• Pull time versus data read: SOCI paper (Amazon), citing Slacker; Grab engineering, lazy loading
• zstd:chunked: Red Hat, faster container image pulls
• Peer-to-peer meshes: Spegel vs Kraken vs Dragonfly
• Sigstore in registries: Sigstore in image registries: adoption and reality; cosign and the referrers API
• tj-actions tag attack: CVE-2025-30066; Legit Security, immutable tags
• Artifactory and Nexus: Codefresh, Artifactory alternatives; PeerSpot, Artifactory improvements