# Cylinder: artifacts, the Oxygen way

Status: Draft 0.3 (proposal revision 3), 2026-10-12. Draft 0.2 added the

object model, deployment leases, inverted vulnerability indexing,

multi-party consensus, agent scopes and revset queries; 0.3 corrects the

MicroVM mount path, the reach of the file index, revocation, encryption,

leases and upstream quarantine, and keeps §7 open. C0 is built; the rest

is not. "Cylinder" is a working name (the artifact-repository idea parked

on 2026-10-02); nothing here is decided until Andy says so.

**Cylinder** keeps and moves everything a build produces: runner images,

container images, packages, binaries, SBOMs and reports. It would be a

service of its own, embedded in OxygenHub by default and shown through

the hub's UI, the same pattern as Jetstream. It is meant to be better

than what exists, not compatible with it; compatibility is an edge we add

for adoption, never the core.

## 1. What exists today

| | Strengths | Weaknesses |

|---|---|---|

| **Docker Hub** | The default namespace; huge public catalogue; official images; scanning (Scout). | Pull limits since April 2025: 10 an hour anonymous, 100 an hour on a free account. SaaS only. Tags are mutable. Signatures and provenance are separate artifacts; referrers are still in beta. Nothing links an image to the source it came from. |

| **GitHub** (GHCR, Packages) | Next to the code and Actions; free container storage for now; the best Sigstore integration; artifact attestations. | Storage billing for private images deferred, not settled. One silo per package format. Auth by personal access tokens. Tags are mutable: the tj-actions attack (March 2025, CVE-2025-30066) moved every tag of a popular action to a malicious commit at once. No self-hosting outside Enterprise Server. |

| **GitLab** | A registry per project; many package formats; a dependency proxy; self-hostable, so it can be sovereign. | Cleanup policies delete tags only; layers stay until an administrator runs garbage collection. The registry is a separate service to run and scale. Signing means bringing cosign yourself. |

| **Bitbucket** (Packages) | GA since December 2025; permissions inherited from the repository; garbage collection runs by itself; Maven and npm since April 2026. | Young; Cloud only; few formats; no provenance story. |

| **Artifactory, Nexus** | Every format; proxies and caches; enterprise features. | Costly; heavy JVM operations; slow UIs. A second system beside the forge, with its own identities and permissions. Provenance and scanning are add-on products. |

| **Harbor, Quay** | Self-hosted OCI with signing (cosign, Notation), replication and scanning. | Many components to run. A central bottleneck: peer-to-peer means adding Dragonfly. |

| **ECR, Artifact Registry, ACR** | Cloud IAM; regional; scanning; lazy pulling on AWS (SOCI). | Lock-in; egress costs; not sovereign. |

And the bolt-ons each of them needs:

- **Distribution:** Dragonfly, Kraken or Spegel, run beside the registry.

- **Lazy pulling:** eStargz, SOCI, Nydus or zstd:chunked, each a format

  plus a snapshotter. It matters: pulling took 76% of container start time

  while only 6.4% of the pulled data was read (Slacker, FAST '16).

- **Signing and provenance:** cosign and Rekor, SLSA generators,

  in-toto, Notation, attached afterwards through the OCI referrers API.

- **Retention:** cleanup policies plus a separate garbage collection.

## 2. The pattern behind the weaknesses

1. **The unit is the tarball layer.** One changed file re-pushes and

   re-pulls a whole layer; layers share nothing below their own size; and a

   layer has two digests (compressed and not), so tools disagree on what

   it is called.

2. **Names are mutable pointers.** A tag moves silently and leaves no

   history. Pinning by digest is safe but unreadable, so most people

   don't.

3. **Trust is bolted on afterwards.** The registry stores bytes; a

   separate tool signs them, a third says how they were built, and each

   brings its own keys and identities. Nothing stops an unsigned image

   from being pushed, and most pulls never verify anything.

4. **Distribution is someone else's problem.** Every registry is a

   central server; peer-to-peer and lazy loading are extra products.

5. **The forge and the registry are strangers.** Permissions, identity,

   retention and "which commit is this?" are kept twice and drift apart.

## 3. What Oxygen already has

Every piece Cylinder needs exists in some form:

- **Content-defined chunks with BLAKE3** (FastCDC, 32 to 256 KiB), shared

  by repositories, release assets, and runner images: deduplication below

  file level, across everything.

- **Hybrid post-quantum signatures** (Ed25519 with ML-DSA-65), identities

  with delegation chains, and rotation recorded in the repository.

- **Signed Flow receipts:** a runner signs what it built, from which

  state, with which pipeline and environment, producing which outputs.

  That is provenance by construction, not a statement added later.

- **flow-casfs:** a content-addressed tree mounted lazily over FUSE or

  virtio-fs. Lazy pulling is native, not a format bolted on.

- **The Jetstream swarm:** peer-to-peer chunks, rarest first, TLS pinned

  by the hub, tickets that only the hub signs.

- **Strata:** storage on disk, S3, GCS or Azure, with no server in between,

  sovereign keys and client-side hash verification.

- **OxygenHub:** one identity and permission model, checks, proposals and

  reports.

- **Scoped delegation:** key scopes (`changes`, `refs`, `seal`) allowing

  principals and AI agents to act within strict cryptographic boundaries.

## 4. The design

### 4.1 One kind of thing: a tree of chunks

An **artifact** is an immutable, content-addressed object in the Oxygen

store naming a file tree:

```

Artifact {

  version     [0]: 1

  kind        [1]: "container" | "runner-image" | "package" | "archive" | "report"

  arch        [2]: text | null              // "x86_64", "riscv64", "aarch64"

  os          [3]: text | null              // "linux", "darwin", "none"

  tree        [4]: ObjectRef                // DirTree blob: paths, modes, blobs, sizes

  entrypoint  [5]: [text] | null            // default exec argv

  env         [6]: map<text, text> | null   // environment variables

  labels      [7]: map<text, text> | null   // metadata / OCI labels

  total_size  [8]: u64                      // uncompressed bytes

  receipt     [9]: ObjectRef | null         // Receipt object that produced it

}

```

Files in the `DirTree` are ordinary Oxygen `blob` objects composed of

FastCDC BLAKE3 `chunk`s. So:

- An image rebuilt with one changed file moves only that file's changed

  chunks: no layers.

- A release tarball, an npm package, and the container image containing

  the same binary share its chunks verbatim. If a 100 MB model or library

  is committed in the repo and then packaged, zero new chunk bytes are

  written to Strata.

- A single file can be fetched independently: a container or job starts

  after fetching only the files it reads.

- Every artifact has exactly one name for its content: the BLAKE3 of its

  canonical CBOR encoding.

- **MicroVMs:** Flow's MicroVM tier runs on Firecracker, which has no

  virtio-fs, so a microVM gets the artifact as a block device: an ext4

  image served on the fly from the chunk tree (a userspace block device,

  ublk, or an NBD server), fetching blocks as the guest reads them and

  duplicating no bytes. That removes copying whole images onto each

  runner's disk without changing hypervisor. The syscall tier and

  containers mount the tree directly over `flow-casfs`; mounting it with

  virtio-fs would need Cloud Hypervisor or QEMU instead of Firecracker.

### 4.2 Names are signed histories, not pointers

A **channel** (`app:stable`, `runner-image:rust`, `pkg:cargo/my-crate`) is an

append-only log of signed moves, recorded as an attestation of kind

`channel`:

```

Attest (kind = "channel") {

  channel   [0]: text             // e.g. "app:stable"

  old_art   [1]: ObjectRef | null // previous artifact hash (CAS check)

  new_art   [2]: ObjectRef        // target artifact hash

  serial    [3]: u64              // monotonically increasing

  receipt   [4]: ObjectRef | null // Receipt that built new_art

  reason    [5]: text             // commit message or promotion note

}

```

A move needs policy: signed by an authorized key under the repository's

trust policy (`spec.md §14.5`), from an approved bookmark, after

required checks or human approvals pass. Clients follow a channel and

verify each step:

- A tag cannot move silently. The tj-actions attack needed one stolen

  token; here it needs a move signed by an authorized key, and the move

  is logged in the append-only history.

- Rolling back is a move to an earlier artifact, recorded and attested

  like any other move.

- Pinning stays readable and tamper-evident: `app:stable@42` names the

  42nd entry of an audited log.

- On Strata stores, updates use compare-and-swap on the channel record

  (`old_art` and `serial`), matching how bookmark `refs` operate.

### 4.3 Provenance by construction and multi-party consensus

An artifact is published by a Flow job, and its record is the unbroken

chain:

> source state → pipeline → runner (identity, tier, image) → signed

> receipt → artifact

Each link is a hash or signature verifiable offline. Beyond what

attestations give today:

- **Assurance levels:** a MicroVM job on a registered runner is worth

  more than a syscall-tier job, which is worth more than a signed local

  build. Trust policies can enforce minimum assurance floors for

  sensitive channels.

- **Multi-party consensus (`reproduced`):** when multiple independent

  runners on disjoint machines or architectures build the same artifact

  hash from the same state, each signs a receipt. A policy rule can

  require threshold consensus:

  `require: "reproduced(2)"` demands two independent runner receipts

  before an artifact can be promoted to a production channel.

- **Upstream lineage:** an artifact built from other artifacts (a base

  image, a package) names their hashes in its receipt inputs, so the chain

  reaches back transitively to their sources.

- **An SBOM with every artifact:** the job that publishes an artifact also

  records what it contains by package and version (an SBOM, itself an

  artifact named in the receipt's outputs), so "which artifacts hold a

  vulnerable version of this package" is a lookup, not a scan (§4.6).

### 4.4 Distribution is built in

- **The store is the seed; the swarm delivers.** Jetstream's swarm,

  generalized: any runner, cluster node or developer machine in a site

  shares the chunks it has, over TLS pinned by the hub, admitted by

  hub-signed tickets.

- **Lazy by default:** artifacts are mounted via `flow-casfs` and fetched

  on demand.

- **Learned prefetch:** each run records which chunks were accessed

  during its initial execution window; subsequent pulls request those

  chunks first.

- **Sites and air gaps:** a site syncs chunks and channel logs from

  another hub as a signed bundle, so a disconnected network gets

  verified artifacts with no live connection.

### 4.5 Sovereign storage, and our own transparency

- Bytes live on any Strata store: local disk, S3, GCS or Azure, in the

  customer's own account and region, encrypted at rest by the store with

  customer-managed keys (for example S3 with a KMS key). Chunks stay named

  by the hash of their plain bytes: encrypting them on the client with

  separate keys would stop them deduplicating and stop peers checking them

  by hash.

- Channel moves and signatures are recorded in the hub's append-only

  Merkle log. Auditing requires no dependence on public ledgers like

  Rekor, though Cylinder can bridge to them when desired.

### 4.6 Policy at point of use, and the inverted vulnerability index

Verification happens where the artifact is used, not only where it is

stored:

- A runner checks the job's image against policy before execution.

- A Kubernetes admission hook checks pod images against channel policies

  and assurance levels.

- `oxy` and `flow` verify what they fetch.

**The inverted file index:** Because artifacts are trees of

content-addressed files rather than opaque tarballs, Cylinder keeps an

index from each file's hash to the artifacts that contain it:

$$\text{FileHash} \longrightarrow [\text{ArtifactHash}]$$

- **Exact files, at once.** Given a known-bad file (a compromised binary,

  a leaked key, the exact `libcrypto.so.3` an advisory names), one lookup

  lists every container image, runner image and package across the fleet

  holding it, with no layers to unpack.

- **Not versions.** A patched or rebuilt library is a different file with

  a different hash, so this index does not answer "which images have a

  vulnerable version of OpenSSL". That comes from package identity: the

  SBOM each artifact carries (§4.3), indexed by package and version. The

  two together answer both questions; the file index alone answers the

  first.

- **Revocation without an outage.** When an artifact, base image or file

  is revoked in policy, admission hooks refuse new pods that use it at

  once. Pods already running are not touched by admission; a controller

  marks them, then replaces or evicts them within a grace period the

  policy sets, so one policy change cannot take production down.

### 4.7 Retention by reachability and deployment leases

What a channel, a release, a pinned pipeline or an active deployment

reaches is kept; unreferenced chunks are collected by `oxy gc`.

**Deployment leases:** Container registries often suffer from garbage

collection deleting images still running on long-lived cluster pods.

To prevent this, nodes or Kubernetes operators register periodic

**deployment leases**:

```

Attest (kind = "lease") {

  artifact [0]: ObjectRef  // artifact hash currently running

  scope    [1]: text       // e.g. "k8s:cluster-lon/ns/workload"

  expires  [2]: u64        // timestamp

}

```

Active leases act as garbage collection roots. If a workload terminates

or a cluster is decommissioned, the lease expires naturally, enabling

safe automated retention without operational risk. A lease is signed by

a key scoped to leases and to its `scope`, and has a maximum length (a

day, renewed while the workload runs), so a compromised node cannot hold

storage forever or pin artifacts outside its own scope.

### 4.8 Compatibility at the edge

- **An OCI registry façade:** `docker pull` and Kubernetes work

  unchanged. Manifests and layers are assembled on the fly from chunks.

  `docker push` is accepted and chunked via FastCDC, applying

  deduplication to legacy images.

- **A containerd snapshotter** speaking the native protocol, mounting

  file trees via `flow-casfs` for instant sub-second container starts

  and P2P delivery on Kubernetes.

- **Package façades (Cargo, npm, PyPI):** Served from the same chunk

  store. Façade endpoints synthesize package metadata (Cargo sparse

  indexes, npm packuments, PyPI simple indexes) dynamically from artifact

  manifests.

- **Upstream dependency quarantine:** When Flow builds fetch third-party

  packages from crates.io or npm, Cylinder proxies them into the

  organization's Strata store, locking the hashes into internal mirror

  channels. Upstream outages or deletions cannot break builds. Packages

  arrive compressed (`.crate` and `.tgz` are gzip), which chunks poorly,

  so the original archive is kept for clients that check its hash and its

  unpacked files are chunked as an artifact tree, where they deduplicate

  with everything else.

### 4.9 AI-native agent integration

Extending Oxygen's scoped delegation model (`ai_native_agent_workflows.md §2`),

agent keys can hold artifact-specific scopes:

```

delegates [8]: [{ key: KeyId, scope: ["changes" | "refs" | "seal" | "artifacts" | "channels"], expires: u64? }]

```

- An AI agent (e.g. Aider, Claude Code) working autonomously can be

  granted `--scope artifacts` and restricted channel rights:

  - Permitted: building artifacts and moving `agents/NAME/**` channels.

  - Forbidden: moving production channels (`app:stable`, `release/*`).

- Agents can test containerized workloads end-to-end in isolated

  channels, while production promotion remains guarded by human landing

  rules or multi-party consensus.

### 4.10 Revset query integration

Oxygen's revset language (`revset_query_language.md`) bridges source

changes and deployed artifacts:

| Expression | Selects |

|---|---|

| `artifact(CHANNEL)` | Changes contained within the artifact currently at `CHANNEL` |

| `deployed(ENV)` | Changes currently active in environment `ENV` (via active leases) |

| `unshipped()` | Equivalent to `main ~ artifact(app:stable)` (changes on `main` that have not reached production) |

| `reproduced()` | Changes whose built artifacts have 2+ independent runner receipts |

| `vulnerable(CVE)` | Changes that touch or introduced files present in an artifact flagged with `CVE` |

Running `oxy log -r 'main ~ artifact(app:stable)'` immediately shows all

changes pending deployment.

### 4.11 The UI (inside the hub)

- **Per repository, an Artifacts tab:** published artifacts, channel

  histories, and the provenance graph from commit to run, artifact, and

  deployment lease.

- **Site-wide:** chunk distribution across hubs and runners, active

  swarm transfers, deduplication savings, and the global CVE inverted index.

- **Runner images** are listed as standard artifacts with their build

  receipts and hardware architecture badges.

```mermaid

flowchart LR

  subgraph Build

    S[Source state] --> P[Pipeline]

    P --> R1[Runner 1: microvm]

    P --> R2[Runner 2: disjoint host]

    R1 -->|receipt 1| A[Artifact: tree of chunks]

    R2 -->|receipt 2| A

  end

  A -->|consensus move| C[Channel log: app:stable@42]

  A --> ST[(Strata store)]

  ST -->|seed| SW{{Swarm}}

  SW --> N1[Runner]

  SW --> N2[K8s node: containerd snapshotter]

  SW --> N3[Developer]

  N2 -->|deployment lease| DL[Active Leases: GC Roots]

  C -->|policy check| N2

  OCI[OCI façade] --- ST

  PKG[Package façades: cargo/npm/pypi] --- ST

  H[OxygenHub UI & CVE Index] --- C

  H --- A

```

### 4.12 Where it stands among the services

- **OxygenHub** owns repositories, proposals, checks, user identity,

  and the UI.

- **Jetstream** owns the fleet: queue, placement, and runner swarms. It

  fetches runner images from Cylinder by artifact hash.

- **Flow** executes pipelines and produces signed execution receipts.

- **Cylinder** owns artifacts: manifests, channel logs, provenance

  chains, distribution, package façades, and deployment leases.

- **Strata** provides the underlying content-addressed chunk storage.

Each service runs embedded in OxygenHub by default, or standalone for

distributed scale.

## 5. Better how, in one table

| | Today's best | Cylinder |

|---|---|---|

| Rebuild with one changed file | Re-push and re-pull the layer | Only that file's changed chunks |

| Dedupe | Whole identical layers | FastCDC chunks across repos, images and packages |

| Names | Mutable tags; digests nobody reads | Signed, append-only channels (`app:stable@42`) |

| Provenance | Attestations added later, if at all | The build's signed receipt is the record |

| Signing | A separate tool and keys | Native Ed25519 + ML-DSA-65 composite signatures |

| Consensus | Single-builder attestation | Multi-party consensus receipts (`reproduced(N)`) |

| Lazy start | Bolted-on snapshotter (SOCI, Nydus) | Native: file trees mounted via `flow-casfs` |

| Peer to peer | Separate mesh daemon (Dragonfly) | Built-in Jetstream swarm, TLS-pinned |

| Vulnerability search | Unpack and scan every layer | Exact files by one index lookup; versions by SBOM package index |

| Retention | Tag rules + manual GC | Reachability + active deployment leases |

| AI workflows | Uncontrolled service tokens | Scoped delegation (`artifacts`, `channels`) |

| Querying | Disconnected CI/CD dashboards | Unified revsets (`main ~ artifact(app:stable)`) |

| Sovereignty | Complex multi-tier server stack | Single binary; customer KMS and Strata bucket |

## 6. How we could build it

- **C0. Images in the hub (built 2026-10-12, oxygenhub #146).** Admin ›

  Runners › Images: every runner image, its size and architecture, where

  it is (the hub, each runner, swarm fetches in progress), the

  repositories whose jobs used it, and an image held nowhere marked.

- **C1. The service boundary and artifact model.** Implement the

  `Artifact` CBOR object type and FastCDC chunk engine in `cylinder`;

  unify runner images with artifacts.

- **C2. Channels, provenance and consensus.** Flow publishes artifacts;

  signed channel move attestations; multi-party `reproduced(N)` consensus;

  revset predicates.

- **C3. The OCI façade and containerd snapshotter.** OCI pull/push

  translation; `containerd` snapshotter over `flow-casfs`.

- **C4. Leases and the indexes.** Kubernetes deployment lease operator

  for safe GC; the inverted file index and the SBOM package index; the

  controller that replaces revoked running workloads within their grace

  period.

- **C5. Reach and package façades.** Package façades for Cargo, npm,

  and PyPI; upstream dependency quarantine mirroring.

## 7. Open questions

These are Andy's decisions. Each carries the position proposed in review,

not an answer.

- **The name.** Proposed: keep **Cylinder**. It fits the elemental

  progression: *Oxygen* (VCS) is carried in *Cylinders* (artifacts) by

  *Jetstream* (the fleet), driven by *Flow* (pipelines).

- **Where it lives.** Proposed: first in `oxygenflow` as

  `crates/oxygen-cylinder`, beside `jet` and `flow`, to share `flow-casfs`

  and the chunking code; its own repository from C3.

- **Local builds.** Proposed: allowed, at the lowest assurance level

  (`local`); production channels require more (`tier-microvm` and

  `reproduced(2)`).

- **A public catalogue and federation.** Proposed: public hubs federate

  by publishing signed Strata bundles of channels and artifacts, checked

  offline with no central intermediary.

- **Its place in the queue.** Proposed: C1 alongside Jetstream's service

  boundary, so Jetstream's image store is built on Cylinder from the

  start rather than on a temporary mechanism. Against: the core work of

  2026-10-03 (Strata's remaining stores, agents, hub polish) waits

  longer.

## Sources

- Docker Hub pull limits: [docs.docker.com, pull usage and limits](https://docs.docker.com/docker-hub/usage/pulls/); [heise, "only 10 pulls per hour"](https://heise.de/-10329047)

- GHCR billing: [GitHub community discussion #183054](https://github.com/orgs/community/discussions/183054)

- GitLab cleanup policies and garbage collection: [GitLab docs, reduce container registry storage](https://docs.gitlab.com/user/packages/container_registry/reduce_container_registry_storage/)

- Bitbucket Packages: [Atlassian blog, Bitbucket Packages](https://www.atlassian.com/blog/bitbucket/bitbucket-packages-container-registry); [GA announcement](https://www.atlassian.com/blog/bitbucket/open-beta-introducing-packages-native-container-registry-in-bitbucket-cloud)

- Pull time versus data read: [SOCI paper (Amazon), citing Slacker](https://cdn.amazon.science/84/96/51ea2e0d40fca8cfaee9328278b9/soci-paper.pdf); [Grab engineering, lazy loading](https://engineering.grab.com/docker-lazy-loading)

- zstd:chunked: [Red Hat, faster container image pulls](https://www.redhat.com/sysadmin/faster-container-image-pulls)

- Peer-to-peer meshes: [Spegel vs Kraken vs Dragonfly](https://www.pistack.xyz/posts/2026-04-28-spegel-vs-kraken-vs-dragonfly-self-hosted-p2p-container-image-distribution-guide-2026/)

- Sigstore in registries: [Sigstore in image registries: adoption and reality](https://jacar.es/en/sigstore-in-image-registries-adoption-and-reality/); [cosign and the referrers API](https://oneuptime.com/blog/post/2026-08-11-cosign-signature-storage-oci-referrers-separate-repositories/markdown)

- tj-actions tag attack: [CVE-2025-30066](https://inventivehq.com/tools/security/cve-lookup/cve-2025-30066); [Legit Security, immutable tags](https://legitsecurity.com/blog/what-are-immutable-tags-and-can-they-protect-you-from-supply-chain-attacks)

- Artifactory and Nexus: [Codefresh, Artifactory alternatives](https://codefresh.io/learn/jfrog-artifactory/top-8-artifactory-alternatives-in-2025); [PeerSpot, Artifactory improvements](https://origin.peerspot.com/questions/what-needs-improvement-with-jfrog-artifactory)