Cylinder: the artifact service proposal d0016e229b04

# Cylinder: artifacts, the Oxygen way

Status: Draft 0.2 (proposal revision 2), 2026-10-12. Incorporates review of

object model, deployment leases, inverted vulnerability indexing,

multi-party consensus, agent scopes, and revset queries. Not built.

"Cylinder" is a working name (the artifact-repository idea parked on

2026-10-02); nothing here is decided until Andy says so.

**Cylinder** keeps and moves everything a build produces: runner images,

container images, packages, binaries, SBOMs and reports. It would be a

service of its own, embedded in OxygenHub by default and shown through

the hub's UI, the same pattern as Jetstream. It is meant to be better

than what exists, not compatible with it; compatibility is an edge we add

for adoption, never the core.

## 1. What exists today

| | Strengths | Weaknesses |

|---|---|---|

| **Docker Hub** | The default namespace; huge public catalogue; official images; scanning (Scout). | Pull limits since April 2025: 10 an hour anonymous, 100 an hour on a free account. SaaS only. Tags are mutable. Signatures and provenance are separate artifacts; referrers are still in beta. Nothing links an image to the source it came from. |

| **GitHub** (GHCR, Packages) | Next to the code and Actions; free container storage for now; the best Sigstore integration; artifact attestations. | Storage billing for private images deferred, not settled. One silo per package format. Auth by personal access tokens. Tags are mutable: the tj-actions attack (March 2025, CVE-2025-30066) moved every tag of a popular action to a malicious commit at once. No self-hosting outside Enterprise Server. |

| **GitLab** | A registry per project; many package formats; a dependency proxy; self-hostable, so it can be sovereign. | Cleanup policies delete tags only; layers stay until an administrator runs garbage collection. The registry is a separate service to run and scale. Signing means bringing cosign yourself. |

| **Bitbucket** (Packages) | GA since December 2025; permissions inherited from the repository; garbage collection runs by itself; Maven and npm since April 2026. | Young; Cloud only; few formats; no provenance story. |

| **Artifactory, Nexus** | Every format; proxies and caches; enterprise features. | Costly; heavy JVM operations; slow UIs. A second system beside the forge, with its own identities and permissions. Provenance and scanning are add-on products. |

| **Harbor, Quay** | Self-hosted OCI with signing (cosign, Notation), replication and scanning. | Many components to run. A central bottleneck: peer-to-peer means adding Dragonfly. |

| **ECR, Artifact Registry, ACR** | Cloud IAM; regional; scanning; lazy pulling on AWS (SOCI). | Lock-in; egress costs; not sovereign. |

And the bolt-ons each of them needs:

- **Distribution:** Dragonfly, Kraken or Spegel, run beside the registry.

- **Lazy pulling:** eStargz, SOCI, Nydus or zstd:chunked, each a format

  plus a snapshotter. It matters: pulling took 76% of container start time

  while only 6.4% of the pulled data was read (Slacker, FAST '16).

- **Signing and provenance:** cosign and Rekor, SLSA generators,

  in-toto, Notation, attached afterwards through the OCI referrers API.

- **Retention:** cleanup policies plus a separate garbage collection.

## 2. The pattern behind the weaknesses

1. **The unit is the tarball layer.** One changed file re-pushes and

   re-pulls a whole layer; layers share nothing below their own size; and a

   layer has two digests (compressed and not), so tools disagree on what

   it is called.

2. **Names are mutable pointers.** A tag moves silently and leaves no

   history. Pinning by digest is safe but unreadable, so most people

   don't.

3. **Trust is bolted on afterwards.** The registry stores bytes; a

   separate tool signs them, a third says how they were built, and each

   brings its own keys and identities. Nothing stops an unsigned image

   from being pushed, and most pulls never verify anything.

4. **Distribution is someone else's problem.** Every registry is a

   central server; peer-to-peer and lazy loading are extra products.

5. **The forge and the registry are strangers.** Permissions, identity,

   retention and "which commit is this?" are kept twice and drift apart.

## 3. What Oxygen already has

Every piece Cylinder needs exists in some form:

- **Content-defined chunks with BLAKE3** (FastCDC, 32 to 256 KiB), shared

  by repositories, release assets, and runner images: deduplication below

  file level, across everything.

- **Hybrid post-quantum signatures** (Ed25519 with ML-DSA-65), identities

  with delegation chains, and rotation recorded in the repository.

- **Signed Flow receipts:** a runner signs what it built, from which

  state, with which pipeline and environment, producing which outputs.

  That is provenance by construction, not a statement added later.

- **flow-casfs:** a content-addressed tree mounted lazily over FUSE or

  virtio-fs. Lazy pulling is native, not a format bolted on.

- **The Jetstream swarm:** peer-to-peer chunks, rarest first, TLS pinned

  by the hub, tickets that only the hub signs.

- **Strata:** storage on disk, S3, GCS or Azure, with no server in between,

  sovereign keys and client-side hash verification.

- **OxygenHub:** one identity and permission model, checks, proposals and

  reports.

- **Scoped delegation:** key scopes (`changes`, `refs`, `seal`) allowing

  principals and AI agents to act within strict cryptographic boundaries.

## 4. The design

### 4.1 One kind of thing: a tree of chunks

An **artifact** is an immutable, content-addressed object in the Oxygen

store naming a file tree:

```

Artifact {

  version     [0]: 1

  kind        [1]: "container" | "runner-image" | "package" | "archive" | "report"

  arch        [2]: text | null              // "x86_64", "riscv64", "aarch64"

  os          [3]: text | null              // "linux", "darwin", "none"

  tree        [4]: ObjectRef                // DirTree blob: paths, modes, blobs, sizes

  entrypoint  [5]: [text] | null            // default exec argv

  env         [6]: map<text, text> | null   // environment variables

  labels      [7]: map<text, text> | null   // metadata / OCI labels

  total_size  [8]: u64                      // uncompressed bytes

  receipt     [9]: ObjectRef | null         // Receipt object that produced it

}

```

Files in the `DirTree` are ordinary Oxygen `blob` objects composed of

FastCDC BLAKE3 `chunk`s. So:

- An image rebuilt with one changed file moves only that file's changed

  chunks: no layers.

- A release tarball, an npm package, and the container image containing

  the same binary share its chunks verbatim. If a 100 MB model or library

  is committed in the repo and then packaged, zero new chunk bytes are

  written to Strata.

- A single file can be fetched independently: a container or job starts

  after fetching only the files it reads.

- Every artifact has exactly one name for its content: the BLAKE3 of its

  canonical CBOR encoding.

- **MicroVMs and virtio-fs:** Jetstream microVM runners mount an artifact's

  tree directly via virtio-fs over `flow-casfs`, eliminating ext4 block

  images, loop devices, and cold disk allocation. Where a raw block device

  is required by legacy hardware or external hypervisors, an ext4 block

  map is synthesized on the fly from the chunk tree without duplicating

  bytes.

### 4.2 Names are signed histories, not pointers

A **channel** (`app:stable`, `runner-image:rust`, `pkg:cargo/my-crate`) is an

append-only log of signed moves, recorded as an attestation of kind

`channel`:

```

Attest (kind = "channel") {

  channel   [0]: text             // e.g. "app:stable"

  old_art   [1]: ObjectRef | null // previous artifact hash (CAS check)

  new_art   [2]: ObjectRef        // target artifact hash

  serial    [3]: u64              // monotonically increasing

  receipt   [4]: ObjectRef | null // Receipt that built new_art

  reason    [5]: text             // commit message or promotion note

}

```

A move needs policy: signed by an authorized key under the repository's

trust policy (`spec.md §14.5`), from an approved bookmark, after

required checks or human approvals pass. Clients follow a channel and

verify each step:

- A tag cannot move silently. The tj-actions attack needed one stolen

  token; here it needs a move signed by an authorized key, and the move

  is logged in the append-only history.

- Rolling back is a move to an earlier artifact, recorded and attested

  like any other move.

- Pinning stays readable and tamper-evident: `app:stable@42` names the

  42nd entry of an audited log.

- On Strata stores, updates use compare-and-swap on the channel record

  (`old_art` and `serial`), matching how bookmark `refs` operate.

### 4.3 Provenance by construction and multi-party consensus

An artifact is published by a Flow job, and its record is the unbroken

chain:

> source state → pipeline → runner (identity, tier, image) → signed

> receipt → artifact

Each link is a hash or signature verifiable offline. Beyond what

attestations give today:

- **Assurance levels:** a MicroVM job on a registered runner is worth

  more than a syscall-tier job, which is worth more than a signed local

  build. Trust policies can enforce minimum assurance floors for

  sensitive channels.

- **Multi-party consensus (`reproduced`):** when multiple independent

  runners on disjoint machines or architectures build the same artifact

  hash from the same state, each signs a receipt. A policy rule can

  require threshold consensus:

  `require: "reproduced(2)"` demands two independent runner receipts

  before an artifact can be promoted to a production channel.

- **Upstream lineage:** an artifact built from other artifacts (a base

  image, a package) names their hashes in its receipt inputs, so the chain

  reaches back transitively to their sources.

### 4.4 Distribution is built in

- **The store is the seed; the swarm delivers.** Jetstream's swarm,

  generalized: any runner, cluster node or developer machine in a site

  shares the chunks it has, over TLS pinned by the hub, admitted by

  hub-signed tickets.

- **Lazy by default:** artifacts are mounted via `flow-casfs` and fetched

  on demand.

- **Learned prefetch:** each run records which chunks were accessed

  during its initial execution window; subsequent pulls request those

  chunks first.

- **Sites and air gaps:** a site syncs chunks and channel logs from

  another hub as a signed bundle, so a disconnected network gets

  verified artifacts with no live connection.

### 4.5 Sovereign storage, and our own transparency

- Bytes live on any Strata store: local disk, S3, GCS or Azure, in the

  customer's own account and region, encrypted with customer-managed KMS.

- Channel moves and signatures are recorded in the hub's append-only

  Merkle log. Auditing requires no dependence on public ledgers like

  Rekor, though Cylinder can bridge to them when desired.

### 4.6 Policy at point of use, and the inverted vulnerability index

Verification happens where the artifact is used, not only where it is

stored:

- A runner checks the job's image against policy before execution.

- A Kubernetes admission hook checks pod images against channel policies

  and assurance levels.

- `oxy` and `flow` verify what they fetch.

**The global inverted vulnerability index:** Because artifacts are trees

of content-addressed files and chunks rather than opaque tarballs,

Cylinder maintains an inverted index from chunk/file hash to artifacts:

$$\text{FileHash} \longrightarrow [\text{ArtifactHash}]$$

When a security advisory or zero-day is disclosed for a binary or

library (e.g., a vulnerable `libcrypto.so.3` or Go binary hash):

- Cylinder performs an immediate $O(1)$ lookup to identify every

  container image, runner image, and package across the fleet containing

  that file.

- There is no need to unpack gigabytes of tarball layers across thousands

  of images to discover exposure.

- **Continuous revocation:** If an upstream library or base image is

  revoked in policy, point-of-use admission hooks immediately block

  downstream pods from running any artifact containing that chunk,

  eliminating the gap between vulnerability discovery and redeployment.

### 4.7 Retention by reachability and deployment leases

What a channel, a release, a pinned pipeline or an active deployment

reaches is kept; unreferenced chunks are collected by `oxy gc`.

**Deployment leases:** Container registries often suffer from garbage

collection deleting images still running on long-lived cluster pods.

To prevent this, nodes or Kubernetes operators register periodic

**deployment leases**:

```

Attest (kind = "lease") {

  artifact [0]: ObjectRef  // artifact hash currently running

  scope    [1]: text       // e.g. "k8s:cluster-lon/ns/workload"

  expires  [2]: u64        // timestamp

}

```

Active leases act as garbage collection roots. If a workload terminates

or a cluster is decommissioned, the lease expires naturally, enabling

safe automated retention without operational risk.

### 4.8 Compatibility at the edge

- **An OCI registry façade:** `docker pull` and Kubernetes work

  unchanged. Manifests and layers are assembled on the fly from chunks.

  `docker push` is accepted and chunked via FastCDC, applying

  deduplication to legacy images.

- **A containerd snapshotter** speaking the native protocol, mounting

  file trees via `flow-casfs` for instant sub-second container starts

  and P2P delivery on Kubernetes.

- **Package façades (Cargo, npm, PyPI):** Served from the same chunk

  store. Façade endpoints synthesize package metadata (Cargo sparse

  indexes, npm packuments, PyPI simple indexes) dynamically from artifact

  manifests.

- **Upstream dependency quarantine:** When Flow builds fetch third-party

  packages from crates.io or npm, Cylinder proxies and chunks them into

  the organization's Strata store, locking the hashes into internal

  mirror channels. Upstream outages or deletions cannot break builds.

### 4.9 AI-native agent integration

Extending Oxygen's scoped delegation model (`ai_native_agent_workflows.md §2`),

agent keys can hold artifact-specific scopes:

```

delegates [8]: [{ key: KeyId, scope: ["changes" | "refs" | "seal" | "artifacts" | "channels"], expires: u64? }]

```

- An AI agent (e.g. Aider, Claude Code) working autonomously can be

  granted `--scope artifacts` and restricted channel rights:

  - Permitted: building artifacts and moving `agents/NAME/**` channels.

  - Forbidden: moving production channels (`app:stable`, `release/*`).

- Agents can test containerized workloads end-to-end in isolated

  channels, while production promotion remains guarded by human landing

  rules or multi-party consensus.

### 4.10 Revset query integration

Oxygen's revset language (`revset_query_language.md`) bridges source

changes and deployed artifacts:

| Expression | Selects |

|---|---|

| `artifact(CHANNEL)` | Changes contained within the artifact currently at `CHANNEL` |

| `deployed(ENV)` | Changes currently active in environment `ENV` (via active leases) |

| `unshipped()` | Equivalent to `main ~ artifact(app:stable)` (changes on `main` that have not reached production) |

| `reproduced()` | Changes whose built artifacts have 2+ independent runner receipts |

| `vulnerable(CVE)` | Changes that touch or introduced files present in an artifact flagged with `CVE` |

Running `oxy log -r 'main ~ artifact(app:stable)'` immediately shows all

changes pending deployment.

### 4.11 The UI (inside the hub)

- **Per repository, an Artifacts tab:** published artifacts, channel

  histories, and the provenance graph from commit to run, artifact, and

  deployment lease.

- **Site-wide:** chunk distribution across hubs and runners, active

  swarm transfers, deduplication savings, and the global CVE inverted index.

- **Runner images** are listed as standard artifacts with their build

  receipts and hardware architecture badges.

```mermaid

flowchart LR

  subgraph Build

    S[Source state] --> P[Pipeline]

    P --> R1[Runner 1: microvm]

    P --> R2[Runner 2: disjoint host]

    R1 -->|receipt 1| A[Artifact: tree of chunks]

    R2 -->|receipt 2| A

  end

  A -->|consensus move| C[Channel log: app:stable@42]

  A --> ST[(Strata store)]

  ST -->|seed| SW{{Swarm}}

  SW --> N1[Runner]

  SW --> N2[K8s node: containerd snapshotter]

  SW --> N3[Developer]

  N2 -->|deployment lease| DL[Active Leases: GC Roots]

  C -->|policy check| N2

  OCI[OCI façade] --- ST

  PKG[Package façades: cargo/npm/pypi] --- ST

  H[OxygenHub UI & CVE Index] --- C

  H --- A

```

### 4.12 Where it stands among the services

- **OxygenHub** owns repositories, proposals, checks, user identity,

  and the UI.

- **Jetstream** owns the fleet: queue, placement, and runner swarms. It

  fetches runner images from Cylinder by artifact hash.

- **Flow** executes pipelines and produces signed execution receipts.

- **Cylinder** owns artifacts: manifests, channel logs, provenance

  chains, distribution, package façades, and deployment leases.

- **Strata** provides the underlying content-addressed chunk storage.

Each service runs embedded in OxygenHub by default, or standalone for

distributed scale.

## 5. Better how, in one table

| | Today's best | Cylinder |

|---|---|---|

| Rebuild with one changed file | Re-push and re-pull the layer | Only that file's changed chunks |

| Dedupe | Whole identical layers | FastCDC chunks across repos, images and packages |

| Names | Mutable tags; digests nobody reads | Signed, append-only channels (`app:stable@42`) |

| Provenance | Attestations added later, if at all | The build's signed receipt is the record |

| Signing | A separate tool and keys | Native Ed25519 + ML-DSA-65 composite signatures |

| Consensus | Single-builder attestation | Multi-party consensus receipts (`reproduced(N)`) |

| Lazy start | Bolted-on snapshotter (SOCI, Nydus) | Native: file trees mounted via `flow-casfs` |

| Peer to peer | Separate mesh daemon (Dragonfly) | Built-in Jetstream swarm, TLS-pinned |

| Vulnerability search | Unpack and scan every layer ($O(N)$) | $O(1)$ inverted index by file and chunk hash |

| Retention | Tag rules + manual GC | Reachability + active deployment leases |

| AI workflows | Uncontrolled service tokens | Scoped delegation (`artifacts`, `channels`) |

| Querying | Disconnected CI/CD dashboards | Unified revsets (`main ~ artifact(app:stable)`) |

| Sovereignty | Complex multi-tier server stack | Single binary; customer KMS and Strata bucket |

## 6. How we could build it

- **C0. Images in the hub, now.** A Flow › Images page: each image, where

  it is (hub, runners, swarm activity) and what pins it.

- **C1. The service boundary and artifact model.** Implement the

  `Artifact` CBOR object type and FastCDC chunk engine in `cylinder`;

  unify runner images with artifacts.

- **C2. Channels, provenance and consensus.** Flow publishes artifacts;

  signed channel move attestations; multi-party `reproduced(N)` consensus;

  revset predicates.

- **C3. The OCI façade and containerd snapshotter.** OCI pull/push

  translation; `containerd` snapshotter over `flow-casfs`.

- **C4. Leases and the inverted index.** Kubernetes deployment lease

  operator for safe GC; global inverted chunk index for instantaneous

  CVE discovery.

- **C5. Reach and package façades.** Package façades for Cargo, npm,

  and PyPI; upstream dependency quarantine mirroring.

## 7. Open questions

- **The name:** Keep **Cylinder**. It fits the elemental progression:

  *Oxygen* (VCS) is transported in *Cylinders* (artifacts) via

  *Jetstream* (fleet transport) driven by *Flow* (pipelines).

- **Where it lives:** Initially in `oxygenflow` / `crates/oxygen-cylinder`

  alongside `jet` and `flow` to share `flow-casfs` and chunking primitives,

  splitting into an independent repository at C3.

- **Local builds:** Allowed, but marked with lower assurance level

  (`local`). Production channel rules require higher assurance levels

  (`tier-microvm` + `reproduced(2)`).

- **A public catalogue / Federation:** Public hubs federate by publishing

  signed Strata bundles of channels and artifacts, verified offline

  without centralized intermediaries.

- **Place in the queue:** C0/C1 should proceed alongside Jetstream's

  service boundary work, ensuring Jetstream's image store is built

  directly on Cylinder rather than creating a temporary mechanism.

## Sources

- Docker Hub pull limits: [docs.docker.com, pull usage and limits](https://docs.docker.com/docker-hub/usage/pulls/); [heise, "only 10 pulls per hour"](https://heise.de/-10329047)

- GHCR billing: [GitHub community discussion #183054](https://github.com/orgs/community/discussions/183054)

- GitLab cleanup policies and garbage collection: [GitLab docs, reduce container registry storage](https://docs.gitlab.com/user/packages/container_registry/reduce_container_registry_storage/)

- Bitbucket Packages: [Atlassian blog, Bitbucket Packages](https://www.atlassian.com/blog/bitbucket/bitbucket-packages-container-registry); [GA announcement](https://www.atlassian.com/blog/bitbucket/open-beta-introducing-packages-native-container-registry-in-bitbucket-cloud)

- Pull time versus data read: [SOCI paper (Amazon), citing Slacker](https://cdn.amazon.science/84/96/51ea2e0d40fca8cfaee9328278b9/soci-paper.pdf); [Grab engineering, lazy loading](https://engineering.grab.com/docker-lazy-loading)

- zstd:chunked: [Red Hat, faster container image pulls](https://www.redhat.com/sysadmin/faster-container-image-pulls)

- Peer-to-peer meshes: [Spegel vs Kraken vs Dragonfly](https://www.pistack.xyz/posts/2026-04-28-spegel-vs-kraken-vs-dragonfly-self-hosted-p2p-container-image-distribution-guide-2026/)

- Sigstore in registries: [Sigstore in image registries: adoption and reality](https://jacar.es/en/sigstore-in-image-registries-adoption-and-reality/); [cosign and the referrers API](https://oneuptime.com/blog/post/2026-08-11-cosign-signature-storage-oci-referrers-separate-repositories/markdown)

- tj-actions tag attack: [CVE-2025-30066](https://inventivehq.com/tools/security/cve-lookup/cve-2025-30066); [Legit Security, immutable tags](https://legitsecurity.com/blog/what-are-immutable-tags-and-can-they-protect-you-from-supply-chain-attacks)

- Artifactory and Nexus: [Codefresh, Artifactory alternatives](https://codefresh.io/learn/jfrog-artifactory/top-8-artifactory-alternatives-in-2025); [PeerSpot, Artifactory improvements](https://origin.peerspot.com/questions/what-needs-improvement-with-jfrog-artifactory)