# Cylinder: artifacts, the Oxygen way

Status: Draft 0.3 (proposal revision 3), 2026-10-12. Draft 0.2 added the
object model, deployment leases, inverted vulnerability indexing,
multi-party consensus, agent scopes and revset queries; 0.3 corrects the
MicroVM mount path, the reach of the file index, revocation, encryption,
leases and upstream quarantine, and keeps §7 open. C0 is built; the rest
is not. "Cylinder" is a working name (the artifact-repository idea parked
on 2026-10-02); nothing here is decided until Andy says so.

**Cylinder** keeps and moves everything a build produces: runner images,
container images, packages, binaries, SBOMs and reports. It would be a
service of its own, embedded in OxygenHub by default and shown through
the hub's UI, the same pattern as Jetstream. It is meant to be better
than what exists, not compatible with it; compatibility is an edge we add
for adoption, never the core.

## 1. What exists today

| | Strengths | Weaknesses |
|---|---|---|
| **Docker Hub** | The default namespace; huge public catalogue; official images; scanning (Scout). | Pull limits since April 2025: 10 an hour anonymous, 100 an hour on a free account. SaaS only. Tags are mutable. Signatures and provenance are separate artifacts; referrers are still in beta. Nothing links an image to the source it came from. |
| **GitHub** (GHCR, Packages) | Next to the code and Actions; free container storage for now; the best Sigstore integration; artifact attestations. | Storage billing for private images deferred, not settled. One silo per package format. Auth by personal access tokens. Tags are mutable: the tj-actions attack (March 2025, CVE-2025-30066) moved every tag of a popular action to a malicious commit at once. No self-hosting outside Enterprise Server. |
| **GitLab** | A registry per project; many package formats; a dependency proxy; self-hostable, so it can be sovereign. | Cleanup policies delete tags only; layers stay until an administrator runs garbage collection. The registry is a separate service to run and scale. Signing means bringing cosign yourself. |
| **Bitbucket** (Packages) | GA since December 2025; permissions inherited from the repository; garbage collection runs by itself; Maven and npm since April 2026. | Young; Cloud only; few formats; no provenance story. |
| **Artifactory, Nexus** | Every format; proxies and caches; enterprise features. | Costly; heavy JVM operations; slow UIs. A second system beside the forge, with its own identities and permissions. Provenance and scanning are add-on products. |
| **Harbor, Quay** | Self-hosted OCI with signing (cosign, Notation), replication and scanning. | Many components to run. A central bottleneck: peer-to-peer means adding Dragonfly. |
| **ECR, Artifact Registry, ACR** | Cloud IAM; regional; scanning; lazy pulling on AWS (SOCI). | Lock-in; egress costs; not sovereign. |

And the bolt-ons each of them needs:

- **Distribution:** Dragonfly, Kraken or Spegel, run beside the registry.
- **Lazy pulling:** eStargz, SOCI, Nydus or zstd:chunked, each a format
  plus a snapshotter. It matters: pulling took 76% of container start time
  while only 6.4% of the pulled data was read (Slacker, FAST '16).
- **Signing and provenance:** cosign and Rekor, SLSA generators,
  in-toto, Notation, attached afterwards through the OCI referrers API.
- **Retention:** cleanup policies plus a separate garbage collection.

## 2. The pattern behind the weaknesses

1. **The unit is the tarball layer.** One changed file re-pushes and
   re-pulls a whole layer; layers share nothing below their own size; and a
   layer has two digests (compressed and not), so tools disagree on what
   it is called.
2. **Names are mutable pointers.** A tag moves silently and leaves no
   history. Pinning by digest is safe but unreadable, so most people
   don't.
3. **Trust is bolted on afterwards.** The registry stores bytes; a
   separate tool signs them, a third says how they were built, and each
   brings its own keys and identities. Nothing stops an unsigned image
   from being pushed, and most pulls never verify anything.
4. **Distribution is someone else's problem.** Every registry is a
   central server; peer-to-peer and lazy loading are extra products.
5. **The forge and the registry are strangers.** Permissions, identity,
   retention and "which commit is this?" are kept twice and drift apart.

## 3. What Oxygen already has

Every piece Cylinder needs exists in some form:

- **Content-defined chunks with BLAKE3** (FastCDC, 32 to 256 KiB), shared
  by repositories, release assets, and runner images: deduplication below
  file level, across everything.
- **Hybrid post-quantum signatures** (Ed25519 with ML-DSA-65), identities
  with delegation chains, and rotation recorded in the repository.
- **Signed Flow receipts:** a runner signs what it built, from which
  state, with which pipeline and environment, producing which outputs.
  That is provenance by construction, not a statement added later.
- **flow-casfs:** a content-addressed tree mounted lazily over FUSE or
  virtio-fs. Lazy pulling is native, not a format bolted on.
- **The Jetstream swarm:** peer-to-peer chunks, rarest first, TLS pinned
  by the hub, tickets that only the hub signs.
- **Strata:** storage on disk, S3, GCS or Azure, with no server in between,
  sovereign keys and client-side hash verification.
- **OxygenHub:** one identity and permission model, checks, proposals and
  reports.
- **Scoped delegation:** key scopes (`changes`, `refs`, `seal`) allowing
  principals and AI agents to act within strict cryptographic boundaries.

## 4. The design

### 4.1 One kind of thing: a tree of chunks

An **artifact** is an immutable, content-addressed object in the Oxygen
store naming a file tree:

```
Artifact {
  version     [0]: 1
  kind        [1]: "container" | "runner-image" | "package" | "archive" | "report"
  arch        [2]: text | null              // "x86_64", "riscv64", "aarch64"
  os          [3]: text | null              // "linux", "darwin", "none"
  tree        [4]: ObjectRef                // DirTree blob: paths, modes, blobs, sizes
  entrypoint  [5]: [text] | null            // default exec argv
  env         [6]: map<text, text> | null   // environment variables
  labels      [7]: map<text, text> | null   // metadata / OCI labels
  total_size  [8]: u64                      // uncompressed bytes
  receipt     [9]: ObjectRef | null         // Receipt object that produced it
}
```

Files in the `DirTree` are ordinary Oxygen `blob` objects composed of
FastCDC BLAKE3 `chunk`s. So:

- An image rebuilt with one changed file moves only that file's changed
  chunks: no layers.
- A release tarball, an npm package, and the container image containing
  the same binary share its chunks verbatim. If a 100 MB model or library
  is committed in the repo and then packaged, zero new chunk bytes are
  written to Strata.
- A single file can be fetched independently: a container or job starts
  after fetching only the files it reads.
- Every artifact has exactly one name for its content: the BLAKE3 of its
  canonical CBOR encoding.
- **MicroVMs:** Flow's MicroVM tier runs on Firecracker, which has no
  virtio-fs, so a microVM gets the artifact as a block device: an ext4
  image served on the fly from the chunk tree (a userspace block device,
  ublk, or an NBD server), fetching blocks as the guest reads them and
  duplicating no bytes. That removes copying whole images onto each
  runner's disk without changing hypervisor. The syscall tier and
  containers mount the tree directly over `flow-casfs`; mounting it with
  virtio-fs would need Cloud Hypervisor or QEMU instead of Firecracker.

### 4.2 Names are signed histories, not pointers

A **channel** (`app:stable`, `runner-image:rust`, `pkg:cargo/my-crate`) is an
append-only log of signed moves, recorded as an attestation of kind
`channel`:

```
Attest (kind = "channel") {
  channel   [0]: text             // e.g. "app:stable"
  old_art   [1]: ObjectRef | null // previous artifact hash (CAS check)
  new_art   [2]: ObjectRef        // target artifact hash
  serial    [3]: u64              // monotonically increasing
  receipt   [4]: ObjectRef | null // Receipt that built new_art
  reason    [5]: text             // commit message or promotion note
}
```

A move needs policy: signed by an authorized key under the repository's
trust policy (`spec.md §14.5`), from an approved bookmark, after
required checks or human approvals pass. Clients follow a channel and
verify each step:

- A tag cannot move silently. The tj-actions attack needed one stolen
  token; here it needs a move signed by an authorized key, and the move
  is logged in the append-only history.
- Rolling back is a move to an earlier artifact, recorded and attested
  like any other move.
- Pinning stays readable and tamper-evident: `app:stable@42` names the
  42nd entry of an audited log.
- On Strata stores, updates use compare-and-swap on the channel record
  (`old_art` and `serial`), matching how bookmark `refs` operate.

### 4.3 Provenance by construction and multi-party consensus

An artifact is published by a Flow job, and its record is the unbroken
chain:

> source state → pipeline → runner (identity, tier, image) → signed
> receipt → artifact

Each link is a hash or signature verifiable offline. Beyond what
attestations give today:

- **Assurance levels:** a MicroVM job on a registered runner is worth
  more than a syscall-tier job, which is worth more than a signed local
  build. Trust policies can enforce minimum assurance floors for
  sensitive channels.
- **Multi-party consensus (`reproduced`):** when multiple independent
  runners on disjoint machines or architectures build the same artifact
  hash from the same state, each signs a receipt. A policy rule can
  require threshold consensus:
  `require: "reproduced(2)"` demands two independent runner receipts
  before an artifact can be promoted to a production channel.
- **Upstream lineage:** an artifact built from other artifacts (a base
  image, a package) names their hashes in its receipt inputs, so the chain
  reaches back transitively to their sources.
- **An SBOM with every artifact:** the job that publishes an artifact also
  records what it contains by package and version (an SBOM, itself an
  artifact named in the receipt's outputs), so "which artifacts hold a
  vulnerable version of this package" is a lookup, not a scan (§4.6).

### 4.4 Distribution is built in

- **The store is the seed; the swarm delivers.** Jetstream's swarm,
  generalized: any runner, cluster node or developer machine in a site
  shares the chunks it has, over TLS pinned by the hub, admitted by
  hub-signed tickets.
- **Lazy by default:** artifacts are mounted via `flow-casfs` and fetched
  on demand.
- **Learned prefetch:** each run records which chunks were accessed
  during its initial execution window; subsequent pulls request those
  chunks first.
- **Sites and air gaps:** a site syncs chunks and channel logs from
  another hub as a signed bundle, so a disconnected network gets
  verified artifacts with no live connection.

### 4.5 Sovereign storage, and our own transparency

- Bytes live on any Strata store: local disk, S3, GCS or Azure, in the
  customer's own account and region, encrypted at rest by the store with
  customer-managed keys (for example S3 with a KMS key). Chunks stay named
  by the hash of their plain bytes: encrypting them on the client with
  separate keys would stop them deduplicating and stop peers checking them
  by hash.
- Channel moves and signatures are recorded in the hub's append-only
  Merkle log. Auditing requires no dependence on public ledgers like
  Rekor, though Cylinder can bridge to them when desired.

### 4.6 Policy at point of use, and the inverted vulnerability index

Verification happens where the artifact is used, not only where it is
stored:

- A runner checks the job's image against policy before execution.
- A Kubernetes admission hook checks pod images against channel policies
  and assurance levels.
- `oxy` and `flow` verify what they fetch.

**The inverted file index:** Because artifacts are trees of
content-addressed files rather than opaque tarballs, Cylinder keeps an
index from each file's hash to the artifacts that contain it:

$$\text{FileHash} \longrightarrow [\text{ArtifactHash}]$$

- **Exact files, at once.** Given a known-bad file (a compromised binary,
  a leaked key, the exact `libcrypto.so.3` an advisory names), one lookup
  lists every container image, runner image and package across the fleet
  holding it, with no layers to unpack.
- **Not versions.** A patched or rebuilt library is a different file with
  a different hash, so this index does not answer "which images have a
  vulnerable version of OpenSSL". That comes from package identity: the
  SBOM each artifact carries (§4.3), indexed by package and version. The
  two together answer both questions; the file index alone answers the
  first.
- **Revocation without an outage.** When an artifact, base image or file
  is revoked in policy, admission hooks refuse new pods that use it at
  once. Pods already running are not touched by admission; a controller
  marks them, then replaces or evicts them within a grace period the
  policy sets, so one policy change cannot take production down.

### 4.7 Retention by reachability and deployment leases

What a channel, a release, a pinned pipeline or an active deployment
reaches is kept; unreferenced chunks are collected by `oxy gc`.

**Deployment leases:** Container registries often suffer from garbage
collection deleting images still running on long-lived cluster pods.
To prevent this, nodes or Kubernetes operators register periodic
**deployment leases**:

```
Attest (kind = "lease") {
  artifact [0]: ObjectRef  // artifact hash currently running
  scope    [1]: text       // e.g. "k8s:cluster-lon/ns/workload"
  expires  [2]: u64        // timestamp
}
```

Active leases act as garbage collection roots. If a workload terminates
or a cluster is decommissioned, the lease expires naturally, enabling
safe automated retention without operational risk. A lease is signed by
a key scoped to leases and to its `scope`, and has a maximum length (a
day, renewed while the workload runs), so a compromised node cannot hold
storage forever or pin artifacts outside its own scope.

### 4.8 Compatibility at the edge

- **An OCI registry façade:** `docker pull` and Kubernetes work
  unchanged. Manifests and layers are assembled on the fly from chunks.
  `docker push` is accepted and chunked via FastCDC, applying
  deduplication to legacy images.
- **A containerd snapshotter** speaking the native protocol, mounting
  file trees via `flow-casfs` for instant sub-second container starts
  and P2P delivery on Kubernetes.
- **Package façades (Cargo, npm, PyPI):** Served from the same chunk
  store. Façade endpoints synthesize package metadata (Cargo sparse
  indexes, npm packuments, PyPI simple indexes) dynamically from artifact
  manifests.
- **Upstream dependency quarantine:** When Flow builds fetch third-party
  packages from crates.io or npm, Cylinder proxies them into the
  organization's Strata store, locking the hashes into internal mirror
  channels. Upstream outages or deletions cannot break builds. Packages
  arrive compressed (`.crate` and `.tgz` are gzip), which chunks poorly,
  so the original archive is kept for clients that check its hash and its
  unpacked files are chunked as an artifact tree, where they deduplicate
  with everything else.

### 4.9 AI-native agent integration

Extending Oxygen's scoped delegation model (`ai_native_agent_workflows.md §2`),
agent keys can hold artifact-specific scopes:

```
delegates [8]: [{ key: KeyId, scope: ["changes" | "refs" | "seal" | "artifacts" | "channels"], expires: u64? }]
```

- An AI agent (e.g. Aider, Claude Code) working autonomously can be
  granted `--scope artifacts` and restricted channel rights:
  - Permitted: building artifacts and moving `agents/NAME/**` channels.
  - Forbidden: moving production channels (`app:stable`, `release/*`).
- Agents can test containerized workloads end-to-end in isolated
  channels, while production promotion remains guarded by human landing
  rules or multi-party consensus.

### 4.10 Revset query integration

Oxygen's revset language (`revset_query_language.md`) bridges source
changes and deployed artifacts:

| Expression | Selects |
|---|---|
| `artifact(CHANNEL)` | Changes contained within the artifact currently at `CHANNEL` |
| `deployed(ENV)` | Changes currently active in environment `ENV` (via active leases) |
| `unshipped()` | Equivalent to `main ~ artifact(app:stable)` (changes on `main` that have not reached production) |
| `reproduced()` | Changes whose built artifacts have 2+ independent runner receipts |
| `vulnerable(CVE)` | Changes that touch or introduced files present in an artifact flagged with `CVE` |

Running `oxy log -r 'main ~ artifact(app:stable)'` immediately shows all
changes pending deployment.

### 4.11 The UI (inside the hub)

- **Per repository, an Artifacts tab:** published artifacts, channel
  histories, and the provenance graph from commit to run, artifact, and
  deployment lease.
- **Site-wide:** chunk distribution across hubs and runners, active
  swarm transfers, deduplication savings, and the global CVE inverted index.
- **Runner images** are listed as standard artifacts with their build
  receipts and hardware architecture badges.

```mermaid
flowchart LR
  subgraph Build
    S[Source state] --> P[Pipeline]
    P --> R1[Runner 1: microvm]
    P --> R2[Runner 2: disjoint host]
    R1 -->|receipt 1| A[Artifact: tree of chunks]
    R2 -->|receipt 2| A
  end
  A -->|consensus move| C[Channel log: app:stable@42]
  A --> ST[(Strata store)]
  ST -->|seed| SW{{Swarm}}
  SW --> N1[Runner]
  SW --> N2[K8s node: containerd snapshotter]
  SW --> N3[Developer]
  N2 -->|deployment lease| DL[Active Leases: GC Roots]
  C -->|policy check| N2
  OCI[OCI façade] --- ST
  PKG[Package façades: cargo/npm/pypi] --- ST
  H[OxygenHub UI & CVE Index] --- C
  H --- A
```

### 4.12 Where it stands among the services

- **OxygenHub** owns repositories, proposals, checks, user identity,
  and the UI.
- **Jetstream** owns the fleet: queue, placement, and runner swarms. It
  fetches runner images from Cylinder by artifact hash.
- **Flow** executes pipelines and produces signed execution receipts.
- **Cylinder** owns artifacts: manifests, channel logs, provenance
  chains, distribution, package façades, and deployment leases.
- **Strata** provides the underlying content-addressed chunk storage.

Each service runs embedded in OxygenHub by default, or standalone for
distributed scale.

## 5. Better how, in one table

| | Today's best | Cylinder |
|---|---|---|
| Rebuild with one changed file | Re-push and re-pull the layer | Only that file's changed chunks |
| Dedupe | Whole identical layers | FastCDC chunks across repos, images and packages |
| Names | Mutable tags; digests nobody reads | Signed, append-only channels (`app:stable@42`) |
| Provenance | Attestations added later, if at all | The build's signed receipt is the record |
| Signing | A separate tool and keys | Native Ed25519 + ML-DSA-65 composite signatures |
| Consensus | Single-builder attestation | Multi-party consensus receipts (`reproduced(N)`) |
| Lazy start | Bolted-on snapshotter (SOCI, Nydus) | Native: file trees mounted via `flow-casfs` |
| Peer to peer | Separate mesh daemon (Dragonfly) | Built-in Jetstream swarm, TLS-pinned |
| Vulnerability search | Unpack and scan every layer | Exact files by one index lookup; versions by SBOM package index |
| Retention | Tag rules + manual GC | Reachability + active deployment leases |
| AI workflows | Uncontrolled service tokens | Scoped delegation (`artifacts`, `channels`) |
| Querying | Disconnected CI/CD dashboards | Unified revsets (`main ~ artifact(app:stable)`) |
| Sovereignty | Complex multi-tier server stack | Single binary; customer KMS and Strata bucket |

## 6. How we could build it

- **C0. Images in the hub (built 2026-10-12, oxygenhub #146).** Admin ›
  Runners › Images: every runner image, its size and architecture, where
  it is (the hub, each runner, swarm fetches in progress), the
  repositories whose jobs used it, and an image held nowhere marked.
- **C1. The service boundary and artifact model.** Implement the
  `Artifact` CBOR object type and FastCDC chunk engine in `cylinder`;
  unify runner images with artifacts.
- **C2. Channels, provenance and consensus.** Flow publishes artifacts;
  signed channel move attestations; multi-party `reproduced(N)` consensus;
  revset predicates.
- **C3. The OCI façade and containerd snapshotter.** OCI pull/push
  translation; `containerd` snapshotter over `flow-casfs`.
- **C4. Leases and the indexes.** Kubernetes deployment lease operator
  for safe GC; the inverted file index and the SBOM package index; the
  controller that replaces revoked running workloads within their grace
  period.
- **C5. Reach and package façades.** Package façades for Cargo, npm,
  and PyPI; upstream dependency quarantine mirroring.

## 7. Open questions

These are Andy's decisions. Each carries the position proposed in review,
not an answer.

- **The name.** Proposed: keep **Cylinder**. It fits the elemental
  progression: *Oxygen* (VCS) is carried in *Cylinders* (artifacts) by
  *Jetstream* (the fleet), driven by *Flow* (pipelines).
- **Where it lives.** Proposed: first in `oxygenflow` as
  `crates/oxygen-cylinder`, beside `jet` and `flow`, to share `flow-casfs`
  and the chunking code; its own repository from C3.
- **Local builds.** Proposed: allowed, at the lowest assurance level
  (`local`); production channels require more (`tier-microvm` and
  `reproduced(2)`).
- **A public catalogue and federation.** Proposed: public hubs federate
  by publishing signed Strata bundles of channels and artifacts, checked
  offline with no central intermediary.
- **Its place in the queue.** Proposed: C1 alongside Jetstream's service
  boundary, so Jetstream's image store is built on Cylinder from the
  start rather than on a temporary mechanism. Against: the core work of
  2026-10-03 (Strata's remaining stores, agents, hub polish) waits
  longer.

## Sources

- Docker Hub pull limits: [docs.docker.com, pull usage and limits](https://docs.docker.com/docker-hub/usage/pulls/); [heise, "only 10 pulls per hour"](https://heise.de/-10329047)
- GHCR billing: [GitHub community discussion #183054](https://github.com/orgs/community/discussions/183054)
- GitLab cleanup policies and garbage collection: [GitLab docs, reduce container registry storage](https://docs.gitlab.com/user/packages/container_registry/reduce_container_registry_storage/)
- Bitbucket Packages: [Atlassian blog, Bitbucket Packages](https://www.atlassian.com/blog/bitbucket/bitbucket-packages-container-registry); [GA announcement](https://www.atlassian.com/blog/bitbucket/open-beta-introducing-packages-native-container-registry-in-bitbucket-cloud)
- Pull time versus data read: [SOCI paper (Amazon), citing Slacker](https://cdn.amazon.science/84/96/51ea2e0d40fca8cfaee9328278b9/soci-paper.pdf); [Grab engineering, lazy loading](https://engineering.grab.com/docker-lazy-loading)
- zstd:chunked: [Red Hat, faster container image pulls](https://www.redhat.com/sysadmin/faster-container-image-pulls)
- Peer-to-peer meshes: [Spegel vs Kraken vs Dragonfly](https://www.pistack.xyz/posts/2026-04-28-spegel-vs-kraken-vs-dragonfly-self-hosted-p2p-container-image-distribution-guide-2026/)
- Sigstore in registries: [Sigstore in image registries: adoption and reality](https://jacar.es/en/sigstore-in-image-registries-adoption-and-reality/); [cosign and the referrers API](https://oneuptime.com/blog/post/2026-08-11-cosign-signature-storage-oci-referrers-separate-repositories/markdown)
- tj-actions tag attack: [CVE-2025-30066](https://inventivehq.com/tools/security/cve-lookup/cve-2025-30066); [Legit Security, immutable tags](https://legitsecurity.com/blog/what-are-immutable-tags-and-can-they-protect-you-from-supply-chain-attacks)
- Artifactory and Nexus: [Codefresh, Artifactory alternatives](https://codefresh.io/learn/jfrog-artifactory/top-8-artifactory-alternatives-in-2025); [PeerSpot, Artifactory improvements](https://origin.peerspot.com/questions/what-needs-improvement-with-jfrog-artifactory)