# Cylinder: artifacts, the Oxygen way

Status: Draft 0.2 (proposal revision 2), 2026-10-12. Incorporates review of
object model, deployment leases, inverted vulnerability indexing,
multi-party consensus, agent scopes, and revset queries. Not built.
"Cylinder" is a working name (the artifact-repository idea parked on
2026-10-02); nothing here is decided until Andy says so.

**Cylinder** keeps and moves everything a build produces: runner images,
container images, packages, binaries, SBOMs and reports. It would be a
service of its own, embedded in OxygenHub by default and shown through
the hub's UI, the same pattern as Jetstream. It is meant to be better
than what exists, not compatible with it; compatibility is an edge we add
for adoption, never the core.

## 1. What exists today

| | Strengths | Weaknesses |
|---|---|---|
| **Docker Hub** | The default namespace; huge public catalogue; official images; scanning (Scout). | Pull limits since April 2025: 10 an hour anonymous, 100 an hour on a free account. SaaS only. Tags are mutable. Signatures and provenance are separate artifacts; referrers are still in beta. Nothing links an image to the source it came from. |
| **GitHub** (GHCR, Packages) | Next to the code and Actions; free container storage for now; the best Sigstore integration; artifact attestations. | Storage billing for private images deferred, not settled. One silo per package format. Auth by personal access tokens. Tags are mutable: the tj-actions attack (March 2025, CVE-2025-30066) moved every tag of a popular action to a malicious commit at once. No self-hosting outside Enterprise Server. |
| **GitLab** | A registry per project; many package formats; a dependency proxy; self-hostable, so it can be sovereign. | Cleanup policies delete tags only; layers stay until an administrator runs garbage collection. The registry is a separate service to run and scale. Signing means bringing cosign yourself. |
| **Bitbucket** (Packages) | GA since December 2025; permissions inherited from the repository; garbage collection runs by itself; Maven and npm since April 2026. | Young; Cloud only; few formats; no provenance story. |
| **Artifactory, Nexus** | Every format; proxies and caches; enterprise features. | Costly; heavy JVM operations; slow UIs. A second system beside the forge, with its own identities and permissions. Provenance and scanning are add-on products. |
| **Harbor, Quay** | Self-hosted OCI with signing (cosign, Notation), replication and scanning. | Many components to run. A central bottleneck: peer-to-peer means adding Dragonfly. |
| **ECR, Artifact Registry, ACR** | Cloud IAM; regional; scanning; lazy pulling on AWS (SOCI). | Lock-in; egress costs; not sovereign. |

And the bolt-ons each of them needs:

- **Distribution:** Dragonfly, Kraken or Spegel, run beside the registry.
- **Lazy pulling:** eStargz, SOCI, Nydus or zstd:chunked, each a format
  plus a snapshotter. It matters: pulling took 76% of container start time
  while only 6.4% of the pulled data was read (Slacker, FAST '16).
- **Signing and provenance:** cosign and Rekor, SLSA generators,
  in-toto, Notation, attached afterwards through the OCI referrers API.
- **Retention:** cleanup policies plus a separate garbage collection.

## 2. The pattern behind the weaknesses

1. **The unit is the tarball layer.** One changed file re-pushes and
   re-pulls a whole layer; layers share nothing below their own size; and a
   layer has two digests (compressed and not), so tools disagree on what
   it is called.
2. **Names are mutable pointers.** A tag moves silently and leaves no
   history. Pinning by digest is safe but unreadable, so most people
   don't.
3. **Trust is bolted on afterwards.** The registry stores bytes; a
   separate tool signs them, a third says how they were built, and each
   brings its own keys and identities. Nothing stops an unsigned image
   from being pushed, and most pulls never verify anything.
4. **Distribution is someone else's problem.** Every registry is a
   central server; peer-to-peer and lazy loading are extra products.
5. **The forge and the registry are strangers.** Permissions, identity,
   retention and "which commit is this?" are kept twice and drift apart.

## 3. What Oxygen already has

Every piece Cylinder needs exists in some form:

- **Content-defined chunks with BLAKE3** (FastCDC, 32 to 256 KiB), shared
  by repositories, release assets, and runner images: deduplication below
  file level, across everything.
- **Hybrid post-quantum signatures** (Ed25519 with ML-DSA-65), identities
  with delegation chains, and rotation recorded in the repository.
- **Signed Flow receipts:** a runner signs what it built, from which
  state, with which pipeline and environment, producing which outputs.
  That is provenance by construction, not a statement added later.
- **flow-casfs:** a content-addressed tree mounted lazily over FUSE or
  virtio-fs. Lazy pulling is native, not a format bolted on.
- **The Jetstream swarm:** peer-to-peer chunks, rarest first, TLS pinned
  by the hub, tickets that only the hub signs.
- **Strata:** storage on disk, S3, GCS or Azure, with no server in between,
  sovereign keys and client-side hash verification.
- **OxygenHub:** one identity and permission model, checks, proposals and
  reports.
- **Scoped delegation:** key scopes (`changes`, `refs`, `seal`) allowing
  principals and AI agents to act within strict cryptographic boundaries.

## 4. The design

### 4.1 One kind of thing: a tree of chunks

An **artifact** is an immutable, content-addressed object in the Oxygen
store naming a file tree:

```
Artifact {
  version     [0]: 1
  kind        [1]: "container" | "runner-image" | "package" | "archive" | "report"
  arch        [2]: text | null              // "x86_64", "riscv64", "aarch64"
  os          [3]: text | null              // "linux", "darwin", "none"
  tree        [4]: ObjectRef                // DirTree blob: paths, modes, blobs, sizes
  entrypoint  [5]: [text] | null            // default exec argv
  env         [6]: map<text, text> | null   // environment variables
  labels      [7]: map<text, text> | null   // metadata / OCI labels
  total_size  [8]: u64                      // uncompressed bytes
  receipt     [9]: ObjectRef | null         // Receipt object that produced it
}
```

Files in the `DirTree` are ordinary Oxygen `blob` objects composed of
FastCDC BLAKE3 `chunk`s. So:

- An image rebuilt with one changed file moves only that file's changed
  chunks: no layers.
- A release tarball, an npm package, and the container image containing
  the same binary share its chunks verbatim. If a 100 MB model or library
  is committed in the repo and then packaged, zero new chunk bytes are
  written to Strata.
- A single file can be fetched independently: a container or job starts
  after fetching only the files it reads.
- Every artifact has exactly one name for its content: the BLAKE3 of its
  canonical CBOR encoding.
- **MicroVMs and virtio-fs:** Jetstream microVM runners mount an artifact's
  tree directly via virtio-fs over `flow-casfs`, eliminating ext4 block
  images, loop devices, and cold disk allocation. Where a raw block device
  is required by legacy hardware or external hypervisors, an ext4 block
  map is synthesized on the fly from the chunk tree without duplicating
  bytes.

### 4.2 Names are signed histories, not pointers

A **channel** (`app:stable`, `runner-image:rust`, `pkg:cargo/my-crate`) is an
append-only log of signed moves, recorded as an attestation of kind
`channel`:

```
Attest (kind = "channel") {
  channel   [0]: text             // e.g. "app:stable"
  old_art   [1]: ObjectRef | null // previous artifact hash (CAS check)
  new_art   [2]: ObjectRef        // target artifact hash
  serial    [3]: u64              // monotonically increasing
  receipt   [4]: ObjectRef | null // Receipt that built new_art
  reason    [5]: text             // commit message or promotion note
}
```

A move needs policy: signed by an authorized key under the repository's
trust policy (`spec.md §14.5`), from an approved bookmark, after
required checks or human approvals pass. Clients follow a channel and
verify each step:

- A tag cannot move silently. The tj-actions attack needed one stolen
  token; here it needs a move signed by an authorized key, and the move
  is logged in the append-only history.
- Rolling back is a move to an earlier artifact, recorded and attested
  like any other move.
- Pinning stays readable and tamper-evident: `app:stable@42` names the
  42nd entry of an audited log.
- On Strata stores, updates use compare-and-swap on the channel record
  (`old_art` and `serial`), matching how bookmark `refs` operate.

### 4.3 Provenance by construction and multi-party consensus

An artifact is published by a Flow job, and its record is the unbroken
chain:

> source state → pipeline → runner (identity, tier, image) → signed
> receipt → artifact

Each link is a hash or signature verifiable offline. Beyond what
attestations give today:

- **Assurance levels:** a MicroVM job on a registered runner is worth
  more than a syscall-tier job, which is worth more than a signed local
  build. Trust policies can enforce minimum assurance floors for
  sensitive channels.
- **Multi-party consensus (`reproduced`):** when multiple independent
  runners on disjoint machines or architectures build the same artifact
  hash from the same state, each signs a receipt. A policy rule can
  require threshold consensus:
  `require: "reproduced(2)"` demands two independent runner receipts
  before an artifact can be promoted to a production channel.
- **Upstream lineage:** an artifact built from other artifacts (a base
  image, a package) names their hashes in its receipt inputs, so the chain
  reaches back transitively to their sources.

### 4.4 Distribution is built in

- **The store is the seed; the swarm delivers.** Jetstream's swarm,
  generalized: any runner, cluster node or developer machine in a site
  shares the chunks it has, over TLS pinned by the hub, admitted by
  hub-signed tickets.
- **Lazy by default:** artifacts are mounted via `flow-casfs` and fetched
  on demand.
- **Learned prefetch:** each run records which chunks were accessed
  during its initial execution window; subsequent pulls request those
  chunks first.
- **Sites and air gaps:** a site syncs chunks and channel logs from
  another hub as a signed bundle, so a disconnected network gets
  verified artifacts with no live connection.

### 4.5 Sovereign storage, and our own transparency

- Bytes live on any Strata store: local disk, S3, GCS or Azure, in the
  customer's own account and region, encrypted with customer-managed KMS.
- Channel moves and signatures are recorded in the hub's append-only
  Merkle log. Auditing requires no dependence on public ledgers like
  Rekor, though Cylinder can bridge to them when desired.

### 4.6 Policy at point of use, and the inverted vulnerability index

Verification happens where the artifact is used, not only where it is
stored:

- A runner checks the job's image against policy before execution.
- A Kubernetes admission hook checks pod images against channel policies
  and assurance levels.
- `oxy` and `flow` verify what they fetch.

**The global inverted vulnerability index:** Because artifacts are trees
of content-addressed files and chunks rather than opaque tarballs,
Cylinder maintains an inverted index from chunk/file hash to artifacts:

$$\text{FileHash} \longrightarrow [\text{ArtifactHash}]$$

When a security advisory or zero-day is disclosed for a binary or
library (e.g., a vulnerable `libcrypto.so.3` or Go binary hash):
- Cylinder performs an immediate $O(1)$ lookup to identify every
  container image, runner image, and package across the fleet containing
  that file.
- There is no need to unpack gigabytes of tarball layers across thousands
  of images to discover exposure.
- **Continuous revocation:** If an upstream library or base image is
  revoked in policy, point-of-use admission hooks immediately block
  downstream pods from running any artifact containing that chunk,
  eliminating the gap between vulnerability discovery and redeployment.

### 4.7 Retention by reachability and deployment leases

What a channel, a release, a pinned pipeline or an active deployment
reaches is kept; unreferenced chunks are collected by `oxy gc`.

**Deployment leases:** Container registries often suffer from garbage
collection deleting images still running on long-lived cluster pods.
To prevent this, nodes or Kubernetes operators register periodic
**deployment leases**:

```
Attest (kind = "lease") {
  artifact [0]: ObjectRef  // artifact hash currently running
  scope    [1]: text       // e.g. "k8s:cluster-lon/ns/workload"
  expires  [2]: u64        // timestamp
}
```

Active leases act as garbage collection roots. If a workload terminates
or a cluster is decommissioned, the lease expires naturally, enabling
safe automated retention without operational risk.

### 4.8 Compatibility at the edge

- **An OCI registry façade:** `docker pull` and Kubernetes work
  unchanged. Manifests and layers are assembled on the fly from chunks.
  `docker push` is accepted and chunked via FastCDC, applying
  deduplication to legacy images.
- **A containerd snapshotter** speaking the native protocol, mounting
  file trees via `flow-casfs` for instant sub-second container starts
  and P2P delivery on Kubernetes.
- **Package façades (Cargo, npm, PyPI):** Served from the same chunk
  store. Façade endpoints synthesize package metadata (Cargo sparse
  indexes, npm packuments, PyPI simple indexes) dynamically from artifact
  manifests.
- **Upstream dependency quarantine:** When Flow builds fetch third-party
  packages from crates.io or npm, Cylinder proxies and chunks them into
  the organization's Strata store, locking the hashes into internal
  mirror channels. Upstream outages or deletions cannot break builds.

### 4.9 AI-native agent integration

Extending Oxygen's scoped delegation model (`ai_native_agent_workflows.md §2`),
agent keys can hold artifact-specific scopes:

```
delegates [8]: [{ key: KeyId, scope: ["changes" | "refs" | "seal" | "artifacts" | "channels"], expires: u64? }]
```

- An AI agent (e.g. Aider, Claude Code) working autonomously can be
  granted `--scope artifacts` and restricted channel rights:
  - Permitted: building artifacts and moving `agents/NAME/**` channels.
  - Forbidden: moving production channels (`app:stable`, `release/*`).
- Agents can test containerized workloads end-to-end in isolated
  channels, while production promotion remains guarded by human landing
  rules or multi-party consensus.

### 4.10 Revset query integration

Oxygen's revset language (`revset_query_language.md`) bridges source
changes and deployed artifacts:

| Expression | Selects |
|---|---|
| `artifact(CHANNEL)` | Changes contained within the artifact currently at `CHANNEL` |
| `deployed(ENV)` | Changes currently active in environment `ENV` (via active leases) |
| `unshipped()` | Equivalent to `main ~ artifact(app:stable)` (changes on `main` that have not reached production) |
| `reproduced()` | Changes whose built artifacts have 2+ independent runner receipts |
| `vulnerable(CVE)` | Changes that touch or introduced files present in an artifact flagged with `CVE` |

Running `oxy log -r 'main ~ artifact(app:stable)'` immediately shows all
changes pending deployment.

### 4.11 The UI (inside the hub)

- **Per repository, an Artifacts tab:** published artifacts, channel
  histories, and the provenance graph from commit to run, artifact, and
  deployment lease.
- **Site-wide:** chunk distribution across hubs and runners, active
  swarm transfers, deduplication savings, and the global CVE inverted index.
- **Runner images** are listed as standard artifacts with their build
  receipts and hardware architecture badges.

```mermaid
flowchart LR
  subgraph Build
    S[Source state] --> P[Pipeline]
    P --> R1[Runner 1: microvm]
    P --> R2[Runner 2: disjoint host]
    R1 -->|receipt 1| A[Artifact: tree of chunks]
    R2 -->|receipt 2| A
  end
  A -->|consensus move| C[Channel log: app:stable@42]
  A --> ST[(Strata store)]
  ST -->|seed| SW{{Swarm}}
  SW --> N1[Runner]
  SW --> N2[K8s node: containerd snapshotter]
  SW --> N3[Developer]
  N2 -->|deployment lease| DL[Active Leases: GC Roots]
  C -->|policy check| N2
  OCI[OCI façade] --- ST
  PKG[Package façades: cargo/npm/pypi] --- ST
  H[OxygenHub UI & CVE Index] --- C
  H --- A
```

### 4.12 Where it stands among the services

- **OxygenHub** owns repositories, proposals, checks, user identity,
  and the UI.
- **Jetstream** owns the fleet: queue, placement, and runner swarms. It
  fetches runner images from Cylinder by artifact hash.
- **Flow** executes pipelines and produces signed execution receipts.
- **Cylinder** owns artifacts: manifests, channel logs, provenance
  chains, distribution, package façades, and deployment leases.
- **Strata** provides the underlying content-addressed chunk storage.

Each service runs embedded in OxygenHub by default, or standalone for
distributed scale.

## 5. Better how, in one table

| | Today's best | Cylinder |
|---|---|---|
| Rebuild with one changed file | Re-push and re-pull the layer | Only that file's changed chunks |
| Dedupe | Whole identical layers | FastCDC chunks across repos, images and packages |
| Names | Mutable tags; digests nobody reads | Signed, append-only channels (`app:stable@42`) |
| Provenance | Attestations added later, if at all | The build's signed receipt is the record |
| Signing | A separate tool and keys | Native Ed25519 + ML-DSA-65 composite signatures |
| Consensus | Single-builder attestation | Multi-party consensus receipts (`reproduced(N)`) |
| Lazy start | Bolted-on snapshotter (SOCI, Nydus) | Native: file trees mounted via `flow-casfs` |
| Peer to peer | Separate mesh daemon (Dragonfly) | Built-in Jetstream swarm, TLS-pinned |
| Vulnerability search | Unpack and scan every layer ($O(N)$) | $O(1)$ inverted index by file and chunk hash |
| Retention | Tag rules + manual GC | Reachability + active deployment leases |
| AI workflows | Uncontrolled service tokens | Scoped delegation (`artifacts`, `channels`) |
| Querying | Disconnected CI/CD dashboards | Unified revsets (`main ~ artifact(app:stable)`) |
| Sovereignty | Complex multi-tier server stack | Single binary; customer KMS and Strata bucket |

## 6. How we could build it

- **C0. Images in the hub, now.** A Flow › Images page: each image, where
  it is (hub, runners, swarm activity) and what pins it.
- **C1. The service boundary and artifact model.** Implement the
  `Artifact` CBOR object type and FastCDC chunk engine in `cylinder`;
  unify runner images with artifacts.
  *Started 2026-10-28 (oxygen#108):* `crates/cylinder` has the
  `Artifact` and `DirTree` encodings of §4.1 (canonical CBOR; a name is
  the BLAKE3 of the encoding; `env` and `labels` as sorted pairs, since
  the profile's maps take integer keys), packing a directory into the
  repository's own blobs and chunks, verifying and unpacking, and the
  roots garbage collection keeps; `oxy artifact pack | list | show |
  unpack | verify`. The artifact is an object type of its own
  (`artifact`, in oxy-core's `artifact` module, checked by `oxy fsck`),
  named by its object hash like every object; its tree is a blob, and
  `.oxy/artifacts/NAME` records the artifacts a repository keeps. Next:
  runner images as artifacts, then moving them between repositories (C2).
- **C2. Channels, provenance and consensus.** Flow publishes artifacts;
  signed channel move attestations; multi-party `reproduced(N)` consensus;
  revset predicates.
- **C3. The OCI façade and containerd snapshotter.** OCI pull/push
  translation; `containerd` snapshotter over `flow-casfs`.
- **C4. Leases and the inverted index.** Kubernetes deployment lease
  operator for safe GC; global inverted chunk index for instantaneous
  CVE discovery.
- **C5. Reach and package façades.** Package façades for Cargo, npm,
  and PyPI; upstream dependency quarantine mirroring.

## 7. Open questions

- **The name:** Keep **Cylinder**. It fits the elemental progression:
  *Oxygen* (VCS) is transported in *Cylinders* (artifacts) via
  *Jetstream* (fleet transport) driven by *Flow* (pipelines).
- **Where it lives:** Initially in `oxygenflow` / `crates/oxygen-cylinder`
  alongside `jet` and `flow` to share `flow-casfs` and chunking primitives,
  splitting into an independent repository at C3.
- **Local builds:** Allowed, but marked with lower assurance level
  (`local`). Production channel rules require higher assurance levels
  (`tier-microvm` + `reproduced(2)`).
- **A public catalogue / Federation:** Public hubs federate by publishing
  signed Strata bundles of channels and artifacts, verified offline
  without centralized intermediaries.
- **Place in the queue:** C0/C1 should proceed alongside Jetstream's
  service boundary work, ensuring Jetstream's image store is built
  directly on Cylinder rather than creating a temporary mechanism.

## Sources

- Docker Hub pull limits: [docs.docker.com, pull usage and limits](https://docs.docker.com/docker-hub/usage/pulls/); [heise, "only 10 pulls per hour"](https://heise.de/-10329047)
- GHCR billing: [GitHub community discussion #183054](https://github.com/orgs/community/discussions/183054)
- GitLab cleanup policies and garbage collection: [GitLab docs, reduce container registry storage](https://docs.gitlab.com/user/packages/container_registry/reduce_container_registry_storage/)
- Bitbucket Packages: [Atlassian blog, Bitbucket Packages](https://www.atlassian.com/blog/bitbucket/bitbucket-packages-container-registry); [GA announcement](https://www.atlassian.com/blog/bitbucket/open-beta-introducing-packages-native-container-registry-in-bitbucket-cloud)
- Pull time versus data read: [SOCI paper (Amazon), citing Slacker](https://cdn.amazon.science/84/96/51ea2e0d40fca8cfaee9328278b9/soci-paper.pdf); [Grab engineering, lazy loading](https://engineering.grab.com/docker-lazy-loading)
- zstd:chunked: [Red Hat, faster container image pulls](https://www.redhat.com/sysadmin/faster-container-image-pulls)
- Peer-to-peer meshes: [Spegel vs Kraken vs Dragonfly](https://www.pistack.xyz/posts/2026-04-28-spegel-vs-kraken-vs-dragonfly-self-hosted-p2p-container-image-distribution-guide-2026/)
- Sigstore in registries: [Sigstore in image registries: adoption and reality](https://jacar.es/en/sigstore-in-image-registries-adoption-and-reality/); [cosign and the referrers API](https://oneuptime.com/blog/post/2026-08-11-cosign-signature-storage-oci-referrers-separate-repositories/markdown)
- tj-actions tag attack: [CVE-2025-30066](https://inventivehq.com/tools/security/cve-lookup/cve-2025-30066); [Legit Security, immutable tags](https://legitsecurity.com/blog/what-are-immutable-tags-and-can-they-protect-you-from-supply-chain-attacks)
- Artifactory and Nexus: [Codefresh, Artifactory alternatives](https://codefresh.io/learn/jfrog-artifactory/top-8-artifactory-alternatives-in-2025); [PeerSpot, Artifactory improvements](https://origin.peerspot.com/questions/what-needs-improvement-with-jfrog-artifactory)