Partial mirror aka download blobs on demand

[rgl]

Is there a way to configure zot mirror to only download blobs on demand?

This could help in:

1. Many images are multi-platform, but we end-up only using small (in fact, for my amd64 cluster, its only one image) subset of the supported image platforms.
2. Cause less load on upstream registry.
3. Since its on-demand, the image would be available faster.

For example, many images are multi-platform, it seems, when zot 2.0.1 mirrors a image, it first downloads all the blobs into the .sync sub-directory, and only when everything is download, it "promotes" the image into a visible image. Downloading everything seems to cause too much load in upstream registry. and things starts to stall/fail (e.g. today was the first time I've noticed this, while it failed (and never worked) while downloading the quay.io/argoproj/argocd:v2.10.3 image from https://quay.io/repository/argoproj/argocd/manifest/sha256:18a1e1b4006c387a6a2037859004be3e6330bfb6b48809d6a342c6daa76a2e9b).

FWIW, Here's I'm configuring zot 2.0.1:

extensions:
  ui:
    enable: true
  metrics:
    enable: true
    prometheus:
      path: /metrics
  search:
    enable: true
    cve:
      updateInterval: 2h
  sync:
    enable: true
    registries:
      - urls:
          - https://quay.io
        content:
          - prefix: "**"
            destination: /mirror/quay.io
        onDemand: true
        tlsVerify: true

[rchincha]

@rgl so maybe add an additional param called platform: under content:
This could mitigate it somewhat, however with large images an ever-present problem though.
You raise a good point that failures cause a worse flood than the good path.

[rgl]

@rchincha where is platform: documented? I'm not finding it at https://zotregistry.dev/v2.0.2/articles/mirroring/.

[rchincha]

Sorry, not present yet. We would have to build this support in. This is just a proposed solution if it makes sense.

[rgl]

I would rather prefer that zot downloads everything on-demand, and only the specific parts that are requested by a client, instead of buffering/staging the entire logical image(s).

The reasoning being, the client knowns which parts/blobs it requires. For example, the client typically only downloads images for its own platform; so those would be the only ones that zot needs to actually mirror. If we really want to preload zot, we could use a tool like crane/skopeo/regctl.

[andaaron]

Normally when an image is pushed to the the registry using the REST API in the spec the manifest/index is only accepted if all references in the descriptors (blobs) are already present in the registry, to avoid inconsistencies.

Allowing indexes to be present with some invalid references (for architectures the user doesn't want) would lead to inconsistencies. If the rest API doesn't allow it, why would sync allow it? This would also mean modifications in the storage logic to handle errors on these missing blobs. It may also need changes on the logic of GC-in invalid images.

Note we cannot have zot managing/modifying the image index to keep only valid references present in zot, as the clients downloading this index would not know there are other architectures available and what digests they have. Not to mention the digest of the index itself would change so any references to it by other artifacts/images would be invalid.

---

We could download individual manifests for different platforms, without the index, but then they would have to have distinguishable tags per platform, which clients would need to know in order to access this content. Would this work?