Support referrers responses in the Image Layout

[sudo-bmitch]

This defines how referrers should be implemented with the Image Layout. Presently, they can be stored in the Layout using the fallback tag. That has issues for tooling that doesn't want to pollute the tag namespace. The proposed solution uses two annotations. One annotation is on the descriptors of the index.json file, org.opencontainers.image.referrer.subject, which indicates the descriptor references a referrers response for the specified subject digest. The other annotation, org.opencontainers.image.referrer.convert, set on the top level manifest of the index.json file, is used as a flag for tooling to know the file has already had any content converted over to use the org.opencontainers.image.referrer.subject annotation, and further conversions may be skipped.

I've tested these annotations in a registry that is based on a filesystem of OCI Image Layout content.

[shizhMSFT]

oras-go (versions >= v2.0.0) and oras (versions >= 1.0.0) support referrers natively for the image layout without the fallback tag. Basically, oras puts the referrer manifests in the index.json file without specifying a tag so that oras can traverse to that node without scanning all blobs. The benefits of this methods are

• Clean: fallback tag is not required.
• Backward compatible: backfill is not required. Backward compatible with existing image layout and tools.
• Organized: index.json is de facto a manifest list where some of the manifests have a tag.
• Performant: scanning the blobs folder is not required.

Here's a sample output using oras 1.2.0-beta.1.

$ echo foo > foo.txt
$ oras push --oci-layout hello:foo foo.txt
✓ Uploaded  foo.txt                                                                                4/4  B 100.00%    2ms
  └─ sha256:b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c
✓ Uploaded  application/vnd.oci.empty.v1+json                                                      2/2  B 100.00%    2ms
  └─ sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
✓ Uploaded  application/vnd.oci.image.manifest.v1+json                                         585/585  B 100.00%  992µs
  └─ sha256:7f1216714a7ecbe4525a55ec07c4bb1d06d32cc862f6e4c6598395aeb89e1be8
Pushed [oci-layout] hello:foo
Digest: sha256:7f1216714a7ecbe4525a55ec07c4bb1d06d32cc862f6e4c6598395aeb89e1be8
$ echo bar > bar.txt
$ oras attach --oci-layout --artifact-type application/bar hello:foo bar.txt
✓ Exists    application/vnd.oci.empty.v1+json                                                      2/2  B 100.00%     0s
  └─ sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
✓ Uploaded  bar.txt                                                                                4/4  B 100.00%  826µs
  └─ sha256:7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730
✓ Uploaded  application/vnd.oci.image.manifest.v1+json                                         897/897  B 100.00%  444µs
  └─ sha256:0570b97df7a9bf4bcef65e724cc60d663f09b7650d5010d03a3ed4571d385e81
Attached to [oci-layout] hello@sha256:7f1216714a7ecbe4525a55ec07c4bb1d06d32cc862f6e4c6598395aeb89e1be8
Digest: sha256:0570b97df7a9bf4bcef65e724cc60d663f09b7650d5010d03a3ed4571d385e81
$ oras discover --oci-layout hello:foo
Discovered 1 artifact referencing foo
Digest: sha256:7f1216714a7ecbe4525a55ec07c4bb1d06d32cc862f6e4c6598395aeb89e1be8

Artifact Type     Digest
application/bar   sha256:0570b97df7a9bf4bcef65e724cc60d663f09b7650d5010d03a3ed4571d385e81
$ tree hello
hello
├── blobs
│   └── sha256
│       ├── 0570b97df7a9bf4bcef65e724cc60d663f09b7650d5010d03a3ed4571d385e81
│       ├── 44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a
│       ├── 7d865e959b2466918c9863afca942d0fb89d7c9ac0c99bafc3749504ded97730
│       ├── 7f1216714a7ecbe4525a55ec07c4bb1d06d32cc862f6e4c6598395aeb89e1be8
│       └── b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c
├── index.json
├── ingest
└── oci-layout

3 directories, 7 files

The resulted hello/index.json after formating is

{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:7f1216714a7ecbe4525a55ec07c4bb1d06d32cc862f6e4c6598395aeb89e1be8",
      "size": 585,
      "annotations": {
        "org.opencontainers.image.created": "2024-02-21T09:33:26Z",
        "org.opencontainers.image.ref.name": "foo"
      },
      "artifactType": "application/vnd.unknown.artifact.v1"
    },
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:0570b97df7a9bf4bcef65e724cc60d663f09b7650d5010d03a3ed4571d385e81",
      "size": 897,
      "annotations": {
        "org.opencontainers.image.created": "2024-02-21T09:34:11Z"
      },
      "artifactType": "application/bar"
    }
  ]
}

/cc @sajayantony

[sudo-bmitch]

Performant: scanning the blobs folder is not required.

Wouldn't oras discover need to read the manifest of every untagged index entry from the blob store? This proposal would eliminate that.

[shizhMSFT]

Wouldn't oras discover need to read the manifest of every untagged index entry from the blob store?

Yes, oras does traverse all manifests to build a graph in the memory. BTW, reading from OCI image layout usually happens on local disk (with system-level file cache). Currently, we don't observe any performance issues yet.

[tianon]

We discussed this on the OCI call quite a few weeks ago now, and I'm going to do my best to summarize some of the concerns several of us on that call had:

• our gut reaction to the added subject annotation was that it must be for objects which have a subject field (to "pull up" their subject)
  • this was on the assumption that the most common way to bundle "subject-having" objects in a layout would be via something akin to docker save which is then used to move them around
• this PR is concretely about adding the actual "referrers API" response directly as an index object within the OCI layout (and adding these annotations to that)
  • Brandon is doing something more ambitious (a registry serving from oci-layout), and makes the argument that clients need to generate this object either way
  • on the flip side, clients do need to generate this, but always in a "merge with existing object from registry" fashion

I'm not entirely convinced we should add this (but I'm also not feeling super strongly that it shouldn't) -- perhaps officially an even softer NACK than my normal soft NACK.

[sudo-bmitch]

The inspiration for the annotation pointing to the referrers response, rather than each entry in the response, was to consolidate client workflows. It allows the OCI Layout to be treated the same as a distribution-spec 1.0 registry. By having the annotation on every entry, the client would need a separate workflow for handling an OCI Layout.

Some of this gets into how clients are treating the OCI Layout, whether it's a static transport for a batch process (export/import), or if it's treated like a repository. I think there will always be the first use case. The latter use case opens up a lot of efficiencies and security opportunities for things like CI pipelines that want to review and possibly add metadata before finally uploading it. For those latter use cases, the more an OCI Layout can look like a repository to the tooling, the easier the tooling is to write.