proposal: support targz-ref conversion and runtime

[imeoer]

Converter

Related PR for nydus-image builder: dragonflyoss/nydus#858
Related PR for nydus snapshotter: #245
Related Branch for acceld conversion tool: https://github.com/imeoer/acceleration-service/tree/nydus-support-ref

To implement pull tar.gz lazily, the converter workflow will be:

1. Create the nydus blob layer (application/vnd.oci.image.layer.nydus.blob.v1) with the tar-like format:

• image.blob.meta: nydus blob meta
• image.boot: nydus bootstrap

| <image.blob.meta> | <image.blob.meta> header | <image.boot> | <image.boot> header |

2. Merge all <image.boot> into a final bootstrap (application/vnd.oci.image.layer.v1.tar+gzip).
3. Acceld/Nydusify support converting all nydus blob layers and bootstrap layer into OCI artifact manifest like:

{
  "mediaType": "application/vnd.oci.artifact.manifest.v1+json",
  "artifactType": "application/vnd.nydus.ref.v1",
  "blobs": [
    {
      "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",
      "size": 123,
      "digest": "sha256:87923725d74f4bfb94c9e86d64170f7521aad8221a5de834851470ca142da630",
      "annotations": {
        "containerd.io/snapshot/nydus-blob": "true"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",
      "size": 456,
      "digest": "sha256:87923725d74f4bfb94c9e86d64170f7521aad8221a5de834851470ca142da630",
      "annotations": {
        "containerd.io/snapshot/nydus-blob": "true"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:aec98c9e3dce739877b8f5fe1cddd339de1db2b36c20995d76f6265056dbdb08",
      "size": 789,
      "annotations": {
        "containerd.io/snapshot/nydus-bootstrap": "true"
      }
    }
  ],
  "subject": {
    "mediaType": "application/vnd.oci.image.manifest.v1+json",
    "size": 123,
    "digest": "sha256:cc06a2839488b8bd2a2b99dcdc03d5cfd818eed72ad08ef3cc197aac64c0d0a0"
  },
  "annotations": {
    "containerd.io/snapshot/nydus-fs-version": "6"
  }
}

• The subject field reference the original OCI image, https://github.com/opencontainers/image-spec/blob/main/artifact.md.
• Perhaps the legacy registry does not support OCI artifact, we need to make it works by using a compatible image manifest format like:

{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.oci.image.config.v1+json",
    "digest": "sha256:e8366802635d22a771339e18ef406d117b5ee5fa86b2373bd99d188c3cba5c5a",
    "size": 16297
  },
  "layers": [
    {
      "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",
      "digest": "sha256:0269b9edd97b7088428e7913fd635ffa6fda08af373a6aa86edc9f181572c8d7",
      "size": 3179520,
      "annotations": {
        "containerd.io/snapshot/nydus-blob": "true",
        "containerd.io/snapshot/nydus-ref": "true"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.layer.nydus.blob.v1",
      "digest": "sha256:748cba355624ad8d5ebed6e47af9bd88bf64061e326e0c5fc7f43ddcf4ccb49d",
      "size": 29696,
      "annotations": {
        "containerd.io/snapshot/nydus-blob": "true",
        "containerd.io/snapshot/nydus-ref": "true"
      }
    },
    {
      "mediaType": "application/vnd.oci.image.layer.v1.tar+gzip",
      "digest": "sha256:787addb807451fbad912aac460e37edf204c2364b81ece348b4a65f6a1cbed02",
      "size": 1121814,
      "annotations": {
        "containerd.io/snapshot/nydus-bootstrap": "true",
        "containerd.io/snapshot/nydus-fs-version": "6",
        "containerd.io/snapshot/nydus-ref": "true"
      }
    }
  ]
}

Runtime

1. Range fetch <image.blob.meta> data from the remote registry layer by layer and then write to the blob cache directory.
2. Pull the final bootstrap data (in tar.gz format) and unpack it to the snapshot directory.
3. Boot nydusd and mount the final bootstrap.

[jiangliu]

Could we also add support for "blob.digest" files too?
Currently a RAFS v5 chunk information array is embedded in the bootstrap, which is not very efficient. It would be better to embed a "blob.digest" array into each data blob.

[imeoer]

Performance test:

image: node:19.0
workload: node -v
registry network: 3MB/s

Version 1:

image type	image size	acceld conversion	acceld push	nerdctl run
OCI v1	353.05MB	-	-	126s
Nydus v6	337.94MB	29s	1m58s	17s
Nydus v6 (Zran)	37.30MB	11s	12s	50s

Version 2 (compress blob meta & download blob meta concurrently):

image type	image size	acceld conversion	acceld push	nerdctl run	read data	read count
OCI v1	353.05MB	-	-	126s	353.05MB	-
Nydus v6	337.94MB	29s	1m58s	11s	21.18MB	39
Nydus v6 (Zran)	18.98MB	11s	12s	15s	28.78MB	49

Version 3 (compress blob meta, compress bootstrap, download blob meta concurrently):

image type	image size	acceld conversion	acceld push	nerdctl run	read data	read count
Nydus v6 (Zran)	14MB	11s	12s	15s	28.78MB	49