images built with docker v25 or later fail to push

[kalensk]

Describe the bug

Images built with docker version 25 or later fail to push as shown in the below error.

For example, my task is pulling the latest docker in docker image (dind) to build my docker image. Using the latest dind tag which is docker version 26 fails, but using version 24-dind succeeds.

(sorry the formatting is misbehaving here...ugh)

 - task: build_image
   privileged: true
   config:
     platform: linux
     image_resource:
       type: registry-image
       source:
         repository: docker
         tag: dind  # using tag "24-dind" succeeds

INFO[0000] pushing tag(s) dev                           
2024/03/28 01:42:37 existing blob: sha256:a7d483a98052b3c5108d6ed6d38caded5d2a787564ba0be1de08a5434f32a9db
2024/03/28 01:42:38 pushed blob: sha256:24c21885ea2bbe64a8a0b21901a61520772c747e98fb013d2cedf5bc23eecd4b
2024/03/28 01:42:39 pushed blob: sha256:617e845ae9918ce7686edff62e4c892c25ac172f75d75c76baef9abcbc8db052
2024/03/28 01:42:39 pushed blob: sha256:1249c494fb021137c410d273b1fbc3cfccf2daa4fb0c13ad223b31e052837235
2024/03/28 01:42:40 pushed blob: sha256:7ade3cec065838642366736d7d269cbefa2fea5eb839621bff0c520e00a18058
2024/03/28 01:42:41 pushed blob: sha256:052ad86d9dbd8a53393b7700d75dbd25764e564663bcc1a763341e3ac8b1c2af

ERRO[0006] pushing image failed: pushing tag(s): PUT https://gcr.io/v2/my-project/image/manifests/dev: multiple errors returned: BLOB_UNKNOWN: Manifest "sha256:b3322b3a5550897191dbfec1f4a7a256f54f8a6874a6de1f3a1bc3da96e1f4d1" blob "sha256:31a3635cf85a9285b9aebc8e97a2e4021fed41b53eecd55e614a89a03fa1daa4" unknown to registry.; BLOB_UNKNOWN: Manifest "sha256:b3322b3a5550897191dbfec1f4a7a256f54f8a6874a6de1f3a1bc3da96e1f4d1" blob "sha256:3951720cdd18c039b98fd52d43853e3caf4c365d2aaca115c7f3a24f6ea6c07a" unknown to registry.; BLOB_UNKNOWN: Manifest "sha256:b3322b3a5550897191dbfec1f4a7a256f54f8a6874a6de1f3a1bc3da96e1f4d1" blob "sha256:4ae1d13ade680629cf4fbf0c7be28bd2442ab0e670883b7112e59f2aa0f58077" unknown to registry.; BLOB_UNKNOWN: Manifest "sha256:b3322b3a5550897191dbfec1f4a7a256f54f8a6874a6de1f3a1bc3da96e1f4d1" blob "sha256:9cbd3f40c29cbdc8801087adff6e90a065aab4fc1517c962c480ef8b8bebc909" unknown to registry.; BLOB_UNKNOWN: Manifest "sha256:b3322b3a5550897191dbfec1f4a7a256f54f8a6874a6de1f3a1bc3da96e1f4d1" blob "sha256:abdf7e6fb3df9898c3fa4b0985e1e967131a4ab884eccefaa9c0f99885d21a64" unknown to registry. 

Reproduction steps

1. Push an image built with the latest docker.

Expected behavior

Pushing an image should succeed.

cc: @xtremerui @cdmessin @pvaramballypivot

[xtremerui]

I think this is related to the image engine used in the registry. Have you tried with a different registry like docker hub?

[kalensk]

I think this is related to the image engine used in the registry. Have you tried with a different registry like docker hub?

Unfortunately, I don't think it will be feasible to try that as I am pushing to a private registry in gcr.

[prafsoni-smarsh]

We are facing the same issue where we are not able to push images built with docker 25 and newer. Initially, we suspected that it could be something related to the remote registry however we confirmed we could manually push new OCI images to the registry successfully. And then found this issue so we suspect something in the resource doesn't like the new format.