Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add 'umoci copy' subcommand #359

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

jdolitsky
Copy link
Member

@jdolitsky jdolitsky commented Feb 8, 2021

  • Support for pulling oci:// URL into local store
  • Support for pushing from local store to registry
  • Downloads blobs only when not found in local store
  • Currently only supports OCI v1 manifest media type
  • Does not support docker:// URLs or Docker manifests
  • Does not yet support chunked blob uploading

Resolves #345


Demo

Get binaries for zot and umoci (modified), add to PATH

mkdir demo && cd demo/
mkdir bin/
git clone --depth 1 --branch v1.1.12 https://github.com/anuvu/zot.git
(cd zot && make binary && mv bin/zot ../bin/)
git clone --depth 1 --branch copy-subcommand https://github.com/bloodorangeio/umoci.git
(cd umoci && make && mv ./umoci ../bin/)
export PATH="${PWD}/bin:${PATH}"

Start zot server (port 8080)

echo "{\"storage\":{\"rootDirectory\":\"${PWD}/storage\"}}" > zot.yaml
zot serve ./zot.yaml

Use skopeo to fetch Docker manifest, convert to OCI, and push to zot

skopeo copy -f oci --dest-tls-verify=false \
  docker://opensuse/amd64:42.2 docker://127.0.0.1:8080/opensuse/amd64:42.2

umoci copy: remote to local

umoci --log=debug copy --plain-http oci://localhost:8080/opensuse/amd64:42.2 opensuse:42.2

output:

   • Registry address: http://localhost:8080, namespace: opensuse/amd64, tag: 42.2
   • Checking if manifest available in registry
   • Registry reports manifest with digest sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed
   • Downloading manifest from registry
   • actual manifest digest matches expected digest (sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed)
   • blob saved to local store, digest: sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed, size: 349
   • Manifest layer list contains 1 item(s)
   • Copying layer 1/1 with digest sha256:f9ee55447d182e2d7a04e1f0dd7138c1de9aac721a07274c4fdbf3f253b800f8
   • actual layer digest matches expected digest (sha256:f9ee55447d182e2d7a04e1f0dd7138c1de9aac721a07274c4fdbf3f253b800f8)
   • blob saved to local store, digest: sha256:f9ee55447d182e2d7a04e1f0dd7138c1de9aac721a07274c4fdbf3f253b800f8, size: 48850323
   • Copying config with digest sha256:2534c61b53d3222d048c2617c784cf568b9b496efd190f2f026c6e97e7ad3bff
   • actual config digest matches expected digest (sha256:2534c61b53d3222d048c2617c784cf568b9b496efd190f2f026c6e97e7ad3bff)
   • blob saved to local store, digest: sha256:2534c61b53d3222d048c2617c784cf568b9b496efd190f2f026c6e97e7ad3bff, size: 720
   • Saving reference '42.2' to index in opensuse

inspect CAS:

$ tree opensuse
opensuse
├── blobs
│   └── sha256
│       ├── 2534c61b53d3222d048c2617c784cf568b9b496efd190f2f026c6e97e7ad3bff
│       ├── 7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed
│       └── f9ee55447d182e2d7a04e1f0dd7138c1de9aac721a07274c4fdbf3f253b800f8
├── index.json
└── oci-layout
$ cat opensuse/index.json | jq
{
  "schemaVersion": 2,
  "manifests": [
    {
      "mediaType": "application/vnd.oci.image.manifest.v1+json",
      "digest": "sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed",
      "size": 349,
      "annotations": {
        "org.opencontainers.image.ref.name": "42.2"
      }
    }
  ]
}

umoci copy: local to remote

umoci --log=debug copy --plain-http opensuse:42.2 oci://localhost:8080/othersuse/amd64:othertag

output:

   • Registry address: http://localhost:8080, namespace: othersusex/amd64, tag: othertag
   • -> ws.recurse             digest=sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed
   • <- ws.recurse             digest=sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed
   • casext.ResolveReference(42.2) got these descriptors refs=[{[{application/vnd.oci.image.manifest.v1+json sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed 349 [] map[org.opencontainers.image.ref.name:42.2] <nil>}]}]
   • Reference '42.2' found in index, points to manifest sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed
   • Manifest successfully loaded from local store
   • Manifest layer list contains 1 item(s)
   • Uploading layer 1/1 with digest sha256:f9ee55447d182e2d7a04e1f0dd7138c1de9aac721a07274c4fdbf3f253b800f8 from local store
   • Uploading config sha256:2534c61b53d3222d048c2617c784cf568b9b496efd190f2f026c6e97e7ad3bff
   • Successfully copied to remote localhost:8080

If you add the --trace-requests flag, raw HTTP output will be displayed:

umoci --log=debug copy --plain-http --trace-requests oci://localhost:8080/opensuse/amd64:42.2 opensuse:42.2

output:

   • Registry address: http://localhost:8080, namespace: opensuse/amd64, tag: 42.2
   • Checking if manifest available in registry
2021/02/08 15:58:15.750531 DEBUG RESTY
==============================================================================
~~~ REQUEST ~~~
HEAD  /v2/opensuse/amd64/manifests/42.2  HTTP/1.1
HOST   : localhost:8080
HEADERS:
	Accept: application/vnd.oci.image.manifest.v1+json
	User-Agent: umoci 0.4.6+dev~gitd2f6056482314a44724ac2151622843881a18c45
BODY   :
***** NO CONTENT *****
------------------------------------------------------------------------------
~~~ RESPONSE ~~~
STATUS       : 200 OK
RECEIVED AT  : 2021-02-08T15:58:15.749825-05:00
TIME DURATION: 4.758014ms
HEADERS      :
	Content-Length: 0
	Date: Mon, 08 Feb 2021 20:58:15 GMT
	Docker-Content-Digest: sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed
BODY         :

==============================================================================
   • Registry reports manifest with digest sha256:7878ba2e769e17d8da0c4720487373a84099e7139786e5c6d5eb50ca91dd3fed
 ...

- Support for pulling oci:// URL into local store
- Support for pushing from local store to registry
- Downloads blobs only when not found in local store
- Currently only supports OCI v1 manifest media type
- Does not support docker:// URLs or Docker manifests
- Does not yet support chunked blob uploading

Signed-off-by: Josh Dolitsky <[email protected]>
fmt.Sprintf("Content-Type header for image manifest invalid: %s", mediaType))
}

d, s, err := engine.PutBlob(engineContext, bytes.NewReader(resp.Body()))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I can tell, resp.Body() here is: https://pkg.go.dev/github.com/go-resty/resty/v2#Response.Body which will end up reading the whole layer into memory. Seems like you want RawBody() here?

https://pkg.go.dev/github.com/go-resty/resty/v2#Response.RawBody

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, derp. I meant to put this on the layer one below. For this one I think it's unavoidable: you definitely want to read the manifest bytes into memory.

@tych0
Copy link
Member

tych0 commented Mar 10, 2021

One small nit on this, but otherwise it looks like it might be useful to us. How close do you feel this is to merging?

@cyphar
Copy link
Member

cyphar commented Mar 11, 2021

@tych0

One small nit on this, but otherwise it looks like it might be useful to us. How close do you feel this is to merging?

I've been chatting to @jdolitsky about this for a little bit -- my current plan is for me to rework this PR so that it's implemented as a cas.Engine backend once I get back from vacation next week.

@tych0
Copy link
Member

tych0 commented Mar 11, 2021

my current plan is for me to rework this PR so that it's implemented as a cas.Engine backend once I get back from vacation next week.

Excellent, thanks!

@tych0
Copy link
Member

tych0 commented Sep 7, 2021

What's the status of this? I'd like to get some features into containers/image soon-ish, but it would be better to have them here, if possible :)

@cyphar
Copy link
Member

cyphar commented Sep 7, 2021

I haven't touched this in a while. I can try to revive my PoC using the CAS API -- what exactly did you want to get into containers/image? Switching to umoci copy would probably be somewhat non-trivial even if I did get it working soon.

@tych0
Copy link
Member

tych0 commented Sep 7, 2021

Mostly I don't want it to compress things automatically that aren't gzip :)

@cyphar
Copy link
Member

cyphar commented Sep 21, 2021

I feel the the only slight complication is going to be chunked uploads and downloads which this PoC didn't implement. I started working on a PoC of this, it shouldn't be that bad on paper but implementing a generic umoci copy API might be slightly tricky and I think the CasEngineExt API needs to be reworked somewhat in order to be able to handle arbitrary CAS implementations to wrap).

@jdolitsky
Copy link
Member Author

@cyphar - shall I close this for now?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

support pulling images with OCI Distribution Spec API
3 participants