How to deploy Quay with RHOCS backend on OCP 4.6

Introduction

Quay is a secure, private container registry that builds, analyzes and distributes container images. In this post we will show how we deploy it on top of OCP 4 and how we use OCS Multi-Cloud Object Gateway object service to provide a storage backend to Quay. We will also enable Clair container vulnerability analysis service.

Lab Virtual Env Overview

OCP 4.6 UPI baremetal deployment
All oc commands are issued from a CentOS 8 bastion node
3 Masters (16Go of RAM) and 3 Workers (32Go of RAM) VMs

[root@bastion ~] oc get nodes
NAME                        STATUS   ROLES    AGE   VERSION
master0.mycluster.lab.local   Ready    master   10h   v1.19.0+7070803
master1.mycluster.lab.local   Ready    master   10h   v1.19.0+7070803
master2.mycluster.lab.local   Ready    master   10h   v1.19.0+7070803
worker0.mycluster.lab.local   Ready    worker   10h   v1.19.0+7070803
worker1.mycluster.lab.local   Ready    worker   10h   v1.19.0+7070803
worker2.mycluster.lab.local   Ready    worker   10h   v1.19.0+7070803

OCS 4.6 has been deployed

[root@bastion ~] oc get cephcluster -n openshift-storage
NAME                             DATADIRHOSTPATH   MONCOUNT   AGE     PHASE   MESSAGE                        HEALTH
ocs-storagecluster-cephcluster   /var/lib/rook     3          5h17m   Ready   Cluster created successfully   HEALTH_OK

Simple deployment

Let's Create a dedicated Namespace and deploy Quay Operator

cat << EOF | oc apply -f -
apiVersion: v1
kind: Namespace
metadata:
  labels:
    openshift.io/cluster-monitoring: "true"
  name: local-quay
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: local-quay
  namespace: local-quay
spec:
  targetNamespaces:
  - local-quay
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: quay-operator
  namespace: local-quay
spec:
  channel: quay-v3.4
  installPlanApproval: Automatic
  name: quay-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF

We wait for the operator to be ready

[root@bastion ~] oc get pod -n local-quay
NAME                                                    READY   STATUS    RESTARTS   AGE
quay-operator-7cb56c7c5f-7dgwk                          1/1     Running   0          6s

Create an OpenShift secret to be able to pull the required container images

oc create -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: redhat-pull-secret
data:
  .dockerconfigjson:  <Quay Key>
type: kubernetes.io/dockerconfigjson
EOF
Get the key here https://access.redhat.com/solutions/3533201

RHOCS Object storage service provides an S3 API and is based on NooBaa project.

To interact with the service, we need to:

Install Noobaa CLI on our Bastion

OS="linux"
VERSION=$(curl -s https://api.github.com/repos/noobaa/noobaa-operator/releases/latest | jq -r '.name')
curl -LO https://github.com/noobaa/noobaa-operator/releases/download/$VERSION/noobaa-$OS-$VERSION
chmod +x noobaa-$OS-$VERSION
mv noobaa-$OS-$VERSION /usr/local/bin/noobaa

Full documentation can be found here: https://github.com/noobaa/noobaa-operator

Retrieve S3 API credentials and endpoint

[root@bastion ~] noobaa status -n openshift-storage
INFO[0000] CLI version: 2.3.0                           
INFO[0000] noobaa-image: noobaa/noobaa-core:5.5.0       
INFO[0000] operator-image: noobaa/noobaa-operator:2.3.0
INFO[0000] Namespace: openshift-storage                 
 .
 .
 .
#----------------#
#- S3 Addresses -#
#----------------#
ExternalDNS : [https://s3-openshift-storage.mycluster.lab.local]
ExternalIP  : []
NodePorts   : [https://10.226.115.10:30795]
InternalDNS : [https://s3.openshift-storage.svc:443]
InternalIP  : [https://172.30.20.203:443]
PodPorts    : [https://10.131.0.90:6443]
#------------------#
#- S3 Credentials -#
#------------------#
AWS_ACCESS_KEY_ID     : FSfXsNiWnva1lmDnzZ1c
AWS_SECRET_ACCESS_KEY : r+RpYkH8Es81ZipMjngCmHUfMvIa4TLYOlTJwLgE
#------------------#
#- Backing Stores -#
#------------------#
NAME                           TYPE            TARGET-BUCKET                             PHASE   AGE         
noobaa-default-backing-store   s3-compatible   nb.1609358827719.mycluster.lab.local   Ready   94h48m18s   
#------------------#
#- Bucket Classes -#
#------------------#
NAME                          PLACEMENT                                                             PHASE   AGE         
noobaa-default-bucket-class   {Tiers:[{Placement: BackingStores:[noobaa-default-backing-store]}]}   Ready   94h48m18s   
#-----------------#
#- Bucket Claims -#
#-----------------#
No OBCs found.

Take good notes of ExternalDNS, InternalDNS, AWS_ACCESS_KEY_ID and AWS_ACCESS_KEY_ID as we will need it to create our Quay Echosystem

Install s3cmd and create .s3cmd config (This is not mandatory but makes manipulation of buckets and objects easier)

[root@bastion ~] dnf install s3cmd -y
[root@bastion ~] cat .s3cfg
[default]
access_key = FSfXsNiWnva1lmDnzZ1c
secret_key = r+RpYkH8Es81ZipMjngCmHUfMvIa4TLYOlTJwLgE
host_base = s3-openshift-storage.mycluster.lab.local  ##We use ExternalDNS retrieved with nooba command
host_bucket = s3-openshift-storage.mycluster.lab.local
check_ssl_certificate = False
check_ssl_hostname = False
use_http_expect = False
use_https = True
signature_v2 = True
signurl_use_https = True

Create a bucket for Quay

[root@bastion ~]# s3cmd mb  s3://quay1
Bucket 's3://quay1/' created

We can now create a QuayEcosystem with the following command:

oc create  -f - <<EOF
apiVersion: redhatcop.redhat.io/v1alpha1
kind: QuayEcosystem
metadata:
  name: quayecosystem1
spec:
  clair:
    enabled: true
    imagePullSecretName: redhat-pull-secret
  quay:
    imagePullSecretName: redhat-pull-secret
    registryBackends:
      - name: rhocs
        rhocs:
          hostname: s3.openshift-storage.svc.cluster.local:443
          accessKey: FSfXsNiWnva1lmDnzZ1c
          secretKey: r+RpYkH8Es81ZipMjngCmHUfMvIa4TLYOlTJwLgE
          bucketName: quay1
EOF

Let's take a closer look :

First we create a QuayEcossytem object and name it quayecosystem1

apiVersion: redhatcop.redhat.io/v1alpha1
kind: QuayEcosystem
metadata:
  name: quayecosystem1

We specify we want to enable Clair and we use redhat-pull-secret so we can pull the necessary image

  spec:
    clair:
      enabled: true
      imagePullSecretName: redhat-pull-secret

This is the most important part (see comments)

quay:
  imagePullSecretName: redhat-pull-secret
  registryBackends:
    - name: rhocs
      rhocs:    # We specify Quay we are using rhcos as storage backend
        hostname: s3.openshift-storage.svc.cluster.local:443 #Make sure InternalDNS and Port are specified here
        accessKey: FSfXsNiWnva1lmDnzZ1c
        secretKey: r+RpYkH8Es81ZipMjngCmHUfMvIa4TLYOlTJwLgE
        bucketName: quay1 # This is the bucket we created previously

hostname field needs to be specified with the port otherwise you'll get the infamous error:

 'Failed to Validate Component: registry-storage Validation Failed: Invalid storage configuration: rhocs: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:727)'    


* After a few minutes we check all pods are up and running
```bash
[root@bastion ~] oc get pod
NAME                                               READY   STATUS    RESTARTS   AGE
quay-operator-7cb56c7c5f-7dgwk                     1/1     Running   0          4d5h
quayecosystem1-clair-77f7844f4f-75lh8              1/1     Running   0          3h7m
quayecosystem1-clair-postgresql-5c69478878-bfmtn   1/1     Running   0          3h7m
quayecosystem1-quay-84cbc66cbd-mdw4j               1/1     Running   0          3h8m
quayecosystem1-quay-config-5bf565888c-dtvpv        1/1     Running   0          3h9m
quayecosystem1-quay-postgresql-c7645f945-nzhrr     1/1     Running   0          3h9m
quayecosystem1-redis-68945f56b7-xk67g              1/1     Running   0          3h10m

We can now find out the route and access Quay GUI at https://quayecosystem1-quay-local-quay.mycluster.lab.local (login: quay pass: password)

[root@esxi-bastion ~] oc get route
NAME                         HOST/PORT                                                      PATH   SERVICES                     PORT   TERMINATION            WILDCARD
quayecosystem1-quay          quayecosystem1-quay-local-quay.mycluster.lab.local                 quayecosystem1-quay          8443   passthrough/Redirect   None
quayecosystem1-quay-config   quayecosystem1-quay-config-local-quay.mycluster.lab.local          quayecosystem1-quay-config   8443   passthrough/Redirect   None

## To be continued

Name		Name	Last commit message	Last commit date
Latest commit History 7 Commits
README.md		README.md
quay-operator.yml		quay-operator.yml

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

README.md

quay-operator.yml

quay-operator.yml

Repository files navigation

How to deploy Quay with RHOCS backend on OCP 4.6

Introduction

Lab Virtual Env Overview

Simple deployment

About

Releases

Packages

latouchek/quayocs

Folders and files

Latest commit

History

README.md

README.md

quay-operator.yml

quay-operator.yml

Repository files navigation

How to deploy Quay with RHOCS backend on OCP 4.6

Introduction

Lab Virtual Env Overview

Simple deployment

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages 0

Packages