Google Artifact Registry Service Account key.json as PASSWORDFILE

[nikola197]

Support guidelines

• I've read the support guidelines

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

I can not get diun to pull image manifests from Google Artifact Registry.

Expected behaviour

I should be able to login with username _json_key and key.json as password to Google Artifact Registry, in my case us-east1-docker.pkg.dev

Actual behaviour

Actual behavior is that I get 403 error with USERNAME=_json_key and PASSWORDFILE=/etc/secret/key.json

Steps to reproduce

1. Create Google SA
2. Create and download Google SA JSON key file
3. Add roles/regisry.reader and roles/iam.serviceAccountTokenCreator to the SA
4. Create GKE/k8s Opaque secret with key.json key and data content of key.json
5. Apply k8s configuration with sample app of your choice

Diun version

4.26.0

Docker info

v1.27.7-gke.1121000
containerd://1.7.7

Docker Compose config

No response

Logs

Tue, 30 Jan 2024 14:26:24 CET INF Starting Diun version=v4.26.0
Tue, 30 Jan 2024 14:26:24 CET DBG No configuration file found
Tue, 30 Jan 2024 14:26:24 CET INF Configuration loaded from 10 environment variable(s)
Tue, 30 Jan 2024 14:26:24 CET DBG {
  "db": {
    "path": "/data/diun.db"
  },
  "watch": {
    "workers": 20,
    "schedule": "0 */6 * * *",
    "jitter": 30000000000,
    "firstCheckNotif": false,
    "runOnStartup": true,
    "compareDigest": true
  },
  "defaults": {
    "watchRepo": false,
    "notifyOn": [
      "new",
      "update"
    ],
    "sortTags": "reverse"
  },
  "regopts": [
    {
      "name": "us-east1-docker.pkg.dev",
      "selector": "name",
      "username": "_json_key",
      "passwordFile": "/etc/secret/key.json",
      "insecureTLS": false,
      "timeout": 0
    }
  ],
  "providers": {
    "kubernetes": {
      "tlsInsecure": false,
      "namespaces": [
        "my-app"
      ],
      "watchByDefault": false
    }
  }
}
Tue, 30 Jan 2024 14:26:24 CET WRN No notifier available
Tue, 30 Jan 2024 14:26:24 CET DBG 0 entries found in manifest bucket
Tue, 30 Jan 2024 14:26:24 CET DBG Current database version: 1
Tue, 30 Jan 2024 14:26:24 CET INF Database migration v2...
Tue, 30 Jan 2024 14:26:24 CET INF Cron triggered
Tue, 30 Jan 2024 14:26:24 CET DBG Creating in-cluster Kubernetes provider client 
Tue, 30 Jan 2024 14:26:24 CET DBG Validate image ctn_image=haproxy:1.7-alpine ctn_name=haproxy pod_annot=null pod_name=db-proxy-port-fwd-8579bc6886-zt5wg provider=kubernetes
Tue, 30 Jan 2024 14:26:24 CET DBG Watch disabled ctn_image=haproxy:1.7-alpine ctn_name=haproxy pod_annot=null pod_name=db-proxy-port-fwd-8579bc6886-zt5wg provider=kubernetes
Tue, 30 Jan 2024 14:26:24 CET DBG Validate image ctn_image=us-east1-docker.pkg.dev/reducted-project-id/my-app-backend/my-app-backend:latest ctn_name=my-app pod_annot=null pod_name=my-app-portal-backend-584f565668-gd9pq provider=kubernetes
Tue, 30 Jan 2024 14:26:24 CET DBG Watch disabled ctn_image=us-east1-docker.pkg.dev/reducted-project-id/my-app-backend/my-app-backend:latest ctn_name=my-app pod_annot=null pod_name=my-app-portal-backend-584f565668-gd9pq provider=kubernetes
Tue, 30 Jan 2024 14:26:24 CET DBG Validate image ctn_image=us-east1-docker.pkg.dev/reducted-project-id/my-app-frontend/my-app-frontend:latest ctn_name=my-app pod_annot=null pod_name=my-app-portal-frontend-77d8f7dcc4-97tlc provider=kubernetes
Tue, 30 Jan 2024 14:26:24 CET DBG Watch disabled ctn_image=us-east1-docker.pkg.dev/reducted-project-id/my-app-frontend/my-app-frontend:latest ctn_name=my-app pod_annot=null pod_name=my-app-portal-frontend-77d8f7dcc4-97tlc provider=kubernetes
Tue, 30 Jan 2024 14:26:24 CET DBG Validate image ctn_image=us-east1-docker.pkg.dev/reducted-project-id/my-app-frontend2/my-app-frontend2:latest ctn_name=my-app pod_annot=null pod_name=my-app-portal-frontend2-5b7c987ffb-6mzd8 provider=kubernetes
Tue, 30 Jan 2024 14:26:24 CET DBG Watch disabled ctn_image=us-east1-docker.pkg.dev/reducted-project-id/my-app-frontend2/my-app-frontend2:latest ctn_name=my-app pod_annot=null pod_name=my-app-portal-frontend2-5b7c987ffb-6mzd8 provider=kubernetes
Tue, 30 Jan 2024 14:26:24 CET DBG Validate image ctn_image=us-east1-docker.pkg.dev/reducted-project-id/my-app-db/my-app-db:latest ctn_name=my-app pod_annot={"diun.enable":"true"} pod_name=my-app-portal-db-58976bbcf4-cs2rr provider=kubernetes
Tue, 30 Jan 2024 14:26:24 CET INF Found 1 image(s) to analyze provider=kubernetes
Tue, 30 Jan 2024 14:26:24 CET DBG [containers/image] Loading registries configuration "/etc/containers/registries.conf"
Tue, 30 Jan 2024 14:26:24 CET DBG [containers/image] No credentials matching us-east1-docker.pkg.dev found in /run/containers/0/auth.json
Tue, 30 Jan 2024 14:26:24 CET DBG [containers/image] No credentials matching us-east1-docker.pkg.dev found in /root/.config/containers/auth.json
Tue, 30 Jan 2024 14:26:24 CET DBG [containers/image] No credentials matching us-east1-docker.pkg.dev found in /root/.docker/config.json
Tue, 30 Jan 2024 14:26:24 CET DBG [containers/image] No credentials matching us-east1-docker.pkg.dev found in /root/.dockercfg
Tue, 30 Jan 2024 14:26:24 CET DBG [containers/image] No credentials for us-east1-docker.pkg.dev found
Tue, 30 Jan 2024 14:26:24 CET DBG [containers/image] Using registries.d directory /etc/containers/registries.d
Tue, 30 Jan 2024 14:26:24 CET DBG [containers/image] Returning credentials for us-east1-docker.pkg.dev/reducted-project-id/my-app-db/my-app-db from DockerAuthConfig
Tue, 30 Jan 2024 14:26:24 CET DBG [containers/image]  No signature storage configuration found for us-east1-docker.pkg.dev/reducted-project-id/my-app-db/my-app-db:latest, using built-in default file:///var/lib/containers/sigstore
Tue, 30 Jan 2024 14:26:24 CET DBG [containers/image] Looking for TLS certificates and private keys in /etc/docker/certs.d/us-east1-docker.pkg.dev
Tue, 30 Jan 2024 14:26:24 CET DBG [containers/image] GET https://us-east1-docker.pkg.dev/v2/
Tue, 30 Jan 2024 14:26:24 CET DBG [containers/image] Ping https://us-east1-docker.pkg.dev/v2/ status 401
Tue, 30 Jan 2024 14:26:24 CET DBG [containers/image] GET https://us-east1-docker.pkg.dev/v2/token?scope=repository%3Areducted-project-id%2Fmy-app-db%2Fmy-app-db%3Apull
Tue, 30 Jan 2024 14:26:24 CET WRN Cannot get remote manifest error="cannot get image digest from HEAD request: Requesting bearer token: invalid status code from registry 403 (Forbidden)" image=us-east1-docker.pkg.dev/reducted-project-id/my-app-db/my-app-db:latest provider=kubernetes
Tue, 30 Jan 2024 14:26:24 CET INF Jobs completed added=0 failed=1 skipped=0 unchanged=0 updated=0
Tue, 30 Jan 2024 14:26:24 CET INF Cron initialized with schedule 0 */6 * * *
Tue, 30 Jan 2024 14:26:24 CET INF Next run in 3 hours 33 minutes (2024-01-30 18:00:07.219993394 +0100 CET)

Additional info

Kubernetes diun configuration:

apiVersion: v1
kind: ServiceAccount
metadata:
  namespace: default
  name: diun
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: diun
rules:
  - apiGroups:
      - ""
    resources:
      - pods
    verbs:
      - get
      - watch
      - list
  - apiGroups:
      - ""
    resources:
      - secrets
    verbs:
      - get
      - list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: diun
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: diun
subjects:
  - kind: ServiceAccount
    name: diun
    namespace: default
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: default
  name: diun
spec:
  replicas: 1
  selector:
    matchLabels:
      app: diun
  template:
    metadata:
      labels:
        app: diun
      annotations:
        diun.enable: "true"
    spec:
      serviceAccountName: diun
      containers:
        - name: diun
          image: crazymax/diun:latest
          imagePullPolicy: Always
          args: ["serve"]
          env:
            - name: TZ
              value: "Europe/Paris"
            - name: LOG_LEVEL
              value: "DEBUG"
            - name: LOG_JSON
              value: "false"
            - name: DIUN_WATCH_WORKERS
              value: "20"
            - name: DIUN_WATCH_SCHEDULE
              value: "0 */6 * * *"
            - name: DIUN_WATCH_JITTER
              value: "30s"
            - name: DIUN_PROVIDERS_KUBERNETES
              value: "true"
            - name: DIUN_PROVIDERS_KUBERNETES_WATCHBYDEFAULT
              value: "false"
            - name: DIUN_PROVIDERS_KUBERNETES_NAMESPACES
              value: "my-app"
            - name: DIUN_REGOPTS_0_NAME
              value: "us-east1-docker.pkg.dev"
            - name: DIUN_REGOPTS_0_USERNAME
              value: "_json_key"
            - name: DIUN_REGOPTS_0_PASSWORDFILE
              value: "/etc/secret/key.json"
          volumeMounts:
            - name: secret-volume
              mountPath: "/etc/secret"
              readOnly: true
          resources:
            limits:
              cpu: "500m"
              memory: "512Mi"
            requests:
              cpu: "100m"
              memory: "128Mi"
      restartPolicy: Always
      volumes:
        - name: secret-volume
          secret:
            secretName: diun-gar-service-account
            items:
              - key: key.json
                path: key.json # Google SA JSON key file - SA have the following roles roles/regisry.reader and roles/iam.serviceAccountTokenCreator

[crazy-max]

Config looks good but are you sure content of /etc/secret/key.json is correct within the container (newlines, ...)?

If it is then this might be an auth issue with upstream module https://github.com/containers/image that we are using here:

diun/pkg/registry/registry.go

Line 33 in 61dac9f

DockerAuthConfig: &opts.Auth,