v0.4.3 vault-secrets-operator crashes when deploying from a manually rendered Helm chart.

[anilpally]

Describe the bug
v0.4.3 vault-secrets-operator crashes, also verbs are missing for hcpauth, hcpvaultsecretsapps clusterrole, i expect these to be created with deployment/ CRDs

To Reproduce
Steps to reproduce the behavior:

1. Deploy 0.4.3 vault-secrets-operator

See error (vault-secrets-operator logs, application logs, etc.)

E0129 18:15:53.073016 1 reflector.go:147] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1beta1.HCPVaultSecretsApp: failed to list *v1beta1.HCPVaultSecretsApp: hcpvaultsecretsapps.secrets.hashicorp.com is forbidden: User "system:serviceaccount:vault-secrets-operator:vault-secrets-operator-controller-manager" cannot list resource "hcpvaultsecretsapps" in API group "secrets.hashicorp.com" at the cluster scope
W0129 18:16:00.727099 1 reflector.go:539] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1beta1.HCPVaultSecretsApp: hcpvaultsecretsapps.secrets.hashicorp.com is forbidden: User "system:serviceaccount:vault-secrets-operator:vault-secrets-operator-controller-manager" cannot list resource "hcpvaultsecretsapps" in API group "secrets.hashicorp.com" at the cluster scope
E0129 18:16:00.727258 1 reflector.go:147] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1beta1.HCPVaultSecretsApp: failed to list *v1beta1.HCPVaultSecretsApp: hcpvaultsecretsapps.secrets.hashicorp.com is forbidden: User "system:serviceaccount:vault-secrets-operator:vault-secrets-operator-controller-manager" cannot list resource "hcpvaultsecretsapps" in API group "secrets.hashicorp.com" at the cluster scope
W0129 18:16:04.331736 1 reflector.go:539] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1beta1.HCPAuth: hcpauths.secrets.hashicorp.com is forbidden: User "system:serviceaccount:vault-secrets-operator:vault-secrets-operator-controller-manager" cannot list resource "hcpauths" in API group "secrets.hashicorp.com" at the cluster scope
E0129 18:16:04.331906 1 reflector.go:147] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1beta1.HCPAuth: failed to list *v1beta1.HCPAuth: hcpauths.secrets.hashicorp.com is forbidden: User "system:serviceaccount:vault-secrets-operator:vault-secrets-operator-controller-manager" cannot list resource "hcpauths" in API group "secrets.hashicorp.com" at the cluster scope

Expected behavior
stable deployment not crashing often, with clusterrole updated for hcpauth/hcpvaultsecretsapps.

Environment
ocp 4.14

• vault-secrets-operator version: 0.4.3

Additional context
Add any other context about the problem here.

[anilpally]

[athangal@marv2257 ~]$ oc logs vault-secrets-operator-controller-manager-7c6fb6cd5d-khtgr| grep ERROR
2024-01-29T18:46:08Z ERROR Could not wait for Cache to sync {"controller": "hcpauth", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "HCPAuth", "error": "failed to wait for hcpauth caches to sync: timed out waiting for cache to be synced for Kind *v1beta1.HCPAuth"}
2024-01-29T18:46:08Z ERROR controller-runtime.source.EventHandler failed to get informer from cache {"error": "Timeout: failed waiting for *v1beta1.HCPVaultSecretsApp Informer to sync"}
2024-01-29T18:46:08Z ERROR Could not wait for Cache to sync {"controller": "vaultauth", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultAuth", "error": "failed to wait for vaultauth caches to sync: timed out waiting for cache to be synced for Kind *v1beta1.VaultAuth"}
2024-01-29T18:46:08Z ERROR Could not wait for Cache to sync {"controller": "hcpvaultsecretsapp", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "HCPVaultSecretsApp", "error": "failed to wait for hcpvaultsecretsapp caches to sync: timed out waiting for cache to be synced for Kind *v1beta1.HCPVaultSecretsApp"}
2024-01-29T18:46:08Z ERROR Could not wait for Cache to sync {"controller": "vaultpkisecret", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultPKISecret", "error": "failed to wait for vaultpkisecret caches to sync: timed out waiting for cache to be synced for Kind *v1beta1.VaultPKISecret"}
2024-01-29T18:46:08Z ERROR Could not wait for Cache to sync {"controller": "vaultconnection", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultConnection", "error": "failed to wait for vaultconnection caches to sync: timed out waiting for cache to be synced for Kind *v1beta1.VaultConnection"}
2024-01-29T18:46:08Z ERROR controller-runtime.source.EventHandler failed to get informer from cache {"error": "Timeout: failed waiting for *v1beta1.HCPAuth Informer to sync"}
2024-01-29T18:46:08Z ERROR error received after stop sequence was engaged {"error": "failed to wait for vaultauth caches to sync: timed out waiting for cache to be synced for Kind *v1beta1.VaultAuth"}
2024-01-29T18:46:08Z ERROR error received after stop sequence was engaged {"error": "failed to wait for hcpvaultsecretsapp caches to sync: timed out waiting for cache to be synced for Kind *v1beta1.HCPVaultSecretsApp"}
2024-01-29T18:46:08Z ERROR error received after stop sequence was engaged {"error": "failed to wait for vaultpkisecret caches to sync: timed out waiting for cache to be synced for Kind *v1beta1.VaultPKISecret"}
2024-01-29T18:46:08Z ERROR error received after stop sequence was engaged {"error": "failed to wait for vaultconnection caches to sync: timed out waiting for cache to be synced for Kind *v1beta1.VaultConnection"}
2024-01-29T18:46:08Z ERROR Could not wait for Cache to sync {"controller": "vaultstaticsecret", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultStaticSecret", "error": "failed to wait for vaultstaticsecret caches to sync: timed out waiting for cache to be synced for Kind *v1beta1.VaultStaticSecret"}
2024-01-29T18:46:08Z ERROR error received after stop sequence was engaged {"error": "failed to wait for vaultstaticsecret caches to sync: timed out waiting for cache to be synced for Kind *v1beta1.VaultStaticSecret"}
2024-01-29T18:46:08Z ERROR Could not wait for Cache to sync {"controller": "vaultdynamicsecret", "controllerGroup": "secrets.hashicorp.com", "controllerKind": "VaultDynamicSecret", "error": "failed to wait for vaultdynamicsecret caches to sync: timed out waiting for cache to be synced for Kind *v1beta1.VaultDynamicSecret"}
2024-01-29T18:46:08Z ERROR error received after stop sequence was engaged {"error": "failed to wait for vaultdynamicsecret caches to sync: timed out waiting for cache to be synced for Kind *v1beta1.VaultDynamicSecret"}
2024-01-29T18:46:08Z ERROR setup problem running manager {"error": "failed to wait for hcpauth caches to sync: timed out waiting for cache to be synced for Kind *v1beta1.HCPAuth"}

[benashz]

HI @anilpally - it looks like something may have gone wrong during the installation. Can you provide more details on how you are installing VSO. Are you using Helm, Kustomize or OLM. Also, was this an upgrade or a fresh install?

In case you are using Helm and this was an upgrade, please see https://developer.hashicorp.com/vault/docs/platform/k8s/vso/installation#updating-crds-when-using-helm

[anilpally]

hi @benashz

we convert helm chart into templates, helm template -f vaules.yaml --include-crds --output-dir /tmp/vault-secrets-operator

Argocd apply manifest under /tmp/vault-secrets-operator

$ ls vault-secrets-operator/templates/
job.yaml metrics-service.yaml secrets.hashicorp.com_vaultconnections.yaml
leader-election-rbac.yaml proxy-rbac.yaml secrets.hashicorp.com_vaultdynamicsecrets.yaml
manager-config.yaml secrets.hashicorp.com_hcpauths.yaml secrets.hashicorp.com_vaultpkisecrets.yaml
manager-rbac.yaml secrets.hashicorp.com_hcpvaultsecretsapps.yaml secrets.hashicorp.com_vaultstaticsecrets.yaml
metrics-reader-rbac.yaml secrets.hashicorp.com_vaultauths.yaml serviceaccount.yaml

$ pwd
/vault-secrets-operator-config
$ ls templates/
default-vault-connection.yaml deployment.yaml namespace.yaml ocp-vault-connection.yaml secret_dockerextnexusread.yaml secret_vault-ca.yaml

[anilpally]

@benashz can you let us know in which order we should apply, so i can annotate them in the order arogocd applies.

[anilpally]

any update?

[benashz]

@anilpally, it looks you are using a non standard installation method by rendering the Helm chart to k8s manifests. In theory that might work, but it is not supported. We currently only support installing VSO from the Helm chart (using helm), the OLM package, or Kustomize.