Error "could not get secret data from provider" on AWS ExternalSecret

[patdechandol]

Hello to everybody,

all the day long I tried to fix a problem with the Helm chart external-secrets deployed on my cluster with ArgoCD... without success so I really need your help.

I have an EKS cluster. I deployed external-secrets via ArgoCD.
I followed the documentation :

• create a specific user
• add the secretsmanager policies
• add the kms:Decrypt policy
• I created a Kubernetes secret with the user access keys
• I added a configuration for my ClusterSecretStore -> everything is OK

NAME AGE STATUS CAPABILITIES READY
global-secret-store 11m Valid ReadWrite True

Now everything seems ready to be able to configure an ExternalSecret but here is the problem.
The ExternalSecret is created but raises the following error in ArgoCD

Here is my ClusterSecretStore manifest :

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"external-secrets.io/v1beta1","kind":"ClusterSecretStore","metadata":{"annotations":{},"labels":{"argocd.argoproj.io/instance":"external-secrets"},"name":"global-secret-store"},"spec":{"provider":{"aws":{"auth":{"secretRef":{"accessKeyIDSecretRef":{"key":"access_key","name":"aws-credentials-secret","namespace":"external-secrets"},"secretAccessKeySecretRef":{"key":"secret_key","name":"aws-credentials-secret","namespace":"external-secrets"}}},"region":"eu-west-1","service":"SecretsManager"}}}}
  creationTimestamp: '2023-01-16T18:19:16Z'
  generation: 1
  labels:
    argocd.argoproj.io/instance: external-secrets
  name: global-secret-store
  resourceVersion: '5217526'
  uid: 509acddd-97c8-4a56-a49a-aef5d09be53e
spec:
  provider:
    aws:
      auth:
        secretRef:
          accessKeyIDSecretRef:
            key: access_key
            name: aws-credentials-secret
            namespace: external-secrets
          secretAccessKeySecretRef:
            key: secret_key
            name: aws-credentials-secret
            namespace: external-secrets
      region: eu-west-1
      service: SecretsManager
status:
  capabilities: ReadWrite
  conditions:
    - lastTransitionTime: '2023-01-16T18:19:17Z'
      message: store validated
      reason: Valid
      status: 'True'
      type: Ready

And here is my ExternalSecret manifest :

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"external-secrets.io/v1beta1","kind":"ExternalSecret","metadata":{"annotations":{},"labels":{"argocd.argoproj.io/instance":"external-secrets"},"name":"postgres-admin-secret","namespace":"external-secrets"},"spec":{"data":[{"remoteRef":{"key":"k8s/staging/postgres","property":"user"},"secretKey":"user"}],"secretStoreRef":{"kind":"ClusterSecretStore","name":"global-secret-store"},"target":{"creationPolicy":"Owner","name":"postgres-admin-secret"}}}
  creationTimestamp: '2023-01-16T18:19:46Z'
  generation: 1
  labels:
    argocd.argoproj.io/instance: external-secrets
  name: postgres-admin-secret
  namespace: external-secrets
  resourceVersion: '5217712'
  uid: 2cbf82eb-c33b-4cfb-8623-aadda93ad295
spec:
  data:
    - remoteRef:
        conversionStrategy: Default
        decodingStrategy: None
        key: k8s/staging/postgres
        property: user
      secretKey: user
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: global-secret-store
  target:
    creationPolicy: Owner
    deletionPolicy: Retain
    name: postgres-admin-secret
status:
  conditions:
    - lastTransitionTime: '2023-01-16T18:19:47Z'
      message: could not get secret data from provider
      reason: SecretSyncedError
      status: 'False'
      type: Ready

I'm completely stucked with this problem and desesperate to find the solution.

[patdechandol]

The problem is solved : I had to use a JWT auth in ClusterSecretRole and everything was OK

[mudasirmirza]

Hi @patdechandol ,

Can you explain in a bit more detail, I are facing somewhat similar issue but unable to figure out. It will be great if you can explain and show some examples.

[github-actions[bot]]

Hello! Reopening this discussion to make it searchable.

[tuwid]

Having similar issue with latest version, below the operator logs

{"level":"error","ts":1714467976.3931026,"logger":"controllers.ExternalSecret","msg":"could not get secret data from provider","ExternalSecret":{"name":"REDACTED-external","namespace":"REDACTED"},"error":"RequestError: send request failed\ncaused by: Post \"https://ssm.eu-north-1.amazonaws.com/\": net/http: invalid header field value for \"Authorization\"","stacktrace":"github.com/external-secrets/external-secrets/pkg/controllers/externalsecret.(*Reconciler).markAsFailed\n\t/home/runner/work/external-secrets/external-secrets/pkg/controllers/externalsecret/externalsecret_controller.go:333\ngithub.com/external-secrets/external-secrets/pkg/controllers/externalsecret.(*Reconciler).Reconcile\n\t/home/runner/work/external-secrets/external-secrets/pkg/controllers/externalsecret/externalsecret_controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1714467976.4082015,"msg":"Reconciler error","controller":"externalsecret","controllerGroup":"external-secrets.io","controllerKind":"ExternalSecret","ExternalSecret":{"name":"REDACTED-external","namespace":"REDACTED"},"namespace":"REDACTED","name":"REDACTED-external","reconcileID":"e6102359-1973-4d59-bb24-ac7245b4ad85","error":"RequestError: send request failed\ncaused by: Post \"https://ssm.eu-north-1.amazonaws.com/\": net/http: invalid header field value for \"Authorization\"","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/home/runner/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}

Already tried the credentials by hand and I'm able to retrieve the secrets but operator is unable to do so