Prometheus scaler unable to connect to Grafana Cloud Prometheus

[tylerauerbeck]

Report

When configuring a Prometheus scaler to run against a Grafana Cloud hosted Prometheus, an error occurs resulting in "https://my-prometheus.grafana.net/api/prom/api/v1/query?query=buildkite_queues_scheduled_jobs_count&time=2025-01-14T05:53:39Z": context canceled

Expected Behavior

According to the documentation, this should be just like any other Prometheus, with the expected api being available on Grafana Cloud at https://my-prometheus.grafana.net/api/prom/.

Actual Behavior

Error occurs.

Get "https://my-cloud-prom.grafana.net/api/prom/api/v1/query?query=buildkite_queues_scheduled_jobs_count&time=2025-01-14T05:53:39Z": context canceled

Steps to Reproduce the Problem

1. Deploy KEDA
2. Apply below ScaledJob
3. Fails immediately

apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  name: my-scaled-job
spec:
  jobTargetRef:
    parallelism: 1
    completions: 1
    activeDeadlineSeconds: 600
    backoffLimit: 6
    template:
      spec:
        containers:
        - name: pi
          image: perl:5.34.0
          command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
        restartPolicy: Never
  pollingInterval: 10
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  minReplicaCount: 1
  maxReplicaCount: 35
  triggers:
    - type: prometheus
      metadata:
        serverAddress: https://my-cloud-prom.grafana.net/api/prom
        threshold: '1'
        query:  buildkite_queues_scheduled_jobs_count
        authModes: "basic"
      authenticationRef:
        name: keda-grafana-cloud-prom-creds

Logs from KEDA operator

2025-01-14T21:18:59Z    ERROR   prometheus_scaler       error executing prometheus query        {"type": "ScaledJob", "namespace": "default", "name": "my-scaled-job", "error": "Get \"https://my-cloud-prom.grafana.net/api/prom/api/v1/query?query=buildkite_queues_scheduled_jobs_count&time=2025-01-14T21:18:56Z\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
github.com/kedacore/keda/v2/pkg/scalers.(*prometheusScaler).GetMetricsAndActivity
        /workspace/pkg/scalers/prometheus_scaler.go:291
github.com/kedacore/keda/v2/pkg/scaling/cache.(*ScalersCache).GetMetricsAndActivityForScaler
        /workspace/pkg/scaling/cache/scalers_cache.go:151
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).getScaledJobMetrics
        /workspace/pkg/scaling/scale_handler.go:847
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).isScaledJobActive
        /workspace/pkg/scaling/scale_handler.go:897
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).checkScalers
        /workspace/pkg/scaling/scale_handler.go:262
github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).startScaleLoop
        /workspace/pkg/scaling/scale_handler.go:182

KEDA Version

2.16.1

Kubernetes Version

1.30

Platform

Any

Scaler Details

Prometheus

Anything else?

No response