Cloudwatch metrics alert is not going to Firing state through Mimir Alertmanager

[Vaibhav-1995]

Issue

We are using prometheus-cloudwatch-exporter to scrape cloudwatch metrics and push them to mimir datasource. We are able to see those metrics in grafana explore under mimir datasource. But when we created alert rule in mimir ruler it is not going to Firing State even though the metric exceeds threshold value.

Please refer below details : -

Alert rule in ruler configmap -

Not going to Firing State from listed rule -

Environment

• Infrastructure: Kubernetes
• Mimir Version: 2.9.0
• Prometheus Cloudwatch Exporter Version: 0.25.1
• Deployment tool: helm

Expected behavior

It should go to the Firing state and alert should get triggered for set rule as all required configurations in alertmanger are done.

Also if when I tested the same alert rule in grafana UI by creating it with Grafana managed alert then its going in firing state. Please see below image -

[pstibrany]

Mimir evaluates rules as instant queries at "now" timestamp (in fact, now - 1m by default). At that point there are no series above 30, so no alert notification is sent.

[pstibrany]

I'm not familiar with cloudwatch exporter. For my understanding, could you please show me what running query aws_ec2_cpuutilization_average (without > 30 filter) looks like?

Instant query aws_ec2_cpuutilization_average > 30 would select samples up to 5 minutes old, so 1 minute scrape interval doesn't seem like a problem to me.

[Vaibhav-1995]

The running query aws_ec2_cpuutilization_average (without > 30 filter) looks like below - it returns the aws instances cpu utilization avg values (in percent) series per instance.

And, may issue is with selecting or displaying samples/metrics/values of now/latest/5 min timeframe - need to select last 1 hour time frame then only samples/metrics/values are displayed on graph as per above image.

[pracucci]

As mentioned by @pstibrany in the original response (above), looks like there are big gaps (or delays) in CloudWatch metric. This looks something more related to the CloudWatch exporter rather than Mimir (at least based on the information provided so far).

My suggestion is to check the CloudWatch exporter configuration, and make sure it's has been configured to reduce the exported metrics delay as much as possible.

I'm assuming you're running https://github.com/prometheus/cloudwatch_exporter (you mentioned prometheus-cloudwatch-exporter above). I'm not familiar with it, so take my feedback with a grain of salt. Looking at the doc, the default delay_seconds is 600s and set_timestamp is true, which should meant that at least metrics are ingested with 10m of delay (but check other timing / delay settings too, cause there may be other factors too). The alert query you're running is aws_ec2_cpuutilization_average > 30 which means only samples in the last 5m (PromQL lookback period) are evaluated. If samples are always ingested at least with 10m delay, that alerting query will never process any sample.

[Vaibhav-1995]

@pracucci Thanks a lot for your reply!

Surely, I will dig out on the findings you have shared and will get back on it.

[DaniB-Claroty]

@Vaibhav-1995, did you manage to send alerts from the cloudwatch exporter?
I am having the exact same problem