Can Mimir AlertManager be used for Loki Ruler?

[hobbsh]

Hi,

I have been unable to get alerts to actually notify when using Alert Rules in the Loki helm chart, pointing alertmanager_url at the Mimir Alertmanager instance. I've searched through the docs extensively but I can't seem to uncover what I'm missing. Is there a catch or is this just not designed to work?

I can see rule evaluating logs in the ruler and the alert is even Firing in the Grafana Alerting dashboard, but nothing is being notified and I have not found evidence of errors/misconfiguration anywhere.

[thmshmm]

hey, it should work, using it in this setup as well.

loki:
  structuredConfig:
    auth_enabled: true
    ruler:
      alertmanager_url: http://lgtm-distributed-mimir-alertmanager:8080/alertmanager

Do you have multi-tenancy enabled everywhere? Loki in that case forwards the X-Scope-OrgID header.

[hobbsh]

So I think remote_write was part of the issue, but now we have a tenant mismatch because the default tenant is anonymous on Mimir and fake on Loki. How is this typically reconciled? Should we set the tenant name to anonymous on the ruler remote_write clients?

[pstibrany]

If your Mimir is configured with multitenancy disabled, I'd expect that Mimr ignores the tenant coming from Loki.

Looking at the remote-write config options in Loki ruler, I'd perhaps try setting remote_write.add_org_id_header option to false, so that Loki doesn't report the tenant to alertmanager. (But I have no direct experience trying to setup Loki ruler with Mimir alertmanager, just guessing here.)

[hobbsh]

It looks like that option is only available in Loki 3.x - we are still on 2.9 because the loki-distributed chart does not support the new version yet.

[zktaiga]

Thought I'd chime in since I've made it work, was in a similar situation: different default tenants between Mimir/Loki. Ended up aligning the tenant names to match, and then just setting alertmanager_url.

By the way, I don't think remote_write has anything to do here. Rules are evaluated by the Loki Ruler and then sent to Mimir's Alertmanager (they don't go through Mimir). As far as I can see, remote_write just generates metrics and pushes them to Mimir for collection.

My 2 cents with the loki chart in simple scalable mode:

loki:
  auth_enabled: true
  rulerConfig:
    wal:
      dir: /var/loki/ruler-wal
    alertmanager_url: http://mimir-alertmanager.mimir.svc:8080/alertmanager
    external_url: https://grafana.com
    enable_alertmanager_v2: true
    enable_api: true
    external_labels:
      origin: loki
    remote_write:
      enabled: true
      client:
        url: http://mimir-nginx.mimir.svc:80/api/v1/push

This way, Grafana can connect to Loki's ruler API, you can create datasource-managed alerts (Loki ruler), they'll be created in object storage and then loaded by the ruler. Hope this helps!

[hobbsh]

Ok so what I had to do since I don't have access to disabling the org id header for the Loki ruler is to just add an AlertManager config for the fake tenant with mimirtool. Maybe I needed this either way, but with this remote_write config and a duplicate config for the fake tenant (just passing --id=fake to mimirtool alertmanager load like we do for anonymous tenant), it works!

        ruler:
          enable_api: true
          alertmanager_url: http://mimir-distributed-alertmanager.mimir:8080/alertmanager
          enable_alertmanager_v2: true
          enable_sharding: true
          remote_write:
            enabled: true
            clients:
              fake:
                url: http://mimir-distributed-gateway.mimir:8080/api/v1/push

[applejag]

We also used Mimir and Loki without multi-tenancy and was struck by this issue. We resolved it by adding this to our mimir-distributed Helm chart values:

gateway:
  nginx:
    config:
      httpSnippet: |
        # Force override the X-Scope-OrgID header
        # as clients like Loki tries to use "fake"
        # but a Mimir without tenants should use "anonymous"
        map $http_x_scope_orgid $http_x_scope_orgid {
          "" "anonymous";
        }

This adds a snippet to the Nginx config used in the gateway so it always overrides the X-Scope-OrgID with anonymous. This overrides the default behavior of using the existing value if it exists:

mimir/operations/helm/charts/mimir-distributed/values.yaml

Lines 3484 to 3492 in 7ef5c82

	{{- with .Values.gateway.nginx.config.httpSnippet }}
	{{ . | nindent 2 }}
	{{- end }}
	# Ensure that X-Scope-OrgID is always present, default to the no_auth_tenant for backwards compatibility when multi-tenancy was turned off.
	map $http_x_scope_orgid $ensured_x_scope_orgid {
	default $http_x_scope_orgid;
	"" "{{ include "mimir.noAuthTenant" . }}";
	}

Then in our Loki config we point loki-ruler to the Mimir gateway:

rulerConfig:
  rule_path: /tmp/loki/rules-temp
  storage:
    type: local
    local:
      directory: /etc/loki/rules
  external_labels:
    category: logs
  ring:
    kvstore:
      store: memberlist
  alertmanager_url: http://mimir-gateway.mimir.svc.cluster.local:80/alertmanager

To verify that this works, I created some Loki alerts that will always fire and then port-forwarded the Mimir gateway (kubectl port-forward svc/mimir-gateway 8080:80) and tried to access the Alertmanager (http://localhost:8080/alertmanager/#/alerts) to see that the alerts show up there as well, together with all of our Prometheus alerts from mimir-ruler.

[hobbsh]

Cool solution, thanks for sharing!