Metadata endpoint does not return data

[Demivan]

I'm trying to setup Mimir with Grafana. Everything is working fine except that Grafana metric dropdown does not work:

I can see that Grafana calls /prometheus/api/v1/metadata and gets just { "status": "success" } in response. So no info about metrics.

I cannot find if I need to enable something in Mimir to make it work. Am I missing something?

[pstibrany]

Hello, Prometheus v2.15+ or Grafana Agent are responsible for pushing metadata into Prometheus, unless you have

metadata_config:
  send: false

in your remote write configuration.

There is no special flag to enable metadata in Mimir.

Another possibility I can think of is querying wrong tenant (not the one where Prometheus pushes samples and metadata)

[Demivan]

Thanks for the info. I'm actually using OpenTelemetry Collector and write to Mimir directly, without writing to Prometheus first. Will investigate whether Collector is pushing metadata. Strange that it was working with Prometheus.

[pstibrany]

Strange that it was working with Prometheus.

Prometheus sends metadata to remote-write targets by default, unless explicitly disabled by option mentioned in #2078 (comment).

[Demivan]

I meant that it is strange that it was working in Grafana when I was using OpenTelemetry Collector with Prometheus instead of Mimir.

[pstibrany]

I meant that it is strange that it was working in Grafana when I was using OpenTelemetry Collector with Prometheus instead of Mimir.

Ah, I see, sorry for misunderstanding. Could it be because Prometheus was doing the scraping, instead of collector pushing data to Prometheus? Prometheus scraping requires that metadata (TYPE and HELP lines) exists.

[Demivan]

I have tested this with clean Prometheus installation - same problem. So I must have used scraping at some point and that populated the metadata.