Mimir Distributor: "received a series with duplicate label name"

[cabrinha]

Describe the bug

Mimir distributors are seeing metrics with duplicate label names when receiving metrics from HA prometheus agents

To Reproduce

Steps to reproduce the behavior:

1. Start Mimir (2.12.0)
2. Perform Operations(Read/Write/Others)
3. Remote Write from Prometheus HA w/ agents

Expected behavior

Expecting not to see duplicated labels.

Environment

• Infrastructure: Kubernetes 1.29 EKS
• Deployment tool: helm

Additional Context

Mimir HA tracker config:

    distributor:
      ha_tracker:
        enable_ha_tracker: true
        kvstore:
          etcd:
            endpoints:
            - mimir-etcd-headless.mimir.svc.cluster.local:2379
          store: etcd
      ring:
        heartbeat_period: 1m
        heartbeat_timeout: 4m
    limits:
      accept_ha_samples: true
      drop_labels:
      - prom_shard
      ha_cluster_label: prom_shard
      ha_replica_label: prometheus_replica

Prometheus config:

global:
  external_labels:
    clustername: < kubernetes cluster name >
    prom_shard: < kubernetes cluster name >-1
    prometheus_replica: prom-agent-< team >-agent-shard-1-0

Distributor Logs:

received a series with duplicate label name, label: 'service' series: kubelet_graceful_shutdown_start_time_seconds
received a series with duplicate label name, label: 'prometheus_replica' series: kube_ingress_created
received a series with duplicate label name, label: 'prom_shard' series: container_cpu_usage_seconds_total
received a series with duplicate label name, label: 'zone' series: coredns_forward_request_duration_seconds_bucket
received a series with duplicate label name, label: 'namespace' series: rest_client_request_duration_seconds_bucket
received a series with duplicate label name, label: 'node' series: container_cpu_usage_seconds_total

All components (nodes/pods) are running in a single AZ.

[56quarters]

This is configuration identical between both HA replicas?

global:
  external_labels:
    clustername: < kubernetes cluster name >
    prom_shard: < kubernetes cluster name >-1
    prometheus_replica: prom-agent-< team >-agent-shard-1-0

If so, the intention of the prometheus_replica label is to differentiate between the two HA replicas. So for example, one replica would have the configuration:

global:
  external_labels:
    clustername: < kubernetes cluster name >
    prom_shard: < kubernetes cluster name >-1
    prometheus_replica: prom-agent-< team >-agent-shard-1-0

While the other would have

global:
  external_labels:
    clustername: < kubernetes cluster name >
    prom_shard: < kubernetes cluster name >-1
    prometheus_replica: prom-agent-< team >-agent-shard-1-1

Mimir would then pick one of them to accept metrics from.