memberlist problem in k8s

[thel0]

Hello
i would like to test mimir in order to replace thanos in my k8s cluster

• k8s 1.25
• cilium for cni
• IPv6 disabled

mimir is installed with helm but i have lot of problem with memberlist

ts=2022-11-20T20:16:39.722165927Z caller=memberlist_logger.go:74 level=info msg="Suspect mimir-distributor-5b5c545445-jsdt5-28b63e57 has failed, no acks received"                                                                                               │
│ ts=2022-11-20T20:16:39.724090092Z caller=memberlist_logger.go:74 level=warn msg="Got ping for unexpected node 'mimir-ruler-bd6fc5d9b-g227l-42b279d4' from=:7946"                                                                                                 │
│ ts=2022-11-20T20:16:49.719162725Z caller=memberlist_logger.go:74 level=info msg="Marking mimir-ruler-bd6fc5d9b-g227l-42b279d4 as failed, suspect timeout reached (0 peer confirmations)"                                                                         │
│ ts=2022-11-20T20:17:19.723273849Z caller=memberlist_logger.go:74 level=info msg="Suspect mimir-ruler-bd6fc5d9b-g227l-42b279d4 has failed, no acks received"                                                                                                      │
│ ts=2022-11-20T20:17:19.725356114Z caller=memberlist_logger.go:74 level=warn msg="Got ping for unexpected node 'mimir-querier-9d9849484-4txfl-da58f0b2' from=:7946"                                                                                               │
│ ts=2022-11-20T20:17:24.721726203Z caller=memberlist_logger.go:74 level=info msg="Marking mimir-ingester-2-b47d4e29 as failed, suspect timeout reached (0 peer confirmations)"                                                                                    │
│ ts=2022-11-20T20:17:59.721572557Z caller=memberlist_logger.go:74 level=info msg="Marking mimir-querier-9d9849484-4txfl-da58f0b2 as failed, suspect timeout reached (0 peer confirmations)"                                                                       │
│ ts=2022-11-20T20:17:59.724885875Z caller=memberlist_logger.go:74 level=info msg="Suspect mimir-querier-9d9849484-4txfl-da58f0b2 has failed, no acks received"                                                             

I dont see others pod in member list

As you can see, i have problem with memberlist. Does someone have an idea ?

For information, all pod can ping other and can resolve mimir-gossip-ring

my values.yaml file

---
global:
  extraEnvFrom:
    - secretRef:
        name: my-bucket-secret

  extraEnv:
    - name: MY_POD_IP
      valueFrom:
        fieldRef:
          fieldPath: status.podIP

minio:
  enabled: false

mimir:
  structuredConfig:
    blocks_storage:
      backend: s3
      s3:
        bucket_name: my-blocks-bucket
        access_key_id: "thanosbucket"
        endpoint: minio.minio.svc.cluster.local:9000
        secret_access_key: "XXXXXX"
        insecure: true

distributor:
  extraArgs:
    memberlist.bind-addr: ${MY_POD_IP}

ingester:
  extraArgs:
    memberlist.bind-addr: ${MY_POD_IP}

querier:
  extraArgs:
    memberlist.bind-addr: ${MY_POD_IP}

compactor:
  extraArgs:
    memberlist.bind-addr: ${MY_POD_IP}

alertmanager:
  extraArgs:
    memberlist.bind-addr: ${MY_POD_IP}

ruler:
  extraArgs:
    memberlist.bind-addr: ${MY_POD_IP}

store-gateway:
  extraArgs:
    memberlist.bind-addr: ${MY_POD_IP}

thanks for your help
thel

[pstibrany]

Hi. memberlist.bind-addr option specifies address that Mimir will listen on for incoming memberlist messages. You may also want to configure memberlist.advertise-addr so that other nodes know what address to talk to.