Communication with query scheduler failed

[pacias1]

Hello, I have a question about query scheduler and his configuration.

This is configuration of mimir:

alertmanager:
  enabled: false
compactor:
  persistentVolume:
    enabled: false
  resources:
    limits:
      cpu: 1200m
      memory: 2Gi
    requests:
      cpu: 1
      memory: 1Gi

distributor:
  replicas: 3
  resources:
    limits:
      memory: 4Gi
    requests:
      cpu: 2
      memory: 2Gi

ingester:
  persistentVolume:
    size: 50Gi
    storageClass: gp3
  replicas: 4
  resources:
    limits:
      memory: 25Gi
    requests:
      cpu: 4
      memory: 15Gi
  topologySpreadConstraints: {}
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: target
                operator: In
                values:
                  - ingester
          topologyKey: 'kubernetes.io/hostname'
        - labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/component
                operator: In
                values:
                  - ingester
          topologyKey: 'kubernetes.io/hostname'
          
chunks-cache:
  enabled: true
  replicas: 2

index-cache:
  enabled: true
  replicas: 3

metadata-cache:
  enabled: true
results-cache:
  enabled: true

minio:
  enabled: false

overrides_exporter:
  replicas: 1
  resources:
    limits:
      memory: 256Mi
    requests:
      cpu: 100m
      memory: 128Mi
querier:
  replicas: 2
  resources:
    limits:
      memory: 24Gi
    requests:
      cpu: 1
      memory: 12Gi
query_frontend:
  replicas: 2
  resources:
    limits:
      memory: 6Gi
    requests:
      cpu: 2
      memory: 2Gi
ruler:
  enabled: false
store_gateway:
  persistentVolume:
    size: 50Gi
    storageClass: gp3
  replicas: 1
  resources:
    requests:
      cpu: 1
      memory: 6Gi
  topologySpreadConstraints: {}
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: target # support for enterprise.legacyLabels
                operator: In
                values:
                  - store-gateway
          topologyKey: 'kubernetes.io/hostname'

        - labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/component
                operator: In
                values:
                  - store-gateway
          topologyKey: 'kubernetes.io/hostname'
mimir:
  structuredConfig:
    no_auth_tenant: ''
    alertmanager:
      enable_api: false
    ingester:
      ring:
        unregister_on_shutdown: true
    limits:
      max_label_value_length: 4096
      ingestion_rate: 200000
      max_global_series_per_metric: 100000
      max_global_series_per_user: 1000000
      ingestion_burst_size: 400000
      max_query_parallelism: 32
      ingestion_tenant_shard_size: 3
      store_gateway_tenant_shard_size: 1
      compactor_tenant_shard_size: 1
      max_queriers_per_tenant: 1
    frontend:
      results_cache:
        backend: memcached
      cache_results: true
      port: 9095
      parallelize_shardable_queries: true
      log_queries_longer_than: 1s
      align_queries_with_step: true
      querier_forget_delay: 1m
    query_scheduler:
      querier_forget_delay: 1m
      max_outstanding_requests_per_tenant: 600

Mimir is running on EKS + S3. All components are working, we're able to use mimir as datasource in grafana and query metrics from it. However, Some errors occured in frontend and querier logs:

level=error ts=2022-10-17T09:25:47.609652967Z caller=scheduler_processor.go:119 msg="error processing requests from scheduler" err="rpc error: code = Unavailable desc = closing transpor │
│ t due to: connection error: desc = \"error reading from server: EOF\", received prior goaway: code: NO_ERROR" addr=10.12.182.202:9095

Also, this error sometimes occured:

level=warn ts=2022-10-19T07:47:18.486485953Z caller=logging.go:86 traceID=115955b83e8799ae msg="GET /prometheus/api/v1/metadata (500) 405.461µs Response: \"failed to enqueue request\" ws: false; Acce │
│ pt: application/json, text/plain, */*; Accept-Encoding: gzip, deflate, br; Accept-Language: en-US,en;q=0.8; Connection: close; Sec-Fetch-Dest: empty; Sec-Fetch-Mode: cors; Sec-Fetch-Site: same-origin │
│ ; Sec-Gpc: 1; User-Agent: Grafana/8.4.6; X-Amzn-Trace-Id: Root=1-634fab86-1e660e4947d1dd864199e041; X-Forwarded-For: 10.200.0.38, 10.12.118.230, 10.12.118.230, 10.10.1.18; X-Forwarded-Host: <host>; X-Forwarded-Port: 80; X-Forwarded-Proto: http; X-Grafana-Org-Id: 1; X-Real-Ip: 10.10.1.18; X-Scope-Orgid: anonymous; "

What could be the reason of those errors?

[dimitarvdimitrov]

Hi, I think this should be fixed with #3262 and will be released in the 3.2 mimir-distributed chart

as a workaround you can set in your helm values file

query_scheduler:
  extraArgs:
    'server.grpc.keepalive.max-connection-age': 2562047h
    'server.grpc.keepalive.max-connection-age-grace': 2562047h

[pacias1]

Thanks, looks like the problem is gone.