rpc error: code = Code(429)

[zhuw0312]

mimir version

2.3.1

config file

multitenancy_enabled: false

blocks_storage:
  backend: filesystem
  bucket_store:
    sync_dir: /tmp/mimir/tsdb-sync
  filesystem:
    dir: /tmp/mimir/data/tsdb
  tsdb:
    dir: /tmp/mimir/tsdb

compactor:
  data_dir: /tmp/mimir/compactor
  sharding_ring:
    kvstore:
      store: memberlist

distributor:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: memberlist
  instance_limits:
    max_inflight_push_requests: 200000
    max_ingestion_rate: 0

ingester:
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: memberlist
    replication_factor: 1

ruler_storage:
  backend: filesystem
  filesystem:
    dir: /tmp/mimir/rules

server:
  http_listen_port: 9009
  log_level: error

store_gateway:
  sharding_ring:
    replication_factor: 1

limits:
  max_global_series_per_user: 1000000
  max_global_series_per_metric: 30000
  ingestion_rate: 13333
  ingestion_burst_size: 3333
  ruler_max_rules_per_rule_group: 20
  ruler_max_rule_groups_per_tenant: 70

get error

level=error ts=2022-10-24T11:52:39.63145316Z caller=push.go:136 user=anonymous msg="push error" err="rpc error: code = Code(429) desc = the request has been rejected because the tenant exceeded the ingestion rate limit, set to 13333 items/s with a maximum allowed burst of 3333. This limit is applied on the total number of samples, exemplars and metadata received across all distributors (err-mimir-tenant-max-ingestion-rate). To adjust the related per-tenant limits, configure -distributor.ingestion-rate-limit and -distributor.ingestion-burst-size, or contact your service administrator."

question

• pls teach me how to set the value of -distributor.ingestion-rate-limit and -distributor.ingestion-burst-size correctly ?

[56quarters]

There are docs here: https://grafana.com/docs/mimir/latest/operators-guide/mimir-runbooks/#err-mimir-tenant-max-ingestion-rate

The expected ingestion rate can be determined by: ingestion_rate = number_of_series / scrape_interval_in_seconds. So for example, if you have 150K series and a scrape interval of 60s, your ingestion rate would be 150000 / 60 = 2500. Burst is typically set to 10x the ingestion rate.