how to set ingestion_rate params

[kwenzh]

i deploy a mimir cluster for prometheus remote write. Apply it to product environment, i find these:

distributor has some error log:

level=error ts=2022-11-08T11:34:18.287564438Z caller=push.go:136 user=anonymous msg="push error" err="rpc error: code = Code(429) desc = the request has been rejected because the tenant exceeded the ingestion rate limit, set to 100000 items/s with a maximum allowed burst of 2000000. This limit is applied on the total number of samples, exemplars and metadata received across all distributors (err-mimir-tenant-max-ingestion-rate). To adjust the related per-tenant limits, configure -distributor.ingestion-rate-limit and -distributor.ingestion-burst-size, or contact your service administrator.

mimir config about limits:

 limits:
      ingestion_burst_size: 2000000
      ingestion_rate: 100000
      max_global_series_per_user: 100000000
      max_label_names_per_series: 50
      max_query_parallelism: 224

prometheus remote wirte confg:

remote_write:
- url: http://sdc-mimir-nginx.mimir.svc:80/api/v1/push
  remote_timeout: 30s
  follow_redirects: true
  queue_config:
    capacity: 2500
    max_shards: 200
    min_shards: 1
    max_samples_per_send: 500
    batch_send_deadline: 5s
    min_backoff: 30ms
    max_backoff: 100ms
  metadata_config:
    send: true
    send_interval: 1m

How should I determine the concurrency rate？ Does it cause some series to be lost when written to mimir?

[kwenzh]

even so these config ,there alway are some error log "the request has been rejected because the tenant exceeded the ingestion rate limit, " in distributors

[Logiraptor]

These limits exist as a way to prevent the Mimir cluster from misbehaving clients. If the cluster is scaled to handle more, it's perfectly safe to increase the limit. You can determine an appropriate rate with the expected number of series and scrape interval.

For example, 5 million series with a 15s scrape interval means 20 million samples per minute or ~334k samples per second. Ingestion burst size is typically set to 10x the ingestion rate.

[kwenzh]

These limits exist as a way to prevent the Mimir cluster from misbehaving clients. If the cluster is scaled to handle more, it's perfectly safe to increase the limit. You can determine an appropriate rate with the expected number of series and scrape interval.

For example, 5 million series with a 15s scrape interval means 20 million samples per minute or ~334k samples per second. Ingestion burst size is typically set to 10x the ingestion rate.

ok thank you， i find doc about How to estimate the rate of samples per second, 1800 w active series,
1800w series with a 30s scrape interval means 60w sample per second.

https://grafana.com/docs/mimir/latest/operators-guide/run-production-environment/planning-capacity/#ingester

[kwenzh]

the full config file mimir.yaml

activity_tracker:
  filepath: /data/metrics-activity.log
alertmanager:
  data_dir: /data
  enable_api: true
  external_url: /alertmanager
alertmanager_storage:
  backend: s3
  s3:
    access_key_id: xxxx
    bucket_name: mimir-product-ruler
    endpoint: xxxxx
    insecure: true
    secret_access_key: xxxxx
blocks_storage:
  backend: s3
  bucket_store:
    chunks_cache:
      backend: memcached
      memcached:
        addresses: dns+sdc-mimir-chunks-cache.mimir.svc:11211
        max_item_size: 5242880
        timeout: 450ms
    index_cache:
      backend: memcached
      memcached:
        addresses: dns+sdc-mimir-index-cache.mimir.svc:11211
        max_item_size: 5242880
    metadata_cache:
      backend: memcached
      memcached:
        addresses: dns+sdc-mimir-metadata-cache.mimir.svc:11211
        max_item_size: 5242880
    sync_dir: /data/tsdb-sync
  s3:
    access_key_id: xxxxx
    bucket_name: mimir-product-tsdb
    endpoint: xxxxx
    insecure: true
    secret_access_key: xxxxxp
  tsdb:
    dir: /data/tsdb
compactor:
  data_dir: /data
frontend:
  align_queries_with_step: true
  cache_results: true
  log_queries_longer_than: 10s
  parallelize_shardable_queries: true
  results_cache:
    backend: memcached
    memcached:
      addresses: dns+sdc-mimir-results-cache.mimir.svc:11211
      max_item_size: 5242880
  scheduler_address: sdc-mimir-query-scheduler-headless.mimir.svc:9095
frontend_worker:
  scheduler_address: sdc-mimir-query-scheduler-headless.mimir.svc:9095
ingester:
  ring:
    final_sleep: 0s
    num_tokens: 512
    tokens_file_path: /data/tokens
    unregister_on_shutdown: false
ingester_client:
  grpc_client_config:
    max_recv_msg_size: 104857600
    max_send_msg_size: 104857600
limits:
  ingestion_burst_size: 2000000
  ingestion_rate: 100000
  max_global_series_per_user: 100000000
  max_label_names_per_series: 50
  max_query_parallelism: 224
memberlist:
  abort_if_cluster_join_fails: false
  compression_enabled: false
  join_members:
  - dns+sdc-mimir-gossip-ring.mimir.svc.cluster.local:7946
querier:
  max_concurrent: 16
query_scheduler:
  max_outstanding_requests_per_tenant: 1600
ruler:
  alertmanager_url: dnssrvnoa+http://_http-metrics._tcp.sdc-mimir-alertmanager-headless.mimir.svc.cluster.local/alertmanager
  enable_api: true
  rule_path: /data
ruler_storage:
  backend: s3
  s3:
    access_key_id: xxxxxxxx
    bucket_name: mimir-product-ruler
    endpoint: xxxxxxxxxxxxx
    insecure: true
    secret_access_key: xxxxxxxxxx
runtime_config:
  file: /var/mimir/runtime.yaml
server:
  grpc_server_max_concurrent_streams: 1000
  grpc_server_max_connection_age: 2m
  grpc_server_max_connection_age_grace: 5m
  grpc_server_max_connection_idle: 1m
store_gateway:
  sharding_ring:
    replication_factor: 2
    tokens_file_path: /data/tokens