mimir does not comes up with error out-of-order series added

[suikast42]

Describe the bug

after some monkey testing with startups and stops on all runngin containers the mimir container give up work with error

ts=2023-03-11T23:05:26.46054743Z caller=ingester.go:2005 level=error msg="error while opening existing TSDBs" err="unable to open TSDB for user anonymous: failed to compact TSDB: /data/tsdb/anonymous: compact head: persist head block: populate block: adding series: asyncBlockWriter doesn't run anymore: add series: out-of-order series added with label set "{name="consul_catalog_service_node_healthy", agent_hostname="master-01", instance="master-01", job="integrations/consul_exporter", node="worker-01", service_id="_nomad-task-0bc0dfa2-6d72-7629-b440-1f567871da20-group-keycloak-keycloak-8080", service_name="keycloak"}""

[pstibrany]

Hello,

Note that out-of-order support is still an experimental feature.

To troubleshoot your problem, we would need to inspect content of your TSDB directory at /data/tsdb/anonymous.

If you want to unstuck your ingester, you can delete content of this directory and restart the ingester. Assuming you use replication factor of 3, this is safe if done on single ingester.

[dhilgarth]

@pstibrany
I had this issue just now. To unstuck, I've removed the data from the ingester directory, but I've retained it, to maybe be able to understand what caused it and prevent it in the future.
You mentioned that you would need to inspect the contents of the TSDB directory. What exactly would you be looking for?
Also, maybe important: I don't have out-of-order support enabled. I'm on Mimir 2.12.0

[suikast42]

It is to late I have already delete it and I have single node ( monotlithic ) dev instance.