I only see 12h/13h data on grafana

[dadoMT]

Hi everyone,

can anyone explain why I can't see old data when I do a query from grafana?

I'm using the mimir-distributed helm chart and have NAS as long-term object storage

[dadoMT]

now, I just refreshed and looked at the mimir-mixin dashboards and the ingester is unhealthy again. how do I fix this? :(

Loki logs:

errors in sequence 6h:

{"log":"ts=2024-02-28T22:05:49.070350109Z caller=ingester.go:2554 level=error msg="error while opening existing TSDBs" err="unable to open TSDB for user default_org: failed to open TSDB: /nas_path/mimir/tsdb/default_org: get segment range: segments are not sequential"\n","stream":"stderr","time":"2024-02-28T22:05:49.092762289Z"}

{"log":"ts=2024-02-28T22:05:49.169287567Z caller=mimir.go:869 level=error msg="module failed" module=ingester-service err="starting module ingester-service: invalid service state: Failed, expected: Running, failure: opening existing TSDBs: unable to open TSDB for user default_org: failed to open TSDB: /nas_path/mimir/tsdb/default_org: get segment range: segments are not sequential"\n","stream":"stderr","time":"2024-02-28T22:05:49.192889917Z"}

{"log":"ts=2024-02-28T22:05:49.718880351Z caller=log.go:103 level=error msg="error running application" err="failed services\ngithub.com/grafana/mimir/pkg/mimir.(*Mimir).Run\n\t/__w/mimir/mimir/pkg/mimir/mimir.go:912\nmain.main\n\t/__w/mimir/mimir/cmd/mimir/main.go:227\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"\n","stream":"stderr","time":"2024-02-28T22:05:49.719067516Z"}

how do I fix please?

[dadoMT]

well I fixed the ingester issue by moving the corrupted block. but I want to figure out how it got corrupted in the first place? any ideas please? and can I fix the corrupted data? I have it in my backup folder

aside from that. why is that I only see 12/13h of data?

there's a lot from yesterday that's not showing which is in the long-term storage

[dadoMT]

not sure why no one replied on this discussion, to fix this one. we just moved to a private s3 storage (dell ecs) instead of using NAS.

This fixed the corrupted data issue, as well as the missing data past 12/13h.

Please update your documentation regarding Object Storage (especially filesystem setup)

on the Prometheus Storage Documentation this is included:

CAUTION: Non-POSIX compliant filesystems are not supported for Prometheus' local storage as unrecoverable corruptions may happen. NFS filesystems (including AWS's EFS) are not supported. NFS could be POSIX-compliant, but most implementations are not. It is strongly recommended to use a local filesystem for reliability.

reference: https://prometheus.io/docs/prometheus/latest/storage/

I only see on the logs to use s3 / cloud storage. There's not much reason why you shouldn't use NFS or NAS or any other file systems for production use

It also say to use distributed filesystem from the logs (which we believe is what NAS is). please explain that properly on your documentation

[lianghuiyuan]

I get the same problem.
I know the time is controled by query_ingesters_within: 13h and query_store_after: 12h,
but I have no idea why grafana can't get more data from mimir than 13 hours ago.