New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
0.35: Panic with query mode distributed #7328
Comments
The issue here was that the querier is not configured to point at query APIs but at store APIs ( we should guard against that better probably ); this leads to the promql-engine distributing to 0 other engines, which exposes a bug where we dont guard against that! Edit: see https://cloud-native.slack.com/archives/CK5RSSC10/p1714654267669009 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Thanos, Prometheus and Golang version used: 0.35.0
Object Storage Provider: Azure Storage Account
What happened:
After enable
query.mode=distributed
my querier get a lot of panics. Removing--query.mode=distributed
stops all pancisWhat you expected to happen:
No panics
How to reproduce it (as minimally and precisely as possible):
At the moment, I'm unable to provide a minimal reproducible environment. However, according ruler logs, all queries like
absent(up{job=\"kube-proxy\"} == 1)
(job can have any label) seems affected.We are using stateless rulers and thanos receive, no sidecars.
Thanos querier arguments:
Full logs to relevant components:
Uncomment if you would like to post collapsible logs:
Ruler Logs
Just a few, they are repeating
Querier Logs
https://gist.github.com/jkroepke/9fc58319bf819866138a8dae4f1c8d92
Anything else we need to know:
Environment:
The text was updated successfully, but these errors were encountered: