-
Notifications
You must be signed in to change notification settings - Fork 794
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add public, read-only access to label names, label values of any metric #689
Comments
Hmm, I will think about this some. Typically the best practice for exposing metrics like this that may disappear/are from some other system (celery in this case) is to use a custom collector and generate the metrics on each scrape. |
Ok I think I understand but want to confirm.
The logic for deciding if a metric should be produced is then moved inside of the custom collector, correct?
So in our celery example it would mean we check if a worker is offline and if it is we do not produce certain metrics for it rather than removing metrics.
I assume then that each of the multiple metrics that we have has to be turned into a custom collector and follow the rule to not produce a metric for offline worker?
Cheers
On 30 Aug 2021, at 15:30, Chris Marchbanks ***@***.***> wrote:
Hmm, I will think about this some. Typically the best practice for exposing metrics like this that may disappear/are from some other system (celery in this case) is to use a custom collector and generate the metrics on each scrape.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<#689 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AGGVI2MVZRBQ55YRGY2WXSLT7OIYPANCNFSM5CO73U7Q>.
|
Correct, you can see the example in the README: https://github.com/prometheus/client_python/#custom-collectors and add a loop over workers inside |
Hi,
I am currently working on Prometheus metrics for Celery workers in Flower project and we use this wonderful library :) - thx for your work on it!
One issue I came across is lack of direct access to metric's labelnames and labelvalues.
I think easiest is to give you an example how it is used/why it is needed.
We have multiple celery workers in a kubernetes cluster, they are monitored by Flower which generates Prometheus metrics.
Now the pods in k8s can die or be replaced if a new release of our software is deployed.
The problem with that then is that the final reading for each metric for that pod is retained forever and it shows in grafana forever.
We want to remove any tuples of label values which contain a celery worker name that is deemed offline/dead.
I used a semi-hack (mher/flower#1135) and get those label values through metric.collect(), and from that iterate over samples and the labels in them and then if they contain an offline worker I remove the whole tuple from a given metric by calling metric.remove(*labelvalues_containg_dead_worker)
There could be a better way - access
metric._metric.keys()
in a thread safe and read-only way.So I propose to add to
MetricWrapperBase
a read only property, sayall_label_values
(name to be improved, suggestions welcome :))I think I am also badly missing access to actual labelnames set for the given metric to know at which position a label with a given name will be in the labelvalues.
Could kill 2 birds with one stone then and add:
Please let me know if such a change is acceptable and if I am missing any nuances.
If you think you would be ok merging it I will make a PR with tests soon.
Cheers,
Tom
The text was updated successfully, but these errors were encountered: