-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Automate generation of Custom Resource configuration #1899
Comments
This is a good idea to generate configurations. However, does it require changing codes to add these annotations? Another idea might be generating configurations using k8s client. |
Yes. This would follow the pattern that controller-runtime/controller-tools uses to generate CRDs (see: https://book.kubebuilder.io/reference/markers/crd.html). Essentially we would add markers for metrics in addition to the ones we already have for the CRD generation.
How would this work? (which k8s client do you mean?) |
This will be an opt-in method alternative to writing this file manually; and as described above, it follow a well established pattern in Kubernetes API development (some of those annotations originate from Kuberentes API itself, kubebuilder added more on top) |
Got it. Maybe we can keep kube-state-metrics API (crd-config.yaml) as same before, and this tool helps generating/merging CRD configurations? The tricky cases I considered before are:
For kuberenetes/kubernetes codes, it needs rebuilding binary after changing codes. For other oss components, sometimes we just use it and want to monitor it without changing codes. |
Yup absolutely, that is the idea :) I think for the cases where the code cannot be adjusted it's probably the easiest to write the metrics configuration manually. What we try to solve is essentially that when you are in control of the CRDs and corresponding Go types you can directly mark the metrics on the fields and then you don't have to write the config. But I think if you can't modify the CRD's go types it's hard to find an easier way to get the the config then just writing the config manually. |
Agree. Once this tool becomes popular in the future, we need to consider the case that |
agreed but from a kube-state-metrics nothing will change, everything will start from one (or potentially more) crd-config.yaml |
We would like to use kube-state-metrics with crossplane where we create dozens of CRD's It would be great if kube-state-metrics could provide With such CRD we can define for our every crossplane component CustomResourceStateMetrics configuration saying how to produce metrics |
I would also love to have a CRD for defining the configuration. However IMHO this should be seen as a separate topic and I'd prefer to use a separate issue to track this. This kind of feature was already mentioned multiple times in slack too 🙂 We should keep the issue here scoped to "generating the input" for kube-state-metrics. If the "read configuration from CRD" feature exists: the here proposed generator should get adjusted/improved to (also?) allow creating the CR's. Edit: I will go forward and create a separate issue for this :-) |
I'd like to propose the following UX for the generator (kudo's to @sbueringer and @fabriziopandini which helped brainstorming and compiling this). We did take the metrics at Cluster API and went through the example metrics to try to catch all use-cases. Note:
Metrics:namePrefix// +Metrics:namePrefix=<string> on API type struct Defines the e.g. // +Metrics:namePrefix=capi_cluster
type Cluster struct { ... } Metrics:labelFromPath// +Metrics:labelFromPath:name=<string>,JSONPath=<string> on API type struct Defines a label that applies to all metrics derived from the struct the markers apply to. e.g. // +Metrics:labelFromPath:name=name,JSONPath=".metadata.name"
// +Metrics:labelFromPath:name=namespace,JSONPath=".metadata.namespace"
// +Metrics:labelFromPath:name=uid,JSONPath=".metadata.uid"
type Cluster struct { ... } Metrics:gauge// +Metrics:gauge:name=<string>,help=<string>,nilIsZero=<bool>,JSONPath:<string>,labelFromPath={map[<string>]<string>} on field When applied to an API field it creates a metric of type gauge for the field.
e.g. // +Metrics:gauge:name="spec_paused",help="Whether the cluster is paused and any of its resources will not be processed by the controllers.",nilIsZero=true
Paused bool `json:"paused,omitempty"`
// +Metrics:gauge:name="created",JSONPath='.creationTimestamp",help="Unix creation timestamp."
metav1.ObjectMeta `json:"metadata,omitempty"` Metrics:stateset// + Metrics:stateset:name=<string>, help=<string>, labelName=<string>, list=[]<string>, JSONPath:<string>, labelFromPath={map[<string>]<string>} on field When applied to an API field it creates a metric of type statest for the field.
e.g. // +Metrics:stateset:name="status_phase", help="The clusters current phase.", labelName=phase, list{"Pending", "Provisioning", "Provisioned", "Deleting", "Failed", "Unknown"}
Phase string `json:"phase,omitempty"`
// +Metrics:stateset:name="status_condition", help="The condition of a cluster.", labelName="status", JSONPath: ".status", list{"True", "False", "Unknown"}, labelsFromPath={type: ".type"}
Conditions Conditions `json:"conditions,omitempty"` Metrics:info// +Metrics:info:name=<string>,help=<string>,JSONPath:<string>, labelFromPath={map[<string>]<string>} on field or struct When applied to an API field it creates a metric of type info for the field.
e.g. // +Metrics:info:name="info",help="Information about a cluster.",labelsFromPath={topology_version: ".spec.topology.version", topology_class: ".spec.topology.class"}
// Cluster is the Schema for the clusters API.
type Cluster struct {
// +Metrics:info:name="annotation_paused", JSONPath='.annotations.['cluster\\.x-k8s.io/paused']", help="Whether the cluster is paused and any of its resources will not be processed by the controllers.", labelsFromPath={paused_value: "."}
metav1.ObjectMeta `json:"metadata,omitempty"`
} |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
/remove-lifecycle stale |
Add alternative kubernetes#1899
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
/remove-lifecycle stale |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
/remove-lifecycle rotten |
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
/remove-lifecycle stale |
What would you like to be added:
As a developer, I would like the kube-state-metrics Custom Resource configuration to be automatically generated from the API code.
Thanks to @sbueringer and @fabriziopandini for helping in driving this efforts 👍
Why is this needed:
kube-state-metrics requires a configuration file for Custom Resources which could grow to a very large file and may be hard to maintain over long-term together with changing Custom Resource definitions.
Instead of writing the configuration file manually it would be awesome if the configuration could get generated from the code where the custom resource gets defined instead.
Describe the solution you'd like
Prior art:
Kubebuilder makes use of a tool called controller-gen to generate the yaml's for e.g. Custom Resource definitions from code.
To do that it makes heavy use of markers inside the go code, which are comments in the form of
// +foo:bar:key1=value1,key2=value2
.Similar to
controller-gen
, kube-state-metrics could provide an additional tool to generate the custom resource metrics configuration file from pre-defined markers at the go code level.Additional context
We already thought about adding a tool like this at the Cluster API project, but we think it would be a better fit for the kube-state-metrics project, as it could help authors of any custom resource and is not specific to Cluster API.
We also have an idea how the markers / design could look like, which builds on top of the currently manually written configuration in Cluster API: kubernetes-sigs/cluster-api#7158 (comment) .
I could also migrate the first implementation design idea over to this issue or however it would be decided to proceed.
If the maintainers of kube-state-metrics also think that the described idea would be worth to implement: I'd be of course happy to volunteer to help or contribute for this efforts.
Small example:
Could result in the following configuration file
The text was updated successfully, but these errors were encountered: