[FeatureRequest] Monitoring metrics for supervisors status

[layoaster]

Description

Currently, there is no way of monitoring the state of supervisors' health via monitoring metrics.

I would like to request a metric (or set of metrics) to report the following:

• Supervisor health status: whether a supervisor is healthy or not. (Druid API).
• Supervisor current state/status: e.g.: RUNNING, SUSPENDED, etc ... (List of possible states)

Motivation

• To be able to monitor the status of Druid supervisors via monitoring metrics.
• Supervisor is a critical piece of Druid when performing streaming ingestion.

[ashwintumma23]

A caveat we should definitely consider in this request is how we can represent string labels as metrics.

• Most of the metrics aggregation systems work great with numerical values, but with string values things start getting complex. For instance, Prometheus supports count, histogram, gauge, and summary type of metrics (https://prometheus.io/docs/concepts/metric_types/).

• One approach can be encode strings as numerical values, like RUNNING state maps to 1, SUSPENDED maps to 2, etc.

• The current supported metrics on Druid as well have numerical format only: https://druid.apache.org/docs/latest/operations/metrics/

[layoaster]

@ashwintumma23 AFAIK this is not an issue for Prometheus.

A sample implementation could rely on a Prometheus gauge metric that have two possible values: 0 or 1 and leverage "dimensions" (Prometheus labels) to store the different states RUNNING, SUSPENDED, etc as strings and also a separate label for the supervisor id/name. (https://prometheus.io/docs/concepts/data_model/)

The value 1 corresponds to the current/active state of any given supervisor.