tap-prometheus

This is a Singer tap that produces JSON-formatted data following the Singer spec.

Module loads PromQL query result and pushes the result as a record.

Sample config to calculate daily online peak by customer and environment:

{
  "endpoint": "http://localhost:9000",
  "auth": {
    "username": "prometheus",
    "password": "123"
  },
  "start_date": "2018-11-01T00:00:00Z",
  "metrics": [
    {
      "name": "online_peak",
      "query": "sum(sessions_count)",
      "batch": "10000",
      "step": "120s", 
      "labels": {
        "type": "object",
        "properties": {
          "customer": {
            "type": "string"
          },
          "environment": {
            "type": "string"
          }
        }
      }
    }
  ]
}

The top-level configuration settings for this tap are

• endpoint required: The Prometheus API endpoint URL - do not include the obligatory /api/v1 suffix in here, the tap will add this itself
• auth optional: HTTP basic auth username and password
• start_date required: Earliest date from which to start collecting data
• metrics required: Definition of individual prometheus metric query streams

This tap emits a stream for every metric definition.

• query: PromQL query, must be a range query
• batch: The maximum number of steps to move forward in the stream in one batch
• step: metrics resolution in seconds
• labels: JSON Schema for labels returned by the query. You can set this to {"type": "null"} if your query does not return any labels.

You can tune batch size to your prometheus instance and singer tap memory requirements. For typical setups it should be fine to fetch up to 10.000 data points per prometheus API query.

Several source code parts copied from:

• tap-stripe: https://github.com/singer-io/tap-stripe

Module based on the patched python Prometheus API client promalyze: https://github.com/meshcloud/promalyze

Capabilities

This tap supports incremental replication. It will run collection until the current extraction time rounded down to the last full step (see configuration). If you're using tap with meltano, configure it with the following capabilities

capabilities:
- catalog
- discover
- state