prometheus

Monitoring Prometheus with OpenTelemetry Collector

This simple example demonstrates monitoring prometheus sources with the OpenTelemetry collector, using the prometheus receiver and sending the data to New Relic via OTLP. A simple prometheus data generator is configured to generate dummy metrics scraped by prometheus receiver.

Requirements

• You need to have a Kubernetes cluster, and the kubectl command-line tool must be configured to communicate with your cluster. Docker desktop includes a standalone Kubernetes server and client which is useful for local testing.
• A New Relic account
• A New Relic license key

Running the example

1. Update the NEW_RELIC_API_KEY value in secrets.yaml to your New Relic license key.

   # ...omitted for brevity
   stringData:
     # New Relic API key to authenticate the export requests.
     # docs: https://docs.newrelic.com/docs/apis/intro-apis/new-relic-api-keys/#license-key
     NEW_RELIC_API_KEY: <INSERT_API_KEY>

   • Note, be careful to avoid inadvertent secret sharing when modifying secrets.yaml. To ignore changes to this file from git, run git update-index --skip-worktree k8s/secrets.yaml.

   • If your account is based in the EU, update the NEW_RELIC_OTLP_ENDPOINT value in collector.yaml the endpoint to: https://otlp.eu01.nr-data.net

   # ...omitted for brevity
   env:
    # The default US endpoint is set here. You can change the endpoint and port based on your requirements if needed.
    # docs: https://docs.newrelic.com/docs/more-integrations/open-source-telemetry-integrations/opentelemetry/best-practices/opentelemetry-otlp/#configure-endpoint-port-protocol
    - name: NEW_RELIC_OTLP_ENDPOINT
      value: https://otlp.eu01.nr-data.net

2. Run the application with the following command.

   kubectl apply -f k8s/

   • When finished, cleanup resources with the following command. This is also useful to reset if modifying configuration.

   kubectl delete -f k8s/

Viewing your data

To review your statsd data in New Relic, navigate to "New Relic -> Query Your Data". To list the metrics reported, query for:

FROM Metric SELECT uniques(metricName) WHERE otel.library.name = 'otelcol/prometheusreceiver' LIMIT MAX

See get started with querying for additional details on querying data in New Relic.

Additional notes

This example monitors a simple prometheus data generator instance defined in prometheus-data-generator.yaml. To use in production, you'll need to modify the .receivers.prometheus.config.scrape_configs value in collector.yaml ConfigMap to point at your prometheus sources.

The prometheus receiver includes service.name and service.instance.id resource attributes derived from job name and target configured in .receivers.prometheus.config.scraep_configs. As documented here, New Relic considers any data with service.name as a service despite the fact that not all prometheus data sources are services. As a result, you can find a prometheus_data_generator entity under "New Relic -> All Entities -> Services - OpenTelemetry", although the panels will not contain data because the scraped metrics do not represent APM data.