-
Notifications
You must be signed in to change notification settings - Fork 752
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Problems with prometheus on OTel 1.8 #5613
Labels
bug
Something isn't working
pkg:OpenTelemetry.Exporter.Prometheus.AspNetCore
Issues related to OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package
Comments
reyang
added
pkg:OpenTelemetry.Exporter.Prometheus.AspNetCore
Issues related to OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package
and removed
area:exporter
labels
May 14, 2024
Today I had the same problem after upgrading from version 1.6.0 to 1.8.1. (.NET 8) |
@reyang Can we reopen this issue? Today I tried to use OTel 1.9 but still I have problems with metrics. Prometheus logs:
When rollback to 1.7 everything looks good. Is this a bug or we have to use new prometheus to handle metrics from 1.8+? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
bug
Something isn't working
pkg:OpenTelemetry.Exporter.Prometheus.AspNetCore
Issues related to OpenTelemetry.Exporter.Prometheus.AspNetCore NuGet package
Area
area:exporter:prometheus
Package Version
Runtime Version
net8.0
Description
After update to new Otel 1.8 we are expirience issues in our prometheus. In one version it says "Error on ingesting samples that are too old or are too far into the future" and after cross check with new version of prometheus it says "Error on ingesting samples with different value but same timestamp". Additionally we observe that on some random queries to /metrics (we are getting otel_scope_info that is missing timestamp like described here prometheus/prometheus#6554). Like below
If I'm getting this right this info should be returned only when "Accept: application/openmetrics-text" is send but maybe I'm wrong. Additionally this info is random - one query return metrics with this info and another without.
If this behavior is by design please share some documentation.
ps. After downgrade to 1.7 we don't have those errors
Steps to Reproduce
Just use standard OTel and serve prometheus endpoint
Expected Result
Prometheus agent is grabbing data with success
Actual Result
Errors from description
Additional Context
No response
The text was updated successfully, but these errors were encountered: