Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: prometheus remote write and read #346

Merged
merged 11 commits into from
Oct 28, 2022
Merged

Conversation

killme2008
Copy link
Contributor

@killme2008 killme2008 commented Oct 25, 2022

Intro

Related to #313 .

It's ready for review.

Main changes:

  • Impl prometheus remote write
  • Impl prometheus remote read, but only samples response. The stream protocol is designed for prometheus itself (communication between prometheus instances), it's not enabled and useful for third-parties remote storage.

Todo:

  • push gateway protocol

How to test

  1. Adds remote write and read config to prometheus.yml:
remote_write:
- url: http://localhost:4000/v1/prometheus/write

remote_read:
- url: http://localhost:4000/v1/prometheus/read
  read_recent: true

Enable read_recent to test remote reading.

  1. Start datanode and frontend:
$ cargo run -- datanode start

$ cargo run -- frontend start
  1. Start prometheus by ./prometheus --config.file=prometheus.yml --log.level=debug.Then frontend will receive the remote write reqeusts and create many tables automatically in datanode.
  2. Access prometheus http://localhost:9090/graph to test querying. If you want to test remote reading, you can delete the data folder in prometheus, and restart it, it will read historical data from greptimedb.

src/servers/src/query_handler.rs Show resolved Hide resolved
src/servers/src/prometheus.rs Outdated Show resolved Hide resolved
src/servers/src/prometheus.rs Outdated Show resolved Hide resolved
src/servers/src/prometheus.rs Outdated Show resolved Hide resolved
src/servers/src/http.rs Outdated Show resolved Hide resolved
src/frontend/src/instance/prometheus.rs Show resolved Hide resolved
@yuanbohan yuanbohan self-requested a review October 26, 2022 03:00
@yuanbohan yuanbohan marked this pull request as ready for review October 26, 2022 03:01
@yuanbohan
Copy link
Contributor

When the table does not exist, such as prometheus_remote_storage_sent_batch_duration_seconds, frontend returns 500 Error.

remote_read: remote server http://localhost:4000/v1/prometheus/remote/read returned HTTP status 500 Internal Server Error: {"error":"Failed to execute query: select * from prometheus_remote_storage_sent_batch_duration_seconds where greptime_timestamp>=1666753346745 and greptime_timestamp<=1666753646745 order by greptime_timestamp, source: Error occurred on the data node, code: 3000, msg: Failed to execute sql, source: Cannot plan SQL: SELECT * FROM prometheus_remote_storage_sent_batch_duration_seconds WHERE greptime_timestamp >= 1666753346745 AND greptime_timestamp <= 1666753646745 ORDER BY greptime_timestamp, source: Error during planning: Table or CTE with name 'prometheus_remote_storage_sent_batch_duration_seconds' not found"}

@codecov
Copy link

codecov bot commented Oct 26, 2022

Codecov Report

Merging #346 (b6a4102) into develop (64dac51) will increase coverage by 0.05%.
The diff coverage is 84.87%.

@@             Coverage Diff             @@
##           develop     #346      +/-   ##
===========================================
+ Coverage    84.28%   84.33%   +0.05%     
===========================================
  Files          350      365      +15     
  Lines        32109    33970    +1861     
===========================================
+ Hits         27063    28649    +1586     
- Misses        5046     5321     +275     
Flag Coverage Δ
rust 84.33% <84.87%> (+0.05%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
src/api/src/lib.rs 100.00% <ø> (ø)
src/frontend/src/instance.rs 86.12% <ø> (ø)
src/frontend/src/instance/opentsdb.rs 88.23% <ø> (ø)
src/frontend/src/lib.rs 100.00% <ø> (ø)
src/frontend/src/server.rs 0.00% <ø> (ø)
src/servers/src/error.rs 25.71% <0.00%> (-3.80%) ⬇️
src/servers/src/lib.rs 100.00% <ø> (ø)
src/servers/src/opentsdb/codec.rs 92.12% <ø> (ø)
src/frontend/src/frontend.rs 32.25% <50.00%> (+1.22%) ⬆️
src/frontend/src/prometheus.rs 70.00% <70.00%> (ø)
... and 77 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@killme2008 killme2008 marked this pull request as draft October 26, 2022 03:38
@killme2008
Copy link
Contributor Author

It's still under development, not ready for review, but thanks for your CR @yuanbohan

@killme2008 killme2008 marked this pull request as ready for review October 27, 2022 13:10
src/frontend/src/instance/prometheus.rs Outdated Show resolved Hide resolved
src/frontend/src/instance/prometheus.rs Outdated Show resolved Hide resolved
src/frontend/src/prometheus.rs Outdated Show resolved Hide resolved
src/servers/src/error.rs Outdated Show resolved Hide resolved
src/servers/src/error.rs Outdated Show resolved Hide resolved
src/servers/src/prometheus.rs Outdated Show resolved Hide resolved
src/servers/src/prometheus.rs Outdated Show resolved Hide resolved
src/servers/src/prometheus.rs Outdated Show resolved Hide resolved
src/servers/src/prometheus.rs Show resolved Hide resolved
src/servers/src/prometheus.rs Show resolved Hide resolved
src/servers/src/prometheus.rs Outdated Show resolved Hide resolved
src/servers/src/query_handler.rs Outdated Show resolved Hide resolved
src/servers/src/prometheus.rs Outdated Show resolved Hide resolved
src/frontend/src/instance/prometheus.rs Show resolved Hide resolved
Copy link
Member

@waynexia waynexia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An exciting feature 🥳

@killme2008 killme2008 merged commit 0604eb7 into develop Oct 28, 2022
@killme2008 killme2008 deleted the feature/write-prometheus branch October 28, 2022 10:47
paomian pushed a commit to paomian/greptimedb that referenced this pull request Oct 19, 2023
* feat: scaffold for prometheus protocol handler

* feat: impl remote write and read for prometheus

* chore: make label matchers working in remote reading

* chore: case senstive regexp matching for labers and tweak restful api

* test: prometheus test

* test: adds test for prometheus handler and http server

* fix: typo in comment

* refactor: move snappy_compress and snappy_decompress

* fix: by code review

* fix: collect_timeseries_ids

* fix: timestamp and value column's value may be null
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants