Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[observability] Prometheus and Grafana as metrics monitoring stack #257

Closed
2 tasks done
michaelkoepf opened this issue Dec 23, 2024 · 4 comments · Fixed by #286
Closed
2 tasks done

[observability] Prometheus and Grafana as metrics monitoring stack #257

michaelkoepf opened this issue Dec 23, 2024 · 4 comments · Fixed by #286
Labels
feature New feature or request

Comments

@michaelkoepf
Copy link
Contributor

michaelkoepf commented Dec 23, 2024 •
edited
Loading

Search before asking

  • I searched in the issues and found nothing similar.

Motivation

Fluss already exposes several metrics via Prometheus, but there is no way to visualize them yet.

Solution

  1. Provide Grafana dashboards for monitored metrics.
  2. Provide a minimal configuration how for Prometheus + Grafana and show how to set it up in the quickstart guide

Dashboards:

  • Overview dashboard with selected metrics
  • Dashboard that shows all metrics

Anything else?

No response

Willingness to contribute

  • I'm willing to submit a PR!
@michaelkoepf michaelkoepf added the feature New feature or request label Dec 23, 2024
@polyzos
Copy link
Collaborator

polyzos commented Dec 24, 2024

This can be quite helpful and we could add Prometheus and Grafana into the docker-compose quickstart guide to make it easy for new users to test things out.

michaelkoepf added a commit to michaelkoepf/fluss that referenced this issue Dec 28, 2024
@michaelkoepf
[WIP][observability] Add Prometheus and Grafana observability stack
cc1264f 
- Adapted Flink quickstart guide
- Adapted monitoring section in docs

Issue alibaba#257
@michaelkoepf michaelkoepf changed the title [Feature] Grafana dashboards for monitored metrics [observability] Prometheus and Grafana as observability stack Dec 28, 2024
@michaelkoepf michaelkoepf mentioned this issue Dec 28, 2024
Merged
michaelkoepf added a commit to michaelkoepf/fluss that referenced this issue Dec 28, 2024
@michaelkoepf
[WIP][observability] Add Prometheus and Grafana observability stack
ff64149 
Fixed messed-up formatting

Issue alibaba#257
michaelkoepf added a commit to michaelkoepf/fluss that referenced this issue Dec 29, 2024
@michaelkoepf
[WIP][observability] Add Prometheus and Grafana observability stack
6d8f949 
Fixed messed-up formatting

Issue alibaba#257
michaelkoepf added a commit to michaelkoepf/fluss that referenced this issue Dec 29, 2024
@michaelkoepf
[WIP][observability] Add Prometheus and Grafana observability stack
e00cb8a 
Fixed messed-up formatting

Issue alibaba#257
michaelkoepf added a commit to michaelkoepf/fluss that referenced this issue Dec 29, 2024
@michaelkoepf
[WIP][observability] Add Prometheus and Grafana observability stack
c6d40cc 
Fixed messed-up formatting

Issue alibaba#257
michaelkoepf added a commit to michaelkoepf/fluss that referenced this issue Dec 29, 2024
@michaelkoepf
[observability] Add Prometheus and Grafana observability stack
62a4cac 
- Adapted Flink quickstart guide
- Provide quickstart config (Prometheus, Grafana) and Grafana dashboards
- Fixed minor error in monitoring doc

Issue alibaba#257
@michaelkoepf
Copy link
Contributor Author

michaelkoepf commented Dec 29, 2024 •
edited
Loading

@polyzos done. i added a quickstart config for prometheus and grafana + 2 grafana dashboards (for convenience, it is provided as a ZIP archive under docs/assets). i also added instructions how to set up the observability stack to the flink quickstart guide. see #286.

@polyzos
Copy link
Collaborator

polyzos commented Dec 30, 2024 •
edited
Loading

@michaelkoepf It looks good and all dashboards are working great .. kudos for this 🚀 👍

What I'm thinking though is to keep the quickstart guide as is to 1. keep it simple and 2. avoid resources for extra containers (for limited environments) and add the observability containers under the monitor-metrics section as complementary to the quickstart, i.e based on the quick start guide you can add the prometheus and grafana ....

WDYT?

michaelkoepf added a commit to michaelkoepf/fluss that referenced this issue Dec 30, 2024
@michaelkoepf
[observability] Add Prometheus and Grafana observability stack
2b94e26 
Move observability instructions from quickstart guide to monitoring doc section

Issue alibaba#257
michaelkoepf added a commit to michaelkoepf/fluss that referenced this issue Dec 30, 2024
@michaelkoepf
[observability] Add Prometheus and Grafana observability stack
7a86d81 
Move observability instructions from quickstart guide to monitoring doc section

Issue alibaba#257
michaelkoepf added a commit to michaelkoepf/fluss that referenced this issue Dec 30, 2024
@michaelkoepf
[observability] Add Prometheus and Grafana observability stack
c31a11c 
Move observability instructions from quickstart guide to monitoring doc section

Issue alibaba#257
michaelkoepf added a commit to michaelkoepf/fluss that referenced this issue Dec 30, 2024
@michaelkoepf
[observability] Add Prometheus and Grafana observability stack
10bd4a1 
Add link to observability guide in quickstart guide

Issue alibaba#257
michaelkoepf added a commit to michaelkoepf/fluss that referenced this issue Dec 30, 2024
@michaelkoepf
[observability] Add Prometheus and Grafana observability stack
add43a7 
Fix broken link to observability ZIP archive

Issue alibaba#257
@michaelkoepf
Copy link
Contributor Author

@polyzos adapted according to your feedback, ptal

@michaelkoepf michaelkoepf changed the title [observability] Prometheus and Grafana as observability stack [observability] Prometheus and Grafana as metrics monitoring stack Dec 30, 2024
polyzos pushed a commit that referenced this issue Dec 31, 2024
@michaelkoepf
[observability] Add Prometheus and Grafana as metrics monitoring stack (
53b6fe5 
#286)

* [WIP][observability] Add Prometheus and Grafana observability stack

- Adapted Flink quickstart guide
- Adapted monitoring section in docs

Issue #257

* [WIP][observability] Add Prometheus and Grafana observability stack

Fixed messed-up formatting

Issue #257

* [observability] Add Prometheus and Grafana observability stack

- Adapted Flink quickstart guide
- Provide quickstart config (Prometheus, Grafana) and Grafana dashboards
- Fixed minor error in monitoring doc

Issue #257

* [observability] Add Prometheus and Grafana observability stack

Move observability instructions from quickstart guide to monitoring doc section

Issue #257

* [observability] Add Prometheus and Grafana observability stack

Add link to observability guide in quickstart guide

Issue #257

* [observability] Add Prometheus and Grafana observability stack

Fix broken link to observability ZIP archive

Issue #257
@michaelkoepf michaelkoepf mentioned this issue Dec 31, 2024
Open
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[observability] Add Prometheus and Grafana as metrics monitoring stack michaelkoepf/fluss
2 participants
@polyzos @michaelkoepf