Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run IDs should probably not be labels in Loki #876

Open
mmurto opened this issue Aug 22, 2024 · 2 comments
Open

Run IDs should probably not be labels in Loki #876

mmurto opened this issue Aug 22, 2024 · 2 comments
Labels
enhancement New feature or request
Milestone

Comments

@mmurto
Copy link
Contributor

mmurto commented Aug 22, 2024

The current logaccess from Loki requires the runId to be a label:

"""run_id="$ortRunId"} | logfmt level | $levelCriterion """ +

This does not seem to be advised with Loki, according to Loki's docs:

If you are dynamically setting labels, never use a label which can have unbounded or infinite values. This will always result in big problems for Loki.

I believe (according to the same doc) that the correct solution would be to not label them and use a regex filtering to get logs for the specific run.

@sschuberth sschuberth added the enhancement New feature or request label Aug 22, 2024
@mnonnenmacher
Copy link
Contributor

@mmurto As part of fixing #168 this will have to be implemented in a more generic way. AFAIK the run_id labels are currently only added in our internal configuration based on the labels of the Kubernetes pods. With #875 it will be possible to take them directly from the log statements.

@mmurto
Copy link
Contributor Author

mmurto commented Aug 22, 2024

AFAIK the run_id labels are currently only added in our internal configuration based on the labels of the Kubernetes pods.

We do that as well, but the problem is more with the Logaccess. The code I linked above filters the correct logs for the run with {run_id="$ortRunId"}, which as far as I understand only filters them based on the labels, so if the label wasn't created in deployment setup, logaccess wouldn't return any logs.

@mnonnenmacher mnonnenmacher added this to the 1.0.0 milestone Oct 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants