filter_rewrite_tag: add and-combination for rules #2399
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rules in rewrite_tag filter were combined with OR combination. In some usecases
an AND-combination is helpful. For instance, when logmessages in kubernetes from
customer namespaces should be dropped, which haven't set a special annotation
field. Without an AND-combination, two filter section are necessary, to get this
done.
Configuration example:
to each rule a fifth field with true|false can be added. 'true' means, that this
rule should be "AND"-combined with the next rule. "false" means default "OR"
behaviour, and is not needed. So its full compatible with old filter configuration.
```
[FILTER]
Name rewrite_tag
Match tail
Rule $log ^(1|3)$ newtag_or false false
Rule $log ^(.*end)$ newtag_and_1 false true
Rule $log ^(1.*)$ newtag_and_2 false false
Rule $log ^(2.*)$ newtag_or false
```
Signed-off-by: Michael Voelker <[email protected]>
|
Example configfiguration fluentbit.conf /var/tmp/loginput.txt rewrite_tag result: |
|
Debug log |
novegit
added a commit
to novegit/fluent-bit-docs
that referenced
this pull request
Jul 29, 2020
description for and-combination for rewrite_tag filter rules PR: fluent/fluent-bit#2399 Signed-off-by: Michael Voelker <[email protected]>
|
thanks for opening this PR. I think it's not the main purpose of rewrite tag filter to discard records, but rewrite tags only, and optionally discard the original matched ones. If you need to "let pass" records that matches a criteria there are other ways to do it. E.g: consider the following test file with two records: {"log": {"kubernetes": {"msg": "test1", "logme": "yes"}}}
{"log": {"kubernetes": {"msg": "test2"}}}Using tail + grep filter you can accomplish the same thing you need: fluent-bit -R parsers.conf \
-i tail -p path=test.log -p parser=json \
-F grep -p "regex=\$log['kubernetes']['logme'] yes" -m '*' \
-o stdout -p format=json_lines -f 1output: {"date":1596059959.811035,"log":{"kubernetes":{"msg":"test1","logme":"yes"}}}another way to accomplish the same is using the stream processor, making the input plugin non-routable and creating a new stream that matches a SQL criteria. |
edsiper
added
the
waiting-for-user
|
thanks for the reply, I didnt' knew so far, that grep can handle nested fields with this regex syntax. (In grep documentation section is only an example about using filter/nest for this case) But i still don't see, that grep can solve my usecase. Task is to drop messages, that fulfill both conditions: input: result: I assume, that the An alternate requirement could be in future, not to drop, but to route them to a seperate output with a customers pool kafka topic, collecting all the 'customer-logs' without a namespace specific kafkatopic (set in annotation loggingkafkatopic) to seperate them from the clusters default kafkatopic. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
rules in rewrite_tag filter were combined with OR combination. In some usecases
an AND-combination is helpful. For instance, when logmessages in kubernetes from
customer namespaces should be dropped, which haven't set a special annotation
field. Without an AND-combination, two filter section are necessary, to get this
done.
Configuration example:
to each rule a fifth field with true|false can be added. 'true' means, that this
rule should be "AND"-combined with the next rule. "false" means default "OR"
behaviour, and is not needed. So its full compatible with old filter configuration.
Signed-off-by: Michael Voelker [email protected]
Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
Documentation
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.