log: Log File size limit and Log rotation for fluent-bit logs #1890

phiremande · 2020-01-18T10:12:22Z

Currently fluent-bit logs can be redirected to a file using Log_File.
But there is no configuration option to limit the size of the file. Additionally, there is no log rotation that can be enabled for fluent-bit logs.
These might be useful when we want to collect fluent-bit process logs itself and also to debug.

Added new configuration parameters:
Log_File_Size - Size limit in MB for the file specified by Log_File.
Log_File_History - Max number of backups to be kept.
These backups are kept as <Log_File>.1, <Log_File>.2 etc.

Sample configuration.
[SERVICE]
Flush 1
Daemon Off
Log_Level debug
Log_File ./fb_log.log
Log_File_Size 1
Log_File_History 2

This PR has the corresponding changes.

src/flb_log.c

edsiper · 2020-06-30T18:28:50Z

src/flb_log.c

+    int ret = -1;
+
+    if (log->max_history == 0) {
+        //no backups.


use /* ... */ for comments

src/flb_log.c

edsiper · 2020-06-30T18:29:13Z

src/flb_log.c

+    }
+
+    //max 2 digits for backup name suffix and 1 each for '.' and '\0'.
+    backup_file_name = flb_malloc(strlen(log->out) + 4);


set the malloc size in a variable before malloc

src/flb_log.c

edsiper · 2020-06-30T18:29:36Z

src/flb_log.c

+    }
+    src_file_name = flb_malloc(strlen(log->out) + 4);
+    if (!src_file_name) {
+        perror("malloc");


flb_errno();

src/flb_log.c

edsiper · 2020-06-30T18:30:06Z

src/flb_log.c

+    }
+
+    for (int l = log->max_history; l > 1; l--) {
+        sprintf(backup_file_name, "%s.%d", log->out, l);


use snprintf instead

src/flb_log.c

phiremande · 2020-09-03T06:30:12Z

@edsiper , kindly let me know if you have any further suggestions.

phiremande · 2022-01-19T04:18:29Z

@edsiper , please let me know if this PR still makes sense and are there any plans to merge this.

edsiper

thanks for this PR.

I added some minor change requests before to get this merged we need to sort out two things:

rebase on top of GIT master
all I/O functions must work on Linux and Windows, .eg: remove(3) is not valid on windows

src/flb_log.c

phiremande · 2022-01-21T12:31:31Z

thanks for this PR.

I added some minor change requests before to get this merged we need to sort out two things:
* rebase on top of GIT master

* all I/O functions must work on Linux and Windows, .eg: remove(3) is not valid on windows

Kindly let me know if the changes look fine now. If you have any more suggestions please do let me know. Also should I use "log" or "engine" in the commit message?

patrick-stephens · 2022-04-07T21:33:28Z

You may need a rebase to solve the fuzzing failures but the unit tests also need resolving. There seems to be quite a few failures which will prevent merging.

phiremande · 2022-04-09T00:46:45Z

You may need a rebase to solve the fuzzing failures but the unit tests also need resolving. There seems to be quite a few failures which will prevent merging.

@patrick-stephens , I have rebased and fixed UT failures. Can you please let me know what should be the commit message? I guess the 1 failure is due to component missing in commit message.

patrick-stephens · 2022-04-09T14:43:07Z

Yeah you need it in the format: component: message

Signed-off-by: Pradeep Hiremande <[email protected]>

edsiper · 2022-08-26T19:08:04Z

@patrick-stephens both ideally

qihonggang · 2022-11-25T07:06:17Z

@patrick-stephens When will this PR merge into the master branch？

patrick-stephens · 2022-11-25T10:15:51Z

Apologies I've had little time to look at this so far but will endeavour to soon.
It can't be merged right now anyway as it has a conflict.

said1995 · 2023-05-29T06:20:14Z

Hi Team,
Any update on this request?
This would be very helpful.

patrick-stephens · 2023-05-30T08:33:29Z

Apologies, I've still not tested this yet.

ustndagsemih · 2023-06-26T08:59:54Z

Hi team,
Any thoughts on when would this be merged?

robert-heinzmann-logmein · 2023-07-18T12:56:52Z

Any update on this one, I think it would be very valuable to have predictable log sizes for fluent bit itself.

wojciechtobis · 2023-10-02T15:53:04Z

Hey @phiremande, are you going to finish this PR? I can continue to work on it as I really need these changes.

hcdmonitor · 2023-11-21T10:40:36Z

Hey everyone, I'm grateful for the work that has already been carried out on this issue.
The log file size limit and rotation would be very useful to us to prevent disk utilisation issues caused by fluent-bit.

Is there an update as to when the changes will be merged?

patrick-stephens · 2023-11-21T10:49:38Z

Hey everyone, I'm grateful for the work that has already been carried out on this issue. The log file size limit and rotation would be very useful to us to prevent disk utilisation issues caused by fluent-bit.

Is there an update as to when the changes will be merged?

It needs rebasing and updating so if you pick it up @hcdmonitor it may speed things up.

etolbakov · 2023-12-22T15:53:06Z

src/flb_log.c

+    }
+
+    /* max 2 digits for backup name suffix and 1 each for '.' and '\0' */
+    size_t filename_len = strlen(log->out) + 4;


is this expected here?

Suggested change

size_t filename_len = strlen(log->out) + 4;

size_t filename_len = snprintf(NULL, 0, "%"PRIu64, log->out);

etolbakov · 2023-12-22T15:53:40Z

src/flb_log.c

+    }
+    src_file_name = flb_malloc(filename_len);
+    if (!src_file_name) {
+        flb_errno();


is this expected here?

Suggested change

flb_errno();

flb_free(backup_file_name);

flb_errno();

etolbakov · 2023-12-22T16:27:45Z

Hello everyone,
I'm also interested in this feature!
Am I correct assuming that the only way of rebasing here is to open a new PR @patrick-stephens @hcdmonitor ?
I don't think anyone except @phiremande has permissions to push to phiremande:log-rotate branch.
Please let me know if I'm wrong.

ashraf133 · 2024-07-16T07:53:35Z

Hello, any news about this feature ? it is fundamental

alanbrito · 2024-07-22T10:57:48Z

This is a very basic feature. We cannot go live with a component that will almost guarantee and outage eventually, due to filling the disk.

Is there any progress?

patrick-stephens · 2024-07-29T09:46:22Z

Hello everyone, I'm also interested in this feature! Am I correct assuming that the only way of rebasing here is to open a new PR @patrick-stephens @hcdmonitor ? I don't think anyone except @phiremande has permissions to push to phiremande:log-rotate branch. Please let me know if I'm wrong.

It may be easier with a new PR, you can generate the patch for this one by just adding a .diff or .patch to this one.

For everyone asking about log rotate, typically I would do this separately via logrotate or something central to the OS rather than relying on each individual process to do it with all the associated configs to then manage too.

klee-it · 2025-01-02T10:47:46Z

Hello, any news about this feature? I really need that (on my Windows machines) :-)

phiremande force-pushed the log-rotate branch 2 times, most recently from 90b8135 to 8e96609 Compare January 18, 2020 10:36

phiremande requested a review from edsiper January 20, 2020 04:52

phiremande force-pushed the log-rotate branch from 1c30efa to 040815f Compare January 24, 2020 17:49

edsiper assigned fujimotos May 5, 2020

edsiper added the work-in-process label May 5, 2020

edsiper requested changes Jun 30, 2020

View reviewed changes

phiremande requested review from fujimotos and koleini as code owners July 1, 2020 12:56

phiremande force-pushed the log-rotate branch 2 times, most recently from 157f164 to b09402c Compare July 1, 2020 13:05

phiremande requested a review from edsiper July 2, 2020 05:10

phiremande force-pushed the log-rotate branch 2 times, most recently from 1cbd281 to 1b8bcc8 Compare July 23, 2020 08:37

lecaros added docs-required waiting-for-user Waiting for more information, tests or requested changes labels Jan 20, 2022

edsiper requested changes Jan 20, 2022

View reviewed changes

src/flb_log.c Outdated Show resolved Hide resolved

src/flb_log.c Show resolved Hide resolved

src/flb_log.c Outdated Show resolved Hide resolved

phiremande force-pushed the log-rotate branch from 1b8bcc8 to bf6b108 Compare January 21, 2022 10:14

phiremande requested a review from edsiper February 1, 2022 05:43

phiremande force-pushed the log-rotate branch 2 times, most recently from 5bc5fb3 to a202f2c Compare April 8, 2022 19:08

phiremande force-pushed the log-rotate branch from a202f2c to e36f5fc Compare April 9, 2022 17:15

log: changes for limiting/rotating fluent-bit logs

e889424

Signed-off-by: Pradeep Hiremande <[email protected]>

phiremande force-pushed the log-rotate branch from e36f5fc to e889424 Compare April 9, 2022 17:19

lecaros added this to the Fluent Bit v1.9.6 milestone Jun 23, 2022

lecaros modified the milestones: Fluent Bit v1.9.6, Fluent Bit v1.9.7 Jul 7, 2022

lecaros modified the milestones: Fluent Bit v1.9.7, Fluent Bit v1.9.8 Aug 11, 2022

lecaros modified the milestones: Fluent Bit v1.9.8, Fluent Bit v1.9.9 Sep 5, 2022

lecaros modified the milestones: Fluent Bit v1.9.9, Fluent Bit v1.9.10 Sep 27, 2022

lecaros modified the milestones: Fluent Bit v1.9.10, Fluent Bit v1.9.11 Nov 30, 2022

lecaros removed this from the Fluent Bit v1.9.11 milestone Jun 1, 2023

etolbakov reviewed Dec 22, 2023

View reviewed changes

nuclearpidgeon mentioned this pull request Dec 10, 2024

log rotation/compressing/purging #215

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

log: Log File size limit and Log rotation for fluent-bit logs #1890

log: Log File size limit and Log rotation for fluent-bit logs #1890

phiremande commented Jan 18, 2020

edsiper Jun 30, 2020

phiremande Jul 1, 2020

edsiper Jun 30, 2020

phiremande Jul 1, 2020

edsiper Jun 30, 2020

phiremande Jul 1, 2020

edsiper Jun 30, 2020

phiremande Jul 1, 2020

phiremande commented Sep 3, 2020

phiremande commented Jan 19, 2022

edsiper left a comment

phiremande commented Jan 21, 2022

patrick-stephens commented Apr 7, 2022

phiremande commented Apr 9, 2022 •

edited

Loading

patrick-stephens commented Apr 9, 2022

edsiper commented Aug 26, 2022

qihonggang commented Nov 25, 2022

patrick-stephens commented Nov 25, 2022

said1995 commented May 29, 2023

patrick-stephens commented May 30, 2023

ustndagsemih commented Jun 26, 2023

robert-heinzmann-logmein commented Jul 18, 2023

wojciechtobis commented Oct 2, 2023

hcdmonitor commented Nov 21, 2023

patrick-stephens commented Nov 21, 2023

etolbakov Dec 22, 2023

etolbakov Dec 22, 2023

etolbakov commented Dec 22, 2023

ashraf133 commented Jul 16, 2024

alanbrito commented Jul 22, 2024

patrick-stephens commented Jul 29, 2024

klee-it commented Jan 2, 2025 •

edited

Loading

	size_t filename_len = strlen(log->out) + 4;
	size_t filename_len = snprintf(NULL, 0, "%"PRIu64, log->out);

log: Log File size limit and Log rotation for fluent-bit logs #1890

Are you sure you want to change the base?

log: Log File size limit and Log rotation for fluent-bit logs #1890

Conversation

phiremande commented Jan 18, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

phiremande commented Sep 3, 2020

phiremande commented Jan 19, 2022

edsiper left a comment

Choose a reason for hiding this comment

phiremande commented Jan 21, 2022

patrick-stephens commented Apr 7, 2022

phiremande commented Apr 9, 2022 • edited Loading

patrick-stephens commented Apr 9, 2022

edsiper commented Aug 26, 2022

qihonggang commented Nov 25, 2022

patrick-stephens commented Nov 25, 2022

said1995 commented May 29, 2023

patrick-stephens commented May 30, 2023

ustndagsemih commented Jun 26, 2023

robert-heinzmann-logmein commented Jul 18, 2023

wojciechtobis commented Oct 2, 2023

hcdmonitor commented Nov 21, 2023

patrick-stephens commented Nov 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

etolbakov commented Dec 22, 2023

ashraf133 commented Jul 16, 2024

alanbrito commented Jul 22, 2024

patrick-stephens commented Jul 29, 2024

klee-it commented Jan 2, 2025 • edited Loading

phiremande commented Apr 9, 2022 •

edited

Loading

klee-it commented Jan 2, 2025 •

edited

Loading