[bug] High memory usage even in low-memory mode

[fukusuket]

Describe the bug
I noticed that even in low-memory mode, since 2.16.0, memory usage is high ... :(
I'll investigate if I can reduce memory usage. 🤔

I compared them with the results of the baseline-evtx and 2.14.0's rule.

Ver	low-memory	Memory usage(peak-reserved)	Memo
2.14.0	Y	2.0 GiB	low-memory mode was released
2.15.0	Y	3.0 GiB	#1368 1GB increase is a crate side issue.
2.16.0-dev	Y	5.2 GiB	d578a89 #1334
2.16.0-dev	Y	5.2 GiB	48fa7e0 #1336
2.16.0-dev	Y	5.2 GiB	69b7793 #1347
2.16.0-dev	Y	5.2 GiB	b8b0f86 #1349
2.16.0-dev	Y	5.2 GiB	310756f #1352
2.16.0	Y	5.2 GiB
2.17.0	Y	5.2 GiB	low-memory mode enabled by default
2.18.0-dev	Y	5.2 GiB
2.18.0-dev	Y	5.2 GiB	-A -a option

The results of the execution are attached.
perf.txt

Expected behavior
Memory usage to be about the same as 2.15.0

[fukusuket]

Released feature in 2.16.0

• feat: add support for Sigma Correlation Value count #1360
• embedded non configurable files into binary #1359
• feat: add support for Sigma Correlation Event count #1357
• chg: output agg condition's JSON result as Details's child element #1354
• feat: add support for Data[x] notation and Provider_Name(field mapping feature's config) #1352
• enable overflow checks #1349
• output - when count rules's output has no content #1347
• feat(main): adjusted splunk api json format #1083 #1346
• chg: disable Channel filter when -J, --JSON-input option #1345
• feat: add support for multiple key grouping in count condition #1341
• feat: adjusted multiple directory option #1335 #1336
• feat: exclude rules/evtx when target Channel does not exists in files #1334

[fukusuket]

I checked out->build commits since 2.15.0 and confirmed that the following PR seems to be the cause.

• feat: exclude rules/evtx when target Channel does not exists in files #1334
• d578a89

The following measurement results show that the problem seems to be that memory usage is high only in low-memory mode.

• feat: exclude rules/evtx when target Channel does not exists in files #1334 (comment)

[YamatoSecurity]

I confirmed there is a memory increase:
I tested with 14GB sample files:

v2.14.0 with low memory mode:

Rule Parse Processing Time: 00:00:06.643
Analysis Processing Time: 00:37:27.972
Output Processing Time: 00:00:00.385

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     4.0 GiB     4.0 GiB     0           4.0 GiB
 committed:     1.6 GiB     4.0 GiB  2175.2 GiB -2171.2 GiB                          ok
     reset:     0
    purged:   151.0 GiB
   touched:   128.5 KiB    35.3 MiB  1546.9 GiB -1546.9 GiB                          ok
  segments:    31         565         553          12                                not all freed!
-abandoned:     1           1           0           1                                not all freed!
   -cached:     0           0           0           0                                ok
     pages:     0           0           7.5 Mi     -7.5 Mi                           ok
-abandoned:     4           4           0           4                                not all freed!
 -extended:     0
 -noretire:     0
     mmaps:     0
   commits:     0
    resets:     0
    purges:    80.8 Ki
   threads:    33          33           1          32                                not all freed!
  searches:     0.0 avg
numa nodes:     1
   elapsed:  2255.164 s
   process: user: 22488.307 s, system: 314.314 s, faults: 0, rss: 2.9 GiB, commit: 1.6 GiB

main branch:

Rule Parse Processing Time: 00:00:06.346
Analysis Processing Time: 00:28:35.124
Output Processing Time: 00:00:00.521

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:     7.3 GiB     7.3 GiB   384.0 MiB     7.0 GiB
 committed:     1.5 GiB     7.3 GiB  2026.6 GiB -2019.3 GiB                          ok
     reset:     0
    purged:   113.9 GiB
   touched:   128.5 KiB    42.3 MiB  1220.4 GiB -1220.4 GiB                          ok
  segments:    31         678         665          13                                not all freed
-abandoned:     1           1           1           0                                ok
   -cached:     0           0           0           0                                ok
     pages:     0           0           7.1 Mi     -7.1 Mi                           ok
-abandoned:     3           3           3           0                                ok
 -extended:     0
 -noretire:     0
    arenas:     7
-crossover:     0
 -rollback:     0
     mmaps:     0
   commits:     0
    resets:     0
    purges:    54.5 Ki
   threads:    33          33           1          32                                not all freed
  searches:     0.0 avg
numa nodes:     1
   elapsed:  1722.236 s
   process: user: 17143.321 s, system: 222.957 s, faults: 0, rss: 2.9 GiB, commit: 1.5 GiB

main branch with sorting events:

Rule Parse Processing Time: 00:00:06.132
Analysis Processing Time: 00:25:09.594
Output Processing Time: 00:04:49.126

Memory usage stats:
heap stats:     peak       total       freed     current        unit       count
  reserved:    21.0 GiB    23.5 GiB     5.5 GiB    18.0 GiB
 committed:     1.6 GiB    23.5 GiB  1965.1 GiB -1941.6 GiB                          ok
     reset:     0
    purged:   101.0 GiB
   touched:   128.5 KiB   206.9 MiB  1122.2 GiB -1122.0 GiB                          ok
  segments:    22           3.2 Ki      3.2 Ki     11                                not all freed
-abandoned:     1           1           1           0                                ok
   -cached:     0           0           0           0                                ok
     pages:     0           0           6.6 Mi     -6.6 Mi                           ok
-abandoned:     3           3           3           0                                ok
 -extended:     0
 -noretire:     0
    arenas:    13
-crossover:     0
 -rollback:     0
     mmaps:     0
   commits:     0
    resets:     0
    purges:    68.4 Ki
   threads:    33          33           1          32                                not all freed
  searches:     0.0 avg
numa nodes:     1
   elapsed:  1808.210 s
   process: user: 18140.995 s, system: 251.570 s, faults: 0, rss: 19.2 GiB, commit: 1.6 GiB

[YamatoSecurity]

@fukusuket I tested things monitoring with Activity Monitor and found out that the maximum memory usage is not in the top lines of reserved and committed but in the bottom line rss.
In v2.14.0 and the latest branch, my maximum memory usage was 2.9GB for the 14GB files so memory usage is actually the same. (When sorted, it goes up to 19GB)

So I don't think there is a problem. But if you want to check on your end, please be sure to also monitor with Activity Monitor or Task Manager.

[fukusuket]

@YamatoSecurity
Thank you so much for checking! I see, I had been checking peak/reserved until now. This does not seem to represent the maximum memory usage now. I am relieved to hear that looks fine...

I 'll check with Windows Task Manager and if everything is OK, close this issue!

[fukusuket]

@YamatoSecurity
I checked the low-memory mode in Windows task manager and compared 2.15.0 with the main branch and found that the low-memory mode worked fine!

As you said, the rss value was close to the maximum memory usage, and both had a maximum usage of 1.4 GB. No memory issue were found and the issue is closed! Sorry for the misunderstanding😅