Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Excessive Memory Usage with Multiple WAF Instances in Coraza Using CRS #975

Open
jptosso opened this issue Feb 2, 2024 · 2 comments
Open

Comments

@jptosso
Copy link
Member

jptosso commented Feb 2, 2024

Issue Summary

It is a well-documented issue that Coraza consumes significant memory when multiple coraza.WAF instances are created with the CRS. Previously, we mitigated this issue by employing memoization. However, a new challenge has emerged: we need to share Coraza's memory across Apache/Nginx workers efficiently.

Problem Statement

In my opinion, it is unreasonable for 50 CRS implementations to occupy 2GB of memory. This raises a critical need to investigate the source of this overhead and implement strategies to reduce the memory footprint of CRS.

Expected Outcome

Identify the root cause(s) of the excessive memory usage.
Explore and implement solutions to minimise memory consumption without compromising functionality.
Additional Context:

The use of memoization previously provided a workaround, but the current scenario demands a more scalable solution, especially for environments with Apache/Nginx workers.
Your insights and suggestions on addressing this issue would be greatly appreciated.

@fzipi
Copy link
Member

fzipi commented Apr 27, 2024

Was there any progress here? Is this repeatable? Can we provide a script, or something, to reproduce the claim that 50 crs implementations use 2GB of memory?

@hseljenes
Copy link

hseljenes commented Jun 21, 2024

I'm also looking at high memory usage for a coraza-caddy deployment.

Flat	Flat%	Sum%	Cum	Cum%	Name	Inlined?
2498.34MB	80.33%	80.33%	2510.34MB	80.71%	github.com/corazawaf/coraza/v3/internal/corazawaf.(*Rule).transformArg	
84.47MB	2.72%	83.04%	84.47MB	2.72%	github.com/corazawaf/coraza/v3/internal/collections.(*Map).FindAll	
80.76MB	2.60%	85.64%	80.76MB	2.60%	github.com/corazawaf/coraza/v3/internal/collections.(*NamedCollectionNames).FindAll	
40.54MB	1.30%	86.94%	58.05MB	1.87%	crypto/internal/bigmod.(*Nat).montgomeryMul	
36.52MB	1.17%	88.12%	36.52MB	1.17%	crypto/internal/bigmod.(*Nat).reset	(inline)
28.42MB	0.91%	89.03%	28.42MB	0.91%	github.com/corazawaf/coraza/v3/internal/collections.(*Map).SetIndex	
26.88MB	0.86%	89.89%	27.88MB	0.90%	github.com/corazawaf/coraza/v3/internal/collections.(*Map).Add	
21.25MB	0.68%	90.58%	212.63MB	6.84%	github.com/corazawaf/coraza/v3/internal/corazawaf.(*Transaction).GetField	
20MB	0.64%	91.22%	20MB	0.64%	github.com/petar-dambovaliev/aho-corasick.AhoCorasick.IterByte	
16.65MB	0.54%	91.75%	2868.21MB	92.22%	github.com/corazawaf/coraza/v3/internal/corazawaf.(*Rule).doEvaluate	
16.50MB	0.53%	92.29%	16.50MB	0.53%	strings.(*Builder).WriteString	(inline)
13.93MB	0.45%	92.73%	23.86MB	0.77%	io.copyBuffer	
13MB	0.42%	93.15%	16.50MB	0.53%	github.com/corazawaf/libinjection-go.isXSS	
10.01MB	0.32%	93.47%	20.80MB	0.67%	github.com/corazawaf/coraza/v3/internal/corazawaf.(*Transaction).MatchRule	
9.77MB	0.31%	93.79%	88.53MB	2.85%	github.com/corazawaf/coraza/v3/internal/collections.(*ConcatCollection).FindAll	
7.38MB	0.24%	94.02%	86.35MB	2.78%	github.com/corazawaf/coraza/v3/internal/collections.(*ConcatKeyed).FindAll	
4.50MB	0.14%	94.17%	24.50MB	0.79%	github.com/petar-dambovaliev/aho-corasick.AhoCorasick.Iter	(inline)
2MB	0.06%	94.23%	29.38MB	0.94%	github.com/corazawaf/coraza/v3/internal/corazawaf.(*Transaction).matchVariable	

Source view:

github.com/corazawaf/coraza/v3/internal/corazawaf.(*Rule).transformArg
/go/path/pkg/mod/github.com/corazawaf/coraza/[email protected]/internal/corazawaf/rule.go

  Total:      2.44GB     2.45GB (flat, cum) 80.71%
    390            .        5MB           ???
    394          8MB        8MB           ???
    398            .          .           ???
    412            .        7MB           ???
    413         94MB       94MB           ???
    415       2.34GB     2.34GB           ???
github.com/corazawaf/coraza/v3/internal/collections.(*Map).FindAll
/go/path/pkg/mod/github.com/corazawaf/coraza/[email protected]/internal/collections/map.go

  Total:     84.47MB    84.47MB (flat, cum)  2.72%
    104      84.47MB    84.47MB           ???
github.com/corazawaf/coraza/v3/internal/collections.(*NamedCollectionNames).FindAll
/go/path/pkg/mod/github.com/corazawaf/coraza/[email protected]/internal/collections/named.go

  Total:     80.76MB    80.76MB (flat, cum)  2.60%
    117      80.76MB    80.76MB           ???
crypto/internal/bigmod.(*Nat).reset
/usr/local/go/src/crypto/internal/bigmod/nat.go

  Total:     36.52MB    36.52MB (flat, cum)  1.17%
     97      36.52MB    36.52MB           ???
crypto/internal/bigmod.(*Nat).resetFor
/usr/local/go/src/crypto/internal/bigmod/nat.go

  Total:    512.25kB   512.25kB (flat, cum) 0.016%
    510     512.25kB   512.25kB           ???
crypto/internal/bigmod.(*Nat).montgomeryMul
/usr/local/go/src/crypto/internal/bigmod/nat.go

  Total:     40.54MB    40.54MB (flat, cum)  1.30%
    596      40.54MB    40.54MB           ???
crypto/internal/bigmod.(*Nat).montgomeryMul
/usr/local/go/src/crypto/internal/bigmod/nat.go

  Total:           0    17.51MB (flat, cum)  0.56%
    651            .    17.51MB           ???

I built with this branch https://github.com/corazawaf/coraza-caddy/tree/dependabot/go_modules/main/github.com/corazawaf/coraza/v3-3.2.0 since it was on the latest coraza.

The top of my go.mod looks like this

require (
	github.com/caddyserver/caddy/v2 v2.7.6
	github.com/caddyserver/replace-response v0.0.0-20231221003037-a85d4ddc11d6
	github.com/corazawaf/coraza-caddy/v2 v2.0.0-rc.3.0.20240620173425-4d5437f2dcb3
	github.com/gr33nbl00d/caddy-revocation-validator v1.0.4
)

I'm fairly new to golang but it seems like it's being used by this transformation cache map?

If it matters this is an API app that uses RSA client certificates for authentication.


Edit: Since the transformation cache is only used when you have transformations I dropped the t:lowercase from the requestBodyProcessor rules in the default config and memory is much more stable at around 300mb.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants