[Bug]: increased RAM usage on cosmos-sdk v50 full nodes

[MSalopek]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

Given recent user reports on the Hub and at least one other network we're looking for guidance about potetntial performance bottlenecks on full nodes running cosmos-sdk v50.

Related issue on Gaia:

• [Bug]: memory leak with cosmos mainnet tendermint full node gaia#3417

This comment is especially concerning as it points to increased RAM usage related to goleveldb and iavl.

If possible, we would like to move this issue to cosmos-sdk. We can also provide information coming from other chains and node operators.

Please advise.

Thank you!

Cosmos SDK Version

v0.50.x

How to reproduce?

Not clear at this point.

It seems that changing iavl-cache-size can help mitigate the issues.

Nodes seem to be used as query nodes.

[tac0turtle]

this is the frist we have heard of this, on my nodes i havent observed any sort of increase. do you know if its reproducible?

[MSalopek]

Thank you for checking it out!

We were not able to reproduce.
I checked with another team that experienced similar issue but those were related to a custom module - fixing the module alleviated the issue.

In the past we have seen this with RPC query nodes with no rate limits/caching. There are some inefficiencies in the staking module that were documented earlier.

I'm still waiting for further details, but as it stands I cannot link this to any previous issues or reproduce it reliably.

[tac0turtle]

amazing, let us know, we are around to help if needed. There are gas on queries so the node should cancel if the query is too large. But it could be a memory leak somewhere. Not sure i think its in iavl as its been running for a while with no issues

[faddat]

@MSalopek personally I do agree that there's a memory leak somewhere. I have not been able to find that memory leak and I'm really curious about exactly what SDK you upgraded from and exactly what SDK you upgraded to because that can of course influence the memory consumption.

So to be clear:

• Yes I think the SDK leaks ram
• Yes I have hunted the leak
• Sadly my hunt failed

.... The version change thing is kind of a new twist for me but I'm happy to help to investigate