Memory usage

[anekdoti]

Hello,

I observed that our operator using KubeOps uses a lot of memory during reconciliation - in our case about 1Gb for each controlled entity - although we do not use so much memory for our logic during the reconciliation. This gets a problem in particular on restart of the operator, when reconciliation for all entities happens at more or less the same time, adding up the memory consumption to large amounts. For example, in our case, we need to have memory limits for our operator beyond 6Gb just to handle a few entities.

Currently, we are at a loss to understand why this is the case, who is the culprit and what to do about it. Do you have any ideas?

[buehler]

Holy moly. This should definitely not be the case.

Are the entities particularly large (i.e. many properties)?
Do you use the requeue event feature?

It could relate to the following two issues: #280 and #264.

If there would be a better way to check if the object only changed in status (to prevent double reconciliation when status objects are updated), we wouldn't need the item cache.

[anekdoti]

Thanks for your reply! I don't think its a piling up of events as this already happens when the operator starts and handles the entities for the first time. Our entities are not trivial, but by far not big enough (< 1kB) to be the main contributor to the memory usage.

Another observation is that the number of threads spikes to about 45 during a reconciliation of 3 entities.

The memory metric that spikes a lot is process_private_memory_bytes. If we reach the memory limits for the operator according to this metric, it gets OOM killed. Otherwise, the memory usage goes down again to spike again on next reconciliation. GC is also very active.

We did a memory dump during a reconciliation. The memory dump is executed after GC, and it only has a small amount of used memory still reachable. So it is also possible that for some reason GC does not affects the metrics that are crucial for deciding about OOM kills fast enough.

These are just some observations - we still struggle to make sense of them.

[anekdoti]

It turned out that there was actually a very high memory usage in a Snowflake client library which we use in every reconciliation. A semaphore helped to make sure that this memory usage does not adds app when multiple entities are reconcilied at the same time. So everything is fine with KubeOps in this regard :-)

[buehler]

phew! glad to hear that!

BUUUUT nevertheless, it would help to have things like #340 and #280 implemented :)

[anekdoti]

Yes :-) I definitely want to contribute more to the project than to just raise issues and complain