Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

rocketpool_node causes significant CPU and IO load every 5 minutes in the besu EC client #471

Open
amegyeri opened this issue Mar 28, 2024 · 0 comments

Comments

@amegyeri
Copy link

version 1.11.9
EC is besu, BN Nimbus
HW: Kingston KC3000 2TB, CPU is i7-8550U (not ideal but meets the minimum benchmar required and does the job), 16 GB DDR4 RAM. Network is perfectly stable 1/1 Gbps fiber.

Discussed a lot on discord, and found out some details.

RP node runs smoothly and validates properly most of the time. But every 5 minutes rockepool_node performs some task that makes the besu EC consume a lot of I/O and CPU time (lasting about a minute). During these spikes attestations fail and I have also missed a proposal during such a load.

For testing purposes (as advised on RP Discord support) I stopped rocketpool_node for 30 mins and confirmed the high load was gone.

The screenshot below illustrates the CPU and I/O reduction with disabled node:

image

Node log shows no errors/warnings.
BN log has a few "Missed multiple heartbeats" entries during peaks.

FUN FACT:

This is a Proxmox virtualization host, with LXC Containers (very low overhead). There are two containers running:

  1. Rocketpool node - all functions (EC,BN,Validator,Node, mev, prometheus...etc)
  2. Stader - Runing just a Validator role (no EC, no BN), connecting to Rocketpool clients using http. It validates perfectly, no misses at all!

So the performance impact is only in the RP container (don't know what to do with this info...)
EC Clients is set to 25, BN to 100 (Tried 160 and 50 as well - but not much change at all. Using 100 for performance reasons)

Any ideas?
Is there something to be configured in the rocketpool_node to prevent these spikes?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant