Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Caplin: to set boundaries - no how much goroutines can be spawned by for loop #12510

Open
AskAlexSharov opened this issue Oct 28, 2024 · 0 comments

Comments

@AskAlexSharov
Copy link
Collaborator

AskAlexSharov commented Oct 28, 2024

Found many patterns like:

wg := WaitGroup{}
for {
  wg.Add(1)
  go func() {
     defer wg.Done(1)
  }
}
wg.Wait()

Examples:
getDutiesProposer
connectWithAllPeers
batchVerifyAttestations
VerifyAgainstIdentifiersAndInsertIntoTheBlobStore

This pattern spawns unlimited amount of goroutines. All "unlimited" things are bad - because:

  • because we have limited cpus/ram/disk/etc...
  • even if today some loop doesn't spawn much goroutines - future PR's or future bugs - may change it and we will not notice
  • some "for loops" are run in-parallel - means it's will be hard for us to reproduce "worst case scenario" - where N loops spawning much goroutines in same time.

let's replace it by next pattern:

g := &errgroup.Group{}
g.SetLimit(runtime.GOMAXPROCS(-1))
for {
  g.Go(func() error {
  })
}
if err := g.Wait(); err != nil {
    return err 
})

Can set any limit you like.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant