-
Notifications
You must be signed in to change notification settings - Fork 206
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Profiling large number of goroutines #748
Comments
With the current tooling we have available from Go runtime and in combination with what Sentry.io expects to receive, I don't see a way to improve profiling for apps with a very large number of goroutines. As such, I think we may want to make some adjustments to automatically disable the profiler when it encounters a given number of goroutines. This would act as a reasonable default behaviour working around this limitation before it's resolved in one way or another. Users could be able to change this configuration or disable it completely if they want to run the profiler unhindered, regardless of the number of routines they launch. |
I've made a PR in Go to land the change to GoroutineProfile() and verified the functionality in the SDK on branch https://github.com/getsentry/sentry-go/tree/feat/goroutine-profile |
Sweet! |
I've created a simple sample application with 5000 concurrent goroutines, tuned to have around 30 % CPU usage (on my desktop PC) without profiling. After profiling was enabled with 5 % sample rate, the CPU usage jumps to around 40 %.
I've run a pprof CPU profile with and without profiling enabled. In the following SVG, you can see that the vast majority of the overhead is coming from calling
Runtime.stack()
. Current profiling implementation calls it periodically to collect stacks for all goroutines. Therefore, the time complexity increases with the number of active goroutines.See the profile output as a tree (.svg so you can zoom in)
I'm investigating how we could improve this.
Note:
runtime.GoroutineProfile()
has lower overhead, possibly because it doesn't produce string output, but it currently doesn't have a goroutine ID so we can't map stacks across multiple calls to a single routine, see golang/go#59663The text was updated successfully, but these errors were encountered: