New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nested Futures Use More Memory Than They Should #709
Labels
Milestone
Comments
A little more information. I don't know much about memory profiling, so apologies if this is not the best way to present the information, but in the hopes it might be helpful...
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I've been running code with nested loops that keeps running into issues with memory usage and I have been trying to come up with a small example that potentially shows the problem. In the example I am just taking a random square matrix and creating a list of the columns. Obviously you wouldn't use a double loop to do this in R but it is hopefully a simple and clear example that shows when using
purrr
the double loop doesn't increase memory usage while withfurrr
andfuture.apply
the memory usage explodes.As you can see, using the double loop actually decreases memory usage for
purrr
, although it stays very similar, but causes memory usage to explode forfurrr
andfuture.apply
. I ran this example on a 2023 MacBook, but the actual code that I am trying to fix has been running on a Linux cluster. I ran this example usingfurrr
andfuture.apply
because yesterday I logged a bug report about nested loops using future.callr and @HenrikBengtsson pointed out that it was only an issue withfurrr
. Please let me know if there is any additional information I can provide or help I can give in solving this issue and thanks for the wonderful collection of packages!The text was updated successfully, but these errors were encountered: