-
-
Notifications
You must be signed in to change notification settings - Fork 32
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Advice on lowering memory usage #215
Comments
So I was looking through the Go lib for libvips and it has this section: https://github.com/davidbyttow/govips#memory-usage-note |
I think this is a memory fragmentation issue. You might consider using a different memory allocator such as jemalloc to prevent memory fragmentation in long-running, multi-threaded, glibc-based Linux processes. See lovell/sharp#955 to learn more about memory allocation and why the choice of a memory allocator is important. For the same reason Redis ships with jemalloc by default since version 2.4.
Please see lovell/sharp-libvips#95 for a future possible enhancement that relates to this. |
Alternatively, you could also consider switching to a Alpine-based Docker image, as the memory allocator in musl, upon which Alpine Linux is based, is generally considered very good in terms of lack of fragmentation and returning freed memory. FWIW, I'm also aware that sharp reduces the default concurrency to 1 when glibc' memory allocator is used, see commit: This can be controlled in NetVips with the net-vips/src/NetVips/NetVips.cs Lines 155 to 162 in 43c043c
|
Thanks for the info. 👍 I also tried running my code in an Alpine Linux docker container and that also fixed the issue. So for anyone reading this thread in the future (and you are also running into memory issues), try the following:
|
Hi, I was hoping to get some tips or tricks on lowering memory usage and fixing a possible memory leak in my code.
My F# code here is a simplified version of what I have in my regular code: https://github.com/Darkle/netvips-memtest/blob/main/Program.fs
You should be able to run it with a
dotnet restore
anddotnet run
Some notes:
Parallel.ForEach
, the memory usage goes past 4GB and slowly keeps going up.Array.Parallel
in case there was an issue withParallel.ForEach
, but the memory keeps rising for that too.Parallel.ForEach
code and uncommenting out theArray.Parallel
code.Enums.Access.Sequential
toEnums.Access.SequentialUnbuffered
helped a bit to lower the memory usage.Details:
Any advice would be appreciated.
Thanks.
The text was updated successfully, but these errors were encountered: