--cleanuprss option to clear caches when memory reaches a certain limit #257
Conversation
|
Perhaps unrelated but my experience with this is that creating a new BrowserTab after each request seems to increase memory usage drastically. If I'm not mistaken, a new BrowserTab is created at every request. Hacky workaround that might be useful for others include
|
|
@AlexIzydorczyk I'm not sure creating BrowserTab objects per se is a problem. There is a basic stress test script which sends 1000 render requests to Splash; very simple HTML pages are rendered (served by a mock server on a localhost). If configured to send only requests which are known to be successful (i had to change MockArgs code to do that), it shows the following resuts: So creating BrowserTab, rendering a basic web page and destroying BrowserTab can be done at 100 requests per second. Memory usage is increasing, but not that much (several MBs after 1000 requests). |
|
But you may be right - memory usage is still increasing, even if not much. |
Hey @kmike can this be related to png rendering and handling of png by PILLOW? In our project we get images as png from splash but than convert to jpeg. We noticed really high memory usage when converting image from png to jpeg, seems like Pillow had some trouble and leaking lots of memory when opening png files. This bug was driving us crazy with @chekunkov :) I described it in following question on SO: http://stackoverflow.com/questions/30411411/why-is-pil-not-releasing-memory-after-opening-certain-png-files It is worth checking how newly added (on branch) render.jpeg endpoint will perform in terms of memory, I wonder if it will improve performance and reduce leaks. |
|
yep can confirm this looks like same bug calling render.png repeatedly caused memory to increase, you can try adding debugging code line 689 of splash/browser tab.py: import resource
mem = lambda:resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
print(mem())
image = render_qwebpage(self.web_page, self.logger,
width=width, height=height,
scale_method=scale_method)
self.store_har_timing("_onScreenshotPrepared")
print(mem())
result = image.to_png()
if b64:
result = base64.b64encode(result)
print(mem())this prints following output: tested with http://localhost:8050/render.png?url=http://www.bankier.pl/&render_all=1&wait=1 |
|
sure @chekunkov I get similar output: |
|
after further research it seems like this happens in _render_qwebpage_full |
See #216 (comment).
I tried it only on simple examples (render several websites with --slots=1). Some findings:
after some time the example above becomes
For me it looks like increasing memory usage is caused by some WebKit or qt or pyqt internals, or maybe by fragmentation of the Python heap. I don't want to dive deeper now because we're going to replace (py)qt4 with (py)qt5 soon. After the switch it worths checking again.