Bug or my machine just too slow?

[Valporaena]

So, I have this issue with archivebox update and I don't know if it's a bug or not. When I start updating, for the first few hours everything works fine. But when I leave it for the night, it starts to use a lot of memory/CPU to the point that my machine suffers significant slowdown and stops archiving properly due to timeouts. It looks like this...:

...with what it seems like a lot of these processes endlessly multiplying:

What also seems strange to me is these processes don't terminate when I shut down archivebox update and archivebox server - they continue to run and hog memory/CPU until I reboot.

Should it work like that? Would appreciate if someone could clear this up for me.

On a sidenote, I would like to thank everyone working on this project. I stumbled on Archivebox a couple weeks ago and it's exactly what I was looking for for years! Invaluable for archiving and accessing research materials for big projects. Your work is greatly appreciated - thanks again!

[Valporaena]

I experimented a bit - turns out I can kill these processes in the middle of archivebox update without any problems. Update keeps on doing its thing and the load on memory/CPU drops immediately. So, maybe it is a bug after all? Should I report it in the "Issues" section or is this already known?

BTW, I see someone here talking about "extra chromes" and "memory usage on the machine would balloon as chrome instance hung around after timing out" pre 0.6.0 update. Seems like the same problem, but I'm on the latest update and it is still here for me.

[pirate]

Yeah, this is a known issue that I thought we fixed. I'm surprised it's still happening to you on >v0.6. Can you post the full output of archivebox --version here?

[Valporaena]

ArchiveBox v0.6.2
Cpython Linux Linux-5.4.0-72-generic-x86_64-with-glibc2.29 x86_64
IN_DOCKER=False DEBUG=False IS_TTY=True TZ=UTC SEARCH_BACKEND_ENGINE=ripgrep

[i] Dependency versions:
 √  ARCHIVEBOX_BINARY     v0.6.2          valid     /usr/bin/archivebox                                                         
 √  PYTHON_BINARY         v3.8.5          valid     /usr/bin/python3.8                                                          
 √  DJANGO_BINARY         v2.2.12         valid     /usr/lib/python3/dist-packages/django/bin/django-admin.py                   
 √  CURL_BINARY           v7.68.0         valid     /usr/bin/curl                                                               
 √  WGET_BINARY           v1.20.3         valid     /usr/bin/wget                                                               
 √  NODE_BINARY           v10.19.0        valid     /usr/bin/node                                                               
 √  SINGLEFILE_BINARY     v0.3.16         valid     ./node_modules/single-file/cli/single-file                                  
 -  READABILITY_BINARY    -               disabled  ./node_modules/readability-extractor/readability-extractor                  
 √  MERCURY_BINARY        v1.0.0          valid     ./node_modules/@postlight/mercury-parser/cli.js                             
 √  GIT_BINARY            v2.25.1         valid     /usr/bin/git                                                                
 -  YOUTUBEDL_BINARY      -               disabled  /usr/bin/youtube-dl                                                         
 √  CHROME_BINARY         v90.0.4430.72   valid     /usr/bin/chromium-browser                                                   
 √  RIPGREP_BINARY        v11.0.2         valid     /usr/bin/rg                                                                 

[i] Source-code locations:
 √  PACKAGE_DIR           23 files        valid     /usr/lib/python3/dist-packages/archivebox                                   
 √  TEMPLATES_DIR         3 files         valid     /usr/lib/python3/dist-packages/archivebox/templates                         
 -  CUSTOM_TEMPLATES_DIR  -               disabled                                                                              

[i] Secrets locations:
 -  CHROME_USER_DATA_DIR  -               disabled                                                                              
 -  COOKIES_FILE          -               disabled                                                                              

[i] Data locations:
 √  OUTPUT_DIR            16 files        valid     /home/.../archivebox                                                     
 √  SOURCES_DIR           35 files        valid     ./sources                                                                   
 √  LOGS_DIR              1 files         valid     ./logs                                                                      
 √  ARCHIVE_DIR           9257 files      valid     ./archive                                                                   
 √  CONFIG_FILE           345.0 Bytes     valid     ./ArchiveBox.conf                                                           
 √  SQL_INDEX             105.7 MB        valid     ./index.sqlite3        

[pirate]

I have an experimental fix to try, but not sure if it'll work. Can you try adding these two flags to your chromium args: --single-process --no-zygote:

Edit this file /usr/lib/python3/dist-packages/archivebox/util.py and change this block on line 229:

@enforce_types
def chrome_args(**options) -> List[str]:
    """helper to build up a chrome shell command with arguments"""

    from .config import CHROME_OPTIONS

    options = {**CHROME_OPTIONS, **options}

    if not options['CHROME_BINARY']:
        raise Exception('Could not find any CHROME_BINARY installed on your system')

-    cmd_args = [options['CHROME_BINARY']]
+    cmd_args = [options['CHROME_BINARY'], '--no-sandbox', '--single-process', '--no-zygote']

Then run it again and let me know if you're still seeing a lot of zombie processes.

[Valporaena]

Edited the file, ran an update for an hour - I don't see any changes with zombie processes or memory/CPU usage.