Troubleshooting with Custom Chrome Binary and Profile

[Shamshiel]

I'm trying to setup ArchiveBox on my Synology-NAS and I'm struggling with getting a custom Chrome profile to work to get rid of consent popups on some sites.

This is the config I use to start my ArchiveBox:

0.7.2
ArchiveBox v0.7.2+editable COMMIT_HASH=db3fee1 BUILD_TIME=2024-03-07 21:13:42 1709846022
IN_DOCKER=True IN_QEMU=False ARCH=x86_64 OS=Linux PLATFORM=Linux-4.4.302+-x86_64-with-glibc2.36 PYTHON=Cpython
FS_ATOMIC=True FS_REMOTE=True FS_USER=911:911 FS_PERMS=644
DEBUG=False IS_TTY=True TZ=UTC SEARCH_BACKEND=ripgrep LDAP=False

[i] Dependency versions:
 √  PYTHON_BINARY         v3.11.8         valid     /usr/local/bin/python3.11                                                   
 √  SQLITE_BINARY         v2.6.0          valid     /usr/local/lib/python3.11/sqlite3/dbapi2.py                                 
 √  DJANGO_BINARY         v3.1.14         valid     /usr/local/lib/python3.11/site-packages/django/__init__.py                  
 √  ARCHIVEBOX_BINARY     v0.7.2          valid     /usr/local/bin/archivebox                                                   

 √  CURL_BINARY           v8.5.0          valid     /usr/bin/curl                                                               
 √  WGET_BINARY           v1.21.3         valid     /usr/bin/wget                                                               
 √  NODE_BINARY           v20.11.1        valid     /usr/bin/node                                                               
 √  SINGLEFILE_BINARY     v1.1.46         valid     /app/node_modules/single-file-cli/single-file                               
 √  READABILITY_BINARY    v0.0.11         valid     /app/node_modules/readability-extractor/readability-extractor               
 √  MERCURY_BINARY        v1.0.0          valid     /app/node_modules/@postlight/parser/cli.js                                  
 √  GIT_BINARY            v2.39.2         valid     /usr/bin/git                                                                
 √  YOUTUBEDL_BINARY      v2023.12.30     valid     /usr/local/bin/yt-dlp                                                       
 √  CHROME_BINARY         v121.0.6158.0   valid     /chromium/chrome-linux/chrome                                               
 √  RIPGREP_BINARY        v13.0.0         valid     /usr/bin/rg                                                                 

[i] Source-code locations:
 √  PACKAGE_DIR           23 files        valid     /app/archivebox                                                             
 √  TEMPLATES_DIR         3 files         valid     /app/archivebox/templates                                                   
 -  CUSTOM_TEMPLATES_DIR  -               disabled  None                                                                        

[i] Secrets locations:
 √  CHROME_USER_DATA_DIR  35 files        valid     /chromium/profile                                                           
 √  COOKIES_FILE          1.1 KB          valid     ./cookies.txt                                                                  

[i] Data locations:
 √  OUTPUT_DIR            7 files @       valid     /data                                                                       
 √  SOURCES_DIR           25 files        valid     ./sources                                                                   
 √  LOGS_DIR              1 files         valid     ./logs                                                                      
 √  ARCHIVE_DIR           4 files         valid     ./archive                                                                   
 √  CONFIG_FILE           81.0 Bytes      valid     ./ArchiveBox.conf                                                           
 √  SQL_INDEX             260.0 KB        valid     ./index.sqlite3           

When I try to archive a site the consent banner/popup still appears. Checking the Cookies file in the Default directory that I uploaded seems to get overwritten with an empty Cookie file.

PS: The profile was created with the same chrome that is located in /chromium/chrome-linux/chrome

[pirate]

Strange, are you sure they were created on the same OS? That's another source of trouble sometimes (if it's used in linux Docker then the profile needs to be created on Linux unfortunately).

/chromium/chrome-linux/chrome --headless=false --user-data-dir=/chromium/profile https://example.com should work to open a headful browser and save that profile.

[Shamshiel]

I created the profile on my Arch Linux system and then copied both the binary and the profile to my NAS.
Do you think the Cookie is overwritten because the Profile is not recognized as created with the same binary?

[pirate]

Yeah that sometimes happens when the OS is different, it just silently overwrites the directory (Arch -> Debian surprises me as they're quite similar, but maybe it's different enough?). It's historically been a really annoying silent behavior by Chrome that makes it hard to catch at the ArchiveBox level, I wish they threw an error when it happened.

I recommend this new approach instead: add a novnc container to your docker-compose.yml and then log in to stuff in using the container browser exposed over VNC (instead of trying to sync a chrome profile with a remote host).

I've updated the docs to include new instructions on how to use Docker + novnc to set up a Chrome Profile:
https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile

---

To improve this UX in the future, I'm working on a new way to export and load credentials as a .json file. Here's my WIP that saves/loads sessions using puppeteer, demonstrated by logging in on TikTok: https://gist.github.com/pirate/a78f8dcf2ade7c2881f2e18386d5c0de