-
After years of playing around with ArchiveBox by creating little archives on removable disks and forgetting about them I finally have an environment capable of maintaining a growing archive: I have a TrueNAS server with lots of free space and a separate server capable of running ArchiveBox within Proxmox. I know that it's common practice to run ArchiveBox in a VM on a NAS directly but I want storage separated from processes (I have a habit of screwing stuff up because there are so many interesting knobs to play around with so I've made an agreement with myself to not run anything on the NAS and not screw with any of the knobs in exchange for running a separate server that I can screw around with as much as I want). At this point the NAS is working great. I have a SMB share that exposes only the archive dataset. And I have an Ubuntu LXC running inside Proxmox that mounts the dataset with CIFS. I know that it is bad practice to store a SQLite database on a network store and indeed the Therefore, what would be your suggestions for laying this out? According to the disk layout we've got:
Clearly the archive should go on the network share. But what about the other stuff? My guess is that both the config and the sources would be fine to put on the network share. Yes? So, how about this layout:
My plan is to rsync |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
Yup! That looks typical for an ArchiveBox NAS setup and is very similar to how I do it with NFS/SMB/ZFS on my TrueNAS box + experiments server at home. My only suggestions are:
If you run into permissions problems or IO errors, the foolproof method is to store everything except Some historical context: In general the latest ArchiveBox versions are deliberately quite strict about their filesystem expectations. Poorly setup NASs have let to tons of subtle bugs in the past, so I've decided to enforce more and more of the FS requirements as our userbase has increased. It has technically always been possible to store everything (including P.S. We have a new TrueNAS chart that's in the incubator train if you ever consider running it directly on your NAS it would help us to have more users beta testing it :) |
Beta Was this translation helpful? Give feedback.
Yup! That looks typical for an ArchiveBox NAS setup and is very similar to how I do it with NFS/SMB/ZFS on my TrueNAS box + experiments server at home.
My only suggestions are:
ArchiveBox.conf
file locally and rsync it (ArchiveBox is picky about only writing to that file atomically to prevent race conditions from multiple processes editing it)index.sqlite3-wal
is also stored locally (you can rsync it but it's not required)compression=lz4
(archived data compresses very well and lz4 uses very little extra CPU)atime=off
xattr=sa
dnodesize=auto
(common performance improvements that p…