Where are the archive logs when you add URLs from the web UI?

[melyux]

The Docker logs are empty and just show the URLs added before they're archived. errors.log only shows errors and schedule.log only shows logs for scheduled adding. Where can I find the logs for normal, manual archiving?

The "progress screen" just says "Check the server log" but not sure if that actually exists anywhere.

[pirate]

Go to the Log page from the navbar or click the Log link in that text ^ (each ArchiveResult listed on that page is a log entry for one extractor run)

If you want the CLI logs, docker manages those, use docker compose logs archivebox.

[melyux]

When I add a URL from the web UI, this is the output it produces in docker compose logs archivebox:

archivebox  | [+] Adding URL: https://google.com
archivebox  | 
archivebox  | 
archivebox  | [*] [2023-08-14 04:06:34] Archiving 1/3083 URLs from added set...
archivebox  | "GET / HTTP/1.1" 302 0
archivebox  | "GET /admin/login/?next=/ HTTP/1.1" 200 11144
archivebox  | "POST /add/ HTTP/1.1" 200 4807

[pirate]

Those are the only logs it produces, were you expecting some more logs than that?

[melyux]

Yeah I would have expected similar logs to the schedule logs, more detailed and reporting the success/failure of each extractor. For example:

[+] [2023-08-15 00:08:45] "Google"
    https://www.google.com
    > ./archive/1692058124.282452
      > favicon
      > headers
      > singlefile
      > screenshot
      > dom
      > wget
      > readability
      > mercury
      > media
        49 files (1.7 MB) in 0:00:17s 

[+] [2023-08-16 00:01:06] "AltaVista"
    https://www.altavista.com
    > ./archive/1692144066.033886
      > favicon
      > singlefile
      > media
        8 files (5.9 MB) in 0:02:02s 

[+] [2023-08-16 00:03:09] "Yahoo"
    https://www.yahoo.com
    > ./archive/1692144066.033843
      > favicon
      > singlefile
      > media
        8 files (59.3 MB) in 0:23:42s 

[pirate]

Ah, the reason subtask progress lines are suppressed in server logs is because the server expects to be able to handle Add tasks from several users in parallel. If it showed subtask progress lines like > favicon interleaved from multiple workers it would be impossible to tell which Snapshot/URL they're for without adding additional logic to prefix each line or change the output somehow. The ArchiveResults entries also already contain all the same information in SQLite.

[melyux]

That makes sense. Still a shame to not see that detail though... If it's possible, maybe only flushing the output after each entire snapshot is done could work?