Unable to Start PDS Service

[thedepartedwhiplash]

Hello,

I'm going through the steps to host a PDS on my Vultr server and got to the point where I need to run the installer.sh file. However, the service related to the PDS refuses to start for some reason:

Is there some permission issue or something I'm missing here?

[bnewbold]

Hi! Could you give more details about your setup? Specifically:

• what specific operating system, distribution, and version are you running?
• what CPU instruction set (eg, x86 vs ARM64)
• what do the service logs say?
• is the server an existing one used for other projects, or set up from scratch just for running the PDS?

[thedepartedwhiplash]

Ubuntu 22.04 LTS, x86_64.

I tried running journalctl pds.service but the logs have no entries. This is a fresh server I set up just for the PDS.

[bnewbold]

Hmmm.

Can you try docker logs <container>, or going to /pds/ and trying to do docker-compose up there and see if anything helpful is logged?

[thedepartedwhiplash]

When I try to list all docker containers, nothing shows up. So there's no container to check the logs of. And going to /pds/ gives me a permission denied error.

[bnewbold]

I'm not sure what to recommend next! Sorry about that, hopefully somebody else can chime in next week.

[thedepartedwhiplash]

No worries. Thanks for trying. Really appreciate it regardless.

[thedepartedwhiplash]

Hello. I have an update. I tried recreating the server and logging in as a root user (I had previously used a limited user with sudo access), before running the script. The PDS service still failed but I actually got logs this time. This is what they say. Is it a firewall issue on my end?

[TheOpponent]

I'm running Debian 12 x86_64 and I can't get the Docker container constructed with the installer script. This is a pre-existing VPS with Docker already installed. I have to use docker-compose up and not docker compose to start containers, so when I run the former in /pds, this is the output:

ERROR: The Compose file './compose.yaml' is invalid because:
services.watchtower.environment.WATCHTOWER_CLEANUP contains true, which is an invalid type, it should be a string, number, or a null

The systemctl status pds output shows the Docker help message for invalid commands, followed by:

Apr 08 11:49:46 *** docker[1731847]: Run 'docker COMMAND --help' for more information on a command.
Apr 08 11:49:46 *** docker[1731847]: To get more help with docker, check out our guides at https://docs.docker.com/go/guides/
Apr 08 11:49:46 *** systemd[1]: pds.service: Main process exited, code=exited, status=125/n/a
Apr 08 11:49:46 *** systemd[1]: pds.service: Failed with result 'exit-code'.
Apr 08 11:49:46 *** systemd[1]: Failed to start pds.service - Bluesky PDS Service.

[TheOpponent]

If I edit compose.yaml and change line 38 to WATCHTOWER_CLEANUP: "true", the installation proceeds, but the container isn't able to start because port 443 is already in use. I have nginx and docker-mailserver running on this VPS. Is it not possible to run a webserver concurrently with PDS?

Attaching to watchtower, pds, caddy
caddy         | {"level":"info","ts":1712592514.191024,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy         | {"level":"info","ts":1712592514.1998582,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy         | {"level":"info","ts":1712592514.2005699,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy         | {"level":"info","ts":1712592514.2009788,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy         | {"level":"warn","ts":1712592514.2025695,"logger":"tls","msg":"unable to get instance ID; storage clean stamps will be incomplete","error":"open /data/caddy/instance.uuid: no such file or directory"}
caddy         | {"level":"info","ts":1712592514.256755,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0001a3580"}
caddy         | {"level":"info","ts":1712592514.25912,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0001a3580"}
caddy         | Error: loading initial config: loading new config: http app module: start: listening on :443: listen tcp :443: bind: address already in use
watchtower    | time="2024-04-08T16:08:34Z" level=info msg="Watchtower 1.7.1"
watchtower    | time="2024-04-08T16:08:34Z" level=info msg="Using no notifications"
watchtower    | time="2024-04-08T16:08:34Z" level=info msg="Checking all containers (except explicitly disabled with label)"
watchtower    | time="2024-04-08T16:08:34Z" level=info msg="Scheduling first run: 2024-04-09 00:00:00 +0000 UTC"
watchtower    | time="2024-04-08T16:08:34Z" level=info msg="Note that the first check will be performed in 7 hours, 51 minutes, 25 seconds"
caddy         | {"level":"info","ts":1712592515.5695665,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy         | {"level":"info","ts":1712592515.5767832,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy         | {"level":"info","ts":1712592515.577814,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy         | {"level":"info","ts":1712592515.578313,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy         | {"level":"error","ts":1712592515.58578,"logger":"tls","msg":"could not clean default/global storage","error":"unable to acquire storage_clean lock: context canceled"}
caddy         | {"level":"info","ts":1712592515.5858023,"logger":"tls","msg":"finished cleaning storage units"}
caddy         | {"level":"info","ts":1712592515.5863602,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00036e600"}
caddy         | {"level":"info","ts":1712592515.5863752,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00036e600"}
caddy         | Error: loading initial config: loading new config: http app module: start: listening on :443: listen tcp :443: bind: address already in use
caddy exited with code 1
caddy         | {"level":"info","ts":1712592516.5577157,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy         | {"level":"info","ts":1712592516.5632362,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//[::1]:2019","//127.0.0.1:2019","//localhost:2019"]}
caddy         | {"level":"info","ts":1712592516.5635664,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy         | {"level":"info","ts":1712592516.5635881,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy         | {"level":"error","ts":1712592516.5654383,"logger":"tls","msg":"could not clean default/global storage","error":"unable to acquire storage_clean lock: context canceled"}
caddy         | {"level":"info","ts":1712592516.5658026,"logger":"tls","msg":"finished cleaning storage units"}
caddy         | {"level":"info","ts":1712592516.5664747,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00041a500"}
caddy         | {"level":"info","ts":1712592516.5668347,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00041a500"}
caddy         | Error: loading initial config: loading new config: http app module: start: listening on :443: listen tcp :443: bind: address already in use
caddy exited with code 1
caddy         | {"level":"info","ts":1712592517.881893,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy         | {"level":"info","ts":1712592517.8902488,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy         | {"level":"info","ts":1712592517.8905945,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy         | {"level":"info","ts":1712592517.890623,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy         | {"level":"error","ts":1712592517.8933687,"logger":"tls","msg":"could not clean default/global storage","error":"unable to acquire storage_clean lock: context canceled"}
caddy         | {"level":"info","ts":1712592517.8970695,"logger":"tls","msg":"finished cleaning storage units"}
caddy         | {"level":"info","ts":1712592517.8990176,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00039e100"}
caddy         | {"level":"info","ts":1712592517.8994105,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00039e100"}
caddy         | Error: loading initial config: loading new config: http app module: start: listening on :443: listen tcp :443: bind: address already in use
caddy exited with code 1
caddy         | {"level":"info","ts":1712592519.9657705,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy         | {"level":"info","ts":1712592519.9736116,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[::1]:2019"]}
caddy         | {"level":"info","ts":1712592519.9743884,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy         | {"level":"info","ts":1712592519.9747581,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy         | {"level":"error","ts":1712592519.9808142,"logger":"tls","msg":"could not clean default/global storage","error":"unable to acquire storage_clean lock: context canceled"}
caddy         | {"level":"info","ts":1712592519.9812784,"logger":"tls","msg":"finished cleaning storage units"}
caddy         | {"level":"info","ts":1712592519.9833608,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00044e200"}
caddy         | {"level":"info","ts":1712592519.9837453,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00044e200"}
caddy         | Error: loading initial config: loading new config: http app module: start: listening on :443: listen tcp :443: bind: address already in use
caddy exited with code 1
caddy         | {"level":"info","ts":1712592522.2955298,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy         | {"level":"info","ts":1712592522.3570905,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy         | {"level":"info","ts":1712592522.3586051,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy         | {"level":"info","ts":1712592522.3590078,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy         | {"level":"info","ts":1712592522.3602543,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00039c280"}
caddy         | {"level":"info","ts":1712592522.364699,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00039c280"}
caddy         | Error: loading initial config: loading new config: http app module: start: listening on :443: listen tcp :443: bind: address already in use
caddy exited with code 1
pds           | {"level":30,"time":1712592524752,"pid":6,"hostname":"jme","name":"pds","msg":"pds has started"}
caddy         | {"level":"info","ts":1712592526.0711071,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy         | {"level":"info","ts":1712592526.0757196,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//127.0.0.1:2019","//localhost:2019","//[::1]:2019"]}
caddy         | {"level":"info","ts":1712592526.0765042,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy         | {"level":"info","ts":1712592526.0768962,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy         | {"level":"error","ts":1712592526.0781076,"logger":"tls","msg":"could not clean default/global storage","error":"unable to acquire storage_clean lock: context canceled"}
caddy         | {"level":"info","ts":1712592526.0784888,"logger":"tls","msg":"finished cleaning storage units"}
caddy         | {"level":"info","ts":1712592526.0790775,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003aa480"}
caddy         | {"level":"info","ts":1712592526.0794213,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0003aa480"}
caddy         | Error: loading initial config: loading new config: http app module: start: listening on :443: listen tcp :443: bind: address already in use
caddy exited with code 1
caddy         | {"level":"info","ts":1712592532.957881,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy         | {"level":"info","ts":1712592532.962767,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy         | {"level":"info","ts":1712592532.9637918,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy         | {"level":"info","ts":1712592532.9641728,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy         | {"level":"error","ts":1712592532.9655178,"logger":"tls","msg":"could not clean default/global storage","error":"unable to acquire storage_clean lock: context canceled"}
caddy         | {"level":"info","ts":1712592532.9659052,"logger":"tls","msg":"finished cleaning storage units"}
caddy         | {"level":"info","ts":1712592532.9665134,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003aa400"}
caddy         | {"level":"info","ts":1712592532.9669194,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0003aa400"}
caddy         | Error: loading initial config: loading new config: http app module: start: listening on :80: listen tcp :80: bind: address already in use
caddy exited with code 1
caddy         | {"level":"info","ts":1712592546.1855721,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy         | {"level":"info","ts":1712592546.1915116,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
caddy         | {"level":"info","ts":1712592546.1923223,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy         | {"level":"info","ts":1712592546.1928625,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy         | {"level":"info","ts":1712592546.1938486,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003a8480"}
caddy         | {"level":"info","ts":1712592546.1942017,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0003a8480"}
caddy         | Error: loading initial config: loading new config: http app module: start: listening on :443: listen tcp :443: bind: address already in use
caddy exited with code 1
caddy         | {"level":"info","ts":1712592572.5820827,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
caddy         | {"level":"info","ts":1712592572.586001,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//[::1]:2019","//127.0.0.1:2019","//localhost:2019"]}
caddy         | {"level":"info","ts":1712592572.5866272,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
caddy         | {"level":"info","ts":1712592572.587016,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy         | {"level":"error","ts":1712592572.58846,"logger":"tls","msg":"could not clean default/global storage","error":"unable to acquire storage_clean lock: context canceled"}
caddy         | {"level":"info","ts":1712592572.5888586,"logger":"tls","msg":"finished cleaning storage units"}
caddy         | {"level":"info","ts":1712592572.5894616,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003ac480"}
caddy         | {"level":"info","ts":1712592572.589842,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0003ac480"}
caddy         | Error: loading initial config: loading new config: http app module: start: listening on :443: listen tcp :443: bind: address already in use
caddy exited with code 1

[devinivy]

You can run a PDS next to a web server. The setup our installer is catered towards assumes that you will run Caddy as a reverse proxy, and it will manage the certs for your PDS.

If you'd rather do this yourself, e.g. with nginx rather than Caddy, you can configure nginx to point at the PDS, which runs locally on port 3000 by default. Typically you'll want to support both yourpdsdomain.com for the PDS itself and *.yourpdsdomain.com to provide handles to your PDS's users. Our Caddy setup deals with all of this, but it's totally possible to do yourself with nginx and letsencrypt or similar.

[TheOpponent]

How do I disable the Caddy?

I think the documentation should be more explicit about support for users to configure their own reverse proxy for the daemon because it isn't made clear that anything other than the default services is permitted.