[Bug]: "Bad Gateway" error on homepage when servers runs out of storage

[RayBB]

Description

"Bad Gateway" error on homepage when servers runs out of storage

Minimal Reproduction (if possible, example repository)

I installed coolify on my very tiny VPS (only 5gb available) with debian.
I tried to setup one small instance of tailscale and something else.
I came back 2 days later and see a "Bad Gateway" error on home page.

When I ssh into the machine I run journalctl -xe and see:
Journal file /var/log/journal/5ceb8f5c910b765bdb9da53f975ab6fb/system.journal is truncated, ignoring file. Jan 07 16:51:48 kvm-WorthwhileBrisk-VM rsyslogd[269]: action 'action-1-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error mess>

When I run df -h I see the storage is at 100% usage. I'm not sure what it was at before but I think there was some space available.

Unfortunately, I already rebooted the VM so I can't start docker to clear things. Fortunately, important is on here so I'll just nuke it in a day or so and try again.

Anyway, this isn't your fault and I don't know if there is really a bug but maybe the experience can be improved.

1. Show something besides bad gateway? Not sure if that's possible
2. Maybe have an easy option to clear of space (though I feel this is coming in some future thing).

Just flagging, let me know if you want me to help you reproduce this or send different logs.

Exception or Error

No response

Version

v4.0.0-beta.182

[IwishIcanFLighT]

I had the almost same error, but only after 2 minutes from a fresh boot of my VPS (1vCore/1Gb RAM/ 10Gb storage). My VPS became very instable, barely able to even perform a simple ls -l.
I had to hard reboot it from my hosting provider's dashboard where I also found out that my instance's disk reads went through the roof as soon as Coolify started.

After doing some research, I found #1482 which hinted me to the fact that my VPS didn't have any swap file in place. After creating a 1Gb swapfile, I restarted Coolify which instantly started using the swap (195M out of 1024M as I type this). My VPS seems to be much stable now.

[RayBB]

I ran into the bad gateway again when storage was full.
Reset my box to defaults, installed coolify but didn't try to setup any services or anything.
My storage currently is sitting at 3.55 GB of 5 GB Used/1.45 GB Free

I'm going to check back in a few days and see if the storage gets all filled up again without it doing anything.

[RayBB]

Right now it says 3.97 GB of 5 GB Used/1.03 GBFree so about 400mb more are being used now without having installed anything.

[RayBB]

Now the VPS is at 4.19 GB of 5 GB Used/826.29 MBFree

[RayBB]

So I think... I figured it out somewhat.
running journalctl I see every second or so a message like this:
run-docker-runtime\x2drunc-moby-b9a4c56d5830a2d1e21bc537b535b3860d3d6ce917329f61f29f1455c51b8521-runc.XDlk09.mount: Succeeded
Running journalctl --disk-usage shows I'm using 456MB of disk for this.

Which is related to Docker flooding syslog with run-docker-runtime logs. Which ultimately links to systemd logs filled with mount unit entries if healtcheck is enabled.

Long story short, I think it'll be addressed with runc 1.2. Which then needs to be released by containerd here.

Until then, you can take your pick of temp fixes with filtering or limiting the log size mentioned in the above links such as:
set SystemMaxUse=50M in /etc/systemd/journald.conf or similar as mentioned here. SystemMaxUse defaults to 10% of the filesystem size.

This leaves an question open for me:
How did a 5GB system (where 500MB can be used for journalctl), start with 1.45GB free and currently have 826MB free? I don't think I've really installed or done anything else. Though perhaps I filled it up last time by trying to update coolify and not having enough space, which took up just enough of the remaining space.

For now, I'll leave everything with the default configs and see if the disk fills up again on its own.

[addvanced]

There's a temporary solution available now:
https://stackoverflow.com/a/78425717