Dokku slow to respond and sometimes errors (502)

[maximilianschmitt]

Hi, I'm a big fan of Dokku and I've been using it for many years.

Recently, I deployed an app that uses a custom Dockerfile for the first time and I'm running into some issues while running it.

When I run HTTP requests against the app, it's sometimes very slow to respond (1+ second, just for a static file) and I sometimes get a 502 error response. When I check my app's logs (dokku logs <appName>) there is no indication of any failures, so I'm suspecting that the issue is somewhere "in front of" my app. Maybe on the NGINX layer or somewhere else?

Where can I look in order to debug this?

Btw, the app is a simple Node.js Express app, mostly serving static files but it has a few routes for Puppeteer which is why I needed the custom Dockerfile.

I'm thankful for any tips!

502 error in the network tab:

Dockerfile:

# Use an official Node.js runtime as a parent image
FROM node:18

# Install puppeteer dependencies
RUN apt-get update && apt-get install -y \
    gconf-service \
    libasound2 \
    libatk1.0-0 \
    libc6 \
    libcairo2 \
    libcups2 \
    libdbus-1-3 \
    libexpat1 \
    libfontconfig1 \
    libgcc1 \
    libgconf-2-4 \
    libgdk-pixbuf2.0-0 \
    libglib2.0-0 \
    libgtk-3-0 \
    libnspr4 \
    libpango-1.0-0 \
    libpangocairo-1.0-0 \
    libstdc++6 \
    libx11-6 \
    libx11-xcb1 \
    libxcb1 \
    libxcomposite1 \
    libxcursor1 \
    libxdamage1 \
    libxext6 \
    libxfixes3 \
    libxi6 \
    libxrandr2 \
    libxrender1 \
    libxss1 \
    libxtst6 \
    ca-certificates \
    fonts-liberation \
    libappindicator1 \
    libnss3 \
    lsb-release \
    xdg-utils \
    wget

WORKDIR /app

COPY . .

RUN yarn

WORKDIR /app/webapp

# Build the app
RUN yarn build

# Run the app when the container launches
CMD ["yarn", "ts-node", "--transpileOnly", "src/server/server.ts"]

App logs:

2024-02-02T08:43:31.357783158Z app[web.1]: ::ffff:172.17.0.1 - GET /?sysLang=en&autofocus=0 HTTP/1.1 304 - - 3.025 ms
2024-02-02T08:43:31.392490277Z app[web.1]: ::ffff:172.17.0.1 - GET /assets/index.959ab157.css HTTP/1.1 304 - - 0.491 ms
2024-02-02T08:43:31.405148742Z app[web.1]: ::ffff:172.17.0.1 - GET /assets/index.e61720f6.js HTTP/1.1 304 - - 3.727 ms
2024-02-02T08:43:31.418855627Z app[web.1]: ::ffff:172.17.0.1 - GET /assets/vendor.ef4e3848.js HTTP/1.1 200 994801 - 1.645 ms
2024-02-02T08:43:34.620391097Z app[web.1]: ::ffff:172.17.0.1 - GET /assets/currencies.json HTTP/1.1 304 - - 1.452 ms
2024-02-02T08:43:34.622013912Z app[web.1]: ::ffff:172.17.0.1 - GET /assets/countryCodes/countryCodes.en.json HTTP/1.1 304 - - 2.243 ms
2024-02-02T08:43:34.668627598Z app[web.1]: ::ffff:172.17.0.1 - GET /assets/countryCodes/countryCodes.en.json HTTP/1.1 304 - - 0.520 ms
2024-02-02T08:43:34.682342081Z app[web.1]: ::ffff:172.17.0.1 - GET /invoices/templates HTTP/1.1 304 - - 0.425 ms
2024-02-02T08:43:35.194436903Z app[web.1]: ::ffff:172.17.0.1 - GET /assets/templateStrings/templateStrings.en.json HTTP/1.1 304 - - 0.499 ms
2024-02-02T08:43:35.268764756Z app[web.1]: ::ffff:172.17.0.1 - GET /assets/invoiceTemplates/_master/template.json HTTP/1.1 304 - - 0.727 ms
2024-02-02T08:43:35.293647592Z app[web.1]: ::ffff:172.17.0.1 - GET /assets/invoiceTemplates/plain/template.ejs HTTP/1.1 304 - - 0.516 ms
2024-02-02T08:43:35.316350317Z app[web.1]: ::ffff:172.17.0.1 - GET /assets/invoiceTemplates/plain/column-width-calculations.ejs HTTP/1.1 304 - - 0.836 ms
2024-02-02T08:43:35.344629964Z app[web.1]: ::ffff:172.17.0.1 - GET /assets/index.959ab157.css HTTP/1.1 304 - - 0.464 ms

[maximilianschmitt]

I discovered in my NGINX error logs that there are error when connecting to the Docker container:

$ dokku2 nginx:error-logs cakedesk-webapp
2024/02/04 06:59:42 [error] 1270284#1270284: *85124 connect() failed (113: Unknown error) while connecting to upstream, client: 84.176.229.64, server: cakedesk-webapp.maxschmitt.me, request: "GET /assets/invoiceTemplates/plain/styles.css HTTP/2.0", upstream: "http://172.17.0.20:5000/assets/invoiceTemplates/plain/styles.css", host: "web.cakedesk.app", referrer: "https://web.cakedesk.app/?sysLang=en&autofocus=0"

I'm able to reproduce this every few requests using cURL:

$ curl http://172.17.0.20:5000;
<!DOCTYPE html>
<html lang="en">
	<head>
		<meta charset="UTF-8" />
		<meta name="viewport" content="width=device-width, initial-scale=1.0" />
		<title>Cakedesk</title>

		<script type="module" crossorigin src="/assets/index.e61720f6.js"></script>
		<link rel="modulepreload" href="/assets/vendor.ef4e3848.js">
		<link rel="stylesheet" href="/assets/index.959ab157.css">
	</head>
	<body>
		<div id="app"></div>

	</body>
</html>
$ curl http://172.17.0.20:5000;
curl: (7) Failed to connect to 172.17.0.20 port 5000 after 359 ms: No route to host

Now I'm trying to figure out why this is happening. Sometimes it takes a few seconds before I get the "Failed to connect", sometimes just a few hundred ms (like above).

I haven't been able to reproduce these errors for any other Dokku apps running on my server.

If I build and run the container locally on my machine, I can't reproduce the errors that I'm experiencing on my server.

I checked the load of my server in my DigitalOcean control panel and it doesn't seem like the load is too high on the server:

If I run docker stats I don't see the container consuming too many resources:

However, if I change the Dockerfile to no longer install the Puppeteer dependencies, the connection issues seem to go away:

# Use an official Node.js runtime as a parent image
FROM node:18

WORKDIR /app

COPY . .

RUN yarn

WORKDIR /app/webapp

# Build the app
RUN yarn build

# Run the app when the container launches
CMD ["yarn", "ts-node", "--transpileOnly", "src/server/server.ts"]

So I do wonder if the issue is that my droplet is too weak. But then I would expect to see significant resource usage for the docker container or the droplet overall in htop but even that looks ok:

---

Using the official Puppeteer Dockerfile makes the Docker container even more unresponsive and the error rate increases.

For a test, I deployed the official Dockerfile and then reverted back to the old one (from the original post above) and now I'm not running into any more issues, strangely.

I wonder if these issues will start occurring after some time...

[josegonzalez]

One thing I see there is that your non-puppeteer image used Node 18, while the Puppeteer one uses node 20. Not sure if thats important.