Perform health check before switching container to avoid downtime in deployment

[louis-l]

Hi there,

Thanks for the awesome package, that helps me a lot as a solo dev.
I have been using Caprover for years and been loving it.

Recently, I started to notice that there are a longer downtime in every deployment. Maybe I did not notice it in the past or my apps/services are getting bigger, but there is a ~5s where I will get a 502 HTTP status after each deployment.

I've been searching around and found #661 and #191. Still no absolute solution for this issue.
I end up with having to deploy my apps in a very early morning to avoid problem.
I understand this is a very edge case, but it would be great if we can avoid that downtime or at least reduce it to minimum.

I came across service https://www.checklyhq.com/ and I think the idea is great. Maybe we can implement something like health check (hit a certain URL and only switch the container if receives 2xx HTTP status).

Cheers,

[githubsaturn]

CapRover doesn't meddle with the deployment as it's fully managed by Docker. In other words, CapRover is just a wrapper for regular docker commands. Having said that, the good news is that Docker already has a built-in healthcheck. See here for an example:

• https://scoutapm.com/blog/how-to-use-docker-healthcheck

CapRover asks docker to update your regular apps (without any persistent directories) using start-first, and apps with persistent directories using stop-first method. Read here:

• https://docs.docker.com/engine/swarm/swarm-tutorial/rolling-update/

For stop-first, you will have to deal with downtime, because a new instance is created only after the old instance is completely killed.

For start-first, however, docker waits for the new instance to be healthy before killing the old instance, in this case, you must see a close to zero downtime.

Another quick win is to edit the DNS caching TTL:

• Log in to your CapRover dashboard
• Go to your app that has a long downtime,
• Go to HTTP Settings (first tab)
• Find this line resolver 127.0.0.11 valid=10s; and change 10s to 2s
• Does this reduce your downtime?

[louis-l]

Hi @githubsaturn

I tried running docker healthcheck but nothing happened, deployment was successful but the app is still running the in old container.

Here is my Dockerfile:

...

EXPOSE 80
COPY ./.deploy/production/entrypoint.sh /
ENTRYPOINT ["sh", "/entrypoint.sh"]

COPY ./.deploy/production/healthcheck.sh /
HEALTHCHECK --interval=2s --timeout=5m --start-period=3s CMD bash healthcheck.sh

and here is my healthcheck.sh script:

#!/bin/sh

code=$(curl -o /dev/null -s -w "%{http_code}\n" http://localhost/ping)
echo "response code: $code"

if [ "$code" == "200" ]
then
  echo "success"
  exit 0;
else
  echo "error"
  exit 1;
fi

The /ping route just simply returns 200 "ok":

Route::get('/ping', function () {
    return 'ok';
});

and here is the build output from CapRover:

Can you please shed some light on here?

Thanks.

[githubsaturn]

Looks okay to me. What do you mean by "app is still running the in old container"? Does it mean that the new version wasn't successfully deployed? Or it means that the HEALTHCHECK wasn't successful and you still had issues with downtime?

[louis-l]

@githubsaturn

Yes, the new version wasn't successfully deployed. Even though it said "Build has finished successfully!". (I've double checked this because I manually open the browser the hit "mydomain.com/ping" and got 404 instead of 200 "ok")
I also have a Slack action in Github that runs after the "deploy action" and that Slack action run as normal (I still can receive a notification after all).

This is my docker ps now:

I intensionally echo something out in the healthcheck.sh to make sure it gets run correctly, but I could not see any of those outputs in the CapRover build logs (see 1 of the screenshots above, line 280 is where the healthcheck.sh run)

[githubsaturn]

I intensionally echo something out in the healthcheck.sh to make sure it gets run correctly, but I could not see any of those outputs in the CapRover build logs (see 1 of the screenshots above, line 280 is where the healthcheck.sh run)

So this is expected I think. Docker doesn't output the healthcheck logs in the build logs or runtime container logs, see here for example.

Having said that, what you said here is a good indication that your image wasn't updated:

(I've double checked this because I manually open the browser the hit "mydomain.com/ping" and got 404 instead of 200 "ok")

Building successfully doesn't mean that your image can be deployed successfully, for example:

FROM nginx
CMD does-not-exist.sh

You can build this dockerfile successfully, but running it fails immediately. Hence you cannot deploy this image.

Now back to your issue. I can't tell what is exactly wrong with your app. But I'll put together some basic example of HEALTHCHECK in the docs at some point as a reference. In the meantime, feel free to post on stackoverflow. As I mentioned, what you're trying to do is not specific and limited to CapRover at all. It's more of a Docker question, and someone familiar with inner workings of HEALTHCHECK might be able to provide an answer.

[louis-l]

Thanks @githubsaturn
Appreciate your help here.

[louis-l]

@githubsaturn

I posted on StackOverflow and still have not receive any feedback.
I guess this is sort of a very edge use case.

In the meantime, is it possible to add a "sleep" or "delay" before switching the container to live?
I guess that "switch" action is performed by CapRover, right?

Maybe an optional env (e.g. CAPROVER_BEFORE_LIVE_DELAY) and if set, will wait x seconds before switching.
This does not actually solve the problem, but I rather wait longer to deploy than having a 502 downtime.

What do you say?

[githubsaturn]

I guess that "switch" action is performed by CapRover, right?

No, it is managed by Docker swarm. Here is a rough timeline of events:

1. CapRover receives the source code
2. CapRover asks Docker to build the image (this can take several minutes)
3. Once image is built, CapRover tags the image properly and it versions it, example img-captain--yourapp:5
4. CapRover asks Docker to update the image, CLI equivalent is docker service update srv-captain--myapp --image img-captain--yourapp:5
5. Docker internally ensures that the image exists (it pulls the image if you used CI/CD to build the image)
6. If the service is using start-first, Docker first spins up the new container, then it kills the old container. Otherwise, if the service is using stop-first, Docker first kills the old container and then start the new one
7. When requests come to the server, Nginx will asks Docker what IP address it should use for srv-captain--yourapp. At this point, Docker is supposed to provide nginx with the Docker container address that is up and healthy

As you see here, there is no step where an artificial delay would make things any better.

There are two cases that could result in 502:

• The current DNS caching (10s) is too long for your use case. As I mentioned above, you can try reducing this to 2s to less by editing nginx configurations
• Another scenario could be that Docker provides the IP address of the container to Nginx before it is fully ready to accept requests. I assume a proper HEALTHCHECK would be the solution to this.

[louis-l]

@githubsaturn

Thanks for the thorough explanation 🙏 I have a clearer picture now.

Im planning to try to change the DSN caching, but will have to wait until early next week (because our app is busy close to the weekend, so trying not to avoid downtime as much as possible).

In the meantime, I will keep digging for the answer of docker healthcheck command.

[devxpy]

The healthcheck command seems to work well for me too

HEALTHCHECK CMD wget 127.0.0.1:8000 || exit 1

alternatively, for a celery service

HEALTHCHECK CMD bash -c 'celery -A celeryapp inspect ping -d celery@$HOSTNAME' || exit 1