Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Traefik keeps restarting all the time #1500

Open
abriginets opened this issue Oct 11, 2024 · 1 comment
Open

[Bug]: Traefik keeps restarting all the time #1500

abriginets opened this issue Oct 11, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@abriginets
Copy link

abriginets commented Oct 11, 2024 •
edited
Loading

Description

I have created a new cluster recently and deployed a couple of web services there along with TLS for each one of them. And finally I put the Hetzner LB behind CloudFlare. And that's when I started getting 525 SSL Handshake Error from CloudFlare. Investigation led me to traefik pod logs which shows that traefik is being restarted quite a lot and I can't seem to understart a true reason behind it.

kubectl describe -n traefik pod traefik-75fd586479-vsgvx

  Type     Reason     Age                   From     Message
  ----     ------     ----                  ----     -------
  Warning  Unhealthy  58m (x69 over 9h)     kubelet  Readiness probe failed: Get "http://10.42.3.14:9000/ping": dial tcp 10.42.3.14:9000: connect: connection refused
  Warning  Unhealthy  3m10s (x114 over 9h)  kubelet  Liveness probe failed: Get "http://10.42.3.14:9000/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  Warning  Unhealthy  3m8s (x301 over 9h)   kubelet  Readiness probe failed: Get "http://10.42.3.14:9000/ping": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

kubectl get pods -n traefik -o wide

NAME                       READY   STATUS    RESTARTS         AGE     IP              NODE                          NOMINATED NODE   READINESS GATES
traefik-75fd586479-cbrkt   1/1     Running   0                3m29s   10.42.102.249   k3s-worker-17440953fcddcf01   <none>           <none>
traefik-75fd586479-k5b5j   1/1     Running   0                3m29s   10.42.102.250   k3s-worker-17440953fcddcf01   <none>           <none>
traefik-75fd586479-mf4wq   1/1     Running   0                3m29s   10.42.131.7     k3s-workload-amd64-xbl        <none>           <none>
traefik-75fd586479-vsgvx   1/1     Running   37 (2m33s ago)   10h     10.42.3.14      k3s-worker-71de43e0505e4dc5   <none>           <none>

Then I execute this command in terminal again after a couple of minutes

NAME                       READY   STATUS    RESTARTS         AGE   IP           NODE                          NOMINATED NODE   READINESS GATES
traefik-75fd586479-vsgvx   1/1     Running   37 (5m39s ago)   10h   10.42.3.14   k3s-worker-71de43e0505e4dc5   <none>           <none>

For some reason, traefik jumps from node to node.

Then I looked into traefik logs and here what it shows

kubectl logs -n traefik traefik-75fd586479-vsgvx --previous

Traefik logs 
kubectl logs -n traefik traefik-75fd586479-vsgvx --previous
2024-10-11T16:30:57Z INF Traefik version 3.1.5 built on 2024-10-02T12:49:07Z version=3.1.5
2024-10-11T16:30:57Z INF 
Stats collection is disabled.
Help us improve Traefik by turning this feature on :)
More details on: https://doc.traefik.io/traefik/contributing/data-collection/

2024-10-11T16:30:57Z INF Enabling ProxyProtocol for trusted IPs [127.0.0.1/32 10.0.0.0/8] entryPointName=web
2024-10-11T16:30:57Z INF Enabling ProxyProtocol for trusted IPs [127.0.0.1/32 10.0.0.0/8] entryPointName=websecure
2024-10-11T16:30:57Z INF Starting provider aggregator aggregator.ProviderAggregator
2024-10-11T16:30:57Z INF Starting provider *traefik.Provider
2024-10-11T16:30:57Z INF Starting provider *ingress.Provider
2024-10-11T16:30:57Z INF ingress label selector is: "" providerName=kubernetes
2024-10-11T16:30:57Z INF Creating in-cluster Provider client providerName=kubernetes
2024-10-11T16:30:57Z INF Starting provider *crd.Provider
2024-10-11T16:30:57Z INF label selector is: "" providerName=kubernetescrd
2024-10-11T16:30:57Z INF Creating in-cluster Provider client providerName=kubernetescrd
2024-10-11T16:30:57Z INF Starting provider *acme.ChallengeTLSALPN
2024-10-11T16:31:09Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.130.0:41942: read: connection reset by peer"
2024-10-11T16:31:13Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:42482: read: connection reset by peer"
2024-10-11T16:31:17Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.130.0:2956: read: connection reset by peer"
2024-10-11T16:31:20Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.131.0:35264: read: connection reset by peer"
2024-10-11T16:31:22Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.3.1:6725: read: connection reset by peer"
2024-10-11T16:31:26Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.102.0:11449: read: connection reset by peer"
2024-10-11T16:31:28Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:60139: read: connection reset by peer"
2024-10-11T16:31:30Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.3.1:26146: read: connection reset by peer"
2024-10-11T16:31:48Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:64538: read: connection reset by peer"
2024-10-11T16:31:50Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.131.0:62160: read: connection reset by peer"
2024-10-11T16:31:54Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.130.0:59993: read: connection reset by peer"
2024-10-11T16:31:58Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:22806: read: connection reset by peer"
2024-10-11T16:32:03Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:19873: read: connection reset by peer"
2024-10-11T16:32:17Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.130.0:40547: read: connection reset by peer"
2024-10-11T16:32:43Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:38871: read: connection reset by peer"
2024-10-11T16:32:50Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.131.0:45781: read: connection reset by peer"
2024-10-11T16:32:52Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.3.1:12022: read: connection reset by peer"
2024-10-11T16:32:54Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.130.0:17638: read: connection reset by peer"
2024-10-11T16:33:07Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.3.1:1189: read: connection reset by peer"
2024-10-11T16:33:24Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.130.0:43642: read: connection reset by peer"
2024-10-11T16:33:39Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.130.0:54101: read: connection reset by peer"
2024-10-11T16:33:41Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.102.0:60014: read: connection reset by peer"
2024-10-11T16:33:47Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.130.0:21735: read: connection reset by peer"
2024-10-11T16:34:18Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:4936: read: connection reset by peer"
2024-10-11T16:34:35Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.131.0:17052: read: connection reset by peer"
2024-10-11T16:34:47Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.130.0:9700: read: connection reset by peer"
2024-10-11T16:34:54Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.130.0:43955: read: connection reset by peer"
2024-10-11T16:35:11Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.102.0:56010: read: connection reset by peer"
2024-10-11T16:35:17Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.130.0:46808: read: connection reset by peer"
2024-10-11T16:35:24Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.130.0:56341: read: connection reset by peer"
2024-10-11T16:35:28Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:37627: read: connection reset by peer"
2024-10-11T16:35:35Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.131.0:23782: read: connection reset by peer"
2024-10-11T16:35:45Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.3.1:58842: read: connection reset by peer"
2024-10-11T16:35:52Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.3.1:29932: read: connection reset by peer"
2024-10-11T16:35:56Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.102.0:21478: read: connection reset by peer"
2024-10-11T16:36:00Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.3.1:18906: read: connection reset by peer"
2024-10-11T16:36:04Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:56715: read: connection reset by peer"
2024-10-11T16:36:05Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.131.0:41407: read: connection reset by peer"
2024-10-11T16:36:11Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.102.0:2593: read: connection reset by peer"
2024-10-11T16:36:13Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:48587: read: connection reset by peer"
2024-10-11T16:36:19Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:12493: read: connection reset by peer"
2024-10-11T16:36:20Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.131.0:55890: read: connection reset by peer"
2024-10-11T16:36:22Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.3.1:4663: read: connection reset by peer"
2024-10-11T16:36:24Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.130.0:59695: read: connection reset by peer"
2024-10-11T16:36:26Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.102.0:23891: read: connection reset by peer"
2024-10-11T16:36:28Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:51389: read: connection reset by peer"
2024-10-11T16:36:30Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.3.1:4682: read: connection reset by peer"
2024-10-11T16:36:32Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.130.0:54670: read: connection reset by peer"
2024-10-11T16:36:34Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:49066: read: connection reset by peer"
2024-10-11T16:36:35Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.131.0:31648: read: connection reset by peer"
2024-10-11T16:36:37Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.3.1:42948: read: connection reset by peer"
2024-10-11T16:36:39Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.130.0:3393: read: connection reset by peer"
2024-10-11T16:36:41Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.102.0:29542: read: connection reset by peer"
2024-10-11T16:36:43Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:47006: read: connection reset by peer"
2024-10-11T16:36:45Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.3.1:59415: read: connection reset by peer"
2024-10-11T16:36:47Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.130.0:46032: read: connection reset by peer"
2024-10-11T16:36:49Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:10751: read: connection reset by peer"
2024-10-11T16:36:50Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.131.0:8004: read: connection reset by peer"
2024-10-11T16:36:52Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.3.1:1877: read: connection reset by peer"
2024-10-11T16:36:54Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.130.0:57597: read: connection reset by peer"
2024-10-11T16:36:56Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.102.0:17810: read: connection reset by peer"
2024-10-11T16:36:58Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:60498: read: connection reset by peer"
2024-10-11T16:37:00Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.3.1:49148: read: connection reset by peer"
2024-10-11T16:37:02Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.130.0:30491: read: connection reset by peer"
2024-10-11T16:37:04Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:9393: read: connection reset by peer"
2024-10-11T16:37:05Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.131.0:13296: read: connection reset by peer"
2024-10-11T16:37:07Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.3.1:53422: read: connection reset by peer"
2024-10-11T16:37:09Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.130.0:37960: read: connection reset by peer"
2024-10-11T16:37:11Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.102.0:17739: read: connection reset by peer"
2024-10-11T16:37:13Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:11345: read: connection reset by peer"
2024-10-11T16:37:15Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.3.1:30884: read: connection reset by peer"
2024-10-11T16:37:17Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.130.0:43100: read: connection reset by peer"
2024-10-11T16:37:19Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:57897: read: connection reset by peer"
2024-10-11T16:37:20Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.131.0:17745: read: connection reset by peer"
2024-10-11T16:37:22Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.3.1:1079: read: connection reset by peer"
2024-10-11T16:37:24Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.130.0:11605: read: connection reset by peer"
2024-10-11T16:37:26Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.102.0:22391: read: connection reset by peer"
2024-10-11T16:37:28Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:30293: read: connection reset by peer"
2024-10-11T16:37:30Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.3.1:13250: read: connection reset by peer"
2024-10-11T16:37:32Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.130.0:7965: read: connection reset by peer"
2024-10-11T16:37:34Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:54181: read: connection reset by peer"
2024-10-11T16:37:35Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.131.0:16583: read: connection reset by peer"
2024-10-11T16:37:37Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.3.1:53389: read: connection reset by peer"
2024-10-11T16:37:39Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.130.0:30789: read: connection reset by peer"
2024-10-11T16:37:41Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.102.0:54004: read: connection reset by peer"
2024-10-11T16:37:43Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:33763: read: connection reset by peer"
2024-10-11T16:37:45Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.3.1:40997: read: connection reset by peer"
2024-10-11T16:37:47Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.130.0:64525: read: connection reset by peer"
2024-10-11T16:37:49Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:36071: read: connection reset by peer"
2024-10-11T16:37:50Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.131.0:12668: read: connection reset by peer"
2024-10-11T16:37:52Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.3.1:10462: read: connection reset by peer"
2024-10-11T16:37:54Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.130.0:18195: read: connection reset by peer"
2024-10-11T16:37:56Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.102.0:29207: read: connection reset by peer"
2024-10-11T16:37:58Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:19648: read: connection reset by peer"
2024-10-11T16:38:00Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.3.1:54209: read: connection reset by peer"
2024-10-11T16:38:02Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.130.0:29710: read: connection reset by peer"
2024-10-11T16:38:04Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:51228: read: connection reset by peer"
2024-10-11T16:38:05Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.131.0:44994: read: connection reset by peer"
2024-10-11T16:38:07Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.3.1:51831: read: connection reset by peer"
2024-10-11T16:38:09Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.130.0:54868: read: connection reset by peer"
2024-10-11T16:38:11Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.102.0:15643: read: connection reset by peer"
2024-10-11T16:38:13Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:4027: read: connection reset by peer"
2024-10-11T16:38:15Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.3.1:44883: read: connection reset by peer"
2024-10-11T16:38:17Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.130.0:64669: read: connection reset by peer"
2024-10-11T16:38:19Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:46981: read: connection reset by peer"
2024-10-11T16:38:20Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.131.0:44315: read: connection reset by peer"
2024-10-11T16:38:22Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.3.1:9132: read: connection reset by peer"
2024-10-11T16:38:24Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.130.0:31548: read: connection reset by peer"
2024-10-11T16:38:26Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.102.0:37932: read: connection reset by peer"
2024-10-11T16:38:28Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:10387: read: connection reset by peer"
2024-10-11T16:38:30Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.3.1:39554: read: connection reset by peer"
2024-10-11T16:38:32Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.130.0:14257: read: connection reset by peer"
2024-10-11T16:38:34Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:40483: read: connection reset by peer"
2024-10-11T16:38:35Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.131.0:30745: read: connection reset by peer"
2024-10-11T16:38:37Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.3.1:41608: read: connection reset by peer"
2024-10-11T16:38:39Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.130.0:64420: read: connection reset by peer"
2024-10-11T16:38:41Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.102.0:32593: read: connection reset by peer"
2024-10-11T16:38:43Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:53297: read: connection reset by peer"
2024-10-11T16:38:45Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.3.1:51223: read: connection reset by peer"
2024-10-11T16:38:47Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.130.0:1576: read: connection reset by peer"
2024-10-11T16:38:49Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:23271: read: connection reset by peer"
2024-10-11T16:38:50Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.131.0:18630: read: connection reset by peer"
2024-10-11T16:38:52Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.3.1:58694: read: connection reset by peer"
2024-10-11T16:38:54Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.130.0:35423: read: connection reset by peer"
2024-10-11T16:38:56Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.102.0:59727: read: connection reset by peer"
2024-10-11T16:38:58Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:55791: read: connection reset by peer"
2024-10-11T16:39:00Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.3.1:39101: read: connection reset by peer"
2024-10-11T16:39:02Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.130.0:23349: read: connection reset by peer"
2024-10-11T16:39:04Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:56860: read: connection reset by peer"
2024-10-11T16:39:05Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.131.0:59433: read: connection reset by peer"
2024-10-11T16:39:07Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.3.1:54419: read: connection reset by peer"
2024-10-11T16:39:09Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.130.0:3194: read: connection reset by peer"
2024-10-11T16:39:11Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.102.0:47675: read: connection reset by peer"
2024-10-11T16:39:13Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:21305: read: connection reset by peer"
2024-10-11T16:39:15Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.3.1:46435: read: connection reset by peer"
2024-10-11T16:39:17Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.130.0:26238: read: connection reset by peer"
2024-10-11T16:39:19Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:65219: read: connection reset by peer"
2024-10-11T16:39:20Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.131.0:63080: read: connection reset by peer"
2024-10-11T16:39:22Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.3.1:21911: read: connection reset by peer"
2024-10-11T16:39:24Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.130.0:14581: read: connection reset by peer"
2024-10-11T16:39:26Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.102.0:16629: read: connection reset by peer"
2024-10-11T16:39:28Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:47622: read: connection reset by peer"
2024-10-11T16:39:30Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.3.1:16943: read: connection reset by peer"
2024-10-11T16:39:34Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:18603: read: connection reset by peer"
2024-10-11T16:39:35Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.131.0:24521: read: connection reset by peer"
2024-10-11T16:39:37Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.3.1:33371: read: connection reset by peer"
2024-10-11T16:39:38Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.131.0:14758: read: connection reset by peer"
2024-10-11T16:39:40Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.3.1:37225: read: connection reset by peer"
2024-10-11T16:39:43Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.102.0:19001: read: connection reset by peer"
2024-10-11T16:39:45Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:18821: read: connection reset by peer"
2024-10-11T16:39:48Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.3.1:25361: read: connection reset by peer"
2024-10-11T16:39:50Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:48834: read: connection reset by peer"
2024-10-11T16:39:53Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.131.0:17658: read: connection reset by peer"
2024-10-11T16:39:55Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.3.1:28881: read: connection reset by peer"
2024-10-11T16:39:58Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.102.0:31312: read: connection reset by peer"
2024-10-11T16:40:00Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:21971: read: connection reset by peer"
2024-10-11T16:40:03Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.3.1:44184: read: connection reset by peer"
2024-10-11T16:40:05Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:15407: read: connection reset by peer"
2024-10-11T16:40:08Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.131.0:52434: read: connection reset by peer"
2024-10-11T16:40:10Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.3.1:25737: read: connection reset by peer"
2024-10-11T16:40:13Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8000->10.42.102.0:29679: read: connection reset by peer"
2024-10-11T16:40:14Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:46834: read: connection reset by peer"
2024-10-11T16:40:16Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.3.1:8303: read: connection reset by peer"
2024-10-11T16:40:19Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:46922: read: connection reset by peer"
2024-10-11T16:40:21Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:63431: read: connection reset by peer"
2024-10-11T16:40:24Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.3.1:36068: read: connection reset by peer"
2024-10-11T16:40:26Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:55451: read: connection reset by peer"
2024-10-11T16:40:29Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:57147: read: connection reset by peer"
2024-10-11T16:40:31Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.3.1:37188: read: connection reset by peer"
2024-10-11T16:40:34Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:32515: read: connection reset by peer"
2024-10-11T16:40:36Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:48865: read: connection reset by peer"
2024-10-11T16:40:39Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.3.1:26165: read: connection reset by peer"
2024-10-11T16:40:41Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:10806: read: connection reset by peer"
2024-10-11T16:40:44Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:49117: read: connection reset by peer"
2024-10-11T16:40:46Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.3.1:26946: read: connection reset by peer"
2024-10-11T16:40:49Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.102.0:17212: read: connection reset by peer"
2024-10-11T16:41:04Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.3.1:65230: read: connection reset by peer"
2024-10-11T16:41:17Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:8443->10.42.131.0:18003: read: connection reset by peer"
2024-10-11T16:41:22Z INF I have to go...
2024-10-11T16:41:52Z INF Stopping server gracefully

It just shuts down gracefuly and that's it. So I kept monitoring it and sometimes I get additional errors use of closed network connection:

2024-10-11T15:57:29Z INF I have to go...
2024-10-11T15:57:35Z INF Stopping server gracefully
2024-10-11T15:57:36Z ERR error="accept tcp [::]:9100: use of closed network connection" entryPointName=metrics
2024-10-11T15:57:42Z ERR error="accept tcp [::]:9000: use of closed network connection" entryPointName=traefik
2024-10-11T15:57:42Z ERR error="accept tcp [::]:8443: use of closed network connection" entryPointName=websecure
2024-10-11T15:57:48Z ERR Error while Peeking first byte error="read tcp 10.42.3.14:9000->10.42.3.1:50102: use of closed network connection"
2024-10-11T15:57:42Z ERR error="accept tcp [::]:8000: use of closed network connection" entryPointName=web
2024-10-11T15:58:16Z INF Shutting down
2024-10-11T15:58:16Z INF Server stopped

I thought maybe this could happen because I have 0 agent_nodepools nodes, only autoscaled ones, where I already deployed high traffic web apps that could easily consume a lof of CPU and RAM. I've added a new node to the agent just to make sure, and traefik pod did occupy that spare node, yet restarted again.

image

I'm completely out of ideas at this point. Traefik is not crashing, it just keeps restarting. Traefik docs says that the stop signal comes from above.

Kube.tf file

locals {
  hcloud_token = "xxxxxxxxxxx"
}

module "kube-hetzner" {
  providers = {
    hcloud = hcloud
  }
  hcloud_token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
  source = "kube-hetzner/kube-hetzner/hcloud"
  ssh_public_key = file("~/.ssh/id_ed25519.pub")
  ssh_private_key = file("~/.ssh/id_ed25519")
  network_region = "eu-central" 
  control_plane_nodepools = [
    {
      name        = "control-plane-fsn1",
      server_type = "cax11",
      location    = "fsn1",
      labels      = ["k8s"],
      taints      = [],
      count       = 1,
      swap_size   = "2G"
    },
    {
      name        = "control-plane-nbg1",
      server_type = "cax11",
      location    = "nbg1",
      labels      = ["k8s"],
      taints      = [],
      count       = 1,
      swap_size   = "2G"
    },
    {
      name        = "control-plane-hel1",
      server_type = "cax11",
      location    = "hel1",
      labels      = ["k8s"],
      taints      = [],
      count       = 1,
      swap_size   = "2G"
    }
  ]

  agent_nodepools = [
    {
      name        = "workload-amd64"
      server_type = "cpx11"
      location    = "fsn1"
      labels      = []
      taints      = []
      kubelet_args = [
        "kube-reserved=cpu=50m,memory=300Mi,ephemeral-storage=1Gi",
        "system-reserved=cpu=250m,memory=300Mi"
      ]
      count     = 1
      swap_size = "2G"
    }
  ]
  load_balancer_type     = "lb11"
  load_balancer_location = "fsn1"
  autoscaler_nodepools = [
    {
      name        = "worker"
      server_type = "cpx21"
      location    = "fsn1"
      min_nodes   = 1
      max_nodes   = 10
      taints      = []
      kubelet_args = [
        "kube-reserved=cpu=250m,memory=1000Mi,ephemeral-storage=1Gi",
        "system-reserved=cpu=250m,memory=200Mi"
      ]
    }
  ]
  system_upgrade_use_drain = true
  extra_firewall_rules = [
    {
      description     = "Allow Outbound TCP MySQL Requests"
      direction       = "out"
      protocol        = "tcp"
      port            = "29867"
      source_ips      = []
      destination_ips = ["78.46.253.145"]
    },
    {
      description     = "Allow Outbound UDP MySQL Requests"
      direction       = "out"
      protocol        = "udp"
      port            = "29867"
      source_ips      = []
      destination_ips = ["78.46.253.145"]
    },
    {
      description     = "Allow Outbound TCP Redis Requests"
      direction       = "out"
      protocol        = "tcp"
      port            = "19321"
      source_ips      = []
      destination_ips = ["91.107.211.207"]
    },
    {
      description     = "Allow Outbound UDP Redis Requests"
      direction       = "out"
      protocol        = "udp"
      port            = "19321"
      source_ips      = []
      destination_ips = ["91.107.211.207"]
    }
  ]
  dns_servers = [
    "1.1.1.1",
    "8.8.8.8",
    "2606:4700:4700::1111",
  ]
}

provider "hcloud" {
  token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
}

terraform {
  required_version = ">= 1.5.0"
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = ">= 1.43.0"
    }
  }
}

output "kubeconfig" {
  value     = module.kube-hetzner.kubeconfig
  sensitive = true
}

variable "hcloud_token" {
  sensitive = true
  default   = ""
}

Screenshots

No response

Platform

Linux
@abriginets abriginets added the bug Something isn't working label Oct 11, 2024
@abriginets
Copy link
Author

It appears that setting ingress_replica_count to 3 solves the issue. From the comment above that option

The default 0 means autoselecting based on number of agent nodes (1 node = 1 replica, 2 nodes = 2 replicas, 3+ nodes = 3 replicas)

And I had 0 agent nodes, but 2-4 autoscaled nodes. Probably a single pod with traefik couldn't handle the amount of traffic apps on my cluster were getting, though kubectl top showed that it's nearly idle. I tried using Lens to see live CPU load, but it just couldn't get the metrics.

I am going to leave this issue open until it's confirmed to be a misconfiguration on my side and not an actual bug.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@abriginets