OTel Collector Container not running

[banandh]

I have installed the open telemetry collector helm, for sometime the collector container was in running mode and now for a long time its not in running and the issue is LivenessProbe and Readiness Probe fails

Warning BackOff 39h (x197 over 41h) kubelet Back-off restarting failed container opentelemetry-collector in pod my-opentelemetry-collector-7744ff5458-k5f4n_default(347bf258-cdff-422c-8f80-68020de42049)
Normal Started 39h (x25 over 41h) kubelet Started container opentelemetry-collector
Warning Unhealthy 38h (x73 over 41h) kubelet Liveness probe failed: Get "http://10.42.0.97:13133/": dial tcp 10.42.0.97:13133: connect: connection refused
Warning Unhealthy 35h (x270 over 41h) kubelet Readiness probe failed: Get "http://10.42.0.97:13133/": dial tcp 10.42.0.97:13133: connect: connection refused
Normal Created 34h (x35 over 41h) kubelet Created container opentelemetry-collector
Warning Unhealthy 4h18m (x39 over 41h) kubelet Readiness probe failed: Get "http://10.42.0.97:13133/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning FailedMount 51m (x3 over 51m) kubelet MountVolume.SetUp failed for volume "kube-api-access-btn95" : object "default"/"kube-root-ca.crt" not registered
Warning FailedMount 51m (x4 over 51m) kubelet MountVolume.SetUp failed for volume "opentelemetry-collector-configmap" : object "default"/"my-opentelemetry-collector" not registered
Normal SandboxChanged 51m kubelet Pod sandbox changed, it will be killed and re-created.
Normal Pulled 51m kubelet Container image "otel/opentelemetry-collector-k8s:0.99.0" already present on machine
Normal Created 51m kubelet Created container opentelemetry-collector
Warning Unhealthy 50m kubelet Liveness probe failed: Get "http://10.42.0.166:13133/": dial tcp 10.42.0.166:13133: connect: connection refused
Warning Unhealthy 45m (x4 over 47m) kubelet Readiness probe failed: Get "http://10.42.0.166:13133/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning Unhealthy 45m (x6 over 47m) kubelet Liveness probe failed: Get "http://10.42.0.166:13133/": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Normal Started 36m (x7 over 51m) kubelet Started container opentelemetry-collector
Warning BackOff 7m54s (x121 over 42m) kubelet Back-off restarting failed container opentelemetry-collector in pod my-opentelemetry-collector-7744ff5458-k5f4n_default(347bf258-cdff-422c-8f80-68020de42049)
Warning Unhealthy 2m52s (x95 over 51m) kubelet Readiness probe failed: Get "http://10.42.0.166:13133/": dial tcp 10.42.0.166:13133: connect: connection refused

Container was in running state earlier

kubectl get all
NAME READY STATUS RESTARTS AGE
pod/opentelemetry-operator-748b475cb9-zrdnl 2/2 Running 63 (3h27m ago) 40d
pod/demo-collector-588ff6b9f5-vg4vj 1/1 Running 0 3h27m
pod/my-opentelemetry-collector-7744ff5458-k5f4n 1/1 Running 86 (3h27m ago) 43h

[TylerHelmuth]

Do you have the healthcheck extension included in your deployment?

[banandh]

Yes it has readiness and liveness probes. If i remove it, will the collector receive all the telemetry data. This questions comes because at times the liveness is failing so the container trying to reinitiate.

[TylerHelmuth]

If you dont have the health check extension then the collector wont startup from this chart. You could check the resource consumption of the pod and make sure it has enough to operate.