Issues running OAP in Kubernetes #10404

pcolazurdo · 2023-02-17T14:25:10Z

pcolazurdo
Feb 17, 2023

I am trying to deploy Skywalking (9.2.0) using Helm on Kubernetes and an external OpenSearch cluster but the oap-helm pods are being terminated with:

java.lang.InterruptedException: null
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(Unknown Source) ~[?:?]
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) ~[?:?]
        at io.kubernetes.client.informer.cache.DeltaFIFO.pop(DeltaFIFO.java:318) ~[client-java-16.0.0.jar:?]
        at io.kubernetes.client.informer.cache.Controller.processLoop(Controller.java:182) ~[client-java-16.0.0.jar:?]
        at io.kubernetes.client.informer.cache.Controller.run(Controller.java:146) ~[client-java-16.0.0.jar:?]
        at java.lang.Thread.run(Unknown Source) [?:?]

I had some problems making OpenSearch work but finally all errors have been solved.

Init Job finishes fine (as shown below) but the oap-helm pods get terminated with the error shown below. Changing the readiness and liveness probes timeouts does not seem to help.
I do not seem to find any reference of what may be going wrong. Any recommendation to debug this?

oap-init log extract:

2023-02-17 12:13:40,364 com.linecorp.armeria.server.Server 807 [armeria-boss-http-*:12800] INFO  [] - Serving HTTP at /0.0.0.0:12800 - http://127.0.0.1:12800/
2023-02-17 12:13:40,365 org.apache.skywalking.oap.server.core.storage.PersistenceTimer 58 [main] INFO  [] - persistence timer start
2023-02-17 12:13:40,367 org.apache.skywalking.oap.server.core.cache.CacheUpdateTimer 46 [main] INFO  [] - Cache updateServiceInventory timer start
2023-02-17 12:14:55,593 org.apache.skywalking.oap.server.starter.OAPServerBootstrap 53 [main] INFO  [] - Version of OAP: 9.2.0-cb8845a
2023-02-17 12:14:55,593 org.apache.skywalking.oap.server.starter.OAPServerBootstrap 56 [main] INFO  [] - OAP starts up in init mode successfully, exit now...

oap-helm log extract:

...
2023-02-17 13:47:20,181 io.kubernetes.client.informer.cache.ReflectorRunnable 87 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO  [] - class io.kubernetes.client.openapi.models.V1Pod#Start listing and watching...
2023-02-17 13:47:20,310 com.linecorp.armeria.common.util.SystemInfo 516 [main] INFO  [] - IPv6: disabled (from /proc/sys/net/ipv6/conf/all/disable_ipv6)
2023-02-17 13:47:20,324 com.linecorp.armeria.common.Flags 1392 [main] INFO  [] - useOpenSsl: true (default)
2023-02-17 13:47:20,381 com.linecorp.armeria.common.Flags 533 [main] INFO  [] - Using OpenSSL: BoringSSL, 0x1010107f
2023-02-17 13:47:20,381 com.linecorp.armeria.common.Flags 1392 [main] INFO  [] - dumpOpenSslInfo: false (default)
2023-02-17 13:47:20,479 com.linecorp.armeria.internal.common.JavaVersionSpecific 45 [main] INFO  [] - Using the APIs optimized for: Java 9+
2023-02-17 13:47:21,141 org.apache.skywalking.library.elasticsearch.ElasticSearch 150 [armeria-common-worker-epoll-2-2] INFO  [] - ElasticSearch version is: ElasticSearch 7.10
2023-02-17 13:48:09,728 org.apache.skywalking.oap.server.library.module.BootstrapFlow 43 [main] INFO  [] - start the provider default in log-analyzer module.
2023-02-17 13:48:09,970 org.apache.skywalking.oap.server.library.module.BootstrapFlow 43 [main] INFO  [] - start the provider default in receiver-sharing-server module.
2023-02-17 13:48:09,970 org.apache.skywalking.oap.server.library.module.BootstrapFlow 43 [main] INFO  [] - start the provider default in receiver-profile module.
2023-02-17 13:48:09,971 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 131 [main] INFO  [] - Bind handler ProfileTaskServiceHandler into gRPC server 0.0.0.0:11800
2023-02-17 13:48:09,974 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 131 [main] INFO  [] - Bind handler ProfileTaskServiceHandlerCompat into gRPC server 0.0.0.0:11800
2023-02-17 13:48:09,975 org.apache.skywalking.oap.server.library.module.BootstrapFlow 43 [main] INFO  [] - start the provider default in alarm module.
2023-02-17 13:48:09,975 org.apache.skywalking.oap.server.library.module.BootstrapFlow 43 [main] INFO  [] - start the provider default in receiver-clr module.
2023-02-17 13:48:16,855 org.apache.skywalking.oap.server.core.analysis.DispatcherManager 117 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.source.oal.rt.dispatcher.ServiceInstanceCLRCPUDispatcher is added into DefaultScopeDefine 19.
2023-02-17 13:48:16,855 org.apache.skywalking.oap.server.core.analysis.DispatcherManager 117 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.source.oal.rt.dispatcher.ServiceInstanceCLRGCDispatcher is added into DefaultScopeDefine 20.
2023-02-17 13:48:16,855 org.apache.skywalking.oap.server.core.analysis.DispatcherManager 117 [main] INFO  [] - Dispatcher org.apache.skywalking.oap.server.core.source.oal.rt.dispatcher.ServiceInstanceCLRThreadDispatcher is added into DefaultScopeDefine 21.
2023-02-17 13:48:16,856 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 131 [main] INFO  [] - Bind handler CLRMetricReportServiceHandler into gRPC server 0.0.0.0:11800
2023-02-17 13:48:16,858 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 131 [main] INFO  [] - Bind handler CLRMetricReportServiceHandlerCompat into gRPC server 0.0.0.0:11800
2023-02-17 13:48:16,859 org.apache.skywalking.oap.server.library.module.BootstrapFlow 43 [main] INFO  [] - start the provider default in receiver-otel module.
2023-02-17 13:50:10,094 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 131 [main] INFO  [] - Bind handler OCMetricHandler into gRPC server 0.0.0.0:11800
2023-02-17 13:50:10,128 org.apache.skywalking.oap.server.core.analysis.meter.MeterSystem 202 [main] INFO  [] - Metric meter_istio_pilot_version is already defined, so skip the metric creation.
....
2023-02-17 13:50:12,076 org.apache.skywalking.oap.server.core.analysis.meter.MeterSystem 202 [main] INFO  [] - Metric meter_vm_udp_inuse is already defined, so skip the metric creation.
2023-02-17 13:50:12,084 org.apache.skywalking.oap.server.core.analysis.meter.MeterSystem 202 [main] INFO  [] - Metric meter_vm_filefd_allocated is already defined, so skip the metric creation.
2023-02-17 13:50:12,084 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 131 [main] INFO  [] - Bind handler OpenTelemetryMetricHandler into gRPC server 0.0.0.0:11800
2023-02-17 13:50:12,087 org.apache.skywalking.oap.server.library.module.BootstrapFlow 43 [main] INFO  [] - start the provider default in receiver-trace module.
2023-02-17 13:50:12,088 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 131 [main] INFO  [] - Bind handler TraceSegmentReportServiceHandler into gRPC server 0.0.0.0:11800
2023-02-17 13:50:12,090 org.apache.skywalking.oap.server.library.server.grpc.GRPCServer 131 [main] INFO  [] - Bind handler TraceSegmentReportServiceHandlerCompat into gRPC server 0.0.0.0:11800
2023-02-17 13:50:12,091 org.apache.skywalking.oap.server.library.server.http.HTTPServer 83 [main] INFO  [] - Bind handler TraceSegmentReportHandler into http server 0.0.0.0:12800
2023-02-17 13:50:12,091 org.apache.skywalking.oap.server.library.module.BootstrapFlow 43 [main] INFO  [] - start the provider default in receiver-jvm module.
2023-02-17 13:50:14,404 io.kubernetes.client.informer.cache.ReflectorRunnable 161 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO  [] - class io.kubernetes.client.openapi.models.V1Pod#Read timeout retry list and watch
2023-02-17 13:50:14,404 io.kubernetes.client.informer.cache.Controller 184 [informer-controller-V1Pod] ERROR [] - [9.2.0-cb8845a] DefaultController#processLoop get interrupted null
java.lang.InterruptedException: null
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(Unknown Source) ~[?:?]
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) ~[?:?]
        at io.kubernetes.client.informer.cache.DeltaFIFO.pop(DeltaFIFO.java:318) ~[client-java-16.0.0.jar:?]
        at io.kubernetes.client.informer.cache.Controller.processLoop(Controller.java:182) ~[client-java-16.0.0.jar:?]
        at io.kubernetes.client.informer.cache.Controller.run(Controller.java:146) ~[client-java-16.0.0.jar:?]
        at java.lang.Thread.run(Unknown Source) [?:?]

kezhenxu94 · 2023-02-17T15:08:45Z

kezhenxu94
Feb 17, 2023
Collaborator

Can you share some information about your Kubernetes cluster?

1 reply

pcolazurdo Feb 17, 2023
Author

Not sure what info would you like - some things I can think of:

# 
kubectl get pods -n skywalking
NAME                                              READY   STATUS             RESTARTS       AGE
skywalking-oap-init-9th9s                         0/1     Completed          0              4h4m
skywalking-skywalking-helm-oap-6677cb7757-4p2tf   0/1     CrashLoopBackOff   47 (21s ago)   4h4m
skywalking-skywalking-helm-oap-6677cb7757-5nb9j   0/1     CrashLoopBackOff   47 (21s ago)   4h4m
skywalking-skywalking-helm-ui-5cf866cb76-tvqsn    1/1     Running            0              4h4m

# kubectl version --short
Client Version: v1.26.1
Kustomize Version: v4.5.7
Server Version: v1.24.8-eks-ffeb93d

wu-sheng · 2023-02-17T15:17:12Z

wu-sheng
Feb 17, 2023
Collaborator

8.2.0 is a very old version, and 8.x is EOL.
Please upgrade to latest first.

4 replies

kezhenxu94 Feb 17, 2023
Collaborator

But the user says it's 9.2.0

pcolazurdo Feb 17, 2023
Author

Running on 9.2.0 - which is the recommended version in https://github.com/apache/skywalking-kubernetes at the moment

wu-sheng Feb 17, 2023
Collaborator

My fault, ignore my reply.

wangchao732 Apr 29, 2024

in 9.2.0 same.

kubectl get pods -n skywalking
NAME READY STATUS RESTARTS AGE
elasticsearch-master-0 1/1 Running 0 18h
elasticsearch-master-1 1/1 Running 0 18h
elasticsearch-master-2 1/1 Running 0 18h
skywalking-es-init-78s89 0/1 Init:Error 0 19h
skywalking-es-init-kmc2l 0/1 Completed 0 18h
skywalking-skywalking-helm-oap-75f55f6bdd-pt4bs 0/1 CrashLoopBackOff 8 (2m26s ago) 18m
skywalking-skywalking-helm-ui-5848fd4cf9-d84vz 1/1 Running 0 20m

wangchao732 · 2024-04-29T03:49:18Z

wangchao732
Apr 29, 2024

t the user says it's 9.2.0

in 9.2.0 same.

kubectl get pods -n skywalking
NAME READY STATUS RESTARTS AGE
elasticsearch-master-0 1/1 Running 0 18h
elasticsearch-master-1 1/1 Running 0 18h
elasticsearch-master-2 1/1 Running 0 18h
skywalking-es-init-78s89 0/1 Init:Error 0 19h
skywalking-es-init-kmc2l 0/1 Completed 0 18h
skywalking-skywalking-helm-oap-75f55f6bdd-pt4bs 0/1 CrashLoopBackOff 8 (2m26s ago) 18m
skywalking-skywalking-helm-ui-5848fd4cf9-d84vz 1/1 Running 0 20m

But the user says it's 9.2.0
2024-04-29 03:48:15,328 org.apache.skywalking.oap.server.core.cache.CacheUpdateTimer 46 [main] INFO [] - Cache updateServiceInventory timer start

2024-04-29 03:48:17,191 org.apache.skywalking.oap.server.starter.OAPServerBootstrap 53 [main] INFO [] - Version of OAP: 9.2.0-cb8845a

2024-04-29 03:48:17,191 org.apache.skywalking.oap.server.starter.OAPServerBootstrap 56 [main] INFO [] - OAP starts up in init mode successfully, exit now...

2024-04-29 03:48:17,194 io.kubernetes.client.informer.cache.ReflectorRunnable 161 [controller-reflector-io.kubernetes.client.openapi.models.V1Pod-1] INFO [] - class io.kubernetes.client.openapi.models.V1Pod#Read timeout retry list and watch

2024-04-29 03:48:17,197 io.kubernetes.client.informer.cache.Controller 184 [informer-controller-V1Pod] ERROR [] - [9.2.0-cb8845a] DefaultController#processLoop get interrupted null

java.lang.InterruptedException: null

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(Unknown Source) ~[?:?]

at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(Unknown Source) ~[?:?]

at io.kubernetes.client.informer.cache.DeltaFIFO.pop(DeltaFIFO.java:318) ~[client-java-16.0.0.jar:?]

at io.kubernetes.client.informer.cache.Controller.processLoop(Controller.java:182) ~[client-java-16.0.0.jar:?]

at io.kubernetes.client.informer.cache.Controller.run(Controller.java:146) ~[client-java-16.0.0.jar:?]

at java.lang.Thread.run(Unknown Source) [?:?]

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issues running OAP in Kubernetes #10404

{{title}}

Replies: 3 comments 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Issues running OAP in Kubernetes #10404

pcolazurdo Feb 17, 2023

oap-init log extract:

oap-helm log extract:

Replies: 3 comments · 5 replies

kezhenxu94 Feb 17, 2023 Collaborator

pcolazurdo Feb 17, 2023 Author

wu-sheng Feb 17, 2023 Collaborator

kezhenxu94 Feb 17, 2023 Collaborator

pcolazurdo Feb 17, 2023 Author

wu-sheng Feb 17, 2023 Collaborator

wangchao732 Apr 29, 2024

wangchao732 Apr 29, 2024

pcolazurdo
Feb 17, 2023

Replies: 3 comments 5 replies

kezhenxu94
Feb 17, 2023
Collaborator

pcolazurdo Feb 17, 2023
Author

wu-sheng
Feb 17, 2023
Collaborator

kezhenxu94 Feb 17, 2023
Collaborator

pcolazurdo Feb 17, 2023
Author

wu-sheng Feb 17, 2023
Collaborator

wangchao732
Apr 29, 2024