-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Deadlock when there is a wrongly configured client #5824
Comments
@AndresSanchezG is it a deadlock, or just that one of your theads is holding a lock that is needed by another? A deadlock isn't clear from what you provided. Also the lock involved "blocked on hudson.model.Hudson@42d24189" isn't coming from fabric8. Since fabric8 can make blocking calls you may want to reconsider how you are holding that lock. Are you expecting the watch call to fail fast? That may have to do with the number of retries that are allowed - if the misconfiguration is producing an error that seems recoverable, then we may keep trying the operation. |
@shawkins, so from the threads and traces it seems that while a pod is trying to start up the lock is not released. May be not a deadlock but there is a situation where the lock is never released and therefore it impacts the system. I have already spoken with Jenkins core developers and it seems to be on the part of the plugin. |
Describe the bug
Deadlock seems to happen when a jenkins kubernetes agent is trying to be created but it cannot due to some error. In this case having wrongly configured a container.
Fabric8 Kubernetes Client version
6.4.1
Steps to reproduce
Example:
"wrong_name" image does not exists and then it retries 100 times.
While this pod is being created, when going to manage/configure and pushing save button; configuration is not saved and an error 504 appears in browser.
Performing a thread dump, this thread appears not to finish:
And blocks other threads:
Checking the calls it seems to be a problem happening Kubernetes client code in https://github.com/fabric8io/kubernetes-client/blob/v6.8.1/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/dsl/internal/BaseOperation.java#L611 introduced by 0ebd164
THis problem started appearing after change in Jenkins core: https://github.com/jenkinsci/jenkins/blame/jenkins-2.414.2/core/src/main/java/jenkins/model/Jenkins.java#L1254 introduced by jenkinsci/jenkins@81bf589
Expected behavior
No deadlock should appear.
Runtime
Kubernetes (vanilla)
Kubernetes API Server version
1.24
Environment
Linux
Fabric8 Kubernetes Client Logs
No response
Additional context
No response
The text was updated successfully, but these errors were encountered: