Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reflector late updates on Kubernetes 1.20 and older (known bug on k8s) and workaround #246

Closed
winromulus opened this issue Dec 19, 2021 · 5 comments
Labels
stale wontfix This will not be worked on

Comments

@winromulus
Copy link
Contributor

Hi,

Due to a number of similar issues, I'm posting this one as a sticky:

On Kubernetes 1.21 and older (up to 1.14 I think) there was an issue with the API server pushing events late or not at all on idle watchers (watchers that have not received any changes). This results in reflector not being aware of changes until the connection times out and the watcher is reset.
Reflector does not poll for changes (due to clusters with large amount of secrets/namespaces etc.). It relies on k8s to push events to the subscribed watchers when changes occur.

If your cluster is on k8s version 1.20.x or older, I suggest you upgrade to the latest version of k8s supported by your cloud or on-premise service. This bug seems to have been fixed in k8s 1.21.x

@winromulus winromulus added the wontfix This will not be worked on label Dec 19, 2021
@winromulus winromulus pinned this issue Dec 19, 2021
@emberstack emberstack locked as resolved and limited conversation to collaborators Dec 19, 2021
@winromulus winromulus changed the title Reflector late updates on Kubernetes 1.20 and older (known bug on k8s) Reflector late updates on Kubernetes 1.20 and older (known bug on k8s) and workaround Dec 25, 2021
@winromulus
Copy link
Contributor Author

Workaround:
I've included in v6.1.16 an option to set the timeout for watchers (the maximum lifetime before reconnecting). Please see the Readme for configuring this value (helm or manually for the vanilla manifests).

From experience k8s 1.20 and older stops sending updates to watchers after 10 minutes. I would suggest setting the timeout to 600 and adjust as needed.
Note: This does not mean that reflector will only sync after 10 minutes. This means that watcher connections are reset every 10 minutes to ensure that k8s sends events as they occur. Within those 10 minutes any changes are reflected immediately.
Word of advice: If you have a very large cluster (namespaces, secrets and configmaps), you might want to tune this value to something larger to prevent performance issues.

I still recommend upgrading to the latest version of k8s available from your provider when possible. The above workaround should be used in case you're currently stuck with the version of k8s you have.

Hope it helps!

@emberstack emberstack unlocked this conversation Dec 25, 2021
@shdwlkr
Copy link

shdwlkr commented Feb 14, 2022

Hello!
Still have this issue on k8s 1.23.x, reflector sometimes doesn' reflect a secret, it can be fixed only by restarting the deployment.
Any fixes on that?

@winromulus
Copy link
Contributor Author

@shdwlkr can you post the logs and sample secrets for it? We're running a lot of clusters on 1.23.x and have not faced any issues so far

@stale
Copy link

stale bot commented Apr 16, 2022

Automatically marked as stale due to no recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Apr 16, 2022
@stale
Copy link

stale bot commented Apr 30, 2022

Automatically closed stale item.

@stale stale bot closed this as completed Apr 30, 2022
@winromulus winromulus unpinned this issue Jun 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stale wontfix This will not be worked on
Development

No branches or pull requests

2 participants