Reflector late updates on Kubernetes 1.20 and older (known bug on k8s) and workaround

[winromulus]

Hi,

Due to a number of similar issues, I'm posting this one as a sticky:

On Kubernetes 1.21 and older (up to 1.14 I think) there was an issue with the API server pushing events late or not at all on idle watchers (watchers that have not received any changes). This results in reflector not being aware of changes until the connection times out and the watcher is reset.
Reflector does not poll for changes (due to clusters with large amount of secrets/namespaces etc.). It relies on k8s to push events to the subscribed watchers when changes occur.

If your cluster is on k8s version 1.20.x or older, I suggest you upgrade to the latest version of k8s supported by your cloud or on-premise service. This bug seems to have been fixed in k8s 1.21.x

[winromulus]

Workaround:
I've included in v6.1.16 an option to set the timeout for watchers (the maximum lifetime before reconnecting). Please see the Readme for configuring this value (helm or manually for the vanilla manifests).

From experience k8s 1.20 and older stops sending updates to watchers after 10 minutes. I would suggest setting the timeout to 600 and adjust as needed.
Note: This does not mean that reflector will only sync after 10 minutes. This means that watcher connections are reset every 10 minutes to ensure that k8s sends events as they occur. Within those 10 minutes any changes are reflected immediately.
Word of advice: If you have a very large cluster (namespaces, secrets and configmaps), you might want to tune this value to something larger to prevent performance issues.

I still recommend upgrading to the latest version of k8s available from your provider when possible. The above workaround should be used in case you're currently stuck with the version of k8s you have.

Hope it helps!

[shdwlkr]

Hello!
Still have this issue on k8s 1.23.x, reflector sometimes doesn' reflect a secret, it can be fixed only by restarting the deployment.
Any fixes on that?

[winromulus]

@shdwlkr can you post the logs and sample secrets for it? We're running a lot of clusters on 1.23.x and have not faced any issues so far

[stale]

Automatically marked as stale due to no recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

Automatically closed stale item.