Kustomize and Helm diff support

[willejs]

Usually, one really valuable step in your CI against a PR is to do a kubectl/helm diff to show users what would change in your cluster (minus secrets).

A great example of this is probably how Terraform Enterprise/Spacelift/Atlantis do this.
They will run a terraform plan and post the diff of the plan back to your PR as a comment and link through to the plan phase for you to inspect. This is really helpful when reviewing PRs, catching errors early on, and communicating the impact of your PRs. In the CiOps world, it is common for people to use helmfile to do a similar thing too.

Currently, you could probably hack this into your CI, reading a helmRelease/kustomize resource and trying to construct diff commands. This would end up devaluing one of the main benefits of gitops, not having k8s credentials in CI tasks. Additionally, it would also mean you have to try and replicate the same logic that the helm (merging values) and kustomize (sops secrets), and even source (repackaging helm values files) controllers carry out.

I am wondering if there is a way to implement this in flux2. Getting the source controller to create an artifact for a PR, letting the controllers do the diff, send a notification to a github pr, and store the diff output in a resource, cleaned up on PR close? Just an idea.

Really, I am wondering if this is on anyone else's radar, and If this could become a feature request?

[stefanprodan]

Posting the diff of decrypted data(Kubernetes secrets) to GitHub or any other external system it’s a no-go IMO.

[shibumi]

Hi, sorry for reviving this thread. I think the mistake is already to have plaintext secrets in any YAML configuration. As far as I know people use Hashicorp Vault or Bitnami's Sealed-Secrets Operator for running around this issue. So flux should always show the diff and any plaintext secrets should not be visible, because SealedSecrets or Vault should handle the rest. Everything else is a cluster misconfiguration IMHO.

[william-salt-cko]

@stefanprodan Have you had any further thoughts or discussions about this?

I am currently doing a flux1 -> flux2 migration and lack of diff support has created a quite complex shell script that diffs just helm releases, which doesn't work quite in the same way as the controller.
We have also on one occasion changed some kustomize and affected a dozen or so releases in production which isn't obvious just from a git diff, whereas a kubectl diff would indicate actually what would happen.

We have other ideas for workarounds, but we would really like some functionality akin to TF, ArgoCD, or Helm in Flux2, or at least indication there is a plan for it. 🙂

[RichiCoder1]

This is something I love about both terraform and ArgoCD, and it's a bit painful having to maintain a massive diff script just to accomplish this for PRs

[william-salt-cko]

@stefanprodan Do you think you could revisit this now? The PR for kubectl diff hiding secrets support has been merged in from the original post. Helm diff already supports this too...

[morremeyer]

I think we are now in a state where we can take a look at how to do this. The PR is merged as @william-salt-cko mentioned, so this will be part of Kubernetes 1.22.

I started #820 (comment) to gather ideas.

[messiahUA]

I think that diff functionality is a must for production use. For example, I've made a mistake in configuration files and flux completely deleted everything including itself with prune enabled. Of course I can disable pruning, but in this case I would need to do it for everything and it defeats the purpose. These mistakes can be prevented only by checking everything before applying...

[morremeyer]

kubectl will have support for hiding secrets in 1.22, see this PR.
I think we could start discussing how this can be implemented to support different git flows, so here is something that I would find useful to have:

In most git flows, there is a production branch (most often named production or main) that flux be configured to apply resources from. If functionality was added that allows the configuration of a staging branch to diff against (e.g. staging or pre-releaes), this would allow to:

• Use gitflow and gitlabflow natively
• Not add any „blocking“ logic to the deployment from the production branch as the diff and the apply workflows would/could be completely seperate from one another

I have not yet found an idea about how to report the output that I am really happy with. My only idea so far would be to add a URL field to specify a webhook to which the diff could then be posted.

[morremeyer]

Another idea I had is for flux to track all branches of the repository and diff changes - this would also enable the usage of githubflow, but probably make the implementation more complex.

However, it would probably also enable easier tracking of diffs by being able to track changes back to a PR (or MR for e.g. GitLab).

[stefanprodan]

In Flux 0.26 we've introduced the flux diff kustomization --path my/local/overlay command which uses the same server-side apply engine as kustomize-controller. The flux diff command detects SOPS encrypted secrets and masks the secret values in the diff output.

[stefanprodan]

You can subscribe to this issue which is a prerequisite for flux diff helm.

[mckennajones]

Hey there @stefanprodan is the above linked issue in helm-controller still a prereq for implementing a flux diff hr command?

[ghobadimhd]

Hi, is there any further update about flux diff HelmRelease? Is this feature going to be added to flux or not? I've seen that the above issue has been completed
@stefanprodan

[onedr0p]

The helm controller just got a pretty big update so hopefully we're not too far away from this functionality.

[TaylorChristie]

Wondering if there is still any interest on the flux diff helmreleases command? Would be quite nice to have native support as currently I have a hacky bash script that doesn't quite get me all the way there. Having output similar to helm diff with the https://github.com/databus23/helm-diff plugin would be ideal

[julienFlowdesk]

The flux diff kustomization --path X is an amazing command. I just wanted to add some early feedbacks:

In my case, I am trying to do flux diffing within a github-action with a specific service account having only read-only access to all kubernetes objects within my cluster to limit risks.

I was expecting flux to be able to do diff by reading only the current cluster objects but in fact it looks like it need more than that:

.  Kustomization diffing...: running dry-run
 errors occurred:
	* Namespace/x dry-run failed, reason: Forbidden, error: namespaces "x" is forbidden: User "github-action@xxxx" cannot patch resource "namespaces" in API group "" in the namespace "x": requires one of ["container.namespaces.update"] permission(s).

Maybe it is the awaited behaviour but I feel like the diff command shouldn't require more than the view roles within kubernetes to output something.

[stefanprodan]

Please raise this with the Kubernetes team, Flux performs a server-side apply dry-run, why dry-run requires write access I have no clue.

[julienFlowdesk]

For posterity purposes: the kubernetes related issues :

kubernetes/kubernetes#95449
kubernetes/kubectl#981