Struggling to use a custom datasource

[jwilkicki]

What would you like help with?

I would like help with my configuration

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

Github Enterprise Renovate version: 37.306.1

Please tell us more about your question or problem

I am using RenovateBot to handle some helm_releases in Terraform, particularly Istio. The problem we have is that Istio has some compatibility constraints with Kubernetes; the latest version of Istio will not handle the version of Kubernetes we are currently using. Unfortunately, Renovate doesn't know this so it keeps suggesting the latest released version, which we can't use.

After a lot of poking around, I settled on this solution: I have a utility that I wrote which grabs Istio's compatibility matrix (which provides the major minor version of istio) and the set of k8s versions that each major-minor is compatible with. The utility then grabs the chart repository for istio and gets the full chart versions for any major-minor versions that are compatible with my desired version of Kubernetes. I then have that utility spit out a JSON file that, as far as I can tell, has the exact scheme that the custom datasource requires. I assumed that way I wouldn't need any JSONata conversions to make this work.

My output looks like this:

{
  "releases": [
    {
      "version": "1.20.6"
    },
    {
      "version": "1.20.5"
    },
    {
      "version": "1.20.4"
    },
    {
      "version": "1.20.3"
    },
    {
      "version": "1.20.2"
    },
    {
      "version": "1.20.1"
    },
    {
      "version": "1.20.0"
    },
    {
      "version": "1.20.0-rc.0"
    },
    {
      "version": "1.20.0-beta.0"
    },
    {
      "version": "1.19.10"
    },
    {
      "version": "1.19.9"
    },
    {
      "version": "1.19.8"
    },
    {
      "version": "1.19.7"
    },
    {
      "version": "1.19.6"
    },
    {
      "version": "1.19.5"
    },
    {
      "version": "1.19.4"
    },
    {
      "version": "1.19.3"
    },
    {
      "version": "1.19.2"
    },
    {
      "version": "1.19.1"
    },
    {
      "version": "1.19.0"
    },
    {
      "version": "1.19.0-rc.0"
    },
    {
      "version": "1.19.0-beta.1"
    },
    {
      "version": "1.19.0-beta.0"
    },
    {
      "version": "1.19.0-alpha.1"
    },
    {
      "version": "1.19.0-alpha.0"
    }
  ]
}

I run this on a schedule in my CI/CD system and check the results into the repo. My assumption was that I could then point renovate.json at that file as the source of my revisions.

I had a few problems that I think I worked through. First, there are two Helm releases that I need to manage: base and istiod. They will always have the same actual istio version and should be in lockstep. The second is that I can't force the terraform manager to use my custom datasource, so I need to tell it to ignore the helm_release resources for the two packages that I need this custom logic for while letting it continue to process other stuff that I might want to change with the default behavior. Here's a cut down version of my renovate.json that deals with these issues:

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "baseBranches": [
    "main"
  ],
  "customManagers": [
    {
      "customType": "regex",
      "fileMatch": [
        "\\.tf$"
      ],
      "matchStrings": [
        "\\s?command\\s?=\\s?\\[\\s?\"[^\\\"]+\"\\s?,\\s?\"(?<currentValue>.*?)\"\\s?\\]",
        "set\\s*{\\s*name\\s*=\\s*\"global.tag\"\\s*value\\s*=\\s*\"(?<currentValue>[^\\\"]+)\"\\s*}",
        "chart\\s+=\\s+\"(?<depName>base|istiod?)\"(.*?\\s)*?version\\s+=\\s*?\"(?<currentValue>.*?)\""
      ],
      "datasourceTemplate": "custom.istio",
      "versioningTemplate": "helm"
    }
  ],
  "customDatasources": {
    "istio": {
      "defaultRegistryUrlTemplate": "file://.github/datasource.json",
      "format": "json"
    }
  },
  "packageRules": [
    {
      "matchManagers": [
        "terraform"
      ],
      "matchDepTypes": [
        "helm_release"
      ],
      "matchPackageNames": [
        "base",
        "istiod"
      ],
      "enabled": false
    },
    {
      "matchPackageNames": [
        "istiod",
        "base"
      ],
      "description": "Istio Charts",
      "groupName": "istio charts"
    }
  ]
}

From looking at the dependency dashboard, this seems to do what I want: terraform is not managing the two helm releases but is managing a module from one of my other git repos. The regex manager is picking up dependencies that seem like they make sense: I have a few set blocks that have the version, a shell script in another resource that needs the version as an argument, and the actual helm_releases.

The problem I have is that Renovate can't seem to find the versions in my datasource JSON file. It doesn't say it can't find the file, it just says it can't find the versions for each package. As far as I can tell, the datasource JSON that Renovate expects doesn't include the package name and I really want the same version information to be used for each of the two packages because they should always be the same version. I'm kind of stumped. I've put what I think is the relevant snippet from my logs below.

For completeness, here's a somewhat sanitized copy of the terraform that I'm parsing:

locals {
  global_hub = "gcr.io/istio-release"
  namespace  = "istio-system"
}

module "preupgrade_job" {
  source = "git::ssh://[email protected]/development/terraform-aws-k8sjob?ref=v2.6.0"

  image = "alpine/k8s:1.25.14"

  name = "istio-preupgrade"

  namespace = "kube-system"

  command = ["/scripts/preupgrade.sh", "1.18.5"]

  files = [
    {
      path     = "/scripts/preupgrade.sh"
      contents = file("${path.module}/preupgrade.sh")
      mode     = "0755"
    }
  ]
}

resource "helm_release" "istio-base" {
  name             = "istio-base"
  repository       = "https://istio-release.storage.googleapis.com/charts"
  chart            = "base"
  version          = "1.18.5"
  namespace        = local.namespace
  create_namespace = true
  atomic           = true
  # CRD's handled by preupgrade
  skip_crds = true
  set {
    name  = "global.hub"
    value = local.global_hub
  }

  set {
    name  = "global.tag"
    value = "1.18.5"
  }
}

resource "helm_release" "istiod" {
  depends_on = [
    helm_release.istio-base
  ]

  name       = "istiod"
  repository = "https://istio-release.storage.googleapis.com/charts"
  chart      = "istiod"
  version    = "1.18.5"
  namespace  = local.namespace
  wait       = true

  set {
    name  = "global.hub"
    value = local.global_hub
  }

  set {
    name  = "global.tag"
    value = "1.18.5"
  }
}

My expectation is that all of those 1.18.5's will become 1.20.6 after RenovateBot does its run.

I'm fairly confident that this is something I'm doing wrong because I don't understand something in the docs. Any help would be greatly appreciated.

Logs (if relevant)

Logs

DEBUG: Found github-actions package files (repository=development/renovatebot-custom-datasource-example, baseBranch=main)
DEBUG: Found terraform package files (repository=development/renovatebot-custom-datasource-example, baseBranch=main)
DEBUG: Found regex package files (repository=development/renovatebot-custom-datasource-example, baseBranch=main)
DEBUG: Found regex package files (repository=development/renovatebot-custom-datasource-example, baseBranch=main)
DEBUG: Found 5 package file(s) (repository=development/renovatebot-custom-datasource-example, baseBranch=main)
 INFO: Dependency extraction complete (repository=development/renovatebot-custom-datasource-example, baseBranch=main)
       "stats": {
         "managers": {
           "github-actions": {"fileCount": 2, "depCount": 3},
           "terraform": {"fileCount": 1, "depCount": 3},
           "regex": {"fileCount": 2, "depCount": 4}
         },
         "total": {"fileCount": 5, "depCount": 10}
       }
DEBUG: baseBranch: main (repository=development/renovatebot-custom-datasource-example, baseBranch=main)
DEBUG: Dependency: base, is disabled (repository=development/renovatebot-custom-datasource-example, baseBranch=main)
DEBUG: Dependency: istiod, is disabled (repository=development/renovatebot-custom-datasource-example, baseBranch=main)
DEBUG: hostRules: no authentication for istio-release.storage.googleapis.com (repository=development/renovatebot-custom-datasource-example, baseBranch=main)
DEBUG: GET https://istio-release.storage.googleapis.com/charts = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=404 retryCount=0, duration=144) (repository=development/renovatebot-custom-datasource-example, baseBranch=main)
DEBUG: Failed to look up custom.istio package base (repository=development/renovatebot-custom-datasource-example, baseBranch=main, packageFile=main.tf, dependency=base)
DEBUG: Failed to look up custom.istio package base (repository=development/renovatebot-custom-datasource-example, baseBranch=main, packageFile=main.tf, dependency=base)
DEBUG: Failed to look up custom.istio package istiod (repository=development/renovatebot-custom-datasource-example, baseBranch=main, packageFile=main.tf, dependency=istiod)
DEBUG: Failed to look up custom.istio package istiod (repository=development/renovatebot-custom-datasource-example, baseBranch=main, packageFile=main.tf, dependency=istiod)
DEBUG: PackageFiles.add() - Package file saved for base branch (repository=development/renovatebot-custom-datasource-example, baseBranch=main)

[jwilkicki]

Not a total answer to my problem, but one thing I found is: the customDatasource JSON is sensitive to whitespace. The example I have in this discussion was pretty-printed so you could read it, but the actual file didn't have any delimiting spaces, just regular JSON with all the braces, etc. but no whitespace for readability. When I checked in the formatted JSON, I started getting the substitutions I expected.

Here's my next problem: now it is only substituting for the chart versions in the helm_resources but not touching the set blocks and the command in my module. Any thoughts on that?

UPDATE: Just so the whole answer is here for somebody, it was the depName template. Two of my matches did not have an available depName. I fixed it by using the depName if it existed in the match group and then just defaulting to 'base' if it wasn't set:

"customManagers": [
    {
      "customType": "regex",
      "fileMatch": [
        "\\.tf$"
      ],
      "matchStrings": [
        "\\s?command\\s?=\\s?\\[\\s?\"[^\\\"]+\"\\s?,\\s?\"(?<currentValue>.*?)\"\\s?\\]",
        "set\\s*{\\s*name\\s*=\\s*\"global.tag\"\\s*value\\s*=\\s*\"(?<currentValue>[^\\\"]+)\"\\s*}",
        "chart\\s+=\\s+\"(?<depName>base|istiod?)\"(.*?\\s)*?version\\s+=\\s*?\"(?<currentValue>.*?)\""
      ],
      "depNameTemplate": "{{#if depName}}{{{depName}}}{{else}}base{{/if}}",
      "datasourceTemplate": "custom.istio",      
      "versioningTemplate": "helm"
    }
  ],

When I did that, I got a single PR with all substitutions where I expected them with the version I expected.

[secustor]

Rather than using a customDatasource you can use helm in case o classic helm repository or docker if it hosted as OCI.

helm_release should also be automatically updated in *.tf files

Attach logs with the extracted with updates block to see what is happening.

[jwilkicki]

I would have liked to use the default behavior you described (I do in other code), but I couldn't in this case because I needed to filter down the list of available chart versions to just the ones that are supported on the version of Kubernetes we are using. Istio's chart repo doesn't include the kubernetes compatibility information so I had to put together a custom data source that combined two sources of data to generate a subset of available chart versions for Renovatebot to consider. I also needed the same value to be inserted in other locations in the same PR.

With the tweaks above, I was able to get what I needed and filled in a few details for myself on custom datasources that I didn't really glean from the documentation.