Renovate bot ignores checks that are required in the ruleset and merges PR regardless when configured to bypass status checks

[effektsvk]

What would you like help with?

I think I found a bug

How are you running Renovate?

Mend Renovate hosted app on github.com

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

No response

Please tell us more about your question or problem

We've encountered an issue where mandatory check runs are missing, which causes the PR to be merged when a check is failing (or maybe not yet finished).

We have 3 mandatory status checks but for some reason the log shows only one (/ / Ensure semantic PR title).
It should contain these three:

[
  {
    "name": "/ / Lint", "status": "completed", "conclusion": "failure"
  },
  {
    "name": "/ / Ensure semantic PR title", "status": "completed", "conclusion": "success"
  },
  {
    "name": "/ / Unit tests", "status": "completed", "conclusion": "failure"
  }
]

Here is the PR link: technology-studio/eslint-config-txo-typescript#498
And here is the ruleset: https://github.com/technology-studio/eslint-config-txo-typescript/rules/146556

Logs (if relevant)

Logs

DEBUG: Pull Request #498 does not need updating (branch="renovate/eslint-txo-packages")
DEBUG: PR is configured for automerge (branch="renovate/eslint-txo-packages")
DEBUG: checking auto-merge (branch="renovate/eslint-txo-packages")
DEBUG: Checking schedule(at any time, null) (branch="renovate/eslint-txo-packages")
DEBUG: No schedule defined (branch="renovate/eslint-txo-packages")
DEBUG: resolveBranchStatus(branchName=renovate/eslint-txo-packages, ignoreTests=false) (branch="renovate/eslint-txo-packages")
DEBUG: getBranchStatus(renovate/eslint-txo-packages) (branch="renovate/eslint-txo-packages")
DEBUG: branch status check result (branch="renovate/eslint-txo-packages")
{
  "state": "pending"
  "statuses": []
}

DEBUG: Successful checks are all internal renovate/ checks, so returning "pending" branch status (branch="renovate/eslint-txo-packages")
DEBUG: check runs result (branch="renovate/eslint-txo-packages")
{
  "checkRuns": [
    {
      "name": "/ / Ensure semantic PR title",
      "status": "completed",
      "conclusion": "success"
    },
    {
      "name": "/ / Ensure semantic PR title",
      "status": "completed",
      "conclusion": "success"
    }
  ]
}

DEBUG: Branch status green (branch="renovate/eslint-txo-packages")
DEBUG: Automerging #498 with strategy auto (branch="renovate/eslint-txo-packages")
DEBUG: mergePr(498, renovate/eslint-txo-packages) (branch="renovate/eslint-txo-packages")
DEBUG: mergePr (branch="renovate/eslint-txo-packages")
{
  "options": {
    "body": {
      "merge_method": "rebase"
    }
  }
  "url": "repos/technology-studio/eslint-config-txo-typescript/pulls/498/merge"
}

DEBUG: PR merged (branch="renovate/eslint-txo-packages")
{
  "automergeResult": {
    "sha": "b9562b5a71794feba65cabb06307cc97b9f4c299",
    "merged": true,
    "message": "Pull Request successfully merged"
  }
  "pr": 498
}

INFO: PR automerged (branch="renovate/eslint-txo-packages")
{
  "pr": 498
  "prTitle": "fix(deps): update eslint txo packages"
}

[rarkins]

The logs show that there were 2 tests present, both of which passed. In other words, the branch is "green" with no pending checks.

The logs also show that GitHub allowed the PR to be merged, which means you don't have the branch protection which you think you have. GitHub would not allow Renovate to merge unless your branch protection was missing, or perhaps you excluded Renovate from branch protection checks.

[effektsvk]

Why do you need Renovate to bypass protections?

We want to remove the renovate-approve bot because there are situations where it doesn't work as expected and we would like renovate to override the minimal PR reviewers protection.

And if you use GitHub's automerge (platformAutomerge=true in Renovate config, which should be the default), does GitHub ignore the mandatory checks too?

We are using Renovate with elevated permissions due to reason described above, so the default protection doesn't stop renovate to merge.

Another question: how long do your checks take to start? Do they set a pending status when they start, or do they only set a passed/failed once they're done?

Yes, they do set a pending status, but there's a brief period until the runners load the workflow and set the pending status for the relevant job. This applies only to simple workflows. For complex ones, where one job depends on another, there can be a significant delay before the relevant job sets the PR status check to pending, so it won't be shown in the check_runs API response.

[rarkins]

We are using Renovate with elevated permissions due to reason described above, so the default protection doesn't stop renovate to merge.

In this case it's not Renovate merging, it's GitHub itself. It may wait for the status checks even if the account has permissions to ignore settings.

How long is the delay for setting pending?

[effektsvk]

In this case it's not Renovate merging, it's GitHub itself. It may wait for the status checks even if the account has permissions to ignore settings.

I don't understand, why GitHub? Clearly, Renovate merged the PR and this is because we configured renovate to have elevated rights to override merge, even when some criteria are not met. In such cases, we expect that renovate would rely on internal checks instead of relying on GitHub.

technology-studio/eslint-config-txo-typescript#498

How long is the delay for setting pending?

It can be seconds, minutes, or even hour, depends on the workflow. As I wrote before, in complex workflows, when one job depends on the another, the pending status won't be set until preceding jobs has been completed.

[rostislav-simonik]

Renovate has been added to the bypass list in the branch's protection rules, which means that GitHub's protections do not apply to Renovate.

The question then arises: what is the purpose of these internal renovate checks if they don't prevent progression to the next stage: the merging, whether it's via GitHub's native tools or otherwise?

[rarkins]

When the platform does the automerge, then the platform (GitHub) decides when it's ok to automerge. That's why I say "it's not Renovate merging". It's Renovate asking the platform to merge when ready, but the decision of when is the platform's.

GitHub has a large number of branch protections (far beyond just mandatory checks), and we don't wish to reverse engineer them all to decide when it's time to merge, because it's a large and moving target. Instead, we rely on GitHub to protect branches against incorrect merges. The check for current status checks and check runs is just a basic/easy one to do, which has existed since before branch protections even existed.

• You have chosen to set Renovate to bypass branch protections because you want it to ignore certain protections
• Renovate automerges
• You're upset that it doesn't know which protections you wanted it to bypass and which not

e.g. what if we enhanced Renovate to read the mandatory status checks and refuse to merge if any aren't completed, but also enhanced Renovate to refuse to merge if approvals aren't met, or some other rule you wanted bypassed isn't met?