pip-compile: Need a way to apply private package registry credentials when compiling setup.py

[mbudnek]

Tell us more.

The pip-compile manager currently looks up credentials for any hosts it finds in --index-url or --extra-index-url flags in its 'requirements.in' in the configured hostRules and passes them to pip-compile when it re-compiles the lock file. That works fine for 'requirements.in' type files, but pip-compile also supports generating lock files from 'setup.py' files. Those don't support embedding any sort of index URL flags, and so we need a different way to specify which credentials to use.

Since pip-compile supports compiling multiple input files to one lock file, one approach would be to create a 'requirements.in' file with just the required --extra-index-url flags and no packages. This currently doesn't work since the pip_requirements manager's extractPackageFile returns null if it finds no dependencies in the file, but it would a fairly simple change to make.

Another approach would be to take the approach that some other managers like poetry or bundler take and pass credentials for all hosts with a matching hostType. Those could be combined with hosts extracted from any 'requirements.in' files.

[mbudnek]

I created #28961 to allow the first option

[craigds]

I'm also hitting this issue - and I'm not using setup.py files at all, I'm using requirements.in files.

We want to be able to update our dependencies locally, in addition to using renovate. Our local update script wraps pip-compile and injects a PIP_INDEX_URL environment variable containing credentials to our private devpi server.

Problem is I can't go and add --index-url in the requirements.txt files for renovate, because that overrides our environment-based config locally. As far as I can tell there's no way to set the credentials for the server separately when using pip locally; the only way is to set PIP_INDEX_URL.

As far as I can tell though, renovate should support setting the index URL via the config without putting it in the lockfiles; the config to do so should be something like:

{
  "packageRules": [
    {
      "matchDatasources": [
        "pypi"
      ],
      "registryUrls": [
        "https://devpi.example.com/root/dev/+simple/"
      ]
    }
  ],
  "hostRules": [
    {
      "matchHost": "devpi.example.com",
      "username": "renovate",
      "encrypted": {
        "password": "<omitted>"
      }
    }
  ]
}

This doesn't actually work unfortunately. I've been hunting through the code and it seems like the pip-compile manager doesn't reference the registryUrls in the config at all. This seems like a bug?

[mbudnek]

Instead of configuring your credentials as part of the index-url you could configure them separately and leave just the URL in your requirements.in.

pip supports two ways of looking up credentials for a package index:

1. A ~/.netrc (or %USERPROFILE%\_netrc on Windows) file

machine devpi.example.com
    login myusername
    password mypassword

2. Pip has support for the keyring module. IIRC it has built-in backends for macOS and Gnome keyrings and the Windows credential store. May not work well with docker and such though. That's actually how Renovate passes credentials to pip-compile internally (using the keyrings.envvars backend).

[craigds]

(1) sounds possible for us as a workaround.

This is a bug though right? renovate supports registryUrls in config, but the pip-compile manager doesn't yet actually use it and it should

[mbudnek]

I would consider it more of an architectural limitation than a bug per say. Basically, the packageRules only apply when looking for updated package versions and grouping related updates. They aren't accessible when actually re-compiling the lock file and the manager wouldn't have enough context to use them properly even if they were.

The pip-compile manager has no way to access the packageRules directly. Renovate only gives the pip-compile manager information about the dependency that's currently being upgraded. That info could include the registry where the package was found (though that info currently seems to get lost somewhere between then the datasource looks it up and when the manager goes to update it), but info about the dependency currently being updated isn't enough. The manager would need info about all of the packages in all of the input files so that it could look up credentials and pass the appropriate --index-url or --extra-index-url flags to pip-compile.