Private networking for GitHub-hosted runners

[stan-spotts]

Select Topic Area

Question

Body

We are looking at containerizing a set of node.js API's (uses loopback.io) that were developed sitting on a VM. I wanted to automate deployment to a container registry first, and it has to be private per our security guidelines. I was going to use a self-hosted runner, but then saw the article Configuring private networking for GitHub-hosted runners in your organization. Having just migrated our GitHub account from a personal pro plan to an organization team plan, I thought wow, this looks promising.

I followed the instructions, mostly - I already had a resource group and a VLAN set up with a container registry, so the only real thing I changed was I didn't create a VNET and update the subnet info, I just created the subnet. So this subnet is in the same VLAN as the registry (in a different subnet). But I created the service principal, set the rbac role, etc., got the networking config in GitHub set up, created the runner group. Happy happy!

Except it won't connect. I put a step in to log what runner group the workflow used, and it was the one I just created.

On the docker/login-action@v3 action, I get this in the log:

Run docker/login-action@v3
  with:
    registry: ***
    username: ***
    password: ***
    ecr: auto
    logout: true
Logging into ***...
Error: Error response from daemon: Get "https://***/v[2](https://github.com/myOrg/myrepo/actions/runs/8791394220/job/24125517103#step:6:2)/": denied: client with IP '20.44.102.[3](https://github.com/myOrg/myrepo/actions/runs/8791394220/job/24125517103#step:6:3)6' is not allowed access. Refer https://aka.ms/acr/firewall to grant access.

I went in with the Cloud CLI and used docker commands to login and connect to the container registry with the clientId and secret of the service principal, same as what is in the repo secrets that the workflow references.

I'm not trying to access anything on our on-premise network, so I didn't think I needed to deal with ExpressRoute or VPN Gateway or anything. But what's the trick here?

[stan-spotts]

Thanks for that response! I am indeed using Azure Container Registry. I checked the results of the json endpoint you supplied and it got back over 3600 IP ranges. Certainly this isn't an answer considering one has to enter them one at a time, although I'm sure I could script that. I didn't see a way to set a conditional access policy for an IP range (just an account). I did see a policy "Configure Container registries with private endpoints" that I didn't use, I set it up per the instructions on the link in the original post.

What I'm not grokking is that I thought the IP whitelisting and firewall ranges are for public access. And what this solution was supposed to do was use private network access. The whole reason I went down this path was to see if I could deploy from GitHub to a private IP without having to set up a self-hosted runner, which I'm already doing for other code. I can see under the Networking section for the Azure Container Registry, under the Private access tab, it has the connection and private endpoint I created, and has the data endpoint. So Why is it trying to go over public IP?

[stan-spotts]

Thanks, this is all interesting. Unfortunately, I'm still having a mental block :). I can't set up a private link because that requires setting up a load balancer, and I can't create a load balancer because I can't add a frontend IP configuration that references the subnet that is already configured through subnet delegation (Delegate subnet to a service) to Github.Network/networkSettings. In the frontend IP configuration, the subnet is grayed out and not selectable. This may be part of #3 in your last response, as it's was done specifically (per the documentation I referenced) so I could use private endpoints.

In case I neglected something important in my description. The workflow did start, and did get past a pre-login azure/login@v1 step then the actions/checkout@main, then the login with azure/login@v1. It's the docker/login-action@v3 action that failed.

on: 
    workflow_dispatch:
    #[push]
name: Linux_Container_Workflow

jobs:
    build-and-deploy:
        runs-on: ubuntu-latest
        strategy:
            matrix:
                runner-group: 
                  - "githubazure"

        steps:
        - name: Log Runner Group
          run: |
            echo "Runner group used: ${{ matrix.runner-group }}"

        # checkout the repo
        - name: 'Checkout GitHub Action'
          uses: actions/checkout@main

        - name: 'Login via Azure CLI'
          uses: azure/login@v1
          with:
            creds: ${{ secrets.AZURE_CREDENTIALS }}

          # fails on this step with the IP address not allowed error
        - name: 'Build and push image'
          uses: docker/login-action@v3
          with:
            registry: ${{ secrets.REGISTRY_LOGIN_SERVER }}
            username: ${{ secrets.REGISTRY_USERNAME }}
            password: ${{ secrets.REGISTRY_PASSWORD }}

        - run: |
            docker build . -t ${{ secrets.REGISTRY_LOGIN_SERVER }}/myRepo:${{ github.sha }}
            docker push ${{ secrets.REGISTRY_LOGIN_SERVER }}/myRepo:${{ github.sha }}

        - name: 'Deploy to Azure Container Instances'
          uses: 'azure/aci-deploy@v1'
          with:
            resource-group: ${{ secrets.RESOURCE_GROUP }}
            dns-name-label: ${{ secrets.RESOURCE_GROUP }}${{ github.run_number }}
            image: ${{ secrets.REGISTRY_LOGIN_SERVER }}/myRepo:${{ github.sha }}
            registry-login-server: ${{ secrets.REGISTRY_LOGIN_SERVER }}
            registry-username: ${{ secrets.REGISTRY_USERNAME }}
            registry-password: ${{ secrets.REGISTRY_PASSWORD }}
            name: myRepo
            location: 'east us'

I was thinking more about what you wrote about whitelisting, and wanted to know what your thoughts were on the documentation here;
https://docs.github.com/en/organizations/managing-organization-settings/configuring-private-networking-for-github-hosted-runners-in-your-organization

Specifically, the bicep file it says to save as actions-nsg-deployment.bicep. There are two NSG definitions with outbound rules defined. Is this where the whitelisted ip ranges you wrote about should be? I would have thought I'd need inbound rules.

I'm sorry that these questions seem like they're all over the place, but I'm trying to wrap my head around following the documentation in the link above and updating my own documentation about what it doesn't say that has to be configured, which you've been so nicely writing about. I think the documentation may need additional info in the "prerequisites" area based on what you're telling me.

[stan-spotts]

I fixed the problem. This all works without public access, with a private IP defined for the VM. It turned out that the documentation did not include enough in the bicep contents to set up the NSG. In order for azure/login to work (and docker/action-login), I had to add the following to the bicep contents:

{
        name: 'AllowAzureCloudOutbound'
        properties: {
          protocol: 'TCP'
          sourcePortRange: '*'
          destinationPortRange: '443'
          destinationAddressPrefix: 'AzureCloud'
          access: 'Allow'
          priority: 100
          direction: 'Outbound'
          destinationAddressPrefixes: []
        }
      }
      {
        name: 'AllowAzureADOutbound'
        properties: {
          protocol: 'TCP'
          sourcePortRange: '*'
          destinationPortRange: '443'
          destinationAddressPrefix: 'AzureActiveDirectory'
          access: 'Allow'
          priority: 110
          direction: 'Outbound'
          destinationAddressPrefixes: []
        }
      }
      {
        name: 'AllowAzureFrontDoorOutbound'
        properties: {
          protocol: 'TCP'
          sourcePortRange: '*'
          destinationPortRange: '443'
          destinationAddressPrefix: 'AzureFrontDoor.Frontend'
          access: 'Allow'
          priority: 120
          direction: 'Outbound'
          destinationAddressPrefixes: []
        }
      }

FWIW, because I was also building services in containers with node.js, I had to add this so I could reach out to registry-1.docker.io, production.cloudflare.docker.com, and registry.npmjs.org:

      {
        name: 'AllowDockerRegistryAndNpmOutbound'
        properties: {
          protocol: 'TCP'
          sourcePortRange: '*'
          destinationPortRange: '443'
          destinationAddressPrefix: '*'
          access: 'Allow'
          priority: 130
          direction: 'Outbound'
          destinationAddressPrefixes: [
            '54.227.20.253'
            '104.16.101.215'
            '104.16.29.34'
          ]
        }
      }