Skip to content

Commit

Permalink
Move setup_environment commands to image generation
Browse files Browse the repository at this point in the history 
The `setup_environment` step involves commands to convert a standard VM
into a runner. Over time, the number of commands has increased in
response to various customer issues, which has also lengthened the run
time.

We previously tried this, but certain commands had to run at runtime
because the runner user was created by cloud-init. Recently, we
separated the runner and runneradmin users. Now, cloud-init creates the
runneradmin user, and we create the runner user in setup_environment.
This allows us to move runner user related commands to image generation.

If the latest image is used, we only run the short version of the
setup_environment script. If an older image is used, we run the long
version of the `setup_environment` script. It allows us to deploy the
new image to some hosts and test it before deploying it to all hosts.

The new version saves the ".setup_info" file using "tee" instead of the
">" redirect, as the file should have different ownership.
  • Loading branch information
@enescakir
enescakir committed May 7, 2024
1 parent 816d182 commit 604cc3d
Show file tree
Hide file tree
Showing 2 changed files with 118 additions and 96 deletions.
168 changes: 91 additions & 77 deletions prog/vm/github_runner.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -191,90 +191,104 @@ def setup_info
command = <<~COMMAND
# To make sure the script errors out if any command fails
set -ueo pipefail
# In case the script is run until we mv to /home/runner but the state
# could not progress because of any reason (e.g. deployment, runner
# failure), the idempotency is broken. The script needs to be copied back
# to the home directory of the runneradmin. More information regarding the
# operation can be found in the middle of the script where we chown the
# actions-runner.
sudo [ ! -d /home/runner/actions-runner ] || sudo mv /home/runner/actions-runner ./
# Since standard Github runners have both runneradmin and runner users
# VMs of github runners are created with runneradmin user. Adding
# runner user and group with the same id and gid as the standard.
# Although userdel command deletes the group as well, separate groupdel
# command is added to make sure that script can run idempotently if failing
# after addgroup but before adduser command below.
sudo userdel -rf runner || true
sudo groupdel -f runner || true
sudo addgroup --gid 1001 runner
sudo adduser --disabled-password --uid 1001 --gid 1001 --gecos '' runner
echo 'runner ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/98-runner
# runner unix user needed access to manipulate the Docker daemon.
# Default GitHub hosted runners have additional adm,systemd-journal groups.
sudo usermod -a -G docker,adm,systemd-journal runner
# runneradmin user on default Github hosted runners is a member of adm and
# sudo groups. Having sudo access also allows us getting journalctl logs in
# case of any issue on the destroy state below by runneradmin user.
sudo usermod -a -G sudo,adm runneradmin
# Some configuration files such as $PATH related to the user's home directory
# need to be changed. GitHub recommends to run post-generation scripts after
# initial boot.
# The important point, scripts use latest record at /etc/passwd as default user.
# So we need to run these scripts before bootstrap_rhizome to use runner user,
# instead of rhizome user.
# https://github.com/actions/runner-images/blob/main/docs/create-image-and-azure-resources.md#post-generation-scripts
sudo su -c "find /opt/post-generation -mindepth 1 -maxdepth 1 -type f -name '*.sh' -exec bash {} ';'"
# Post-generation scripts write some variables at /etc/environment file.
# We need to reload environment variables again.
source /etc/environment
# We placed the script in the "/usr/local/share/" directory while generating
# the golden image. However, it needs to be moved to the home directory because
# the runner creates some configuration files at the script location. Since the
# github runner vm is created with the runneradmin user, directory is first moved
# to runneradmin user's home directory. At the end of this script, it will be moved
# to runner user's home folder. We are checking first whether actions-runner exists
# under "usr/local/share to make sure that the script can be run multiple times idempotently.
sudo [ ! -d /usr/local/share/actions-runner ] || sudo mv /usr/local/share/actions-runner ./
sudo chown -R runneradmin:runneradmin actions-runner
# ./env.sh sets some variables for runner to run properly
./actions-runner/env.sh
# Include /etc/environment in the runneradmin environment to move it to the
# runner environment at the end of this script, it's otherwise ignored, and
# this omission has caused problems.
# See https://github.com/actions/runner/issues/1703
cat <<EOT > ./actions-runner/run-withenv.sh
echo "image version: $ImageVersion"
echo "started at: $(date)"
if [ "$ImageVersion" == "20240422.1.0" ]; then
# runneradmin user on default Github hosted runners is a member of adm and
# sudo groups. Having sudo access also allows us getting journalctl logs in
# case of any issue on the destroy state below by runneradmin user.
sudo usermod -a -G sudo,adm runneradmin
# The `imagedata.json` file contains information about the generated image.
# I enrich it with details about the Ubicloud environment and placed it in the runner's home directory.
# GitHub-hosted runners also use this file as setup_info to show on the GitHub UI.
jq '. += [#{setup_info.to_json}]' /imagegeneration/imagedata.json | sudo -u runner tee /home/runner/actions-runner/.setup_info
else
# In case the script is run until we mv to /home/runner but the state
# could not progress because of any reason (e.g. deployment, runner
# failure), the idempotency is broken. The script needs to be copied back
# to the home directory of the runneradmin. More information regarding the
# operation can be found in the middle of the script where we chown the
# actions-runner.
sudo [ ! -d /home/runner/actions-runner ] || sudo mv /home/runner/actions-runner ./
# Since standard Github runners have both runneradmin and runner users
# VMs of github runners are created with runneradmin user. Adding
# runner user and group with the same id and gid as the standard.
# Although userdel command deletes the group as well, separate groupdel
# command is added to make sure that script can run idempotently if failing
# after addgroup but before adduser command below.
sudo userdel -rf runner || true
sudo groupdel -f runner || true
sudo addgroup --gid 1001 runner
sudo adduser --disabled-password --uid 1001 --gid 1001 --gecos '' runner
echo 'runner ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/98-runner
# runner unix user needed access to manipulate the Docker daemon.
# Default GitHub hosted runners have additional adm,systemd-journal groups.
sudo usermod -a -G docker,adm,systemd-journal runner
# runneradmin user on default Github hosted runners is a member of adm and
# sudo groups. Having sudo access also allows us getting journalctl logs in
# case of any issue on the destroy state below by runneradmin user.
sudo usermod -a -G sudo,adm runneradmin
# Some configuration files such as $PATH related to the user's home directory
# need to be changed. GitHub recommends to run post-generation scripts after
# initial boot.
# The important point, scripts use latest record at /etc/passwd as default user.
# So we need to run these scripts before bootstrap_rhizome to use runner user,
# instead of rhizome user.
# https://github.com/actions/runner-images/blob/main/docs/create-image-and-azure-resources.md#post-generation-scripts
sudo su -c "find /opt/post-generation -mindepth 1 -maxdepth 1 -type f -name '*.sh' -exec bash {} ';'"
# Post-generation scripts write some variables at /etc/environment file.
# We need to reload environment variables again.
source /etc/environment
# We placed the script in the "/usr/local/share/" directory while generating
# the golden image. However, it needs to be moved to the home directory because
# the runner creates some configuration files at the script location. Since the
# github runner vm is created with the runneradmin user, directory is first moved
# to runneradmin user's home directory. At the end of this script, it will be moved
# to runner user's home folder. We are checking first whether actions-runner exists
# under "usr/local/share to make sure that the script can be run multiple times idempotently.
sudo [ ! -d /usr/local/share/actions-runner ] || sudo mv /usr/local/share/actions-runner ./
sudo chown -R runneradmin:runneradmin actions-runner
# ./env.sh sets some variables for runner to run properly
./actions-runner/env.sh
# Include /etc/environment in the runneradmin environment to move it to the
# runner environment at the end of this script, it's otherwise ignored, and
# this omission has caused problems.
# See https://github.com/actions/runner/issues/1703
cat <<EOT > ./actions-runner/run-withenv.sh
#!/bin/bash
mapfile -t env </etc/environment
exec env -- "\\${env[@]}" ./actions-runner/run.sh --jitconfig "\\$1"
EOT
chmod +x ./actions-runner/run-withenv.sh
# runner script doesn't use global $PATH variable by default. It gets path from
# secure_path at /etc/sudoers. Also script load .env file, so we are able to
# overwrite default path value of runner script with $PATH.
# https://github.com/microsoft/azure-pipelines-agent/issues/3461
echo "PATH=$PATH" >> ./actions-runner/.env
# The `imagedata.json` file contains information about the generated image.
# I enrich it with details about the Ubicloud environment and placed it in the runner's home directory.
# GitHub-hosted runners also use this file as setup_info to show on the GitHub UI.
jq '. += [#{setup_info.to_json}]' /imagegeneration/imagedata.json > ./actions-runner/.setup_info
sudo mv ./actions-runner /home/runner/
sudo chown -R runner:runner /home/runner/actions-runner
chmod +x ./actions-runner/run-withenv.sh
# runner script doesn't use global $PATH variable by default. It gets path from
# secure_path at /etc/sudoers. Also script load .env file, so we are able to
# overwrite default path value of runner script with $PATH.
# https://github.com/microsoft/azure-pipelines-agent/issues/3461
echo "PATH=$PATH" >> ./actions-runner/.env
# The `imagedata.json` file contains information about the generated image.
# I enrich it with details about the Ubicloud environment and placed it in the runner's home directory.
# GitHub-hosted runners also use this file as setup_info to show on the GitHub UI.
jq '. += [#{setup_info.to_json}]' /imagegeneration/imagedata.json > ./actions-runner/.setup_info
sudo mv ./actions-runner /home/runner/
sudo chown -R runner:runner /home/runner/actions-runner
fi
echo "finished at: $(date)"
COMMAND

# Remove comments and empty lines before sending them to the machine
vm.sshable.cmd(command.gsub(/^(# .*)?\n/, ""))
vm.sshable.cmd(command.gsub(/^(\s*# .*)?\n/, ""))

hop_register_runner
end
Expand Down
46 changes: 27 additions & 19 deletions spec/prog/vm/github_runner_spec.rb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -373,29 +373,37 @@
expect(github_runner.installation).to receive(:project).and_return(instance_double(Project, ubid: "pjwnadpt27b21p81d7334f11rx", path: "/project/pjwnadpt27b21p81d7334f11rx")).at_least(:once)
expect(sshable).to receive(:cmd).with(<<~COMMAND)
set -ueo pipefail
sudo [ ! -d /home/runner/actions-runner ] || sudo mv /home/runner/actions-runner ./
sudo userdel -rf runner || true
sudo groupdel -f runner || true
sudo addgroup --gid 1001 runner
sudo adduser --disabled-password --uid 1001 --gid 1001 --gecos '' runner
echo 'runner ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/98-runner
sudo usermod -a -G docker,adm,systemd-journal runner
sudo usermod -a -G sudo,adm runneradmin
sudo su -c "find /opt/post-generation -mindepth 1 -maxdepth 1 -type f -name '*.sh' -exec bash {} ';'"
source /etc/environment
sudo [ ! -d /usr/local/share/actions-runner ] || sudo mv /usr/local/share/actions-runner ./
sudo chown -R runneradmin:runneradmin actions-runner
./actions-runner/env.sh
cat <<EOT > ./actions-runner/run-withenv.sh
echo "image version: $ImageVersion"
echo "started at: $(date)"
if [ "$ImageVersion" == "20240422.1.0" ]; then
sudo usermod -a -G sudo,adm runneradmin
jq '. += [{"group":"Ubicloud Managed Runner","detail":"Name: #{github_runner.ubid}\\nLabel: ubicloud-standard-4\\nArch: \\nImage: \\nVM Host: vhfdmbbtdz3j3h8hccf8s9wz94\\nVM Pool: \\nLocation: hetzner-hel1\\nDatacenter: FSN1-DC8\\nProject: pjwnadpt27b21p81d7334f11rx\\nConsole URL: http://localhost:9292/project/pjwnadpt27b21p81d7334f11rx/github"}]' /imagegeneration/imagedata.json | sudo -u runner tee /home/runner/actions-runner/.setup_info
else
sudo [ ! -d /home/runner/actions-runner ] || sudo mv /home/runner/actions-runner ./
sudo userdel -rf runner || true
sudo groupdel -f runner || true
sudo addgroup --gid 1001 runner
sudo adduser --disabled-password --uid 1001 --gid 1001 --gecos '' runner
echo 'runner ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/98-runner
sudo usermod -a -G docker,adm,systemd-journal runner
sudo usermod -a -G sudo,adm runneradmin
sudo su -c "find /opt/post-generation -mindepth 1 -maxdepth 1 -type f -name '*.sh' -exec bash {} ';'"
source /etc/environment
sudo [ ! -d /usr/local/share/actions-runner ] || sudo mv /usr/local/share/actions-runner ./
sudo chown -R runneradmin:runneradmin actions-runner
./actions-runner/env.sh
cat <<EOT > ./actions-runner/run-withenv.sh
#!/bin/bash
mapfile -t env </etc/environment
exec env -- "\\${env[@]}" ./actions-runner/run.sh --jitconfig "\\$1"
EOT
chmod +x ./actions-runner/run-withenv.sh
echo "PATH=$PATH" >> ./actions-runner/.env
jq '. += [{"group":"Ubicloud Managed Runner","detail":"Name: #{github_runner.ubid}\\nLabel: ubicloud-standard-4\\nArch: \\nImage: \\nVM Host: vhfdmbbtdz3j3h8hccf8s9wz94\\nVM Pool: \\nLocation: hetzner-hel1\\nDatacenter: FSN1-DC8\\nProject: pjwnadpt27b21p81d7334f11rx\\nConsole URL: http://localhost:9292/project/pjwnadpt27b21p81d7334f11rx/github"}]' /imagegeneration/imagedata.json > ./actions-runner/.setup_info
sudo mv ./actions-runner /home/runner/
sudo chown -R runner:runner /home/runner/actions-runner
chmod +x ./actions-runner/run-withenv.sh
echo "PATH=$PATH" >> ./actions-runner/.env
jq '. += [{"group":"Ubicloud Managed Runner","detail":"Name: #{github_runner.ubid}\\nLabel: ubicloud-standard-4\\nArch: \\nImage: \\nVM Host: vhfdmbbtdz3j3h8hccf8s9wz94\\nVM Pool: \\nLocation: hetzner-hel1\\nDatacenter: FSN1-DC8\\nProject: pjwnadpt27b21p81d7334f11rx\\nConsole URL: http://localhost:9292/project/pjwnadpt27b21p81d7334f11rx/github"}]' /imagegeneration/imagedata.json > ./actions-runner/.setup_info
sudo mv ./actions-runner /home/runner/
sudo chown -R runner:runner /home/runner/actions-runner
fi
echo "finished at: $(date)"
COMMAND

expect { nx.setup_environment }.to hop("register_runner")
Expand Down

0 comments on commit 604cc3d

Please sign in to comment.