Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Move setup_environment commands to image generation #1533

Merged
merged 1 commit into from
May 7, 2024
Merged

Move setup_environment commands to image generation #1533

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
168 changes: 91 additions & 77 deletions prog/vm/github_runner.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,90 +191,104 @@ def setup_info
command = <<~COMMAND
# To make sure the script errors out if any command fails
set -ueo pipefail

# In case the script is run until we mv to /home/runner but the state
# could not progress because of any reason (e.g. deployment, runner
# failure), the idempotency is broken. The script needs to be copied back
# to the home directory of the runneradmin. More information regarding the
# operation can be found in the middle of the script where we chown the
# actions-runner.
sudo [ ! -d /home/runner/actions-runner ] || sudo mv /home/runner/actions-runner ./

# Since standard Github runners have both runneradmin and runner users
# VMs of github runners are created with runneradmin user. Adding
# runner user and group with the same id and gid as the standard.
# Although userdel command deletes the group as well, separate groupdel
# command is added to make sure that script can run idempotently if failing
# after addgroup but before adduser command below.
sudo userdel -rf runner || true
sudo groupdel -f runner || true
sudo addgroup --gid 1001 runner
sudo adduser --disabled-password --uid 1001 --gid 1001 --gecos '' runner
echo 'runner ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/98-runner

# runner unix user needed access to manipulate the Docker daemon.
# Default GitHub hosted runners have additional adm,systemd-journal groups.
sudo usermod -a -G docker,adm,systemd-journal runner

# runneradmin user on default Github hosted runners is a member of adm and
# sudo groups. Having sudo access also allows us getting journalctl logs in
# case of any issue on the destroy state below by runneradmin user.
sudo usermod -a -G sudo,adm runneradmin

# Some configuration files such as $PATH related to the user's home directory
# need to be changed. GitHub recommends to run post-generation scripts after
# initial boot.
# The important point, scripts use latest record at /etc/passwd as default user.
# So we need to run these scripts before bootstrap_rhizome to use runner user,
# instead of rhizome user.
# https://github.com/actions/runner-images/blob/main/docs/create-image-and-azure-resources.md#post-generation-scripts
sudo su -c "find /opt/post-generation -mindepth 1 -maxdepth 1 -type f -name '*.sh' -exec bash {} ';'"

# Post-generation scripts write some variables at /etc/environment file.
# We need to reload environment variables again.
source /etc/environment

# We placed the script in the "/usr/local/share/" directory while generating
# the golden image. However, it needs to be moved to the home directory because
# the runner creates some configuration files at the script location. Since the
# github runner vm is created with the runneradmin user, directory is first moved
# to runneradmin user's home directory. At the end of this script, it will be moved
# to runner user's home folder. We are checking first whether actions-runner exists
# under "usr/local/share to make sure that the script can be run multiple times idempotently.
sudo [ ! -d /usr/local/share/actions-runner ] || sudo mv /usr/local/share/actions-runner ./
sudo chown -R runneradmin:runneradmin actions-runner

# ./env.sh sets some variables for runner to run properly
./actions-runner/env.sh

# Include /etc/environment in the runneradmin environment to move it to the
# runner environment at the end of this script, it's otherwise ignored, and
# this omission has caused problems.
# See https://github.com/actions/runner/issues/1703
cat <<EOT > ./actions-runner/run-withenv.sh
echo "image version: $ImageVersion"
echo "started at: $(date)"
if [ "$ImageVersion" == "20240422.1.0" ]; then
# runneradmin user on default Github hosted runners is a member of adm and
# sudo groups. Having sudo access also allows us getting journalctl logs in
# case of any issue on the destroy state below by runneradmin user.
sudo usermod -a -G sudo,adm runneradmin

# The `imagedata.json` file contains information about the generated image.
# I enrich it with details about the Ubicloud environment and placed it in the runner's home directory.
# GitHub-hosted runners also use this file as setup_info to show on the GitHub UI.
jq '. += [#{setup_info.to_json}]' /imagegeneration/imagedata.json | sudo -u runner tee /home/runner/actions-runner/.setup_info
else
byucesoy marked this conversation as resolved.
Show resolved Hide resolved
# In case the script is run until we mv to /home/runner but the state
# could not progress because of any reason (e.g. deployment, runner
# failure), the idempotency is broken. The script needs to be copied back
# to the home directory of the runneradmin. More information regarding the
# operation can be found in the middle of the script where we chown the
# actions-runner.
sudo [ ! -d /home/runner/actions-runner ] || sudo mv /home/runner/actions-runner ./

# Since standard Github runners have both runneradmin and runner users
# VMs of github runners are created with runneradmin user. Adding
# runner user and group with the same id and gid as the standard.
# Although userdel command deletes the group as well, separate groupdel
# command is added to make sure that script can run idempotently if failing
# after addgroup but before adduser command below.
sudo userdel -rf runner || true
sudo groupdel -f runner || true
sudo addgroup --gid 1001 runner
sudo adduser --disabled-password --uid 1001 --gid 1001 --gecos '' runner
echo 'runner ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/98-runner

# runner unix user needed access to manipulate the Docker daemon.
# Default GitHub hosted runners have additional adm,systemd-journal groups.
sudo usermod -a -G docker,adm,systemd-journal runner

# runneradmin user on default Github hosted runners is a member of adm and
# sudo groups. Having sudo access also allows us getting journalctl logs in
# case of any issue on the destroy state below by runneradmin user.
sudo usermod -a -G sudo,adm runneradmin

# Some configuration files such as $PATH related to the user's home directory
# need to be changed. GitHub recommends to run post-generation scripts after
# initial boot.
# The important point, scripts use latest record at /etc/passwd as default user.
# So we need to run these scripts before bootstrap_rhizome to use runner user,
# instead of rhizome user.
# https://github.com/actions/runner-images/blob/main/docs/create-image-and-azure-resources.md#post-generation-scripts
sudo su -c "find /opt/post-generation -mindepth 1 -maxdepth 1 -type f -name '*.sh' -exec bash {} ';'"

# Post-generation scripts write some variables at /etc/environment file.
# We need to reload environment variables again.
source /etc/environment

# We placed the script in the "/usr/local/share/" directory while generating
# the golden image. However, it needs to be moved to the home directory because
# the runner creates some configuration files at the script location. Since the
# github runner vm is created with the runneradmin user, directory is first moved
# to runneradmin user's home directory. At the end of this script, it will be moved
# to runner user's home folder. We are checking first whether actions-runner exists
# under "usr/local/share to make sure that the script can be run multiple times idempotently.
sudo [ ! -d /usr/local/share/actions-runner ] || sudo mv /usr/local/share/actions-runner ./
sudo chown -R runneradmin:runneradmin actions-runner

# ./env.sh sets some variables for runner to run properly
./actions-runner/env.sh

# Include /etc/environment in the runneradmin environment to move it to the
# runner environment at the end of this script, it's otherwise ignored, and
# this omission has caused problems.
# See https://github.com/actions/runner/issues/1703
cat <<EOT > ./actions-runner/run-withenv.sh
#!/bin/bash
mapfile -t env </etc/environment
exec env -- "\\${env[@]}" ./actions-runner/run.sh --jitconfig "\\$1"
EOT
chmod +x ./actions-runner/run-withenv.sh

# runner script doesn't use global $PATH variable by default. It gets path from
# secure_path at /etc/sudoers. Also script load .env file, so we are able to
# overwrite default path value of runner script with $PATH.
# https://github.com/microsoft/azure-pipelines-agent/issues/3461
echo "PATH=$PATH" >> ./actions-runner/.env

# The `imagedata.json` file contains information about the generated image.
# I enrich it with details about the Ubicloud environment and placed it in the runner's home directory.
# GitHub-hosted runners also use this file as setup_info to show on the GitHub UI.
jq '. += [#{setup_info.to_json}]' /imagegeneration/imagedata.json > ./actions-runner/.setup_info

sudo mv ./actions-runner /home/runner/
sudo chown -R runner:runner /home/runner/actions-runner
chmod +x ./actions-runner/run-withenv.sh

# runner script doesn't use global $PATH variable by default. It gets path from
# secure_path at /etc/sudoers. Also script load .env file, so we are able to
# overwrite default path value of runner script with $PATH.
# https://github.com/microsoft/azure-pipelines-agent/issues/3461
echo "PATH=$PATH" >> ./actions-runner/.env

# The `imagedata.json` file contains information about the generated image.
# I enrich it with details about the Ubicloud environment and placed it in the runner's home directory.
# GitHub-hosted runners also use this file as setup_info to show on the GitHub UI.
jq '. += [#{setup_info.to_json}]' /imagegeneration/imagedata.json > ./actions-runner/.setup_info

sudo mv ./actions-runner /home/runner/
sudo chown -R runner:runner /home/runner/actions-runner
fi
echo "finished at: $(date)"
COMMAND

# Remove comments and empty lines before sending them to the machine
vm.sshable.cmd(command.gsub(/^(# .*)?\n/, ""))
vm.sshable.cmd(command.gsub(/^(\s*# .*)?\n/, ""))

hop_register_runner
end
Expand Down
46 changes: 27 additions & 19 deletions spec/prog/vm/github_runner_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -373,29 +373,37 @@
expect(github_runner.installation).to receive(:project).and_return(instance_double(Project, ubid: "pjwnadpt27b21p81d7334f11rx", path: "/project/pjwnadpt27b21p81d7334f11rx")).at_least(:once)
expect(sshable).to receive(:cmd).with(<<~COMMAND)
set -ueo pipefail
sudo [ ! -d /home/runner/actions-runner ] || sudo mv /home/runner/actions-runner ./
sudo userdel -rf runner || true
sudo groupdel -f runner || true
sudo addgroup --gid 1001 runner
sudo adduser --disabled-password --uid 1001 --gid 1001 --gecos '' runner
echo 'runner ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/98-runner
sudo usermod -a -G docker,adm,systemd-journal runner
sudo usermod -a -G sudo,adm runneradmin
sudo su -c "find /opt/post-generation -mindepth 1 -maxdepth 1 -type f -name '*.sh' -exec bash {} ';'"
source /etc/environment
sudo [ ! -d /usr/local/share/actions-runner ] || sudo mv /usr/local/share/actions-runner ./
sudo chown -R runneradmin:runneradmin actions-runner
./actions-runner/env.sh
cat <<EOT > ./actions-runner/run-withenv.sh
echo "image version: $ImageVersion"
echo "started at: $(date)"
if [ "$ImageVersion" == "20240422.1.0" ]; then
sudo usermod -a -G sudo,adm runneradmin
jq '. += [{"group":"Ubicloud Managed Runner","detail":"Name: #{github_runner.ubid}\\nLabel: ubicloud-standard-4\\nArch: \\nImage: \\nVM Host: vhfdmbbtdz3j3h8hccf8s9wz94\\nVM Pool: \\nLocation: hetzner-hel1\\nDatacenter: FSN1-DC8\\nProject: pjwnadpt27b21p81d7334f11rx\\nConsole URL: http://localhost:9292/project/pjwnadpt27b21p81d7334f11rx/github"}]' /imagegeneration/imagedata.json | sudo -u runner tee /home/runner/actions-runner/.setup_info
else
sudo [ ! -d /home/runner/actions-runner ] || sudo mv /home/runner/actions-runner ./
sudo userdel -rf runner || true
sudo groupdel -f runner || true
sudo addgroup --gid 1001 runner
sudo adduser --disabled-password --uid 1001 --gid 1001 --gecos '' runner
echo 'runner ALL=(ALL) NOPASSWD:ALL' | sudo tee /etc/sudoers.d/98-runner
sudo usermod -a -G docker,adm,systemd-journal runner
sudo usermod -a -G sudo,adm runneradmin
sudo su -c "find /opt/post-generation -mindepth 1 -maxdepth 1 -type f -name '*.sh' -exec bash {} ';'"
source /etc/environment
sudo [ ! -d /usr/local/share/actions-runner ] || sudo mv /usr/local/share/actions-runner ./
sudo chown -R runneradmin:runneradmin actions-runner
./actions-runner/env.sh
cat <<EOT > ./actions-runner/run-withenv.sh
#!/bin/bash
mapfile -t env </etc/environment
exec env -- "\\${env[@]}" ./actions-runner/run.sh --jitconfig "\\$1"
EOT
chmod +x ./actions-runner/run-withenv.sh
echo "PATH=$PATH" >> ./actions-runner/.env
jq '. += [{"group":"Ubicloud Managed Runner","detail":"Name: #{github_runner.ubid}\\nLabel: ubicloud-standard-4\\nArch: \\nImage: \\nVM Host: vhfdmbbtdz3j3h8hccf8s9wz94\\nVM Pool: \\nLocation: hetzner-hel1\\nDatacenter: FSN1-DC8\\nProject: pjwnadpt27b21p81d7334f11rx\\nConsole URL: http://localhost:9292/project/pjwnadpt27b21p81d7334f11rx/github"}]' /imagegeneration/imagedata.json > ./actions-runner/.setup_info
sudo mv ./actions-runner /home/runner/
sudo chown -R runner:runner /home/runner/actions-runner
chmod +x ./actions-runner/run-withenv.sh
echo "PATH=$PATH" >> ./actions-runner/.env
jq '. += [{"group":"Ubicloud Managed Runner","detail":"Name: #{github_runner.ubid}\\nLabel: ubicloud-standard-4\\nArch: \\nImage: \\nVM Host: vhfdmbbtdz3j3h8hccf8s9wz94\\nVM Pool: \\nLocation: hetzner-hel1\\nDatacenter: FSN1-DC8\\nProject: pjwnadpt27b21p81d7334f11rx\\nConsole URL: http://localhost:9292/project/pjwnadpt27b21p81d7334f11rx/github"}]' /imagegeneration/imagedata.json > ./actions-runner/.setup_info
sudo mv ./actions-runner /home/runner/
sudo chown -R runner:runner /home/runner/actions-runner
fi
echo "finished at: $(date)"
COMMAND

expect { nx.setup_environment }.to hop("register_runner")
Expand Down
Loading