This repository contains the code used to build images containing the base operating system and tools that are required by all Linux resources. Images can be created for Hyper-V or Azure.
The current process will install Ubuntu 18.04.3 Server, i.e. without UI, on the disk and will then configure the following tools and services:
- Consul - Provides service discovery for the environment as well as a distributed key-value store.
- Consul-Template - Renders template files based on
information stored in the
Consul
key-value store and the Vault secret store. - Syslog-ng - Captures logs send to the syslog stream and stores them both locally and forwards them onto the central log storage server.
- Telegraf - Captures metrics for the resource and forwards them onto the time series database for storage and processing.
- Unbound - A local DNS resolver to allow resolving DNS requests via Consul for the environment specific requests and external DNS servers for all other requests.
- A single network interface is configured,
eth0
, with DHCP enabled. - SSH is enabled on port 22.
- The firewall is enabled and blocks all ports except the ports that are explicitly opened.
- All available updates will be applied.
- A single administrator level user is added called
thebigkahuna
. - A set of standard applications are installed as mentioned above.
- Configurations for
Consul
andUnbound
should be provided via the provisioning CD when a new machine is created from the base image. All other services and applications should obtain their configuration viaConsul-Template
and theConsul
key-value store.
For provisoning reasons a systemd daemon
called provision
is added which:
- Read the files on the DVD drive (for Hyper-V) or from the
/run/cloud-init
directory (for Azure using cloud-init) and:- Disable SSH if the
allow_ssh.json
file does not exist - Copy the configuration files and certificates for consul, syslog-ng, telegraf and unbound
- Enable all the deamons for the afore mentioned services
- Execute the resource specific provisioning steps found in the
f_provisionImage
function in/etc/init.d/provision_image.sh
file.
- Disable SSH if the
- Sets the host name to
cv<SHORT_NAME>-<MAJOR>-<MINOR>-<PATCH>-<16_CHARACTER_RANDOM_STRING>
where<SHORT_NAME>
- Is, in general, the name of the resource without theResource-
section<MAJOR>
- The major version number<MINOR>
- The minor version number<PATCH>
- The patch version number<16_CHARACTER_RANDOM_STRING>
- A cryptographically random string of 16 characters
- Eject the DVD if the provisioning files were obtained from DVD
- Restart the machine to ensure that all changes are locked in and so that the machine comes up with the new machine name
For Consul there are a number of configuration files that are expected in the provisioning location. For server and client nodes they are:
- consul/consul_region.json - Contains the Consul datacenter and domain information
- consul/consul_secrets.json - Contains the gossip encrypt key
- [Optional] consul/consul_connect.json - Contains the configuration for Consul Connect
- [Optional] consul/certs/consul_cert.key - The key file for the certificate that Consul is going to use to encrypt node to node communication.
- [Optional] consul/certs/consul_cert.crt - The certificate file for the certificate that Consul is going to use to encrypt node to node communication
- [Optional] consul/certs/consul_cert_bundle.crt - The certificate bundle containing the root certificates for the node to node communication encryption
For client nodes also provide:
- consul/client/consul_client_location.json - Contains the configuration entries that tell Consul how to connect to the cluster
For server nodes specifically also provide:
- consul/server/consul_server_bootstrap.json - Contains the Consul bootstrap information
- consul/server/consul_server_location.json - Contains the configuration entries that tell Consul how to connect to the other cluster nodes
For examples on how to configure for Hyper-V please look at the configuration folder in the
calvinverse.configuration repository. For examples on how to configure when using Azure review the cloud-init
files in the infrastructure.azure.core.servicediscovery repository.
For Unbound one configuration file is expected. This file is expected to be found in the provisioning location at: unbound/unbound_zones.conf
and it is expected to contain the unbound zone information.
For examples on how to configure for Hyper-V please look at the configuration folder in the
calvinverse.configuration repository. For examples on how to configure when using Azure review the cloud-init
files in the infrastructure.azure.core.servicediscovery repository.
Logs are collected via the Syslog-ng which will normally write the logs to
disk. If the Consul-Template service has been provided with the appropriate credentials then it will
generate additional configuration for the syslog service that allows the logs to be pushed to a
RabbitMQ exchange. The exchange the log messages are pushed to is determined by the
Consul Key-Value key at config/services/queue/logs/syslog/exchange
on the
vhost defined by the config/services/queue/logs/syslog/vhost
K-V key. The syslog
routing key is applied to each log message.
Metrics are collected through different means.
- Metrics for Consul are collected by Consul sending StatsD metrics to Telegraf.
- Metrics for Unbound are collected by Telegraf pulling the metrics.
- System metrics, e.g. CPU, disk, network and memory usage, are collected by Telegraf.
The build process follows the standard procedure for building Calvinverse images. Because the base image is build during this process the following differences exist.
-
In order to build a Hyper-V image the following properties need to be specified as part of the command line used to build the image:
ShouldCreateHyperVImage
should be set totrue
- The Ubuntu Server 18.04.3 ISO is obtained from the internal storage as defined by the MsBuild
property
IsoDirectory
.
-
A number of additional scripts and configuration files have to be gathered. Amongst these files is the Ubuntu
preseed.cfg
file. The preseed file contains the OS configuration and it is provided to the machine when booting from the ISO initially. -
Once Packer has created the VM it will additionally
- Add the OS ISO as a secondary DVD drive
- Start the machine and provide the boot command which points the machine to the ISO and the location of the preseed
file. The OS installation will start and during this process the preseed file is read leading the machine to be
configured with
- A US english culture
- In the UTC timezone
- A single administrator user called
thebigkahuna
- Four partitions on the hard drive for:
- BIOS boot
- EFI boot
- OS boot (mounted as
/boot
) - Swap
- General use (mounted as
/
)
- The Hyper-V packages necessary for Hyper-V to connect to Linux
- Once the OS is installed the standard process will be followed
- In order to build an Azure image the following properties need to be specified as part of the
command line used to build the image:
ShouldCreateAzureImage
should be set totrue
AzureClientId
- The client ID of the service principalAzureClientCertPath
- The path to the certificate for the service principalAzureLocation
- The name of the Azure region in which the image should be createdAzureImageResourceGroup
- The name of the resource group into which the image should be storedAzureSubscriptionId
- The subscription ID
The base image should never be deployed to live running infrastructure hence it will not be needing deploy information.