Logging and monitoring a service with AWS Cloudwatch

[ Monitoring Linux Bash AWS_Cloudwatch ] Create a solution to monitor a user's folder and processes launched on a server.

Task:

Monitor the existing solution. The recently added features make the existing solution expected to behave in the following way:

Independent processes will be started up to serve individual client requests. These should all be started by the transmogrifier user.
Each of these processes may run for an extended period, depending on how much work is required to serve their specific request. This workload is expected to be highly variable.
A dedicated Transmogrified/ folder has been created on each server, which these processes will occasionally write to.

You have been asked to create a solution which monitors the following at regular intervals:

Processes currently running on the machine.
Contents of the Transmogrified/ folder.

Write a script in the language of your choosing, which will be used to achieve the above.

Application layer:

To monitor the running processes and the contents of the Transmogrified/ folder as described in the scenario, you can launch the following script:

Please note that sudo priveleges are only granted for the smooth testing. In production they should be adjusted accordingly.

0. Set of commands to create a user and Transmogrified/ folder

Details

sudo useradd transmogrifier
usermod -a -G ec2-user transmogrifier
sudo mkdir /home/transmogrifier/Transmogrified/
sudo passwd transmogrifier

1. Script transmogrifier-monitor.sh to write in intervals to .log files

Details

#!/bin/bash

# This script is to run indefinitely and periodically log information about the transmogrifier process and its associated files.

while true; do  # Start an infinite loop

  # Log the list of processes running the transmogrifier command, along with the hostname and current date/time, to a file called transmogrifier_process.log
  sudo bash -c "sudo printf '\n%s %s %s\n\n%s\n' 'Processes lists for transmogrifier:' '$(hostname)' '$(date +'%Y-%m-%d %H:%M:%S')' '$(ps -u transmogrifier -f)' >> /var/log/transmogrifier_process.log"

  # Log the list of files in the Transmogrified directory, along with the hostname and current date/time, to a file called transmogrifier_files.log
  sudo bash -c "sudo printf '\n%s %s %s\n\n%s\n' 'File list of transmogrifier:' '$(hostname)' '$(date +'%Y-%m-%d %H:%M:%S')' '$(sudo ls -la /home/transmogrifier/Transmogrified/)' >> /var/log/transmogrifier_files.log"


  sleep 300  # Wait for 300 seconds (5 minutes) before running the loop again
done

2. Commands to start the script in the background

Start in the background for testing

# This line grants execute permission to the transmogrifier-monitor.sh script, allowing it to be run as a command
sudo chmod +x /usr/local/bin/transmogrifier-monitor.sh  

# This line runs the transmogrifier-monitor.sh script in the background as a root user using the Bash shell
sudo nohup ./transmogrifier-monitor.sh &

Start as a "systemd" (daemon) service for production (starts automatically at a system boot time)

Grant the executable privileges to your script

sudo chmod +x /usr/local/bin/transmogrifier-monitor.sh

Create a service file in the /etc/systemd/system/ directory. You can use any name you like for the file, but it must end with the .service extension. For example, you can create a file called transmogrifier.service using the following command:

sudo vi /etc/systemd/system/transmogrifier.service

Paste this in transmogrifier.service file. It sets the description for the service, specifies that it should start after the network is available, and sets the ExecStart command to run the /usr/local/bin/transmogrifier-monitor.sh Bash script with elevated privileges using the root user. The Restart option ensures that the service will be restarted if it crashes or stops running for any reason, and the WantedBy option specifies that the service should be enabled for all users who have a multi-user.target session.

[Unit]
Description=Transmogrifier Logging Service
After=network.target

[Service]
ExecStart=/bin/bash /usr/local/bin/transmogrifier-monitor.sh 
Restart=always
User=root

[Install]
WantedBy=multi-user.target

Save and close the file. ":wq!"
Reload systemctl to read the new service file:

sudo systemctl daemon-reload

Start the service:

sudo systemctl start transmogrifier.service

This will start the service and the script will run indefinitely in the background.

Enable the service to start automatically at boot time:

sudo systemctl enable transmogrifier.service

You can check the status of the service using the following command:

sudo systemctl status transmogrifier.service

You can check the status of all running daemons using the following commands:

sudo systemctl list-units

sudo systemctl list-unit-files

Representation layer:

3. Setting up the logs to be regularly sent to AWS CloudWatch by installing the CloudWatch agent to the controlled machine, for metrics monitoring, visualization and triggering notifications if needed

This is how to set up

To configure the CloudWatch agent to send the required logs to CloudWatch, follow these steps:

SSH into the EC2 instance hosting the servers that will run the Transmogrifier.
Install the awslogs package. This is the recommended method for installing awslogs on Amazon Linux instances.

sudo yum update -y

sudo yum install -y awslogs

Once installed, open the CloudWatch Logs agent configuration file located at /etc/awslogs/awslogs.conf.
Add log files that you want to monitor to the configuration file, specifying the log file location, log format, and destination log group in CloudWatch. Here is an example configuration entry:

[/var/log/transmogrifier_process.log]
datetime_format = %b %d %H:%M:%S
file = /var/log/transmogrifier_process.log
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = transmogrifier_demo_processes

[/var/log/transmogrifier_files.log]
datetime_format = %b %d %H:%M:%S
file = /var/log/transmogrifier_files.log
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = transmogrifier_demo_files

In this example, we're monitoring the /var/log/transmogrifier_process.log file and sending its contents to log groups named transmogrifier_demo_processes and transmogrifier_demo_files in CloudWatch. The log_stream_name parameter will automatically include the instance ID in the log stream name, allowing you to distinguish between logs from different instances.

By default, the /etc/awslogs/awscli.conf points to the us-east-1 Region. To push your logs to a different Region, edit the awscli.conf file and specify that Region.
If you are running Amazon Linux 2, start the awslogs service with the following command: