DevSecOps pipeline project

Phase 1: Initial Setup

Step 1: Install Terraform on the local machine

Here are some Installation steps for MacOS, Windows, Linux & Chocolatey package manager.
Install Terraform

Step 2: Clone the GitHub Repository.

Clone this Git Repo on the locale machine.

https://github.com/darjidhruv26/DevSecOps-Pipeline.git

Open this repository in the code editor.
Open the terminal and change the directory to jenkins_terraform.

cd jenkins_terraform

The terraform init command initializes a working directory for Terraform configuration files.

terraform init

The Terraform plan command compares the current state of resources with the desired state and generates a plan of action.

terraform plan

The Terraform apply command executes the actions proposed in a Terraform plan. It is used to deploy infrastructure.

terrafrom apply

There is one file install_jenkins.sh puts all commands for installing jdk,Jenkins, Docker, SonarQube, trivy, aws cli,kubectl and eksctlin this directory.
So that, when Terraform provisions all resources at that time all the tools will install automatically on EC2.

Go to EC2 Instance details and connect with ssh or Putty.
run all commands --

jenkins --version
docker --version
trivy --version
aws --version
kubectl version --client
eksctl version

Also SonarQube is running in a Docker container.
To check this run docker ps and see sonarqube docker container is running.
After that, access SonarQube in a web browser using public IP of your EC2 instance.

<EC2-Public-IP:9000>
After, Popup one massage for Username and Password.
Username: admin
password: admin

SonarQube Dashboard

Access Jenkins in a web browser using EC2 public IP.

<EC2-Public-IP:8080>
Unlock Jenkins
Run this below command.

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Run this command, After that you will see the Administrator password
Copy and paste pop message and local in a notepad.

Now, Install the suggested plugins.
Jenkins will now get installed and install all the libraries.
After, Create an admin user (Optional step)

Jenkins Dashboard

Install Necessary Plugins in Jenkins:

Goto Manage Jenkins -> Plugins -> Available Plugins -> Install the below plugins

Eclipse Temurin Installer
SonarQube Scanner
Sonar Quality Gates
Quality Gates
NodeJS
Docker
Docker Commons
Docker Pipeline
Docker API
docker-build-step And then click Install

Configure Java JDK, NodeJs, SonarQube Scanner and Docker in Jenkins Global tool Configuretion.

Goto Manage Jenkins -> Tools -> Install JDK(17), NodeJs(16), SonarQube Scanner and Docker.

-> Click on Apply and Save

Configure Sonar Server in Manage Jenkins

Goto SonarQube Dashboard home page
Click on Administration -> Security -> Users -> Click on Tokens and Update Token -> Give it a name -> Generate Token.

Click on Generate Token
Copy Token

Goto Jenkins Dashboard -> Manage Jenkins -> Credentials -> Add Secret Text

Now, go to Dashboard -> Manage Jenkins -> System and Add SoanarQube server credentials
Name: SonarQube-Server,
Server URL: http://<EC2-Public-IP:9000>
Server authentication token: SonarQube-Token

Click on Apply and Save

Create a Quality gate

Goto SonarQube dashboard and Click on Quality Gates
Click on Create -> name SonarQube-Quality-Gate -> Save

Now create a webhook between SonarQube and Jenkins

Goto SonarQube dashboard -> Administration -> Configuration -> Webhooks -> Click on create
Name: jenkins
URL: http://<ec2-public-ip:8080>/sonarqube-webhook/
And click on Create

Create a project on the SonarQube server

Goto SonarQube dashboard -> click on Manually
Create a project
Project display name: Youtube-CICD
Project key name: Youtube-CICD
Main branch name: main
Click on Set-up

Now you can see Analyze your project page

Click on Generate -> Continue -> Other (for JS, TS, Go, Python, PHP,...) -> OS Linux -> Copy commands for the script.

Create a Jenkins pipeline

Goto Jenkins dashboard -> click on +New Item
Job Name: Youtube-CICD
Click on Pipeline -> OK

Configuration

Click on Discard old builds -> Max# build to keep 2
Now apply & save this script
Click on Build Now

pipeline{
    agent any
    tools{
        jdk 'jdk17'
        nodejs 'node16'
    }
    environment {
        SCANNER_HOME=tool 'sonar-scanner'
    }
    stages {
        stage('clean workspace'){
            steps{
                cleanWs()
            }
        }
        stage('Checkout from Git'){
            steps{
                git branch: 'main', url: 'https://github.com/darjidhruv26/YouTube-DevSecOps.git'
            }
        }
        stage("Sonarqube Analysis "){
            steps{
                withSonarQubeEnv('SonarQube-Server') {
                    sh ''' $SCANNER_HOME/bin/sonar-scanner -Dsonar.projectName=Youtube-CICD \
                    -Dsonar.projectKey=Youtube-CICD '''
                }
            }
        }
        stage("quality gate"){
           steps {
                script {
                    waitForQualityGate abortPipeline: false, credentialsId: 'SonarQube-Token' 
                }
            } 
        }
        stage('Install Dependencies') {
            steps {
                sh "npm install"
            }
        }
        stage('TRIVY FS SCAN') {
            steps {
                sh "trivy fs . > trivyfs.txt"
            }
        }
     }
  }
}

1st pipeline output

SonarQube scan output

Set the Trigger

Got to Jenkins

Pipeline -> Configuration
Click on GitHub Project -> Select GitHub project URL
And Build Triggers -> select GitHub hook trigger for GITScm polling
Now go to the Repository settings -> Webhooks -> Add webhook -> add Payload URL http://<jenkins-ec2-public-ip:8080>/github-webhook/ -> Add webhook.

Docker Image Build and Push

Create DockerHub access token

Goto DockerHub -> My Account -> Security -> Create a New access token and save it.

Add DockerHub Credentials

Goto Jenkins Dashboard -> Manage Jenkins -> Manage Credentials
Click on System and then Global Credentials.
Click on Add Credentials -> Secret text -> Enter your DockerHub credentials (Username & Password)
And Save it.

Create an API key from RapidAPI

- Rapid API

Create an account

Now in the search bar search for YouTube and select YouTube v3

Copy API and use it in the file.

docker build --build-arg REACT_APP_RAPID_API_KEY=<API-KEY> -t ${imageName} .

Now add Docker Build and Push commands in the pipeline script.

pipeline{
    agent any
    tools{
        jdk 'jdk17'
        nodejs 'node16'
    }
    environment {
        SCANNER_HOME=tool 'sonar-scanner'
    }
    stages {
        stage('clean workspace'){
            steps{
                cleanWs()
            }
        }
        stage('Checkout from Git'){
            steps{
                git branch: 'main', url: 'https://github.com/darjidhruv26/YouTube-DevSecOps.git'
            }
        }
        stage("Sonarqube Analysis "){
            steps{
                withSonarQubeEnv('SonarQube-Server') {
                    sh ''' $SCANNER_HOME/bin/sonar-scanner -Dsonar.projectName=Youtube-CICD \
                    -Dsonar.projectKey=Youtube-CICD '''
                }
            }
        }
        stage("quality gate"){
           steps {
                script {
                    waitForQualityGate abortPipeline: false, credentialsId: 'SonarQube-Token' 
                }
            } 
        }
        stage('Install Dependencies') {
            steps {
                sh "npm install"
            }
        }
        stage('TRIVY FS SCAN') {
            steps {
                sh "trivy fs . > trivyfs.txt"
            }
        }
        stage("Docker Build & Push"){
            steps{
                script{
                   withDockerRegistry(credentialsId: 'dockerhub', toolName: 'docker'){   
                       sh "docker build --build-arg REACT_APP_RAPID_API_KEY=a578815c0fmsh92bed2dp1b322c4b3260 -t youtube ."
                       sh "docker tag youtube dhruvdarji123/youtube:latest "
                       sh "docker push dhruvdarji123/youtube:latest "
                    }
                }
            }
        }
        stage("TRIVY Image Scan"){
            steps{
                sh "trivy image dhruvdarji123/youtube:latest > trivyimage.txt" 
            }
         }
       }
    }
}

Click Apply and Save
Click Build Now

second pipeline output

DockerHub output

Setup Prometheus and Grafana for monitoring

For installing Prometheus and Grafana go to the monitoring-server directory

cd monitoring-server

The terraform init command initializes a working directory for Terraform configuration files.

terraform init

The Terraform plan command compares the current state of resources with the desired state and generates a plan of action.

terraform plan

The Terraform apply command executes the actions proposed in a Terraform plan. It is used to deploy infrastructure.

terraform apply

Now copy the EC2 instance public IP and connect via putty.
After connection run sudo apt update command.

Now check Prometheus status

For that, Run this command

sudo systemctl status prometheus

Check <EC2-public-ip:9090>

Now check the Grafana Server status

sudo systemctl status grafana-server

Access Grafana web Interface on <EC2-Public-IP:3000>

Now check Node_exporter status

sudo systemctl status node_exporter

Now go to the terminal and run this command

cd /etc/prometheus/

list of all files

ls

Configure Prometheus Plugin Integration:

Prometheus Configuration: To configure Prometheus to scrape metrics from Node Exporte, You need to modify the prometheus.yml file.
run this command to open prometheus.yml in nano editor.

sudo nano prometheus.yml

modify like this.

  - job_name: 'node_exporter'
    static_configs:
      - targets: ['IP-Address:9100']

Check the validity of the configuration file:

promtool check config /etc/prometheus/prometheus.yml

Reload the Prometheus configuration without restarting

curl -X POST http://localhost:9090/-/reload

Now you can access Prometheus targets at:

Add Prometheus Data Source

To visualize metrics, You need to add a data source.

Click on the gear icon (⚙️) in the left sidebar to open the "Configuration" menu.
Select "Data Sources."
Click on the "Add data source" button.
Choose "Prometheus" as the data source type.
In the "HTTP" section:
- Set the "URL" to (http://<Ec2-public-ip:9090) (assuming Prometheus is running on the same server).
- Click the Save & Test button to ensure the data source is working.

Import a Dashboard

To make it easier to view metrics, you can import a pre-configured dashboard. Follow these steps:

Click on the + (plus) icon in the left sidebar to open the Create menu.
Select Dashboard.
Click on the Import dashboard option.
Enter the dashboard code you want to import (e.g., code 1860).
Click the Load button.
Select the data source you added (Prometheus) from the dropdown.
Click on the Import button.

You should now have a Grafana dashboard set up to visualize metrics from Prometheus.

Grafana is a powerful tool for creating visualizations and dashboards, and you can further customize it to suit your specific monitoring needs.

That's it! You've successfully installed and set up Grafana to work with Prometheus for monitoring and visualization.

Configure Prometheus Plugin Integration for Jenkins

Integrate Jenkins with Prometheus to monitor the CI/CD pipeline.

Goto Manage Jenkins -> Plugins -> Available Plugins -> Prometheus metrics -> Install
Restart Jenkins
After that, go to Manage Jenkins -> System -> Prometheus
Configuration Path: Prometheus
Default Namespace: default
Collecting metrics period in seconds 120
Job attribute name: jenkins_job
Click on apply and save

Prometheus Configuration:

To configure Prometheus to scrape metrics from Jenkins, You need to modify the prometheus.yml file.

run this command to open prometheus.yml in nano editor.

cd /etc/prometheus/ & $ sudo nano prometheus.yml

- job_name: 'jenkins'
    metrics_path: '/prometheus'
    static_configs:
      - targets: ['IP-Address:8080']

Make sure to replace and with the appropriate values for your Jenkins setup.

Check the validity of the configuration file:

promtool check config /etc/prometheus/prometheus.yml

Reload the Prometheus configuration without restarting:

curl -X POST http://localhost:9090/-/reload

Import a Dashboard

To make it easier to view metrics, you can import a pre-configured dashboard. Follow these steps:

Click on the + (plus) icon in the left sidebar to open the Create menu.
Select Dashboard.
Click on the Import dashboard option.
Enter the dashboard code you want to import (e.g., code 9964).
Click the Load button.
Select the data source you added (Prometheus) from the dropdown.
Click on the Import button.

Setup Email Notification through Jenkins

Install Email Extension Plugin in Jenkins
Go to your Gmail and Click on Profile
Then click on Manage Your Google Account -> click on the security tab on the left side panel you will get this page(provide mail password).
2-step verification should be enabled.
Search for the app in the search bar you will get app passwords like the below image

Click on Generate and copy the password.
Once the plugin is installed in Jenkins,
click on manage Jenkins --> configure system there under the E-mail Notification section configure the details.
E-mail Notification
SMTP server: smtp.gmail.com
Check Use SMTP Authentication and give your Email and password.
Check Use SSL
SMTP port: 465
Then, Click on Apply and Save
After that, Click on Manage Jenkins -> credentials and add your mail username and generated password -> ID: mail -> Description: mail. Now under the Extended E-mail Notification section configure the details.
SMTP server: smtp.gmail.com
SMTP Port: 465
Advanced ^
- Credentials
- Use SSL
Default Content-Type: HTML
Triggers: Always & Failure-Any & Success
Now click Apply and Save
Go to pipeline and add this script

post {
     always {
        emailext attachLog: true,
            subject: "'${currentBuild.result}'",
            body: "Project: ${env.JOB_NAME}<br/>" +
                "Build Number: ${env.BUILD_NUMBER}<br/>" +
                "URL: ${env.BUILD_URL}<br/>",
            to: '[email protected]',
            attachmentsPattern: 'trivyfs.txt,trivyimage.txt'
        }
    }

Output

Create AWS EKS Cluster

Update packages in the Ubuntu instance

sudo apt update

Check kubectl version

kubectl version --client

Check eksctl version

eksctl version

Creating Role for EC2 instance

After that, Go to AWS IAM (Identity and Access Management)
Roles -> Create role -> AWS service -> select EC2 -> Next
Select AdministratorAccess -> Next
Role Name eksctlEC2Role -> Create Role.

Update IAM role

Now go to eksctl's installed EC2 -> Actions -> Security -> Modify IAM role -> select eksctlEC2Role -> Update IAM role

cd ..

Create an EKS cluster using eksctl command

eksctl create cluster --name youtube-cluster \
--region ap-south-1 \
--node-type t2.small \
--nodes 3 \

Cluster created in AWS

Run this command to check running nodes

kubectl get nodes

Integrate Prometheus with EKS

Check the helm version by using this command

helm version

Installing Prometheus on the EKS cluster using helm

Add Helm stable chart for a local client by using this command

helm repo add stable https://charts.helm.sh/stable

Install Prometheus using helm chart by using this command

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

Create a separate namespace for Prometheus using this command

kubectl create namespace prometheus

Install Prometheus by using this command

helm install stable prometheus-community/kube-prometheus-stack -n prometheus

Check Pods for Prometheus

kubectl get pods -n prometheus

Check services for prometheus

kubectl get svc -n prometheus

These pods are not connected with the external world.
So that, edit Prometheus service file.

kubectl edit svc stable-kube-prometheus-sta-prometheus -n prometheus

Edit the Prometheus service file
In type Cluster IP -> LoadBalancer
And Port expose to 9090

kubectl get svc -n prometheus

Copy the Load Balancer URL and type in Browser.

Goto Prometheus dashboard -> Status -> Targets

Import Grafana Monitoring dashboard for Kubernetes

Goto Grafana Dashboard
Grafana Dashboard -> Connections -> Data sources
+ Add new data source -> Name Prometheus-EKS -> URL http://<LoadBalancer:9090-> Save

Import Dashboard

Create a Dashboard for Kubernetes pods
Goto Grafana -> Dashboards -> Add ID 15760 Click Load -> Data Source Prometheus-EKS -> Click Import

Import Dashboard

Create a Dashboard for the Kubernetes EKS Cluster
Goto Grafana -> Dashboards -> Add ID 17119 Click Load -> Data Source Prometheus-EKS -> Click Import

View all Grafana Dashboards

AWS architecture diagram: EKS Cluster

Configure the Jenkins Pipeline for Deploy Application on AWS EKS

Install the Kubernetes plugin

Go to Jenkins Dashboard -> Manage Jenkins -> Plugins
Kubernetes
Kubernetes Client API
Kubernetes Credentials
kubernetes CLI
Click on Install

Go to Terminal and run ls -a

Go to .kube directory and after running cat config

Copy and Paste all content save in the local secret.txt file
Now add this secret.txt file in Jenkins
Go to Manage Jenkins -> credentials -> System -> Global credentials
New credentials
kind Secret file
upload a secret.txt file
ID Kubernetes Add Kubernetes steps in the pipeline.

pipeline{
    agent any
    tools{
        jdk 'jdk17'
        nodejs 'node16'
    }
    environment {
        SCANNER_HOME=tool 'sonar-scanner'
    }
    stages {
        stage('clean workspace'){
            steps{
                cleanWs()
            }
        }
        stage('Checkout from Git'){
            steps{
                git branch: 'main', url: 'https://github.com/darjidhruv26/YouTube-DevSecOps.git'
            }
        }
        stage("Sonarqube Analysis "){
            steps{
                withSonarQubeEnv('SonarQube-Server') {
                    sh ''' $SCANNER_HOME/bin/sonar-scanner -Dsonar.projectName=Youtube-CICD \
                    -Dsonar.projectKey=Youtube-CICD '''
                }
            }
        }
        stage("quality gate"){
           steps {
                script {
                    waitForQualityGate abortPipeline: false, credentialsId: 'SonarQube-Token' 
                }
            } 
        }
        stage('Install Dependencies') {
            steps {
                sh "npm install"
            }
        }
        stage('TRIVY FS SCAN') {
            steps {
                sh "trivy fs . > trivyfs.txt"
            }
        }
        stage("Docker Build & Push"){
            steps{
                script{
                   withDockerRegistry(credentialsId: 'dockerhub', toolName: 'docker'){   
                       sh "docker build --build-arg REACT_APP_RAPID_API_KEY=a578815c0fmsh92bedc0fa0c572dp1b3ea3jsnd22c4b326093 -t youtube ."
                       sh "docker tag youtube dhruvdarji123/youtube:latest "
                       sh "docker push dhruvdarji123/youtube:latest "
                    }
                }
            }
        }
        stage("TRIVY Image Scan"){
            steps{
                sh "trivy image dhruvdarji123/youtube:latest > trivyimage.txt" 
            }
        }
        stage('Deploy to Kubernets'){
        steps{
            script{
                dir('Kubernetes') {
                     withKubeConfig(caCertificate: '', clusterName: '', contextName: '', credentialsId: 'kubernetes', namespace: '', restrictKubeConfigAccess: false, serverUrl: '')  {
                     sh 'kubectl delete --all pods'
                     sh 'kubectl apply -f deployment.yml'
                     sh 'kubectl apply -f service.yml'
                    }   
                }
            }
        }
    }
    }
    
    post {
     always {
        emailext attachLog: true,
            subject: "'${currentBuild.result}'",
            body: "Project: ${env.JOB_NAME}<br/>" +
                "Build Number: ${env.BUILD_NUMBER}<br/>" +
                "URL: ${env.BUILD_URL}<br/>",
            to: '[email protected]',
            attachmentsPattern: 'trivyfs.txt,trivyimage.txt'
        }
    }
}

Add this pipeline script
Apply and Save
Click on Build Now

Grafana dashboard for

final Grafane dashboard for Kubernetes Pods

Run this command

kubectl get svc

Now copy the LoadBalancer URL and paste in a Web browser

Now enjoy your video streaming application

Clean-Up Environment

This command will delete all the pods in the Prometheus namespace

kubectl delete --all pods -n prometheus

This Command will delete Prometheusnamespace .

kubectl delete namespace prometheus

This command will show all the deployments, pods & services in the default namespace

kubectl get all

Delete deployment in your Kubernetes cluster

kubectl delete deployment.apps/youtube-cluster

Delete service for your deployment of Kubernetes cluster

kubectl delete service/youtube-service

This command will delete your EKS cluster

eksctl delete cluster youtube-cluster --region ap-south-1

OR

eksctl delete cluster --region=ap-south-1 --name=youtube-cluster

Second way

Go to AWS CloudFormation

Select Stacks and Delete that

Destroy terraform infrastructure as

Goto jenkins_terraform directory and run this terraform destroy command.

terraform destroy or terraform destroy -auto-approve

Goto monitoring-server directory and run this terraform destroy command.

terraform destroy or terraform destroy -auto-approve

Name		Name	Last commit message	Last commit date
Latest commit History 29 Commits
Kubernetes		Kubernetes
jenkins_terraform		jenkins_terraform
monitoring-server		monitoring-server
public		public
src		src
.dockerignore		.dockerignore
.env.example		.env.example
.gitignore		.gitignore
Dockerfile		Dockerfile
README.md		README.md
package-lock.json		package-lock.json
package.json		package.json

darjidhruv26/DevSecOps-Pipeline

Folders and files

Latest commit

History

Repository files navigation

DevSecOps pipeline project

Phase 1: Initial Setup

Step 1: Install Terraform on the local machine

Step 2: Clone the GitHub Repository.

SonarQube Dashboard

Jenkins Dashboard

Install Necessary Plugins in Jenkins:

Configure Java JDK, NodeJs, SonarQube Scanner and Docker in Jenkins Global tool Configuretion.

Configure Sonar Server in Manage Jenkins

Create a Quality gate

Now create a webhook between SonarQube and Jenkins

Create a project on the SonarQube server

Create a Jenkins pipeline

Configuration

1st pipeline output

SonarQube scan output

Set the Trigger

Docker Image Build and Push

Create DockerHub access token

Add DockerHub Credentials

Create an API key from RapidAPI

- Rapid API

second pipeline output

DockerHub output

Setup Prometheus and Grafana for monitoring

Now check Prometheus status

Now check the Grafana Server status

Now check Node_exporter status

Configure Prometheus Plugin Integration:

Add Prometheus Data Source

Import a Dashboard

Configure Prometheus Plugin Integration for Jenkins

Prometheus Configuration:

Import a Dashboard

Setup Email Notification through Jenkins

Output

Create AWS EKS Cluster

Creating Role for EC2 instance

Update IAM role

Create an EKS cluster using eksctl command

Cluster created in AWS

Integrate Prometheus with EKS

Installing Prometheus on the EKS cluster using helm

Import Grafana Monitoring dashboard for Kubernetes

Import Dashboard

Import Dashboard

AWS architecture diagram: EKS Cluster

Configure the Jenkins Pipeline for Deploy Application on AWS EKS

Install the Kubernetes plugin

Grafana dashboard for

final Grafane dashboard for Kubernetes Pods

Now enjoy your video streaming application

Clean-Up Environment

Second way

Destroy terraform infrastructure as

About

Topics

Resources

Stars

Watchers

Forks

Languages