Build a complete CI/CD Pipeline and its infrastructure with AWS — Jenkins — Bitbucket — Docker — Terraform → Part 6

In this final part of our journey, we are going to finally implement the CI / CD pipeline using a Jenkinsfile. As it was mentioned in the first part of the tutorial, this pipeline will consist of several parts which will be triggered once a push on the bitbucket repo is made. We will also setup the webhook to let bitbucket inform Jenkins to start a new build on each push.

Part 1 (here)→ Set up the project by downloading and looking at the Web App which will be used to test our infrastructure and pipeline. Here we also create & test suitable Dockerfiles for our project and upload everything to Bitbucket.

Part 2 (here)→ Set up Slack and create a Bot which will be used by Jenkins to send notifications on the progression/status of the pipeline.

Part 3 (here)→ Create the first part of the AWS Infrastructure with Terraform. Here we will create the EC2 instances / SSH keys and the actual Network infrastructure plus the basis for the IAM roles.

Part 4 (here)→ Create the second part of the AWS Infrastructure with Terraform. We are going to create the S3 buckets, the ECR repositories and complete the definition of the IAM roles by adding the correct policies.

Part 5 (here)→ Complete the configuration of the Jenkins and Web App instances by implementing the correct user data.

Part 6 (current article) → Implement the Pipeline in a Jenkinsfile and try out the pipeline, see how eveything fit together and lay down some final comments.

Let’s take another look at the actual pipeline:

After a push on bitbucket, the Jenkins multibranch pipeline will be triggered. The actual pipeline will be extracted from a Jenkinsfile in the root folder of our application ( simple-web-app ). It will consist of several steps which we have already summarized in the first part of the tutorial. Let’s hop over to VSCode in the simple-web-app folder and create a Jenkinsfile :

The Jenkinsfile is coded in Groovy (you can check it here) and it will consist of a pipeline and multiple stages:

pipeline {
  agent any
  stages {
    stage("stage 1") {
      [...]
    }
    stage("stage 2") {
      [...]
    }
    [...]
  }
}

The agent any will tell Jenkins to execute the pipeline on any available agent.

Each stage will then have various steps:

stage("stage 1") {
  steps{
    echo "starting stage 1"
    scripts {
      [...]
    }
  }
}

You can look at the Jenkins documentation here to dig deeper into the syntax.

Let’s first lay down the skeleton of our pipeline. In the Jenkinsfile write the following:

Best thing is that we can already test the pipeline! This will also make sure that the Jenkins setup has really gone well. First of all, we need to create the webhook.

Login into Bitbucket and enter the simple-web-app repo. On the left click on Repository Settings :

and then click on Webhooks :

At this point let’s add a new webhook with Add Webhook :

In the Title let’s put Jenkins and in the URL you need to put the following:

http://<public_dns_jenkins_instance>:8080/bitbucket-hook/

So, in my case, I will fill that field with:

http://ec2-35-168-176-186.compute-1.amazonaws.com:8080/bitbucket-hook/

Let’s keep the Status to Active and the Triggers on Repository Push:

Wonderful, click save and now we need to add the SSH public key. Click on Access Key on the left:

and then Add key . Let’s call this key jenkins and grab the public ssh bitbucket key from the terraform.tfvars (remembering to delete a slash / if you escaped that) in the secrets variable:

Wonderful! let’s try to push some local changes to the repo to see whether the pipeline triggers. First, let’s login into Jenkins by visiting http://<jenkins_url>:8080 and then click on Open Blue Ocean :

At this point, we should see something like this:

We can click the CI-CD Pipeline and then wait there.

Cool, let’s hop over to the terminal in the simple-web-app folder and add , commit and push the changes:

After the push, go to Jenkins and see what happens. After a couple of seconds, it should appear the main branch:

Click on that and you should be redirected to the following:

If the pipeline has already been completed and you didn’t have time to look at this in real time, don’t worry, when it will be implemented, it will be slower and we will be able to look at it in real time. What we can do now is to click on Artifacts on the top right corner and then click on the pipelin.log . Here you should see everything that Jenkins has done under the hood plus the actual stages that we programmed in the Jenkinsfile. Moreover, you can click on the different stages in the pipeline:

And see what happened there:

This is so cool!

Let’s begin with the actual implementation of the Pipeline!

Stage 0 — Environment Variables

Before starting to code the different stages, we will need some variables to keep our builds constistent and keep track of our Docker images that we are going to build. Before the start of the pipeline, in the Jenkinsfile, we are going to define the following variables:

• testImage , stagingImage and productionImage → Docker images for the corresponding stages.
• REPOSITORY , REPOSITORY_TEST and REPOSITORY_STAGING → These will store the URLs of their corresponding AWS ECR repositories. The actual URLs, if you remember, are saved in the Jenkins instance in 3 files: repository_url , repository_staging_url and repository_test_url which are placed in $HOME/opt/ (where $HOME is referred to the Jenkins user which is $HOME == /var/lib/jenkins ). These files are generated by the Jenkins user_data.sh (you can check that!).
• GIT_COMMIT_HASH → This variable is going to contain the GIT commit hash of the current commit. This will be used to uniquely identify the builds and will allow to find them easily in the future when needed given a particular commit. The Docker images will be tagged using that Hash.
• INSTANCE_ID → Will store the Web App instance ID, which is stored in the Jenkins instance in a file called instance_id in /var/lib/jenkins/opt/ . This file too was generated by the user_data.sh of the Jenkins instance.
• ACCOUNT_REGISTRY_PREFIX → Contains the account registry prefix which will be used to log into the AWS ECR.
• S3_LOGS → Will store the S3 bucket name for the logs.
• DATE_NOW → Contains the current date to correctly save the logs in the bucket.
• SLACK_TOKEN →Will store the Slack Bearer Token to let Jenkins POST messages on the Slack Channel
• CHANNEL_ID →Initialized with the ID of the channel. Clearly your channel is going to have a different ID and you can check again the Part 2 of this tutorial to refresh how we actually got that value.

Stage 1 — Set up

Let’s start to implement our stages. First of all we are going to set the environment variables. The syntax is the following:

VARIABLE = sh (script: "echo 'hello'", returnStdout: true)

What this will be is to store hello in the variable VARIABLE . The returnStdout: true is necessary to tell Jenkins to actually return the stdout.

In the ‘Set Up’ stage, in the ‘script’ part, let’s put the following:

• The GIT_COMMIT_HASH is set to git log -n 1 --pretty=format: '%H' and this will output the last git commit hash. You can try this command in the terminal:

• The REPOSITORY , REPOSITORY_TEST , REPOSITORY_STAGING , INSTANCE_ID and S3_LOGS are set as the content of their corresponding files.
• The DATE_NOW is set using the shell command date +%Y%m%d which will output the current date in the format YYYYMMDD (in linux):

After that, to initialize the SLACK_TOKEN variable, we employ a script similar to the one we have already used in the create_credentials.sh , namely the first line:

# Retrieve Secrets and Extract the Private key using a python command
python -c "import sys;import json;print(json.loads(json.loads(raw_input())['SecretString'])['private'])" <<< $(aws secretsmanager get-secret-value --secret-id simple-web-app --region us-east-1) > ssh_tmp

However, in this case, we would like to extract the secret which is called slackToken and we do not need to write that value to an auxiliary file. So, in this case, we will have the following:

After that, we actually need to trim these variables to remove trailing spaces:

After that we define the ACCOUNT_REGISTRY_PREFIX and we log into the AWS ECR:

In the pipeline, as you may have noticed, when we want to execute a shell command we need to use the syntax (for multiline scripts — istead, for singleline script one could use sh “#command“ ):

sh """
# my command
"""

And here, we execute the command to log into the AWS ECR. At this point, the ‘Set Up’ stage should look like the following:

Stage 2 — Build Test Image

In this stage, we just build the Docker test image and we push it to the remote AWS ECR repository:

The syntax docker.build is from the Docker plugin that we installed when configuring Jenkins. As you may notice we provide "$REPOSITORY_TEST:$GIT_COMMIT_HASH" as the first argument which will be the TAG of the image. The second argument, instead, provides additional optional parameters and in this case we specify the correct Dockerfile ( Dockerfile.test ) to build the Test image and the folder . (current). After that we simply testImage.push() to the remote ECR Test Repository.

Stage 3 — Run Unit Tests

Wonderful, we are ready to let the pipeline run some unit tests!

We need to spin up a docker container from the image created before and run the mocha unit tests inside it. Since we also would like to save the tests logs, we need to mount a volume in the docker container to be able to share a portion of the file system to save the reports to. The Jenkins docker plugin allows us to do that by simply using the following syntax:

testImage.inside('-v $WORKSPACE:/output -u root') {
  ...
}

Here we specify the volume with the -v and we mount the $WORKSPACE (which is a global variable in the Jenkins pipeline storing the path to the workspace, it will be something like: /var/lib/jenkins/workspace/CI-CD_Pipeline_main/ ), namely we mount the $WORKSPACE of the Jenkins instance to the /output in the docker container. We also enter the docker container as root and inside the brackets { } we can put the code that will be executed inside the docker container.

The code that we are going to put inside the brackets is the following:

Inside the Docker container we enter the Web App folder which will be in /opt/app/server , then we run the tests with npm run test:unit and finally we check whether a directory named unit in /output exists , if it does we delete it and we move the report from mochawesome to that folder.

Once that code is executed inside the docker container, this container will be stopped and removed.

If the tests fail, the pipeline will stop, preventing us to both notify the user on Slack and upload the result on the S3 bucket. In order to prevent that, we’ll wrap the code in a try {} catch(e) {} finally {} block. Also, we’ll have two variables, a textMessage which will contain the Slack Message (will differ between failing and succeeding builds) and a inError which is going to store whether there has been an error, so that after the Slack notification, we can stop the Pipeline avoiding it to continue with the other stages. In the script { we are going to have a try-catch statement. The try part will look like:

The catch part will simply contain:

and then the finally part will contain the code that needs to be executed regardless the presence of errors:

Here we can see that first we upload the results to the S3 bucket. This is done through the following command:

sh "aws s3 cp ./unit/ s3://$S3_LOGS/$DATE_NOW/$GIT_COMMIT_HASH/unit/ --recursive"

We use the AWS CLI command s3 cp which will copy the content of ./unit/ into the S3 bucket at s3://$S3_LOGS/$DATE_NOW/GIT_COMMIT_HASH/UNIT/ (with the --recursive we tell the command to bring in the bucket every element). The format of the bucket will then be:

<first_name><last_name>-simple-web-app-logs/YYYYMMDD/<git_hash>/unit

After that, we make a POST request to the Slack API, using the Token SLACK_TOKEN and channel ID CHANNEL_ID defined before with the content of the message taken from the variable textMessage .

At the end we check whether there was actually an error and eventually we terminate the Pipeline with error("Failed unit tests") .

The final ‘Run Unit Tests’ stage should look like:

Stage 4 — Run Integration Tests

This part is completely analogous to the previous one, the only changes are that we are going to run npm run test:integration instead of the unit tests. Also we will upload the integration tests in a different folder in the bucket. This stage will then be:

Stage 5 — Build Staging Image

At this point, we can build and push the Staging image. This step will be analogous to the building of the test image. It will look like:

Stage 6 — Run Load Balancing tests / Security checks

This stage could be divided in two stages, but we will condense these two together here.

First we run the load tests as we have done in the first part of the tutorial, but inside the container:

Then we Upload the result to the S3 bucket:

After that we need to run some security checks with Arachni. In order to do that, we need to have the Web App active but we need to be outside the docker container. In order to do that, we use the stagingImage.withRun method, which allows us to execute commands while the docker container is running (and so the server is actually serving the Web App), in order to expose the Web App which will run on the port 8000 inside the docker container, we will map that port to the external port 8000 of the jenkins instanceby making use of the the syntax: -p 8000:8000 . The code will then be:

As you may notice, we are using http://$(hostname):8000 as the URL for Arachni to scan. This is necessary since Arachni does not allow us to specify localhost or 127.x.x.x as the url to scan. Since the Web App is served to the port 8000 , we have previously allowed the traffic to that port in the Jenkins Network Interface, in this way, arachni can successfully make its requests. In the second part we use the arachni_reporter to save the report in a zipped html.

At this point, we can upload the report to the S3 Bucket:

Finally we Slack notify the completion of these tests:

The final complete stage should then look like:

Stage 7 — Deploy to Fixed Server

This is absolutely not the ideal way to make a deployment. During this ‘deployment’ the server will be down and this is not optimal for a Web App (it may be okay for other kind of services). What we are doing in this stage is:

• Build the production;
• Push the production image to the AWS ECR;
• Reboot the EC2 instance serving the Web App.

Notice that we tag the image :release , so that the EC2 instance will grab and run it when rebooting.

stage 8 — Clean Up

At the end of the pipeline we should do some housekeeping, in particular we need to get rid of old images, avoiding them to stack up and occupy memory:

First we tag our images with :latest and then we remove the old ones. This will allow us to have only the last images available. The previous images which were tagged :latest will becoms dangling images with TAG <none> and in order to get rid of these, we use the docker image prune -f.

We also delete the config.json file in the .docker folder which would store the docker credentials. This is just a security precaution so that we are not leaving any key in the instance.

Wonderful! We should have eveything setup and we can now try for the first time the completed pipeline! We just have to add , commit and push the changes. Enter the jenkins instance at http://<jenkins_instance_dns>:8080 , login and click on the Open Blue Ocean (as at the beginning of this part). Enter the CI-CD Pipeline and then in the terminal let’s push the changes to Bitbucket:

git add . ; git commit -a -m “Completed Jenkinsfile” ; git push

In the Jenkins instance (in the blue ocean plugin), once appeared, you can click on the branch main and see the pipeline slowly completing all the tasks. At the end, you should have every step in green:

You can click on each step to see the details of the operations that have been performed in that given stage.

We can always click on the Artifacts button on the top right to see the pipeline.log text file with the details of all the commands run in the pipeline.

Also, you can check Slack to see that our Bot correctly informed us about the progression of the pipeline:

Amazing! Let’s check in the AWS console if everything has been uploaded correctly (from the logs to the docker images). Login to the console and navigate to the ECR (Elastic Container Registry) service. You should see something like:

Let’s check the simple-web-app one! Click it and then you should see the following:

On the right, we can see that there are two medium vulnerabilities in the image. This scan was enabled via the scan_on_push = true in the resource creation of the repository in terraform ( ecr.tf ). Recall that we used a precise node image and hash in the Dockerfiles for the node image to pull from? Well, if you were to use the FROM node:14 for example, you would have something like 500 vulnerabilities from which some of them would be critical!

Alright, Cool, we have the images correctly pushed into the remote ECR. Let’s check the S3 buckets for the logs then! Into the <first_name><last_name>-simple-web-app-logs , you should have a folder with the current date in the format YYYYMMDD , entering it, there should be another folder with the commit hash and inside it there should be the following:

Wonderful, we could now download these files and check these reports if we would like to.

And the Web App?

Upon the completion of the pipeline, the EC2 instance hosting the Web App should have been rebooted and it should have grabbed the production image from the repo and run a container serving the Web App. Let’s check that! In the AWS EC2 service, navigate to instances and click on the Simple Web App instance to then be able to grab the public dns :

Navigate to that address prefixed with http:// and boom! You should see our Web App:

But wait, wasn’t it supposed to have a gray background and the text be centered? If you navigate to /users , this page is okay, and you can check the source code of the web app in simple-web-app/server/src/views , here if you look at the the users.html , you can see that we have the <style> tag with the CSS inside. Instead, in the index.html we have a <link> tag, bringing in the css/styles.css . When we navigate to http://<instance_dns> , the web browser makes a request to https://<instance_dns>/css/styles.css , namely via HTTPS, which we didn’t enable. We can see that in the Chrome Developer Tools in the Network tab:

In order to work around this issue (for this demo purpose only) and also to check if the pipeline works well and the EC2 instance correctly pulls down the last release image, let’s modify the index.html to the following:

And now, add , commit and push the changes!

git add .
git commit -a -m "Enable CSS in home page"
git push

You can check the Jenkins pipeline and once it is finished and the EC2 instance hosting the Web App has been rebooted, we can navigate to the Web App URL and see whether something has changed. Indeed we find:

Wonderful, our pipeline works like a charm and the EC2 instance hosting the Web App correctly grab the release image and run it.

Enhancements

In addition to the improvements that were mentioned in the first part of the tutorial, there are some minor changes that I incourage you to try making. These will allow you to test your understanding of the infrastructure and its implementation.

Clean Up After a Fail

If the pipeline fails, there isn’t a cleaning phase, the docker images would pile up without being automatically removed.

Clean Up the Web App Instance Docker Images

After the EC2 instance is rebooted, it will pull down the new release image from the remote ECR repository, however, the old one will not be cleaned up.

Hard-Coded Secret Name

In create_credentials.sh in the following $(aws secretsmanager get-secret-value --secret-id simple-web-app --region us-east-1) we have hard-coded the secret’s name simple-web-app . Best would be to define it as a variable and pass it to the scripts as we have done with the other variables.

ECR Repositories Clean Up

Right now, we do not have any limit on the number of images in the AWS ECR repositories. Since old images might not be used anymore, it should be better to clean them up. This can be done by considering the elapsed time between the current and the uploaded time or by considering the number of images pushed on top of that. Either way, we could store these ‘expired’ images in an S3 Glacier, this is a super cheap type of Bucket which allows to store Artifacts which shouldn’t be touched often. With the command docker save one can create a tar archive and then upload to that S3 Glacier, while deleting that image from the remote ECR.

S3 Bucket Clean Up

As for the ECR repositories, the S3 bucket storing the logs should be cleaned up after a given amount of time has passed or a given amount of commits have been performed. The old logs could be placed in an S3 Glacier to still keep them for any eventuality.

Send Reports to Slack

We could send the unit/integration/load balancing tests results and Arachni reports to the Slack channel to make them available immediately to the developers. If we give our Bot an additional permission: files:write , then we can use the API https://slack.com/api/files.upload to upload a file to the channel (more here).

Slack Bearer Token in Plain Text

If you look at the pipeline.log (the logs of execution of the pipeline), you might see that the Slack Bearer Token is in plain text in the curl requests. This may be a security concern and it would be better to avoid printing that token in the logs. This may be achieved by wrapping the command in a /bin/bash -c "curl [...] .

Final Considerations

As already mentioned in the first part of the tutorial, this infrastructure is not optimal for a real Web Application for a number of critical points: there is neither a load balancer nor an auto scaling group and the server will be down for the deployment time. However, this tutorial is not intended to build a complete project, but to introduce curious readers to these technologies and to some of the techniques that may be used in more serious projects. If you want to keep enhancing this project, you can first start by implementing the above improvements. After these, you can start working on the Infrastructure and try to add all the elements already mentioned various times: Load Balancer, Auto Scaling group, Blue/Green Deployment, Cloud Watch, Handling of multiple branches, Jira integration (this is almost trivial since we are already using Bitbucket, you only need to create a project in Jira and link the repo. Then if the GIT commits will start with the required characters, the commit will be displayed in Jira in the right project), and more.

If you managed to follow the tutorial all the way until the end, congrats to you!! It has been quite a long and fun journey, we have touched a great number of technologies and techniques which I hope you found interesting! The completed project can be found here:

KevinDeNotariis/pipeline-template

If you have any kind of feedback or you found an error or you have a suggestion to improve this tutorial, please — please — please let me know!

Thanks for staying with me and I’ll see you around!

Kevin