August 1, 2019

CI/CD with Kubernetes

Kubernetes is compatible with the majority of Continuous Integration / Continuous Delivery (CI/CD) tools which allows developers to run tests, deploy builds in Kubernetes and update applications with no downtime. One of the most popular CI/CD tools now is Jenkins. This article will focus on configuring a CI/CD pipeline with Jenkins and Helm on Kubernetes.

Jenkins CI/CD Tool for the Cloud

Jenkins is a quite easy to configure, modify and extend. It deploys code instantly and generates test reports. Jenkins can be configured according to your requirements for CI/CD.

CI/CD Steps

The CI/CD process with Jenkins generally

  • checks out code,
  • runs unit tests,
  • Dockerizes an application,
  • pushes the Dockerized application to the Docker Registry, and
  • uses Ansible Playbooks to deploy the Dockerized application on K8s.

To see how it works, let’s start with the installation for Jenkins. We will use a CentOS 7 machine with Docker and Kubernetes already installed.

Installing Jenkins

Step one: update your CentOS 7 system.

sudo yum install epel-release nodejs
sudo yum update

Step two: install Java.

sudo yum install java-1.8.0-openjdk.x86_64
sudo cp /etc/profile /etc/profile_backup
echo 'export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk' | sudo tee -a /etc/profile 
echo 'export JRE_HOME=/usr/lib/jvm/jre' | sudo tee -a /etc/profile source /etc/profile

Step three: install Jenkins

sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
sudo yum install -y jenkins

Step four: start Jenkins and check if it is running.

sudo systemctl start jenkins
sudo systemctl status jenkins

Step five: set up Jenkins.

To start setting up Jenkins, we need to visit its web dashboard running on port 8080. Open your browser and see your public IP address or your domain name followed by the port number.

http://YOUR_IP_OR_DOMAIN:8080

You will see a page like the one below.

To obtain the initial admin password, run:

sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Paste the password in the “Administrator password” field and hit continue. If you are new to Jenkins, we recommend you select “Install suggested plugins”. Now you can see that Jenkins is installing some plugins.

After that, you will be redirected to a page where you have to create your first admin user.

Preparing the Jenkins Server

Jenkins offers a simple way to set up a CI/CD environment for almost any combination of languages and source code repositories. Let’s configure Jenkins Server, which involves Docker, Ansible, Helm, and Docker plugins.

Configuring Docker

Docker is hotter than hot because it makes it possible to get far more apps running on the same old servers and it also makes it very easy to package and ship programs. Let’s create jenkins user for the docker group.

$ sudo groupadd docker
$ sudo usermod -aG docker jenkins
$ chmod root:docker /var/run/docker.sock

Go to /etc/passwd.

Find

jenkins:x:996:993:Jenkins Automation Server:/var/lib/jenkins:/bin/false

and change it to

jenkins:x:996:993:Jenkins Automation Server:/var/lib/jenkins:/bin/bash.

Add the Ansible host to /etc/ansible/hosts.

[localhost]
127.0.0.1

Make sure Jenkins can access the Kubernetes cluster using kubectl.

mv .kube/config to /var/lib/jenkins

Finally, add jenkins to sudo users.

$ visudo -f /etc/sudoers
jenkins ALL=NOPASSWD: ALL

Ansible is an open source automation platform. It is very, very simple to setup and yet powerful. Ansible can help you with configuration management, application deployment, and task automation. It can also do IT orchestration, where you can run tasks in sequence and create a chain of events to run on several different servers or devices.

$ sudo yum install python-pip
$ sudo pip install ansible

Installing and Configuring the Helm Package Manager

Helm helps you manage Kubernetes applications. With Helm Charts you can define, install, and upgrade even the most complex Kubernetes applications.

$ wget https://storage.googleapis.com/kubernetes-helm//var/lib/jenkins/ansible/demo-deploy/deploy.yml-v2.8.1-linux-amd64.tar.gz
$ tar -xzvf helm-v2.8.1-linux-amd64.tar.gz
$ sudo mv linux-amd64/helm /usr/local/bin
$ sudo -i -u jenkins
$ mkdir .kube ; $ touch .kube/config

Copy the contents of /etc/kubernetes/admin.conf to ~/.kube/config. In doing this, you can access the Kubernetes cluster under user jenkins, if necessary. Now let’s initialise Helm.

$ helm init --upgrade

Installing the Jenkins Docker Plugin

The Docker plugin allows us to use Docker to dynamically provision build agents, run a single build, and then push an image to the registry. Navigate to http://your-ip:8080/pluginManager/available and search for the plug-in “CloudBees Docker Build and Publish”. Click “Download Now” and check the box to restart.

Creating the Jenkins Pipeline

Go to Jenkins and select “New Item” on the left side, enter the name “POC”, select Pipeline then click OK.

Generating Pipeline Syntax for Git and the Docker Registry

The Pipeline Syntax section (/job/PIPELINE/pipeline-syntax/) will help you generate the Pipeline Script code which can be used to define various steps. Pick a step you are interested in, configure it, click Generate Pipeline Script, and you will see a Pipeline Script statement that would call a step with that configuration.

Select git and provide the repository URL and user-name/password. If the repository is private, it will generate the syntax for you. If you use Docker Hub, select withdocker-registry which we installed before and provide the credentials of the registry (https://index.docker.io/v1/ for Docker Hub) . Click “Generate Pipeline Script” and you will get a script like that that you will use as credentials.

withDockerRegistry([credentialsId: '85f99fe6-cff4–9064-a85s-a77de72ad87h', url: 'https://index.docker.io/v1/'])

Cloning a Helm chart

Let’s clone a chart for our sample project.

$ sudo su - jenkins 
$ mkdir ansible
$ git clone https:URL
$ cp -r CI/CD-K8s/ansible/Demo ansible/

This will clone a sample project with a hello-world type of application. The Helm charts for our project are located at ansible/Demo/templates. You can replace the YAML with your own files for deployment and services. Here’s the deployment.yml we will use:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: welcome to kubernetes
spec:
  replicas: 3
  selector:
    matchLabels:
      app:welcome to kubernetes
  template:
    metadata:
      labels:
        app: welcome to kubernetes
    spec:
      containers:
      - name: welcome to kubernetes
        image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
        ports:
        - containerPort: 5000
        imagePullPolicy: Always

And this is the service.yml:

apiVersion: v1
kind: Service
metadata:
  name: welcome to kubernetes
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 8080
  selector:
    app: welcome to kubernetes

Configure Ansible to Deploy the Helm Chart

Let’s create an Ansible playbook to call the helm chart.

$ cp -r CI/CD-K8s/ansible/Demo-deploy /var/lib/jenkins/

The playbook we will use in this article looks like this:

- hosts: localhost
  vars:
    ImageName: ""
    Namespace: ""
    imageTag: ""
  #remote_user: ansible
  #become: true
  gather_facts: no
  connection: local
  tasks:
    - name: Create Namespace {{ Namespace }}
      command: "kubectl create namespace {{ Namespace }}"
      ignore_errors: yes
    - name: Deploy Demo
      command: "/usr/local/bin/helm install --name=Demo-{{ Namespace }}  --namespace={{ Namespace }} ../Demo --set image.repository={{ ImageName }} --set image.tag={{ imageTag }} --set namespace={{ Namespace }}"
      delegate_to: localhost
      ignore_errors: yes
    - name: Update K8s App
      command: "/usr/local/bin/helm upgrade --wait --recreate-pods --namespace={{ Namespace }} --set image.repository={{ ImageName }} --set image.tag={{ imageTag }} --set namespace={{ Namespace }} Demo-{{ Namespace }} ../K8s"
      delegate_to: localhost
      ignore_errors: yes

CI/CD with Jenkins Pipeline

The Jenkinsfile that we will use for the pipeline looks like this:

node{
  def Namespace = "default"
  def ImageName = "Demo/K8s"
  def Creds = "3dhf6hhd-a300-78ee-kjg5-7j3dfhjg7764"
  try{
  stage('Checkout'){
      git 'CI/CD-K8s.git
      sh "git rev-parse --short HEAD > .git/commit-id"
      imageTag= readFile('.git/commit-id').trim()
}
stage('RUN Unit Tests'){
      sh "npm install"
      sh "npm test"
  }
  stage('Docker Build, Push'){
    withDockerRegistry([credentialsId: "${Creds}", url: 'https://index.docker.io/v1/']) {
      sh "docker build -t ${ImageName}:${imageTag} ."
      sh "docker push ${ImageName}"
        }
}
    stage('Deploy on K8s'){
sh "ansible-playbook /var/lib/jenkins/ansible/Demo-deploy/deploy.yml  --user=jenkins --extra-vars ImageName=${ImageName} --extra-vars imageTag=${imageTag} --extra-vars Namespace=${Namespace}"
    }
     } catch (err) {
      currentBuild.result = 'FAILURE'
    }
}

Let’s look deeper into the Jenkinsfile. Step one: defining variables.

def Namespace = "default"  
//default namespace on k8s
def ImageName = "Demo/K8s" 
// image name which will be pushed to docker registry
    def Creds = "2dfd9d0d-a300-49ee-vhaf-0a5fgcaa5279" 
// Creds of docker registry

Step two: pull/clone updates from our version control.

git 'URL'
      sh "git rev-parse --short HEAD > .git/commit-id"
      imageTag= readFile('.git/commit-id').trim()

Step three: run unit tests.

stage('RUN Unit Tests'){
         sh "npm install"
         sh "npm test"
    }

Step four: Docker build and push to the Docker Registry.

stage('Docker Build, Push'){
withDockerRegistry([credentialsId: "${Creds}", url: 'https://index.docker.io/v1/']) {
         sh "docker build -t ${ImageName}:${imageTag} ."
         sh "docker push ${ImageName}"
           }
      }

Step five: call the Ansible playbook to deploy on K8s.

stage('Deploy on K8s'){
sh "ansible-playbook ./ansible/demo-deploy/deploy.yml  --user=jenkins --extra-vars ImageName=${ImageName} --extra-vars imageTag=${imageTag} --extra-vars Namespace=${Namespace}"
 }

Access the application running in Kubernetes.

$ kubectl get svc // to get the IP/Port of the application

Now cURL http://public-node-ip:node-port.

Update the Code

Now let’s see if we got it right. Let’s change our YAML files a little. In CI/CD-K8s/app/routes/root.js change “welcome to K8s” to “update k8s” in line 3.

module.exports = function(req, res, next) {
  res.contentType = "json";
  res.send({ message: "welcome to  K8s" });
  next();
};

In CI/CD-K8s/app/test/root.test.js change “Thanks for learning K8s to “update k8s” in line 27.

const chai = require("chai");
const sinon = require("sinon");
var rootResponder = require("../routes/root");
// const expect = chai.expect;
// const assert = chai.assert;
chai.should();
describe("Root Directory Test", function() {
  describe("Should Behave properly on GETing /", function() {
    const nextSpy = sinon.spy();
    const resSpy = { send: sinon.spy() };
    beforeEach(function() {
      nextSpy.resetHistory();
      resSpy.send.resetHistory();
    });
    it("should call next", function() {
      rootResponder({}, resSpy, nextSpy);
      nextSpy.calledOnce.should.be.true;
    });
it("should call send on resp", function() {
      rootResponder({}, resSpy, nextSpy);
      resSpy.send.calledOnce.should.be.true;
    });
    it("should call send on resp with Hello World as a message", function() {
      rootResponder({}, resSpy, nextSpy);
      resSpy.send.calledWith({ message: "Thanks for learning K8s" }).should.be.true;
    });
    it("should have json as the content type of the respones", function() {
      rootResponder({}, resSpy, nextSpy);
      resSpy.contentType.should.exist;
      resSpy.contentType.should.equal("json");
    });
  });
});

Run the pipeline again and cURL http://public-node-ip:node-port. There we have it – we’ve demonstrated a simple CI/CD workflow with Jenkins, Docker, Ansible, Helm and Kubernetes!

By the way, check out our best AWS deal: http://avmconsulting.net/well-architected-review