August 1, 2019
CI/CD with Kubernetes
Kubernetes is compatible with the majority of Continuous Integration / Continuous Delivery (CI/CD) tools which allows developers to run tests, deploy builds in Kubernetes and update applications with no downtime. One of the most popular CI/CD tools now is Jenkins. This article will focus on configuring a CI/CD pipeline with Jenkins and Helm on Kubernetes.
Jenkins CI/CD Tool for the Cloud
Jenkins is a quite easy to configure, modify and extend. It deploys code instantly and generates test reports. Jenkins can be configured according to your requirements for CI/CD.
CI/CD Steps
The CI/CD process with Jenkins generally
- checks out code,
- runs unit tests,
- Dockerizes an application,
- pushes the Dockerized application to the Docker Registry, and
- uses Ansible Playbooks to deploy the Dockerized application on K8s.
To see how it works, let’s start with the installation for Jenkins. We will use a CentOS 7 machine with Docker and Kubernetes already installed.
Installing Jenkins
Step one: update your CentOS 7 system.
sudo yum install epel-release nodejs
sudo yum update
Step two: install Java.
sudo yum install java-1.8.0-openjdk.x86_64
sudo cp /etc/profile /etc/profile_backup
echo 'export JAVA_HOME=/usr/lib/jvm/jre-1.8.0-openjdk' | sudo tee -a /etc/profile
echo 'export JRE_HOME=/usr/lib/jvm/jre' | sudo tee -a /etc/profile source /etc/profile
Step three: install Jenkins
sudo rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key
wget -O /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
sudo yum install -y jenkins
Step four: start Jenkins and check if it is running.
sudo systemctl start jenkins
sudo systemctl status jenkins
Step five: set up Jenkins.
To start setting up Jenkins, we need to visit its web dashboard running on port 8080. Open your browser and see your public IP address or your domain name followed by the port number.
You will see a page like the one below.
To obtain the initial admin password, run:
sudo cat /var/lib/jenkins/secrets/initialAdminPassword
Paste the password in the “Administrator password” field and hit continue. If you are new to Jenkins, we recommend you select “Install suggested plugins”. Now you can see that Jenkins is installing some plugins.
After that, you will be redirected to a page where you have to create your first admin user.
Preparing the Jenkins Server
Jenkins offers a simple way to set up a CI/CD environment for almost any combination of languages and source code repositories. Let’s configure Jenkins Server, which involves Docker, Ansible, Helm, and Docker plugins.
Configuring Docker
Docker is hotter than hot because it makes it possible to get far more apps running on the same old servers and it also makes it very easy to package and ship programs. Let’s create jenkins user for the docker group.
$ sudo groupadd docker
$ sudo usermod -aG docker jenkins
$ chmod root:docker /var/run/docker.sock
Go to /etc/passwd.
Find
jenkins:x:996:993:Jenkins Automation Server:/var/lib/jenkins:/bin/false
and change it to
jenkins:x:996:993:Jenkins Automation Server:/var/lib/jenkins:/bin/bash.
Add the Ansible host to /etc/ansible/hosts.
[localhost]
127.0.0.1
Make sure Jenkins can access the Kubernetes cluster using kubectl.
mv .kube/config to /var/lib/jenkins
Finally, add jenkins to sudo users.
$ visudo -f /etc/sudoers
jenkins ALL=NOPASSWD: ALL
Ansible is an open source automation platform. It is very, very simple to setup and yet powerful. Ansible can help you with configuration management, application deployment, and task automation. It can also do IT orchestration, where you can run tasks in sequence and create a chain of events to run on several different servers or devices.
$ sudo yum install python-pip
$ sudo pip install ansible
Installing and Configuring the Helm Package Manager
Helm helps you manage Kubernetes applications. With Helm Charts you can define, install, and upgrade even the most complex Kubernetes applications.
$ wget https://storage.googleapis.com/kubernetes-helm//var/lib/jenkins/ansible/demo-deploy/deploy.yml-v2.8.1-linux-amd64.tar.gz
$ tar -xzvf helm-v2.8.1-linux-amd64.tar.gz
$ sudo mv linux-amd64/helm /usr/local/bin
$ sudo -i -u jenkins
$ mkdir .kube ; $ touch .kube/config
Copy the contents of /etc/kubernetes/admin.conf to ~/.kube/config. In doing this, you can access the Kubernetes cluster under user jenkins, if necessary. Now let’s initialise Helm.
$ helm init --upgrade
Installing the Jenkins Docker Plugin
The Docker plugin allows us to use Docker to dynamically provision build agents, run a single build, and then push an image to the registry. Navigate to http://your-ip:8080/pluginManager/available and search for the plug-in “CloudBees Docker Build and Publish”. Click “Download Now” and check the box to restart.
Creating the Jenkins Pipeline
Go to Jenkins and select “New Item” on the left side, enter the name “POC”, select Pipeline then click OK.
Generating Pipeline Syntax for Git and the Docker Registry
The Pipeline Syntax section (/job/PIPELINE/pipeline-syntax/) will help you generate the Pipeline Script code which can be used to define various steps. Pick a step you are interested in, configure it, click Generate Pipeline Script, and you will see a Pipeline Script statement that would call a step with that configuration.
Select git and provide the repository URL and user-name/password. If the repository is private, it will generate the syntax for you. If you use Docker Hub, select withdocker-registry which we installed before and provide the credentials of the registry (https://index.docker.io/v1/ for Docker Hub) . Click “Generate Pipeline Script” and you will get a script like that that you will use as credentials.
withDockerRegistry([credentialsId: '85f99fe6-cff4–9064-a85s-a77de72ad87h', url: 'https://index.docker.io/v1/'])
Cloning a Helm chart
Let’s clone a chart for our sample project.
$ sudo su - jenkins
$ mkdir ansible
$ git clone https:URL
$ cp -r CI/CD-K8s/ansible/Demo ansible/
This will clone a sample project with a hello-world type of application. The Helm charts for our project are located at ansible/Demo/templates. You can replace the YAML with your own files for deployment and services. Here’s the deployment.yml we will use:
apiVersion: apps/v1
kind: Deployment
metadata:
name: welcome to kubernetes
spec:
replicas: 3
selector:
matchLabels:
app:welcome to kubernetes
template:
metadata:
labels:
app: welcome to kubernetes
spec:
containers:
- name: welcome to kubernetes
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
ports:
- containerPort: 5000
imagePullPolicy: Always
And this is the service.yml:
apiVersion: v1
kind: Service
metadata:
name: welcome to kubernetes
spec:
type: NodePort
ports:
- port: 80
targetPort: 8080
selector:
app: welcome to kubernetes
Configure Ansible to Deploy the Helm Chart
Let’s create an Ansible playbook to call the helm chart.
$ cp -r CI/CD-K8s/ansible/Demo-deploy /var/lib/jenkins/
The playbook we will use in this article looks like this:
- hosts: localhost
vars:
ImageName: ""
Namespace: ""
imageTag: ""
#remote_user: ansible
#become: true
gather_facts: no
connection: local
tasks:
- name: Create Namespace {{ Namespace }}
command: "kubectl create namespace {{ Namespace }}"
ignore_errors: yes
- name: Deploy Demo
command: "/usr/local/bin/helm install --name=Demo-{{ Namespace }} --namespace={{ Namespace }} ../Demo --set image.repository={{ ImageName }} --set image.tag={{ imageTag }} --set namespace={{ Namespace }}"
delegate_to: localhost
ignore_errors: yes
- name: Update K8s App
command: "/usr/local/bin/helm upgrade --wait --recreate-pods --namespace={{ Namespace }} --set image.repository={{ ImageName }} --set image.tag={{ imageTag }} --set namespace={{ Namespace }} Demo-{{ Namespace }} ../K8s"
delegate_to: localhost
ignore_errors: yes
CI/CD with Jenkins Pipeline
The Jenkinsfile that we will use for the pipeline looks like this:
node{
def Namespace = "default"
def ImageName = "Demo/K8s"
def Creds = "3dhf6hhd-a300-78ee-kjg5-7j3dfhjg7764"
try{
stage('Checkout'){
git 'CI/CD-K8s.git
sh "git rev-parse --short HEAD > .git/commit-id"
imageTag= readFile('.git/commit-id').trim()
}
stage('RUN Unit Tests'){
sh "npm install"
sh "npm test"
}
stage('Docker Build, Push'){
withDockerRegistry([credentialsId: "${Creds}", url: 'https://index.docker.io/v1/']) {
sh "docker build -t ${ImageName}:${imageTag} ."
sh "docker push ${ImageName}"
}
}
stage('Deploy on K8s'){
sh "ansible-playbook /var/lib/jenkins/ansible/Demo-deploy/deploy.yml --user=jenkins --extra-vars ImageName=${ImageName} --extra-vars imageTag=${imageTag} --extra-vars Namespace=${Namespace}"
}
} catch (err) {
currentBuild.result = 'FAILURE'
}
}
Let’s look deeper into the Jenkinsfile. Step one: defining variables.
def Namespace = "default"
//default namespace on k8s
def ImageName = "Demo/K8s"
// image name which will be pushed to docker registry
def Creds = "2dfd9d0d-a300-49ee-vhaf-0a5fgcaa5279"
// Creds of docker registry
Step two: pull/clone updates from our version control.
git 'URL'
sh "git rev-parse --short HEAD > .git/commit-id"
imageTag= readFile('.git/commit-id').trim()
Step three: run unit tests.
stage('RUN Unit Tests'){
sh "npm install"
sh "npm test"
}
Step four: Docker build and push to the Docker Registry.
stage('Docker Build, Push'){
withDockerRegistry([credentialsId: "${Creds}", url: 'https://index.docker.io/v1/']) {
sh "docker build -t ${ImageName}:${imageTag} ."
sh "docker push ${ImageName}"
}
}
Step five: call the Ansible playbook to deploy on K8s.
stage('Deploy on K8s'){
sh "ansible-playbook ./ansible/demo-deploy/deploy.yml --user=jenkins --extra-vars ImageName=${ImageName} --extra-vars imageTag=${imageTag} --extra-vars Namespace=${Namespace}"
}
Access the application running in Kubernetes.
$ kubectl get svc // to get the IP/Port of the application
Now cURL http://public-node-ip:node-port.
Update the Code
Now let’s see if we got it right. Let’s change our YAML files a little. In CI/CD-K8s/app/routes/root.js change “welcome to K8s” to “update k8s” in line 3.
module.exports = function(req, res, next) {
res.contentType = "json";
res.send({ message: "welcome to K8s" });
next();
};
In CI/CD-K8s/app/test/root.test.js change “Thanks for learning K8s“ to “update k8s” in line 27.
const chai = require("chai");
const sinon = require("sinon");
var rootResponder = require("../routes/root");
// const expect = chai.expect;
// const assert = chai.assert;
chai.should();
describe("Root Directory Test", function() {
describe("Should Behave properly on GETing /", function() {
const nextSpy = sinon.spy();
const resSpy = { send: sinon.spy() };
beforeEach(function() {
nextSpy.resetHistory();
resSpy.send.resetHistory();
});
it("should call next", function() {
rootResponder({}, resSpy, nextSpy);
nextSpy.calledOnce.should.be.true;
});
it("should call send on resp", function() {
rootResponder({}, resSpy, nextSpy);
resSpy.send.calledOnce.should.be.true;
});
it("should call send on resp with Hello World as a message", function() {
rootResponder({}, resSpy, nextSpy);
resSpy.send.calledWith({ message: "Thanks for learning K8s" }).should.be.true;
});
it("should have json as the content type of the respones", function() {
rootResponder({}, resSpy, nextSpy);
resSpy.contentType.should.exist;
resSpy.contentType.should.equal("json");
});
});
});
Run the pipeline again and cURL http://public-node-ip:node-port. There we have it – we’ve demonstrated a simple CI/CD workflow with Jenkins, Docker, Ansible, Helm and Kubernetes!
By the way, check out our best AWS deal: https://www.avmconsulting.net/well-architected-review
