In this post, I will demonstrate how we can remove or drain a node from the Kubernetes cluster. Before proceeding I would recommend you to go through the first part of this blog, where you can understand more about Kubernetes architecture, components, and installation.
Kubernetes tutorial (CKA certification) – Part1
My demo cluster has one Control Plain node and two worker nodes, more details are given below.
| Machine name | OS | Size | IP address |
| control-node-1 | Ubuntu 18.04 LTS | 2cpu, 4g memory | 50.1.0.5 |
| worker-node-1 | Ubuntu 18.04 LTS | 2cpu, 4g memory | 50.1.0.7 |
| worker-node-2 | Ubuntu 18.04 LTS | 2cpu, 4g memory | 50.1.0.8 |
Sample container deployment
Before we removing a node from the cluster we will deploy a container (Pods) to our cluster. This will help us to understand what will happen to the Pods when we remove a node from the cluster.
Current cluster node status.
techies@control-node-1:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION control-node-1 Ready control-plane,master 2d v1.21.0 worker-node-1 Ready <none> 2d v1.21.0 worker-node-2 Ready <none> 2d v1.21.0
To create an Nginx pod, copy and paste the below code to a YAML file. My file name is deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
labels:
app: nginx
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
In this deployment, we are using 2 replicas, which means there will be 2 containers for Nginx. To deploy the pod execute the below command.
techies@control-node-1:~$ kubectl apply -f deployment.yaml deployment.apps/nginx created techies@control-node-1:~$
To get details about running nodes execute the below command.
techies@control-node-1:~$ kubectl get pods NAME READY STATUS RESTARTS AGE nginx-66b6c48dd5-d8864 1/1 Running 0 57s nginx-66b6c48dd5-lqt9t 1/1 Running 0 57s
Here the command will not show the node name in which pods are running. To get node details, execute the below command.
kubectl get pods -o wide
Here one Nginx pod is running in worker-node-1 and the other is running on worker-node-2.
For maintenance, we will remove worker-node-2 from the cluster, and let’s see what will happen to the pods running on worker-node-2.
Drain a node from the cluster
The command to drain a node from the cluster is
kubectl drain worker-node-2
Execute the command in the cluster and see what will happen!.
The draining process is failed because the node contains DaemonSet-manage Pods. These are Kubernetes system Pods which we are installed as part of Calico plugin installation and kubeadm init process. To get around this error we have to use the –ignore-daemonstes flag as mentioned in the error message.
Execute the command again with flag.
kubectl drain worker-node-2 --ignore-daemonsets
techies@control-node-1:~$ kubectl drain worker-node-2 --ignore-daemonsets node/worker-node-2 already cordoned WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-vw84b, kube-system/kube-proxy-9nrsf evicting pod default/nginx-66b6c48dd5-d8864 pod/nginx-66b6c48dd5-d8864 evicted node/worker-node-2 evicted techies@control-node-1:~$
Execute kubectl get pods command to know what happens to the pods which were running on worker-node-2
Both pods are running on worker-node-1. As we mentioned the number of replicas is 2, Kubernetes schedule to run the pods on available nodes.
now, we have to check what happened to the removed node. To get the status of nodes, execute the below command.
techies@control-node-1:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION control-node-1 Ready control-plane,master 2d1h v1.21.0 worker-node-1 Ready <none> 2d1h v1.21.0 worker-node-2 Ready,SchedulingDisabled <none> 2d1h v1.21.0
Here the node status is “SchedulingDisabled” which means Kubernetes is no longer schedules any pods on this node.
It’s time to perform the node maintenance. Once it is done we can add back the node to the cluster with the help of the below command.
techies@control-node-1:~$ kubectl uncordon worker-node-2 node/worker-node-2 uncordoned techies@control-node-1:~$
List the node status again.
techies@control-node-1:~$ kubectl get nodes NAME STATUS ROLES AGE VERSION control-node-1 Ready control-plane,master 2d1h v1.21.0 worker-node-1 Ready <none> 2d1h v1.21.0 worker-node-2 Ready <none> 2d1h v1.21.0
You can see the node is now in the “Ready” state. Next, execute the list pods command to understand did pods are scheduled in the worker-node-2.
Here you can see the pods running in the same worker-node-1 because once the node is added to the cluster, the Kubernetes will not automatically schedule the pods, for this we have to increase the “replicaset” or we have to delete and re-deploy. I have increased the replicaset to 3 and applied the changes.
techies@control-node-1:~$ kubectl apply -f deployment.yaml deployment.apps/nginx configured techies@control-node-1:~$ kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-66b6c48dd5-jrp74 1/1 Running 0 20m 192.168.212.66 worker-node-1 <none> <none> nginx-66b6c48dd5-lqt9t 1/1 Running 0 45m 192.168.212.65 worker-node-1 <none> <none> nginx-66b6c48dd5-xv2v2 1/1 Running 0 6s 192.168.19.130 worker-node-2 <none> <none> techies@control-node-1:~$
Yes, the 3rd replica is now running on worker-node-2.
