How to Remove or Drain a Node From Kubernetes cluster – Part2

In this post, I will demonstrate how we can remove or drain a node from the Kubernetes cluster. Before proceeding I would recommend you to go through the first part of this blog, where you can understand more about Kubernetes architecture, components, and installation.

Kubernetes tutorial (CKA certification) – Part1

My demo cluster has one Control Plain node and two worker nodes, more details are given below.

Machine name OS Size IP address
control-node-1 Ubuntu 18.04 LTS 2cpu, 4g memory 50.1.0.5
worker-node-1 Ubuntu 18.04 LTS 2cpu, 4g memory 50.1.0.7
worker-node-2 Ubuntu 18.04 LTS 2cpu, 4g memory 50.1.0.8

Sample container deployment

Before we removing a node from the cluster we will deploy a container (Pods) to our cluster. This will help us to understand what will happen to the Pods when we remove a node from the cluster.

Current cluster node status.

techies@control-node-1:~$ kubectl get nodes
NAME             STATUS   ROLES                  AGE   VERSION
control-node-1   Ready    control-plane,master   2d    v1.21.0
worker-node-1    Ready    <none>                 2d    v1.21.0
worker-node-2    Ready    <none>                 2d    v1.21.0

To create an Nginx pod, copy and paste the below code to a YAML file. My file name is deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

In this deployment, we are using 2 replicas, which means there will be 2 containers for Nginx. To deploy the pod execute the below command.

techies@control-node-1:~$ kubectl apply -f deployment.yaml 
deployment.apps/nginx created
techies@control-node-1:~$

To get details about running nodes execute the below command.

techies@control-node-1:~$ kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
nginx-66b6c48dd5-d8864   1/1     Running   0          57s
nginx-66b6c48dd5-lqt9t   1/1     Running   0          57s

Here the command will not show the node name in which pods are running. To get node details, execute the below command.

kubectl get pods -o wide

 

Here one Nginx pod is running in worker-node-1 and the other is running on worker-node-2.

For maintenance, we will remove worker-node-2 from the cluster, and let’s see what will happen to the pods running on worker-node-2.

Drain a node from the cluster

The command to drain a node from the cluster is

kubectl drain worker-node-2

Execute the command in the cluster and see what will happen!.

The draining process is failed because the node contains DaemonSet-manage Pods. These are Kubernetes system Pods which we are installed as part of Calico plugin installation and kubeadm init process. To get around this error we have to use the –ignore-daemonstes flag as mentioned in the error message.

Execute the command again with flag.

 kubectl drain worker-node-2 --ignore-daemonsets
techies@control-node-1:~$ kubectl drain worker-node-2 --ignore-daemonsets
node/worker-node-2 already cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/calico-node-vw84b, kube-system/kube-proxy-9nrsf
evicting pod default/nginx-66b6c48dd5-d8864
pod/nginx-66b6c48dd5-d8864 evicted
node/worker-node-2 evicted
techies@control-node-1:~$

Execute kubectl get pods command to know what happens to the pods which were running on worker-node-2

Both pods are running on worker-node-1. As we mentioned the number of replicas is 2, Kubernetes schedule to run the pods on available nodes.

now, we have to check what happened to the removed node. To get the status of nodes, execute the below command.

techies@control-node-1:~$ kubectl get nodes
NAME             STATUS                     ROLES                  AGE    VERSION
control-node-1   Ready                      control-plane,master   2d1h   v1.21.0
worker-node-1    Ready                      <none>                 2d1h   v1.21.0
worker-node-2    Ready,SchedulingDisabled   <none>                 2d1h   v1.21.0

Here the node status is “SchedulingDisabled” which means Kubernetes is no longer schedules any pods on this node.

It’s time to perform the node maintenance. Once it is done we can add back the node to the cluster with the help of the below command.

techies@control-node-1:~$ kubectl uncordon worker-node-2
node/worker-node-2 uncordoned
techies@control-node-1:~$

List the node status again.

techies@control-node-1:~$ kubectl get nodes
NAME             STATUS   ROLES                  AGE    VERSION
control-node-1   Ready    control-plane,master   2d1h   v1.21.0
worker-node-1    Ready    <none>                 2d1h   v1.21.0
worker-node-2    Ready    <none>                 2d1h   v1.21.0

You can see the node is now in the “Ready” state. Next, execute the list pods command to understand did pods are scheduled in the worker-node-2.

Here you can see the pods running in the same worker-node-1 because once the node is added to the cluster, the Kubernetes will not automatically schedule the pods, for this we have to increase the “replicaset” or we have to delete and re-deploy. I have increased the replicaset to 3 and applied the changes.

techies@control-node-1:~$ kubectl apply -f deployment.yaml 
deployment.apps/nginx configured

techies@control-node-1:~$ kubectl get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP               NODE            NOMINATED NODE   READINESS GATES
nginx-66b6c48dd5-jrp74   1/1     Running   0          20m   192.168.212.66   worker-node-1   <none>           <none>
nginx-66b6c48dd5-lqt9t   1/1     Running   0          45m   192.168.212.65   worker-node-1   <none>           <none>
nginx-66b6c48dd5-xv2v2   1/1     Running   0          6s    192.168.19.130   worker-node-2   <none>           <none>
techies@control-node-1:~$ 

Yes, the 3rd replica is now running on worker-node-2.

Leave a Reply

Your email address will not be published.