How to build a Kubernetes mutating webhook

This blog post will demonstrate what a Kubernetes mutating webhook is and how to create a mutating webhook to inject some environment variable or configmap to all pods based on some condition.

Mutating webhook

So, what is a mutating webhook? According to the Kubernetes official documentation Admission, webhooks are HTTP callbacks that receive admission requests and do something with them. You can define two types of admission webhooks, validating admission webhook and mutating admission webhook. Mutating admission webhooks are invoked first and can modify objects sent to the API server to enforce custom defaults. Please refer to the below diagram to understand the workflow.

In this example, we will create an environment variable “username = demo-user” to all the pods (which contains the label “hello = test”) automatically through a Mutating admission webhook, so that every time we send a request to the API to create a Pod, the Pod spec is mutated before being saved to storage. Then when the Kubelet creates our Pod on worker nodes, it should have the environment variable included automatically.

Code repository

The complete code required to run this sample project is uploaded to my Github repository The repository also contains all the Kubernetes manifest files for deployment. I recommend you to pull this code to your machine as I will refer file names from this repository.


A working Kubernetes cluster is required for this tutorial. You can set up a Kubernetes cluster on VirtualBox with the help of vagrant, click here for more details.

Create k8s resources


The first Kubernetes resource is configmap (filename = configmap.yaml), in this configmap we will create the environment variable “username = demo-user”. I have created a namespace called ns-webhook to deploy all our required resources.

kubectl create ns ns-webhook
apiVersion: v1
kind: ConfigMap
  name: hello-configmap
  namespace: ns-webhook
  username: demo-user
kubectl apply -f configmap.yaml

Webhook code

Next, we need a webhook that we will create using a simple Go API server. To create the webhook docker image, go to the webhook folder from the Git repository and execute the below command.

vagrant@master:~/webhook-repo/webhook$ sudo docker build .

If everything goes fine you will get a final docker image and the below message.

Step 12/13 : COPY --from=builder /app/main ./main
 ---> cacddf9d08d2
Step 13/13 : CMD ["./main"]
 ---> Running in 9ba1feef8baa
Removing intermediate container 9ba1feef8baa
 ---> cf6395ca261d
Successfully built cf6395ca261d

Tag the image and upload it to your docker hub repository, later we will use this image for Kubernetes deployment.

sudo docker tag cf6395ca261d techiescorner/webhook:latest
sudo docker login
sudo docker push techiescorner/webhook:latest

TLS certificates

To secure the communication between the webhook and API server we need to deploy a certificate resource. This “Certificate” (filename = cert-manager.yaml) resource provides the required certificates for our webhook. To generate a certificate first we need to install cert-manager in our kubernetes cluster.

Execute the below command to install Cert-manger.

helm repo add bitnami
helm install   cert-manager bitnami/cert-manager  --namespace cert-manager --set installCRDs=true
kind: Issuer
  name: self-signer
  namespace: ns-webhook
  selfSigned: {}
kind: Certificate
  name: my-webhook-certificate
  namespace: ns-webhook
  secretName: my-webhook-certificate-secret
  #  - my-webhook
  #  - my-webhook.temp-webhook
    - hello-webhook-service.ns-webhook.svc
    name: self-signer

webhook API server deployment

We have already created a webhook API server docker image and uploaded it to the Docker hub repository. Next, we have to deploy it to the Kubernetes cluster as a deployment, so create a deployment manifest file (file name = deployment.yaml). [Refer to the file from my Github repository].

Cluster IP service for the deployment

We have to create a Cluster IP service for our webhook API server deployment. The service will use port 443.

apiVersion: v1
kind: Service
  name: hello-webhook-service
  namespace: ns-webhook
  type: ClusterIP
    app: hello-webhook
  - protocol: TCP
    port: 443
    targetPort: 8000

Next, create a manifest file (file name = webhook.yaml) to register our mutating webhook configuration with the Kubernetes API server.

kind: MutatingWebhookConfiguration
  name: ""
  namespace: ns-webhook
  annotations: ns-webhook/my-webhook-certificate
- name: ""
      hello: "true"
  - apiGroups:   [""]
    apiVersions: ["v1"]
    operations:  ["CREATE"]
    resources:   ["pods"]
    scope:       "Namespaced"
      namespace: "ns-webhook"
      name: "hello-webhook-service"
      path: /mutate
  admissionReviewVersions: ["v1", "v1beta1"]
  sideEffects: None
  timeoutSeconds: 10

Deploy all above manifest files to the Kubernetes cluster and make sure that it is running without any errors.

kubectl apply -f deployment.yaml 
kubectl apply -f service.yaml 

kubectl apply -f cert-manager.yaml created created

kubectl  apply -f webhook.yaml created
$ kubectl get all -n ns-webhook
NAME                                            READY   STATUS    RESTARTS   AGE
pod/hello-webhook-deployment-69c844d9c4-zhpzj   1/1     Running   0          13s

NAME                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/hello-webhook-service   ClusterIP   <none>        443/TCP   13s

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/hello-webhook-deployment   1/1     1            1           13s

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/hello-webhook-deployment-69c844d9c4   1         1         1       13s


We have completed all the Kubernetes deployment and next is testing. To test the mutating webhook deployment I will deploy 2 “busybox” containers or pods, one with the label “hello = test” and the other without the label “hello = test”. The pod with the label “hello=test” return the username and the other should return an error message. Execute below two commands.

kubectl run busybox-1 -n ns-webhook --image=busybox  --restart=Never -l=app=busybox,hello=test -- sleep 3600

kubectl exec busybox-1 -it -n ns-webhook -- sh -c "cat /etc/config/username"
kubectl run busybox-2 -n ns-webhook --image=busybox  --restart=Never -l=app=busybox,hello=false -- sleep 3600
pod/busybox-2 created

 kubectl exec busybox-2 -it -n ns-webhook -- sh -c "cat /etc/config/username"
cat: can't open '/etc/config/username': No such file or directory
command terminated with exit code 1

From the above outputs, we can confirm that the mutating webhook, which we installed is working as expected.

Leave a Reply

Your email address will not be published.