Apply Mutual TLS over kubernetes/nginx ingress controller

In this article, I will demonstrate how to apply mutual TLS connection between a client and a kubernetes endpoint exposed through nginx…

Muhammad Badawy

~5 min read · June 29, 2022 (Updated: September 25, 2023) · Free: Yes

In this article, I will demonstrate how to apply mutual TLS connection between a client and a kubernetes endpoint exposed through nginx ingress controller.

Mutual TLS concept lies under the umbrella of Zero Trust Policy where strict identity verification is required for any client/person/device trying to access any resources in a private network.

Now let's jump to the demo which will consist of 3 main sections:

Deploying nginx ingress controller
Enabling TLS through self signed certificates
Enabling MTLS through self signed certificates

Prerequisites: - To have a kubernetes cluster up and running and kubectl and openssl installed. For the sake of the demo, I will use minikube to quickly initialize a cluster.

Steps-Deploying nginx ingress controller:

1- First let's apply nginx ingress controller resources to kubernetes. Usually I prefer to have a look on the nginx official website to check the installation steps according to your kubernetes environment. In this demo, I'm using minikube as my cluster environment so nginx ingress controller can be enabled as below:

$minikube addons enable ingress

A pre-flight check to make sure all is up and running:

$kubectl get pods -n ingress-nginx

2- After making sure that all pods in ingress-nginx namespace are up and running, Let's deploy http application as below

$kubectl create deployment test --image=httpd --port=80

3- Then expose this deployment locally through a service of type ClusterIP which is the default in case it is not mentioned:

$kubectl expose deployment test

4- Then expose this service to be accessible from outside the cluster through ingress resource which will use nginx as ingress class and will be hosted through test.localdev.me which is our localhost machine.

$kubectl create ingress test-localhost --class=nginx --rule="test.localdev.me/*=test:80"

Note: The entire wildcard domain entries of *.localdev.me points to 127.0.0.1 and this can be tested by doing "nslookup test.localdev.me"

5- Last step in this section, is to test the connection to our endpoint from outside the cluster, which will be made through port-forward from our local machine on port 8080 to the ingress controller service on port 80:

$kubectl port-forward -n ingress-nginx service/ingress-nginx-controller 8080:80

From another terminal, let's try to curl, and the response should be as below:

$curl http://test.localdev.me:8080/
<html><body><h1>It works!</h1></body></html>

Now we need to be able to call this endpoint with https protocol:

Steps-Enable TLS:

1- Generate self-signed server certificate for domain "test.localdev.me":

$openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout server.key -out server.crt -subj "/CN=test.localdev.me/O=test.localdev.me"

2- Apply the cert to kubernetes through secret resource:

$kubectl create secret tls self-tls --key server.key --cert server.crt

3- Modify the ingress controller to add tls ection as below and refer to the created secret:

$kubectl edit ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-localhost
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: test.localdev.me
    http:
      paths:
      - backend:
          service:
            name: test
            port:
              number: 80
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - test.localdev.me
    secretName: self-tls

4- Now let's try to port-forward on port 443 then hit the endpoint with https connection:

$sudo kubectl port-forward -n ingress-nginx service/ingress-nginx-controller 443:443

Note that sudo is required here to allow incoming traffic on port 443, Also -k is used to skip self-signed certificate verification and -v to see some logs.

$curl -k -v https://test.localdev.me/
<html><body><h1>It works!</h1></body></html>

You can see through the logs that only one certificate has been verified which is the server one:

* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):

Now we have a successful TLS connection, Let's enable the mutual TLS ;)

Steps: Enable Mutual TLS:

1- First let's create CA "Certificate Authority" cert to be used as our verification gate to the client requests:

$openssl req -x509 -sha256 -newkey rsa:4096 -keyout ca.key -out ca.crt -days 356 -nodes -subj '/CN=My Cert Authority'

2- Then apply the CA as secret to kubernetes cluster

$kubectl create secret generic ca-secret --from-file=ca.crt=ca.crt

3- Next we need to generate a client Cert Signing Request and client key:

$openssl req -new -newkey rsa:4096 -keyout client.key -out client.csr -nodes -subj '/CN=My Client'

4- Now we need to sign this CSR "Certificate Signing Request" with the CA to generate the client certificate which can be used to call the endpoint:

$openssl x509 -req -sha256 -days 365 -in client.csr -CA ca.crt -CAkey ca.key -set_serial 02 -out client.crt

At this point, we have a client key "client.key" and client certificate "client.crt"

5- Finally edit the ingress ingress resource to add client verification annotations, one of them is to refer to the CA secret "default/ca-secret" which is our verification authority:

metadata:
  annotations:
    nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
    nginx.ingress.kubernetes.io/auth-tls-secret: default/ca-secret
    nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
    nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1"

Testing our work:

Now let's try to curl without client cert, you will get an error mentioning that no SSL certificate provided, which means that annotations in the previous step are working correctly.

$curl -k https://test.localdev.me/
<html>
<head><title>400 No required SSL certificate was sent</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<center>No required SSL certificate was sent</center>
<hr><center>nginx</center>
</body>
</html>

Try the same call with client key and cert and use '-v' to get more logs:

curl -k -v https://test.localdev.me/ --key client.key --cert client.crt

As you can see in the output of the command that certificate verification has been made twice, one for server certificate and another one for the client certificate and the call works as expected.

* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Request CERT (13):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Certificate (11):
* (304) (OUT), TLS handshake, CERT verify (15):
* (304) (OUT), TLS handshake, Finished (20):
...
<html><body><h1>It works!</h1></body></html>

Complete ingress file for reference:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
    nginx.ingress.kubernetes.io/auth-tls-secret: default/ca-secret
    nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
    nginx.ingress.kubernetes.io/auth-tls-verify-depth: "1"
  name: test-localhost
  namespace: default
spec:
  ingressClassName: nginx
  rules:
  - host: test.localdev.me
    http:
      paths:
      - backend:
          service:
            name: test
            port:
              number: 80
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - test.localdev.me
    secretName: self-tls

Final thoughts for production use of Mutual TLS over kubernetes ingress controller:

As mentioned before nginx ingress controller will be installed according to your kubernetes environment.
Once we have the ingress controller installed, you need to configure DNS record for our domain name to point to the public IP of the ingress controller.
Server certificate and CA certificate should be issued from a trusted certificate provider and you just need to apply them into kubernetes as secrets.

At this point we reached the end of our tutorial. I hope you enjoyed reading this article, feel free to add your comments, thoughts or feedback and don't forget to get in touch on linkedin.

Not a Medium member yet? Sign up now for just $5 and get unlimited access to Medium-world!

#devsecops #mutual-tls #zero-trust #kubernetes #ingresscontroller

Apply Mutual TLS over kubernetes/nginx ingress controller

In this article, I will demonstrate how to apply mutual TLS connection between a client and a kubernetes endpoint exposed through nginx…

Reporting a Problem