Layering NGINX for more Inference security

AI inference networking
Part 1	Open WebUI as front door for ChatGPT
Part 2	Inference security using Open WebUI
Part 3	Layering NGINX for more Inference security

Adding NGINX Ingress Controller for mTLS auth.

Why add NGINX to our architecture?

In the previous posts we covered Open WebUI and F5 AI Guardrails. Now we’ll add NGINX ingress controller so that we can do some mTLS auth. After that we’ll experiment with adding other functionality.

Pre-requisites

• Follow blog posts 1 & 2. By now you should be able to deploy and destroy our environment on demand.

Instructions to deploy NGINX Ingress Controller

1. Deploy Open WebUI + CalypsoAI integration on K8s following previous posts.
2. I’ve cloned the repo for NGINX Ingress Controller at version 5.3.1
   • I have added the relevant manifests to the repo accompanying this series.
   • The instructions to deploy now include all manifests in the correct order. (The linked instructions are a point-in-time version.)
3. Now we should have NGINX I.C. running. You should be able to access Open WebUI via public IP address, and your traffic should be traversing NGINX I.C.

Instructions to deploy cert-manager, server cert, and mTLS certs.

1. Install cert-manager via manifests:

     kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.19.2/cert-manager.yaml
   

2. Create a ClusterIssuer
3. Create a CACert
4. Create a (namespaced) Issuer using the secret from the CACert
5. Create a secret of type nginx.org/ca where the only data field is “ca.crt” and the value is the certificate material.

     # Extract the tls.crt from the ca-cert and create secret of type nginx.org/ca
     # this is so we have a secret in the correct format for NGINX's Policy CRD, which will implement mTLS.
     kubectl get secret open-webui-ca-cert -n open-webui -o json | jq '.type="nginx.org/ca" | .data={"ca.crt": .data."tls.crt"} | .metadata={"name": "open-webui-ca-cert-nginx", "namespace": "open-webui"}' | kubectl apply -f - 
   

6. Create a secret for TLS for NGINX, and multiple mTLS client certs

Instructions to configure NGINX Ingress Controller

1. Create a Policy CR that requires mTLS verification. My example.
2. Create a VirtualServer CR that implements TLS with our server cert and references our Policy impelemting mTLS. My example.

Now, test your access to Open WebUI

1. Export CA Cert into trusted root store of your VM (unless using public CA)
2. Export mTLS client cert/key into your VM
3. Access the website, using a DNS name that matches your TLS cert (required by Chrome to prompt for mTLS certs)

If you can load the Open WebUI site only by providing a mTLS client cert, you have correctly configured NGINX to require and verify client certificates.

Conclusion

We’ve added mTLS authentication to our environment, by using NGINX Ingress Controller. Next we’ll use the identity learned via mTLS to make decisions at the time of inference within Open WebUI.