This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Community Hub.
By default, the Kubernetes Nginx Ingress Controller writes logs and traces to stdout / stderr.
These logs and traces are stored in the ContainerLog table. You can navigate to an autogenerated query from inside the Container Insights blade from the Azure Portal.
During an engagement, we faced an issue with viewing these logs and traces. We could view the logs in the “Live Logs” tab inside the Portal, but none of these got sent to Log Analytics workspace and we were unable to retain and run Kusto queries against that data.
As it turned out, after a troubleshooting session with a colleague, the reason the Nginx Ingress Controller logs and traces were not sent in Log Analytics workspace was the fact that the controller had been deployed to the kube-system namespace, which happens to be excluded by default from the container log collection, based on the below note.
To solve this, the recommended approach, based also on the example in the official Microsoft Learn documentation, is to redeploy the Ingress Controller to a different namespace.
Taking it to the next level, we wanted to do this with zero downtime. For that, a secondary Ingress Controller needs to be created. Then, the ingress routes need to be duplicated to use the secondary ingress controller. Finally, you should do a swap in the Firewall Appliance to redirect traffic to the new Ingress IP address.
Below you can find the detailed steps to deploy and configure a secondary ingress controller:
- Create an additional ingress controller on a different / appropriate namespace, providing a custom ingressClass (e.g., “nginx-v2”), given the default “nginx” ingressClass will already be in use by the existing ingress controller. The helm install command is simplified for readability, you will probably have additional parameters as per your script, but the highlighted ones are required:
|
helm install nginx-ingress ingress-nginx/ingress-nginx \ --namespace ingress-nginx-v2 \ --create-namespace \ --set controller.ingressClass=nginx-v2 \ --set controller.ingressClassResource.name=nginx-v2 \ --set controller.ingressClassResource.controllerValue="k8s.io/nginx-v2" |
- At this point, there should be two ingress controllers on different namespaces and two different ingressClass definitions:
|
kubectl get svc --selector=app.kubernetes.io/name=ingress-nginx -A
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE ingress-nginx-v2 nginx-ingress-ingress-nginx-controller LoadBalancer 11.0.247.238 51.1.1.2 80:32746/TCP,443:31787/TCP 20m ingress-nginx-v2 nginx-ingress-ingress-nginx-controller-admission ClusterIP 11.0.114.217 <none> 443/TCP 20m ingress-nginx ingress-nginx-controller LoadBalancer 11.0.220.7 51.1.1.1 80:32040/TCP,443:30183/TCP 15d ingress-nginx ingress-nginx-controller-admission ClusterIP 11.0.161.45 <none> 443/TCP 15d
kubectl get ingressclass
NAME CONTROLLER PARAMETERS AGE nginx k8s.io/ingress-nginx <none> 15d nginx-v2 k8s.io/nginx-v2 <none> 23m |
- Duplicate the existing ingress routes, as they will be referencing the original ingress controller:
|
k get ingress -A
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE contoso contoso-ingress nginx * 51.138.206.238 80 15d |
- This means you need to create an exact copy of all ingress routes, changing the “ingressClassName” to the newly created ingressClass:
|
cat << EOF | kubectl apply -f - apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: contoso-ingress-v2 namespace: contoso annotations: nginx.ingress.kubernetes.io/ssl-redirect: "false" nginx.ingress.kubernetes.io/use-regex: "true" nginx.ingress.kubernetes.io/rewrite-target: /$2 spec: ingressClassName: nginx-v2 rules: - http: paths: - path: /contoso(/|$)(.*) pathType: Prefix backend: service: name: contoso-webapp port: number: 80 EOF
ingress.networking.k8s.io/contoso-ingress-v2 created
kubectl get ingress -A
NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE contoso contoso-ingress nginx * 1.1.1.1 80 15d contoso contoso-ingress-v2 nginx-v2 * 1.1.1.2 80 85s |
At this point, you can access the applications using both ingress controllers, and you should be able to redirect all traffic to the new ingress after performing the necessary validations.