Handling Configuration and Server Snippets When Upgrading to ingress-nginx 1.12.0

As part of our ongoing Sustainability and Maintenance efforts, we are upgrading our Kubernetes clusters to version 1.32. Much like a major Django upgrade, this process requires us to update other key components to ensure they remain compatible. One of these is the ingress-nginx controller, which manages external access to the cluster.

While upgrading our staging environment's ingress-nginx Helm chart from version 4.11.x to 4.12.x, we hit a snag. Immediately after the upgrade, the controller began returning a 404 Not Found response for every single request.

The Investigation

Our initial troubleshooting confirmed the controller pods were healthy, but the logs showed that no requests were reaching our application pods. The controller itself was intercepting all traffic and serving 404s. The breakthrough came after some digging led us to a GitHub issue titled "404 Not Found nginx after upgrading Ingress-NGINX version from v1.11.6 to v1.12.2". The issue described our exact problem and identified the root cause.

The 4.12.x Helm chart upgrades the underlying ingress-nginx controller to version 1.12.0. A look into the controller's release notes revealed a subtle but significant change that was easy to overlook: the default value for the annotations-risk-level setting was lowered from Critical to High.

With this new information, we went back to the controller logs. This time, we found a warning that we had previously overlooked amidst the noise:

E0702 17:38:58.386255 8 store.go:938] annotation group ConfigurationSnippet contains risky annotation based on ingress configuration

Understanding annotations-risk-level

The annotations-risk-level setting in the ingress-nginx ConfigMap determines the risk level of annotations the controller will accept on an Ingress resource. Annotations that allow for custom NGINX configuration, such as nginx.ingress.kubernetes.io/configuration-snippet, are powerful but are classified as a Critical risk because they can inject arbitrary commands into the NGINX configuration.

With the new default annotations-risk-level set to High, our Ingress objects, which used the Critical-level configuration-snippet annotation, were now being rejected by the controller, leading to the 404s.

We had been using configuration-snippet to increase the NGINX send_timeout value. This was necessary on some projects to prevent premature timeouts on long-running requests.

Resolution and Lessons Learned

This configuration-snippet had been copied and pasted between projects over time, becoming a piece of institutional knowledge that we had stopped questioning. The upgrade gave us a good opportunity to reassess whether this custom setting was still necessary for every project where it was found.

For the projects that still required the snippet, the fix was to explicitly set the annotations-risk-level back to Critical in our Helm chart values. For others, we removed the now-unnecessary annotation, simplifying our configuration.

We hope this post helps if you've run into similar 404 errors after the upgrade!