This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Tech Community.
If you are using Azure Application Gateway as Layer 7 WAF for End to End SSL connectivity , you might have come across Certificate related issues most of the times. In this article I am going to talk about one most common issue "backend certificate not whitelisted"
If you check the backend health of the application gateway you will see the error like this "The root certificate of the server certificate used by the backend does not match the trusted root certificate added to the application gateway. Ensure that you add the correct root certificate to whitelist the backend"
The error says that Root cert is not whitelisted on the AppGW , but you might have a valid Third party certificate on the backend , and more over if you try to access the backend directly bypassing the Application Gateway you will not see any issues related to certificate in the browser.
If your certificate is working on browser directly hitting the app and not with AppGW then what is the exact problem?
To Answer we need to understand what happens in any SSL/TLS negotiation. During SSL negotiation , Client sends "Client Hello" and Server Responds with "Server Hello" with its Certificate to the Client. Now Clients will check the Server certificate and confirm if the certificate is issued by Trusted root or not.
For the server certificate to be trusted we need the Root certificate in Trusted Root Cert Store , usually if you are having certs issued by Godaddy,Digicert,Vergion like Third party Vendors you don’t have to do anything because they are automatically trusted by your client/browser.
If your cert is issued by Internal Root CA , you would have export the root cert and import it the Trust Root Store in the Client.
This is the exact thing what we do when import .CER file in the HTTP Settings of the Application Gateway. In Azure docs, it is clearly documented that you don’t have import Auth certificate in HTTP settings of the backend if your backend application has Global trusted certificate. You should do this only if the backend has cert which is issued by internal CA
I hope we are clear till now on why we import Authenticate cert in the HTTP settings of the AppGW and when we use the option "Use Well Known CA"
But the actual problem arises if you are using a Third party Cert or Internal CA Cert which has Intermediate CA and then Leaf certificate
Most of the orgs for security reasons use Root Cert----> Intermediate Cert ------> Leaf Cert , even Microsoft follows the same for bing , you can check yourself as below
Now lets discuss what exactly is the confusion here if we have multiple Chain Cert
Check this below when you have single chain certificate , then there will be no confusion with appgw , if your root CA is Global trusted just select "Use Trusted Root CA" option in HTTPsettings
If you root CA is Internal CA , then import that Top root cert in .cer format and upload it in the HTTP settings
But when we have multiple chain certificate and your backend application is sending the Application Gateway only the leaf the certificate , AppGW will not be able to trust the cert up to the top level domain root.
Most of the browsers are thick clients , so it may work in the new browsers but PRODUCTs like Application Gateway will not be able to trust the cert unless the backend sends the complete chain
Now how can find if my application sending the complete chain , the easy way to find is running openssl from either windows client or Linux client which is present in the same network/subnet of the backend application.
We should get one Linux machine which is in the same subnet/VNET of the backend application and run the following commands. We are actually trying to simulate the Linux box as AppGW
here is the sample command you need to run, from the linux box that can connect to the backend application.
OpenSSL s_client -connect 10.0.0.4:443 -servername www.example.com -showcerts
here is the IP is your backend Application IP , it changes as per your backend pool you can use even use the hostname directly here.
If the output doesn't show the complete chain of the certificate being returned, export the certificate again with the complete chain, including the root certificate. Configure that certificate on your backend server.
You can find more details about this issue in our Azure docs, there is a solution already documented in Troubleshoot backend health issues in Azure Application Gateway | Microsoft Docs , Look for the sub topic "Trusted root certificate mismatch"