This post has been republished via RSS; it originally appeared at: IIS Support Blog articles.
In our first and second posts about troubleshooting the TLS / SSL problems, we worked to fix a "The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel" error message.
In our first scenario, we made some basic tests, such as a "browser test" and found that the certificate used was not valid.
In our second scenario, we compared a network trace taken from a failing scenario with trace taken from a successful scenario. We saw that the TLS negotiation between the client and the server was failing.
We fixed both issues and we know that the TLS versions match for the client and the server and the certificate is fine but we are still getting an error message.
If you recall the outcome of the scenario 2, we were seeing a FIN package coming from the client and the server was resetting the TCP connection:
This time the error we are getting is different:
Server Error in '/' Application.
--------------------------------------------------------------------------------
The remote certificate is invalid according to the validation procedure.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
The exception message tells us that the remote certificate is not valid. We know that we passed the TLS and Cipher negotiation so there should be some other validation failing. What could it be and how can we troubleshoot this issue?
Here the SChannel ETL logging or System.Net tracing comes in the picture. We will not focus on SChannel ETL logging here as it could be another post's topic. Instead we will use System.Net tracing to troubleshoot this problem.
A quick word about System.Net tracing
A System.Net tracing is a diagnostic option availabile in .NET Framework since the version 2.0. This is a part of System.Diagnostic namespace and it could be used to capture the details of System.Net classes including the System.Net.Socket operations. You can enable it in your configuration file. A sample usage is seen below:
<system.diagnostics> <sources> <source name="System.Net" tracemode="includehex" maxdatasize="1024"> <listeners> <add name="System.Net"/> </listeners> </source> <source name="System.Net.Sockets"> <listeners> <add name="System.Net"/> </listeners> </source> <source name="System.Net.Cache"> <listeners> <add name="System.Net"/> </listeners> </source> </sources> <switches> <add name="System.Net" value="Verbose"/> <add name="System.Net.Sockets" value="Verbose"/> <add name="System.Net.Cache" value="Verbose"/> </switches> <sharedListeners> <add name="System.Net" type="System.Diagnostics.TextWriterTraceListener" initializeData="c:\tracelogs\network_trace.log"/> </sharedListeners> <trace autoflush="true"/> </system.diagnostics>
To enable the System.Net tracing you can simply copy the configuration above and paste it in your ASP.NET application's web.config file. Please pay attention to the folder the traces will be created:
<add name="System.Net" type="System.Diagnostics.TextWriterTraceListener" initializeData="c:\tracelogs\network_trace.log"/>
Here in the example above, the trace will be written in the c:\tracelogs\ directory and the application pool's identity should have the write permission on that folder.
Once we reproduce the issue after enabling the System.Net trace, a log file will be created in the folder we specify in the configuration. The trace file is "human readable" and can be opened in any text editor. Let's take a look at it and if you look carefully you would see the following reported at the end of the trace:
[Public Key]
Algorithm: RSA
Length: 2048
...
...
...
System.Net Information: 0 : [2504] SecureChannel#26517107 - Remote certificate has errors:
System.Net Information: 0 : [2504] SecureChannel#26517107 - A certificate chain processed, but terminated in a root certificate which is not trusted by the trust provider.
System.Net Information: 0 : [2504] SecureChannel#26517107 - Remote certificate was verified as invalid by the user.
System.Net.Sockets Verbose: 0 : [2504] Socket#16754362::Dispose()
System.Net Error: 0 : [2504] Exception in HttpWebRequest#46228029:: - The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel..
System.Net Error: 0 : [2504] Exception in HttpWebRequest#46228029::GetResponse - The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel...
The error message "A certificate chain processed, but terminated in a root certificate which is not trusted by the trust provider" tells us that there is a problem with the CA's (certification authority) root certificate and it is not trusted, so most probably the certificate store for trusted root CAs does not contain the root CA for the remote certificate.
But,remember the browser test we made in our first scenario, it was working fine so the user's trusted root CA certificate store should have the root CA for the remote certificate.
Let's check the certificate once more with "browser test": open a browser on the server where your ASP.NET application works, browse the endpoint URL (which is https://iis85.buggybits.com/ in our case) and click the details of the certificate and move to the "Certification Path" tab:
As seen the certificate is from an internal CA called amborp-AMBDC-CA. Let's take a look at the user's certificate store. We confirm that the root certificate is already there:
But this is user's certificate store. ASP.NET client will use the "local computer" account's "Trusted Root Certification Authorities" store. If we open and check that store we see that the CA's root certificate is not installed there. Once we import the CA's root certificate in computer account's trusted root CA store, the application starts to work fine. Finally!!! :smileyhappy:
Summary:
- In our first post, we learnt that the certificate must be a valid one. Unlike "human clients", an ASP.NET client cannot click the "Ok, I trust this certificate / web site" if a certificate error occurs, unless you explicitly tell the application to ignore the certificate errors, which is actually not a recommended approach.
- In our second post, we learnt the TLS / Cipher negotiation is a requirement for a successful SSL handshake.
- In this third and final post, we learnt how System.Net tracing could be helpful when troubleshooting TLS / SSL related issues.
If you have any comment or suggestion, please use the comments section. If you have faced with another problem when making WCF, web service or HTTP web request calls to an SSL endpoint, please let me know and maybe we can prepare another blog post for that issue.
Happy troubleshooting...
--
AMB