The missing Server Hello in TLS handshake

This post has been republished via RSS; it originally appeared at: IIS Support Blog articles.

During TLS handshake, client machine makes a request to IIS server by sending Client Hello package. Then the server responds it with Server Hello.

 

If the server is not sending Server Hello package, the communication won’t be established and the customer is likely to see error messages below:

 

Internet Explorer:

This page can’t be displayed
Turn on TLS 1.0, TLS 1.1, and TLS 1.2 in Advanced settings and try connecting to again

Nedim_0-1596841596713.jpeg

 

Chrome:

This site can’t provide a secure connection
The website sent an invalid response
ERR_SSL_PROTOCOL_ERROR

Nedim_1-1596841596719.jpeg

 

Solution

Keep in mind that the TLS protocol errors above might be misleading. However, you should start the troubleshooting by analyzing TLS negotiation first.

 

Here are the steps to go through:

  • Make sure all TLS versions are enabled in Internet Explorer (This is for testing. You can later disable the unsecure versions)
  • Check SCHANNEL registry keys to make sure what you set in Internet Explorer is applied at registry level
    HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols
  • Collect a network trace. Check if the client and server are agreeing on a cipher suite. If they are not, make sure the client’s cipher suite list matches with the server's list. Related Group Policy setting:
    Computer Configuration > Administrative Templates > Network > SSL Configuration Settings > SSL Cipher Suite Order
  • If the issue sill exists, look for any network device (proxy, firewall, load balancer etc.) in between that might be intercepting or blocking TLS traffic

 

If you don’t see any issue with TLS protocol and network devices, continue the troubleshooting by focusing on the web server:

  • Check the website binding in IIS. The certificate should be valid
  • Make sure the port 443 is being listened (netstat -an -p TCP | find /I "listening")
  • Change the port number to 444 and test. If it is accessible, it means there is a software blocking 443 port
  • Disable Windows Firewall (If it works, you can enable it back and set up rules accordingly)
  • Look for any third-party applications in the server such as Symantec Endpoint Security and Symantec Data Center Security Server Agent (Security Server Agent uses port 443). Uninstall them (Don’t just disable them. Uninstall them completely. If it works, you can install them back and configure them accordingly)
  • Check if there is any Microsoft software that is using port 443. Applications like SQL Server Reporting Services (SSRS) and Windows Admin Center might interfere the port 443.

REMEMBER: these articles are REPUBLISHED. Your best bet to get a reply is to follow the link at the top of the post to the ORIGINAL post! BUT you're more than welcome to start discussions here:

This site uses Akismet to reduce spam. Learn how your comment data is processed.