This post has been republished via RSS; it originally appeared at: IIS Support Blog articles.

So you want your site to be available over secure HTTP and you already know that you need an SSL certificate. Still, many things can go wrong. I’m not going to document a specific error; rather I’d say how SChannel, TLS, keys and CAPI fit in, and where to look for problems.

Because we set the HTTPS binding and certificate for a site in IIS Manager, we might be tempted to think that IIS is managing the Secure Channel, or SChannel, through which HTTP traffic takes place. But that’s wrong. IIS Manager is going to configure the HTTP.SYS driver, specifying the certificates to be used for HTTPS traffic. And then, HTTP.SYS is going to take care of the TLS channel for HTTPS communication. Look:

IIS sits on top of HTTP.SYS, which in turn relies on SChannel provider for HTTPS communication

Before any HTTPS traffic can happen, a TLS SChannel has to be established. Then, through that channel, normal HTTP requests and responses would travel. The SChannel is established by HTTP.SYS, which sits below IIS. If the SChannel fails, then requests don’t even reach to IIS – there is no point looking for them in IIS logs. In fact, with SChannel failures, HTTPS requests don’t even make it to HTTP.SYS.­

The HTTP.SYS driver is going to call the SChannel provider to create the secure link. SChannel will use the CAPI, the Cryptographic API of Windows. And the certificate’s public and private keys are going to be used by CAPI2. Notice my highlight: the private key of the certificate is needed for the SChannel. If that private key is missing or inaccessible due to ACL (Access Control List), then “game over” – no secure channel can be created, HTTPS traffic would fail. In fact, there are other reasons too for which the certificate may be rejected by the SChannel provider. See below why a certificate may be dismissed.

A certificate that is paired with its private key

The SChannel provider is logging into the Windows Events – look inside the System log with the Event Viewer. Here are some common SChannel events and SSL/TLS protocol alerts.

The logging of the Crypto API is not turned on by default. If you want to troubleshoot, you’ll need to turn on the CAPI2 logging in the Windows Events Viewer.

The TLS handshake

Let’s see the steps of the TLS negotiation, how the secure channel is being established between the client and the server before any HTTPS traffic can happen.

1. TCP handshake is performed between client and server, then
2. Client Hello: the client sends to server the versions of TLS protocol supported, as well as the suite of ciphers (encryption/decryption algorithms it supports)
3. Server Hello: After the server compares what the client sends with its own supported TLS versions and ciphers, the server decides which ones will be used and sends the selection to the client.
4. Server certificate: The server has to authenticate itself to the client, so it sends its certificate. The keys will be used for encryption.
5. Client switches to encrypted; all following messages from client will be encrypted from this point.
6. Server switches to encrypted, too. From this point, normal HTTP requests and response can travel via the encrypted channel.

In fact, the TLS handshake is a bit more complicated, and I recommend the Pluralsight training Analyzing and Decrypting TLS Traffic for details. But the above steps are enough to illustrate what could go wrong and where to look for problems.

TCP handshake

The client (browser) has to be able to resolve the name of a site (the hostname) to an IP address. For that, the DNS records must be in place. Do you have a DNS A or CNAME record in place to help determine the IP address of the server/site/host name from the browser’s address bar? You can check with nslookup:

C:> nslookup duckduckgo.com

Client Hello and Server Hello

The client is sending to the server its supported TLS versions and cipher suites. The server compares the list with its own “records”. If the server does not support any of the TLS versions, or none of its cipher suites matches on from the client’s list, boom! Look into Windows Events, in the System log, for entries sourced SChannel.

A network trace capturing TLS negotiation, Client Hello

If you suspect problems with the encryption, then that’s the domain of CAPI2 - Cryptographic API logging must be enabled, then events must be studied.

Server Certificate

The server authenticates first; then, optionally, the client. But before the server sends its certificate, that must be usable. The SChannel provider performs some validation on the certificate, before using it.

The private key of the certificate must be present on the server, and accessible to SChannel provider. There is common misconception that the private key of a certificate stays within the certificate. That’s wrong; the private key belongs to a certificate, but it is stored separately. Certificates are held in Registry, while their private keys have various repositories, depending on where the certificate is found in the Certificate Manager.

The issuing authority of the certificate must be trusted by the server. The certificate of the issuing authority must be present in the Computer’s Trusted Root Certification Authorities. If there are intermediate certification authorities, their certificates must be present in the Computer’s Intermediate Certification Authorities. The trust chain must be OK.

Both client and server should trust the certification chain. Issuing root and intermediate certification autorities

The certificate must not be revoked. And its Key Usage flags must allow for Digital Signature and Key Encipherment. Working with SSL certificates using IIS Manager helps avoid many of the pitfalls commonly seen when setting up HTTPS.

Encrypted communication

Finally, before the communication is being switched to encrypted – Change Cipher Spec – the client has to accept the server’s certificate. Some browsers, like Internet Explorer, would display warnings for sites presenting improper certificates; but would still allow you to navigate. Other browsers or other client applications might be more… stubborn, should I say? They would not allow continuing navigation or sending requests if the certificate is “smelly”.

If the certificate is expired, not valid anymore: boom. If it was revoked by its issuing authority: boom. If the site/server/host name of the URL does not match with the certificate’s subject (issued to): boom. If the client does not trust the issuing authority of a certificate: boom (on Windows, the certificate of the issuing authority must be present in the Computer’s Trusted Root Certification Authorities / Intermediate Certification Authorities). In all these cases, the encryption of the secure channel would work, technically; but the clients/browsers simply don’t like the certificate presented by the server.

On a side note, only the key exchange relies on the computationally expensive asymmetric encryption, during the TLS handshake (let’s say with the certificate’s public and private key). Once the client and the server have established the secure channel, the data exchange will rely on symmetric encryption. This is why the certificate’s Key Usage flag must include Key Encipherment – it’s for the key exchange.

Handling certificates and their private keys

I keep saying about certificate private keys because their absence is one of the most common issues. Failing to have the private key of a certificate leads to frustration: IIS Manager would not even bother to display a server certificate if its private key is not present. So, in the HTTPS binding, one cannot pick a certificate if its private key is missing. Further on, if the private key is accidentally deleted, corrupted or becomes inaccessible due to rights, SChannel issues immediately appear.

So how do we handle this private key?

The “birth” of a certificate starts with a Certificate Request, a CSR. One creates a CSR, then a certification authority (CA) approves it and generates a certificate file.

When creating CSR, a key pair is generated: public and private key. The public key is, well, public, anyone could see it. But the private key remains securely stored on the machine where the CSR was generated.

Once the CA issues the certificate, we could install that certificate on a different machine. But wait, where is the private key? Oh, the certificate must be paired with its private key on the machine where the CSR was created. Once this pairing is done, both certificate and its private key can be exported in a PFX container. The PFX file may contain the certificate, its private key, and all the trust chain – certificate of the root authority and any intermediate authorities. Then the PFX can be imported on other machines if needed.

I recommend using IIS Manager only for creating certificates to be used for sites and web applications. IIS Manager ensures that the private key is exportable once the certificate was paired with it and it sets the proper key usage flags. It also ensures that the certificate is placed in the proper repository – Computer’s certificates.