Troubleshooting Third Party Updates in MEM CM Hierarchy

This post has been republished via RSS; it originally appeared at: Core Infrastructure and Security Blog articles.

The intention of this blog is not to go into the details of every aspect of third party patching feature in MEM CM as that is covered in a lot of blogs and our own documentation itself - Enable third party updates - Configuration Manager | Microsoft Docs. This is to run through some of the basic requirements which could help troubleshoot issues that you may run into while setting up the feature in a MEM CM hierarchy.

Although much of this is true for a single Primary, I am going to cover the scenario when you have a CAS and Primary in place as that makes it a bit more complicated. To add to it, in my case I have the SUP role running on a remote site system as that has a bearing on the set up.

High level data flow diagram

Catalog List

There are two types of Catalogs in the MEM CM third party patching, Partner Catalogs and Custom Catalogs. Partner Catalogs are currently HP, Dell and Lenovo. Custom Catalog for eg:- is Adobe. Both are catalogs published by third parties but the there is a difference in the way the communication flows for each and hence why I am calling it out.

General Pre-requisites

I know most of it is duplicate from the Microsoft docs site but I will call it out just so you have it for reference while reading this blog.

1. Enough disk space on the top-level update point’s WSUS Content folder to store the source binary for third party updates. In our case the top level is the CAS and the SUP on CAS is configured to store the content.
2. The third-party software update synchronization requires internet access.
• For partner catalog list, download.microsoft.com over HTTPS port 443 is needed. Only if this is allowed the HP, DELL and Lenovo entries will appear in the console. This connection goes out from the SUP and if the SUP can get to the above said URL, the next full sync of the SUP, which is typically every 7 days will populate the catalog entries.
• Third party updates use the same proxy settings as SUP and the proxy should allow the required URLs for the respective update catalog for the sync to work.

Additional Pre-Requisites on the SUP as the SUP is remote in this case

1. Configure SSL on WSUS/SUP (assumption is that is remote.)
2. Choose between Configuration Manager Generated Self-Signed WSUS certificate for signing third party updates and for this to work or choose your own code signing cert issued by your Organisation.
• Remote registry should be enabled on SUP server
• The WSUS server connection account should have remote registry permissions on the SUP/WSUS server.
3. HKLM\Software\Microsoft\Update Services\Server\Setup, create a new DWORD named EnableSelfSignedCertificates with a value of 1.
4. If you are using the self-signed cert the cert needs to be Trusted Publisher store and Trusted root store on the CAS and the WSUS. If the WSUS server connection account has remote admin permissions on the WSUS this will be done by the system itself but make sure these certs exist. The client will also get the cert added to its trusted root with the next MEM CM client policy update. If using a PKI issued cert, make sure that authority is trusted all on relevant systems including the client.
5. The machine where the console is running is used to download the updates from WSUS and add it to the updates package if you are doing it manually. The WSUS signing certificate must be trusted on the console machine. If it isn't, you may see issues with the signature check during the download/sync of third-party updates.
6. When the third-party software update synchronization service on the top-level software update point requires a proxy server for internet access, digital signature checks may fail. To mitigate this issue, configure the WinHTTP proxy settings on the site system. For more information, see Netsh commands for WinHTTP.

Issue 1:- Cannot see the partner catalogs list in console.

1. The synchronization of the partner catalogs is triggered by the WSUS via the SMS_ISVUPDATES_SYNCAGENT component. The component connects to download.microsoft.com using the proxy configured for your site system (in this case the WSUS). The go.microsoft.com link in the log re-directs to this page. Make sure your site system can connect to this URL. Check the SMS_ISVUPDATES_SYNCAGENT.log for similar entries as in the image above for a success.
2. At this point if you see errors along the lines of “Unable to connect to site database“in the ISV sync log on WSUS and the partner catalogs not being populated, check the permissions for the SUP machine account on the SCCM database.
3. Once all is working in terms of connectivity it takes a full sync for this to appear which is by default every 7 days.

Issue 2:- Console does not show signing certificate information in WSUS Signing Certificate Configuration.

SUP Config for CAS

In the above image I have chosen the option to use WSUS self-signing certificates. Once you choose that option the area that shows WSUS signing certificate details will be empty. If that option doesn't populate after a few WSUS syncs check the below steps.
Note:- If you are manually managing the certificate please refer to Enable third party updates - Configuration Manager | Microsoft Docs

1. For the above information to be published and be available you need to turn on https for WSUS. Remember this is only for the CAS SUP as that is where this config gets added and synced. If you don't turn on the https as below you would see an error in the logs.
   

Require SSL communication to WSUS server must be selected.

2. Once the option for "Configuration Manager manages the certificate", and WSUS is enabled on the SUP if the SUP is remote, it would need to replicate the certificate around, before it shows the details on the console. Refer to Wsyncmgr.log for entries like below.

Wsyncmgr.log on CAS

3. Once generated the cert will show up in the certificates node under security on the console as below. Make sure the valid certificate is unblocked.
   

Issue 3:- What certificates do I need and where, for this solution to work.

As I called out earlier, your SUP on CAS should be running on https which by default is on port 8531. This would mean you would have a binding of a certificate with server authentication capability on the WSUS Administration site IIS.

1. On CAS Site Server
• In trusted root certificate authorities, you need the issuing authority of the code-signing cert you used as listed in Issue 2. If you are manually managing the certificate for the code-signing part, the chances are your PKI is already in the trusted root.
• Your CAS should have the authority who issued the certificate for the SSL binding as in the above image, listed in the root certificate authorities list. If you are using self-generated cert for this (which is not the best secure way of doing it), you would have to manually do this.
2. On the CAS SUP
• Make sure the code signing cert ("WSUS Publishers Self Signed" if using Configuration Manager generated certificate) is in the Trusted Publisher's store on the SUP. If you are using Configuration Manager generated certificate the site should add the cert by itself, but if you haven't met all the pre-requisites this could fail and might need a manual copy.
3. On Primary SUP
• The primary SUP, which is a child of the CAS SUP, you need the issuing authority for the SSL binding cert on the CAS SUP added to the Trusted Root Certificates store. As we have set the CAS SUP to require SSL for all connections, the Primary SUP will now make an SSL connection to the parent update server.
4. On Client
• Assuming your Primary SUP is not running SSL, the only certificate that you would need on the client for specifically this is the issuing authority of the code signing certificate in the trusted roots certificates store and the trusted publishers store of the client. This applies automatically if you are using config manager generated cert for this and when you enable third party patching in client settings, via the normal policy refresh.

Issue 4:- I have everything above in place but not able to download Custom Catalog eg:- Adobe.

1.   The first step in adding a Custom Catalog is providing the details to the URL and once you set up , the process downloads the cab file and adds the certificate to MEM CM for the publisher (Adobe). This connection to the internet goes out from the machine where you are running the console using the account which launched the console. So a requirement would be for that combination to be able to get to the specified URL as in below image. This is only the first time you add the catalog. Subsequent update and syncs work from the SUP when run from schedule.
   

Note: - You would be prompted to accept a certificate to complete the subscription process, as below.

2.  If you are seeing issues publishing updates after they have synced make sure the certificate for the chosen vendor is approved in the certificates node in SCCM console.
   
   Note: - Any certs that have expired should be blocked but there are instances where you might bump into one that requires one of these expired certificates.
3. After your catalog has been subscribed you need to wait for catalog synchronization and WSUS synchronization to complete, before you can select new products in the Software Update Point (SUP) properties tab.

Issue 5:- After I configured all this I notice my Primary SUP has stopped syncing with CAS SUP.

1. If your WSUS infrastructure was not configured for SSL prior to this, one of the key changes you have made is for CAS SUP to enforce SSL connection. This means the connection from Primary to CAS will be using SSL and primary needs to trust the certificate authority that has issued the certificate used for SSL binding on CAS SUP.

Issue 6:- "The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.." error in the log

If you see this error in the ISV SyncAgent logs the chances are you have a root certificate missing. Most of the Microsoft hosted endpoints are configured with Baltimore CyberTrust Root Certificate. If this is missing,expired or corrupted you would see the above text in the log. More details here.

The details in the article are applicable to this case too. To resolve this problem it is documented here.

Administrative Tasks

Now that we have the back end set up I would like to invite your attention to a couple of key facts

1. The initial sync after all the set up is done and category has been selected within Products within Software Update Configuration, will bring only the metadata. You need to publish the ones of your choice before you can deploy.
2. With the client settings enabled and configurations required in place clients will scan against the metadata and start showing results of the scan in the console, without you publishing the update.
    

   
   

3. If you have SCUP already configured in your estate, needs additional considerations to make sure it does not break the SCUP process which is not covered in this blog.

Hope this helps you in understanding the config better and troubleshoot some of the issues you might see in your environment.