This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Tech Community.
Hello everyone! My name is Mike Kammer, and I'm a Platforms PFE with Microsoft, dealing with all versions of Windows Server, and other duties as assigned. Sometimes those duties are fun, and everything works perfectly, like playing with kittens or patches applying smoothly. Sometimes those duties bring on tasks, like cleaning a litterbox or figuring out why patches won’t apply to some of your servers. I had a customer who was having one of those litterbox moments and needed help so they could get back to petting kittens.
Here's our issue, and our story:
We had one node of a physical two node Server 2008 R2 cluster which would not accept four updates: KB4474419, KB4516046, KB4507456, and KB4516065.
The other node updated normally and installing the patches on the problem machine was attempted by both running the .MSI files and through Windows Software Update Services (WSUS). This customer mainly ran updates through WSUS as a standard practice but had downloaded the installers for these KBs as part of troubleshooting the issue. As part of our troubleshooting, we had also disabled the installed virus scan to ensure it was not getting in the way.
Our main issue was receiving the error that "This Update is not applicable to this computer." We verified the updates did apply to our machine and looked through our C:\Windows\Logs\CBS.log file for clues. The CBS is the Component Based Servicing log and it contains information on components when they get installed or uninstalled during updates and is a good place to start investigating update errors.
Normally I would grab all the details and be as specific as possible on what I’m looking through. Being at a customer site, I am only able to paraphrase what we found in the log, and not provide screenshots of the log itself. Sometimes I forget to write down exact phrasing. Essentially what we found in that log file was that the system thought there were updates that were still being processed. Since we had rebooted to start fresh, a good 10-minute process because of our physical hardware, we knew there were not any updates being processed at this point, and the log was not reporting correctly.
Next, we moved on to the C:\Windows\winsxs directory, which is the Windows side by side directory and holds component store files, basically all our update files. Here we found several Pending.XML files, dating back over the last year. In general, it is a very bad idea to delete files from the winsxs directory. However, the Pending.xml file is created by updates when there is a reboot needed to complete the update. And as these files were old, we felt it safe to remove them. To remove the files, we had to take ownership of them, as administrators only have read/execute permissions by default. After deleting all the Pending.XML files, we attempted to install the updates again. And, once again, we received the error that "This Update is not applicable to this computer."
Getting a little bit frustrated at this point, we opened the registry editor and loaded up the COMPONENTS hive. This Hive holds data associated with Windows Update configurations and status and works closely with the winsxs directory. Unless you are dealing with update issues, there’s really no need to load this key. As we just deleted files from the winsxs directory, though, it’s a good time to verify everything is ok in here. To do this, we needed to load it into HKEY_LOCAL_MACHINE, so we highlighted that hive and clicked File -> Load Hive.
We browsed to C:\Windows\System32\Config, select COMPONENTS, and clicked Open.
We were prompted to enter a Key name, so we put in COMPONENTS and clicked Ok.
Here we expanded the COMPONENTS hive and looked for the following keys to give us some kind of indication of the issue:
- PendingXmlIdentifier
- SetupExecute
- PoqexecFailure
- StoreDirty
- AdvancedInstallersNeedResolving
My customer only had a PoqexecFailure key. Again, I apologize for no screenshots, I was not able to gather them from the customer site. This key is usually tied closely to the SetupExecute key, and the pocexec.exe displays error messages from updates, so seeing it without seeing SetupExecute was very odd. We took another close look at the C:\Windows\Logs\CBS.log file and found this error (sometimes I DO write them down!):
- Install updates failed with error: 0x80070bc9 - ERROR_FAIL_REBOOT_REQUIRED
After rebooting, we returned to the COMPONENTS key, and this time we found an ExecutionState entry, with a value of 5.
Oooo, what's this key doing? This key, with a value of 5, is telling us that we are in the middle of an execution. Since we have just come back from a full reboot again and have not attempted another install, we should not be in the middle of an execution. This may also indicate that we have corrupt transaction files (hint: we did). So, we deleted the ExecutionState key and proceeded to look for these transaction files.
The transaction files are in C:\Windows\System32\Config\TxR. The only way to remove them is to boot into recovery mode. We used a Windows 10 Diagnostics and Recovery Toolkit (DaRT) disk and browsed the filesystem to the folder and deleted all the files in that folder and rebooted back to our full desktop.
We verified that the PoqexecuteFailure and ExecutionState keys were still gone from the registry, and looking in the TxR folder, we found new files that had been created in the 15 minutes since we rebooted. All good signs, so we went ahead and attempted an install of KB4474419 through WSUS. Our result? SUCCESS!!
We rebooted to complete the install, and upon reaching the desktop confirmed that KB4474419 was in our "Installed Updates" section of Windows Update.
Feeling good, we installed KB4516046 through WSUS again. And our result - SUCCESS!
Feeling great, we went on to install KB 4507456. More success!
Doing a little dance at this point, we installed KB4516065 successfully.
Wait, no. That didn't happen. KB4516065 failed to install with ERROR: 80092004 - CRYPT_E_NOT_FOUND. Huh? We were doing so well!
Back to our basics! What’s in the CBS.log? ERROR: 80092004 - CRYPT_E_NOT_FOUND. Cool, thanks. Winsxs folder? Looking good! COMPONENTS registry – clear! Rebooted (another great 10-minute process with the old 2008 R2 hardware) with DaRT and cleared out the TxR folder again.
Attempted an install. Same error. Grrrrr.
Checking the requirements for KB4516065, we noticed there was a requirement for the latest servicing stack update (SSU) (KB4516655). Cool. We can install that. Wait, no, no we cannot. We get the old "this Update is not applicable to this system" error again. Come on, we just fixed that!
We checked the requirements for that KB, verified that it was indeed applicable to our system and scratched our heads for a bit. Looking closer, we noticed that this KB replaced KB4990628. We tried installing that update and were happy to see a success!
Following that, we proceeded to install our last KB, 4516065, and were successful. All subsequent updates were successful too.
The big takeaway was learning about the COMPONENTS registry hive and the TxR folder. Load that Hive and verify any of the subkeys, then make sure you only have recent files (if any) in the TxR folder.