Difficulty Generating a Memory Dump

This post has been republished via RSS; it originally appeared at: Ask The Performance Team articles.

Hi there!

My name is Teeda, and I am a Support Escalation Engineer on the Windows Performance Team at Microsoft. This blog post provides several suggestions and workarounds when there is difficulty generating a memory dump for bugcheck issues (or even hang scenarios). Special thanks to my colleague, Alisse, for assembling this documentation.

 

Think about the goal…

Is a bugcheck occurring and you are trying to get a memory dump from that?  If so, you can skip the parts about manually triggering a dump.  However, you may want to use these settings to test out if you can get a memory dump.  This will be faster than waiting for the next bugcheck.

 

Do you need to crash the machine manually?  If so, pay attention to the type of machine (virtualized or physical) and the situation we are working with.

 

Is this a virtual machine?

VMware machines allow to create a snapshot which can then be converted to a memory dump.  Often, this is easier than trying to generate the memory dump manually.
  1. Capture the snapshot in the VMWare console with “Take Snapshot” either at the bugcheck screen or if another issue, at the time of the issue.
  2. Go to the following website: https://labs.vmware.com/flings/vmss2core
    • On the left-hand side, check the Agree and Download box.
    • Change the Dropdown to the appropriate OS (vmss2core-sb-8456865.exe).
    • Click on download.
  1. Once you have downloaded the file, save it on the C drive to a folder called c:\Snapshot
  2. Copy the vmss or vmsn/vmem file that you wish to convert to that folder.
  3. Open an elevated command prompt and run the following command:
    1. cd c:\Snapshot
      • For VMs OS until Windows 7/2008R2 use: vmss2core-sb-8456865 –W <snapshot.vmsn/Suspend.vmss> <snapshot.vmem>
      • For VMs OS Windows 8.1/2012 and above use: vmss2core-sb-8456865 –W8 <snapshot.vmsn/Suspend.vmss> <snapshot.vmem>
    2. Replace the '<snapshot.vmsn/Suspend.vmss> <snapshot.vmem>’ with the name of the snapshot.
    3. This process may take a few minutes depending on the size of the snapshot, but it will create a memory.dmp file in the c:\snapshots folder.
There is also the option to use the NMI switch in VMWare as an alternative if taking a snapshot is not an option.  Please note you will still need to configure for a memory dump whether it be kernel or complete: https://kb.vmware.com/s/article/2149185

 

Hyper-V Machines allow to save the state of the machine which can then be converted to a memory dump.

  • To do this, please right click the VM from Hyper-V manager and click "save" in state.  There will be saved state files at the location of the hard disk.
  • To allow the VM to continue running, you will need to right click the server and click start. Please OS version of the host machine as this will be needed to use the correct tool for conversion.
  • You will need to engage Microsoft to convert the Save state files (.bin/.vsv or .vmrs).

Alternatively, you can also configure for a manual Hyper-V crash using: 
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\hyperkbd\crashdump
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\hyperkbd\Parameters

Configuration information found here: Forcing a System Crash from the Keyboard - Windows drivers | Microsoft Docs

 

For Azure machines, Azure engineers can grab a memory dump or use NMI:

Configure for complete memory dump:

Step 1: Change page file size

  • Verify the machine has enough free space for 2x the RAM before continuing.
  • Launch File Explorer, then right-click This PC. Select Properties
  • Click Advanced system settings on the System page. Make sure you are on the Advanced tab.
  • Click Settings under the Performance area.
  • Click the Advanced tab, and then click Change under the Virtual memory area.
    • Note: To enable the system partition, you must uncheck “Automatically manage paging file size for all drives check box.”

TeedaN_0-1620920230012.png

  • Select the C:\ drive for page file location.
  • Click Custom Size. Set the value of Initial size and Maximum size to the amount of physical RAM that is installed plus 256 megabyte (MB) under the Custom Size button. (RAM*1024 + 256MB = Size in MB)
  • Click Set, and then click OK.

Step 2: Configure for a complete memory dump file

  • Go back to Advanced system settings page
  • Click Settings under the Startup and Recovery, and then make sure complete memory dump is selected.
    • Note: If you want to enable the complete memory dump option, manually set the CrashDumpEnabled registry entry to 0x1 under the following registry subkey and restart Windows: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl\
  • Ensure the path is C:\Windows\MEMORY.DMP (%SystemRoot%\MEMORY.DMP)

TeedaN_1-1620920230022.png

  • Click OK
  • Reboot VM for settings to take effect

Step 3: Enable Boot Diagnostics for NMI Crash

  • Login to Azure portal > select VM > Serial Console

TeedaN_2-1620920230027.png

  • Note: Serial Console requires boot diagnostics enabled

TeedaN_3-1620920230030.png

  • So, if not enabled, go to Boot Diagnostics > click Settings > Turn On > Save

TeedaN_4-1620920230035.png

 

Step 4: Send NMI during issue

  • When computer is in problem state > Serial Console > click Send Command [1] > click Send Non-Maskable Interrupt (NMI) [2]

TeedaN_5-1620920230040.png

  • Click Send NMI

TeedaN_6-1620920230046.png

  • Dump will be generated.

TeedaN_7-1620920230053.png

  • After completes login to VM and dump will be in C:\Windows\MEMORY.DMP

TeedaN_8-1620920230058.png

 

 

For AWS machines, try using these steps: https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/diagnostic-interrupt.html

For Nutanix machines, please engage the vendor to capture the memory dump.


Do you have the correct configuration?

Step 1: Change your page file size

  • Verify the machine has enough free space for 2x the RAM before continuing.
  • Go to Advanced system settings
  • On the System page, click the Advanced tab.
  • Click Settings under the Performance area.
  • Click the Advanced tab, and then click Change under the Virtual memory area.
    • Note: To enable the system partition, you must click to clear the Automatically manage paging file size for all drives check box.
  • Select the C:\ drive for pagefile location.
  • Click Custom Size. Set the value of Initial size and Maximum size to the amount of physical RAM that is installed plus 256 megabytes (MB) under the Custom Size button.
  • Click Set, and then click OK three times

Step 2: Configure for a complete memory dump file

  • Go back to Advanced system settings
  • On the System page, click the Advanced tab.
  • Click Settings under the Writing debugging information area (Startup and Recovery), and then make sure complete memory dump is selected.
    • If the complete memory dump is not an option here, to enable the complete memory dump option, manually set the CrashDumpEnabled registry entry to 0x1 under the following registry subkey: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl

Step 3: Apply the settings 

  • Ensure there is more space available on the C drive than there is RAM on the machine.
  • Please restart the machine for the settings to take effect

 

More Options

Try to use DedicatedDumpFile.sys - How to use the DedicatedDumpFile registry value to overcome space limitations on the system drive when capturing a system memory dump | Microsoft Docs

Manual Dump Trigger Options

NMI

Does this machine have a NMI switch? This would be in the Integrated Lights Out (iLO) Web interface. Create a DWORD value called NMICrashDump under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl and set it to a 1.  Then reboot the machine for the setting to take effect.

 

Keyboard initiated

For a USB keyboard, create the following registry entry:

  • In HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\kbdhid\Parameters, create a value named CrashOnCtrlScroll, and set it equal to a REG_DWORD value of 0x01. 

For a PS/2 Keyboard, create the following registry entry:

  • In HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\i8042prt\Parameters, create a value named CrashOnCtrlScroll, and set it equal to a REG_DWORD value of 0x01.

Then reboot the machine for the setting to take effect.

Note: you will need to use the Right Ctrl key + press the ScrLk key twice to trigger the dump with the above settings. If the machine does not have those available, there are other options. Forcing a System Crash from the Keyboard - Windows drivers | Microsoft Docs

Ex: Left Ctrl + Space Bar:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\kbhid\CrashDump

Create DWORD value Dump1keys set to 20 (hex)

Create DWORD value Dump2key (note no s here) set to 3d (hex)

 

NotMyFault

Use NotMyFault to initiate a crash: NotMyFault - Windows Sysinternals | Microsoft Docs

 

Change the Settings

  • Ensure there is enough space to capture the memory dump.  We need enough space for the page file, and for the memory dump itself which will be the size of the page file.
  • Disable the Autoreboot:(HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\CrashControl\AutoReboot)
  • Change the memory dump location to another spot on a local drive
  • Ensure the option "Overwrite Any Existing File" (found in Control Panel System) is selected. It is a good idea to leave this box checked and to move or copy the current Memory.dmp file.

 

There is dump logging

You can create a DWORD registry key HKLM\SYSTEM\CurrentControlSet\Control\CrashControl\EnableLogFile set to 1.  You will need to crash the machine twice, then you will see a dumpstack.log file on the root of the C drive which will keep track of what occurs during the action of writing to the page file.

 

Is ASR enabled?

Hardware vendors, such as HP, IBM, and Dell, may provide an Automatic System Recovery (ASR) feature. You should disable this feature during troubleshooting. For example, if HP and Compaq's ASR feature is enabled in the BIOS, disable this feature while you are troubleshooting to generate a complete memory.dmp file. For the exact steps, contact your hardware vendor.


Antivirus and Encryption

  • Check for any dump filter drivers.
  • Remove the encryption to test.

 

What else?

  • It is possible the paging file on the boot drive is not large enough. To use the "Write Debugging Information To" feature to obtain a complete memory dump file, the paging file on the boot drive must be at least as large as physical memory + 100 MB. When you create a kernel memory dump file, the file is usually around one-third the size of the physical memory on the system. Of course, this quantity will vary, depending on your circumstances.
  • Also possible there is not room for the Memory.dmp file in the path specified for writing the memory dump.
  • It is possible that the SCSI controller is bad, or the system crash is caused by a bad SCSI controller board.
  • If you specify a non-existent path, a dump file will not be written. For example, if you specify the path as C:\Dumpfiles\Memory.dmp and no C:\Dumpfiles folder exists, a dump file will not be written.
  • Is the Host Guardian Service enabled on either the host or the guest?  There are several settings which may prevent dumps from writing.  Managing the Host Guardian Service | Microsoft Docs

Grab that Page file!

Ensure the Autoreboot key is set to 0, and when the bugcheck occurs, boot into winre.  Grab the pagefile.sys and rename to memory.dmp

 

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.