SAP NetWeaver HA deployment with File Share running on Windows failover to DR Region using ASR

This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Community Hub.

Overview

 

The high availability of SAP NetWeaver systems on Windows can be achieved by using Windows Failover Clustering. The resources configured in the cluster are considered highly available, if the node that host the resources are up. But the Windows cluster generally requires more than half of the nodes to be running to avoid a split-brain scenario. To achieve this objective, quorum is required to be set in the Windows failover cluster. Quorum is designed to prevent split-brains scenarios. There are different quorum options available for SAP NetWeaver on Azure VMs.

 

Setting up high availability of SAP workload on Azure protect application from infrastructure maintenance or failure within region. But it doesn’t provide protection from widespread regional disaster. For Disaster Recovery (DR), protection of application running on Azure VMs can by enabled by replicating components using Azure Site Recovery (ASR) to another Azure region. In this article, we will talk about achieving high availability configuration on DR region using ASR, when SAP ASCS/ERS with file share running on Windows is configured in productive region.

 

NOTE: The article only focuses on the SAP NetWeaver HA deployment with file share cluster. If your system is setup using “cluster shared disk” configuration, this article is not applicable.

 

IMPORTANT NOTE:

 

  • The example shown in this article is exercised on the below OS version, cluster share and quorum option -
    • SAP ASCS/ERS OS version: Windows Server 2022 Datacenter
    • Quorum: Cloud witness
    • Cluster share: File share (SMB on Azure Files)
  • Depending on the type of file share storage used for SAP workload, you have to adopt an appropriate method to replicate the storage data to the DR site. In this example, SMB on Azure Files is used and its DR setup can be achieved by configuring separate SMB on Azure Files on DR region and copy the data periodically using robocopy.
    NOTE: In case you are using SMB on Azure NetApp Files (ANF), the DR setup can be achieved using cross-region replication. ANF is not available in all regions. Refer Azure Products by Region to see if ANF is available in your DR region.
  • Failover of other dependent services like DNS or Active directory is not covered in this article.
  • To replicate VMs using ASR for DR setup, review supported regions.
  • SMB Azure file shares are available in every Azure region, including all public and sovereign regions. Premium SMB file shares, which is recommended for SAP workload are available in a subset of regions.
  • ASR doesn’t replicate Azure load balancer that is used as virtual IP for the SAP ASCS/ERS cluster configuration in source site. In the DR site, you need to create load balancer manually beforehand or at the time of the failover event.
  • The cloud witness is configured using Azure blob storage, so you need to configure a separate storage account in DR region beforehand or at the time of the failover event.
  • If you configured your SAP system using Windows DFS-N to support flexible SAPMNT share creation for SMB-based file share, then some of the post-steps like editing UNC path at multiple areas won’t be required. You can directly modify SMB endpoint in DFS management console. But this article covers SAP system installed without DFS-N namespace for SMB storage volumes.
  • The procedure described here has not been tested with different OS releases. So, it might need some additional works based on your implementations or with future OS releases. So, make sure you test and document the entire procedure thoroughly in your environment.

SAP ASCS/ERS with SMB on Azure Files Disaster Recovery architecture

 

In the figure below, SAP ASCS/ERS with SMB on Azure files high availability cluster is configured in the primary region. The cluster uses cloud witness as quorum option. To establish DR for the setup, Azure site recovery (ASR) is used to replicate the SAP ASCS and SAP ERS VMs across site. For file share, separate SMB on Azure Files is created on DR region and data is copied periodically using robocopy.

 

NOTE: You can also leverage SMB on Azure NetApp Files (ANF) for SAP ASCS/ERS cluster. But in this blog, details on SMB on ANF is not covered.

 

SAP ASCS/ERS with SMB on Azure Files DR ArchitectureSAP ASCS/ERS with SMB on Azure Files DR Architecture

 

 

As described in the example, to achieve HA setup in the DR site for SAP ASCS/ERS we need to make sure that all components that are part of the solution are replicated.

 

Components

DR setup

SAP ASCS/ERS VMs

Replicate VMs using Azure Site Recovery

SMB on Azure premium files

Create separate SMB on Azure premium Files. Copy data using robocopy.

Storage used for Cloud Witness

Create a separate storage in DR region.

Load balancer used for cluster virtual IP.

Create a separate Load balancer in DR region.

 

Disaster Recovery (DR) site preparation

 

To achieve similar highly available setup of SAP ASCS/ERS on the DR site, you need to make sure that all the components are replicated and available in the event of failover.

 

Configure ASR for SAP ASCS/ERS and Application server VMs

 

  1. Deploy Resource Group, Virtual Network, Subnet and Recovery Service Vault in secondary region. For more information on networking in Azure VM disaster recovery, refer to prepare networking for Azure VM disaster recovery.
  2. Follow the instructions in Tutorial to set up disaster recovery for Azure VMs document to configure ASR for SAP ASCS and ERS VMs.
  3. On enabling Azure site recovery for VM to setup DR, the OS and local data disk that are attached to VMs get replicated to DR site. During replication, the VM disk writes are sent to a cache storage account in the source region. Data is sent from there to the target region, and recovery points are generated from the data. When you fail over a VM during DR, a recovery point is used to restore the VM in the target region.
  4. After the VMs are replicated, the status will turn into “Protected” and the replication health will be “Healthy”.
    Replicated VMs in Recovery Service VaultReplicated VMs in Recovery Service Vault

 

Configure robocopy to replicate SMB on Azure files data to DR Region

 

Robocopy, short for “Robust File Copy”, is a command-line utility in Windows for copying files and directories from one location to another. It is available as standard feature in Windows sines Windows Server 2008. So, you do not have to install this tool separately on your Windows server from where you will be executing the command.

 

  1. To copy the data of SMB on Azure premium files to DR region, you need to create a separate SMB on Azure premium files on the DR region.
  2. Follow the same post steps (like Active directory integration, assign required access and roles) on the new SMB on Azure files that you have followed in Primary region.
  3. Before you can use Robocopy, you need to make sure the SMB on Azure files is accessible. The easiest way is to mount the share as local network drive to the Windows servers from where you are planning on using for Robocopy.
    NOTE: You can have UNC path as your source and target location in robocopy command. It is not necessary to have SMB on Azure files mounted at all times on server from where you are executing command.  
  4. You can execute Robocopy from any Windows servers, but make sure it can access both source and target SMB on Azure files.
  5. Execute Robocopy command. Below screenshot is just an example, you must make sure to set appropriate copy option with the command as per your requirement.
    Example:  	Robocopy command to replicate data between two SMB on Azure Files storageExample: Robocopy command to replicate data between two SMB on Azure Files storage

NOTE: You can schedule a job to execute Robocopy command periodically to copy the content of SMB on Azure Files.

 

Configure cloud witness for SAP ASCS/ERS in DR Site

 

Tip: Based on your DR strategy, you can either execute this step when you are preparing your DR site like setting up ASR or you can execute at the time of failover event.

 

  • Create an Azure storage account on DR site to use as cloud witness.

    Region

    Storage account name

    Primary

    ts3clusteastn

    Disaster Recovery

    ts3drwestwitness

 

Configure Standard Load Balancer for SAP ASCS/ERS in DR Site

 

Tip: Based on your DR strategy, you can either execute this step when you are preparing your DR site like setting up ASR or you can execute at the time of failover event.

 

Deploy standard load balancer on the DR site, similar to the one you have deployed in primary region. If you are creating load balancer beforehand on the DR site, you won’t be able to assign VMs to the backend pool. You can create the backend pool as empty which allows you to define the load balancing rules, but you cannot assign the DR VMs to the backend pool. Also, keep following points in mind -

  • Keep the probe port in DR load balancer same as primary.
  • When VMs without public IP addresses are placed in the backend pool of internal (no public IP address) Standard Azure load balancer, there will be no outbound internet connectivity, unless additional configuration is performed to allow routing to public end points. For details on how to achieve outbound connectivity see public endpoint connectivity for Virtual Machines using Azure Standard Load Balancer in SAP high-ava....

    Site

    Frontend IP

    Primary Region - ASCS

    10.19.0.97

    Primary Region - ERS

    10.19.0.105

    DR Region - ASCS

    10.5.0.9

    DR Region - ERS

    10.5.0.10

 

Disaster Recovery (DR) failover event

 

[A] - Applicable to SAP ASCS Node, [B] - Applicable to SAP ERS Node, [C] - Applicable to SAP Dialog Nodes.

 

In case of a production DR failover event, the following procedure needs to be followed for SAP ASCS/ERS and SAP Application server. If you are using different Azure services with your SAP system, you many need to adjust your procedure accordingly. The failover procedure described here assumes that the system running on primary region in not reachable or unavailable. So, when initiating failover to DR region the system on primary region will remain down at all times. 

 

  1. Perform the failover of SAP ASCS/ERS and all application server VMs that are configured in ASR to the DR region. For more details on how to failover, refer Tutorial to fail over Azure VMs to a secondary region for disaster recovery with Azure Site Recovery document.
    NOTE: Use of Azure site recovery for databases isn’t recommended. For more details on the DR recommendation for databases, refer to SAP database servers DR guidelines.
  2. After the failover is completed, the status of the replicated items in the recovery service vault would be like below -
    Failover completion status in Recovery Service VaultFailover completion status in Recovery Service Vault
  3. Update the IP address of VMs in DNS or in host files (if maintained). In this example, update the IP address for SAP ASCS/ERS, and all application servers. The ASCS/ERS server name registered in cluster is also maintained in DNS. So, you need to update the IP address of ASCS/ERS server name in DNS or in host files as well.

    Component

    Primary region IP address

    DR region IP address

    ASCS Instance

    10.19.0.103

    10.5.0.8

    ERS Instance

    10.19.0.104

    10.5.0.5

    Dialog Instance

    10.19.0.106

    10.5.0.7

    ASCS Virtual Server Name (cluster resource)

    10.19.0.97

    10.5.0.9

    ERS Virtual Server Name (cluster resource)

    10.19.0.105

    10.5.0.10


    NOTE: The ASCS/ERS virtual Server Name IP address needs to be updated in DNS and in cluster configuration. Check point 9 for information on how to update Server Name resource IP address in cluster.
  4. [A] [B] Login to SAP ASCS and SAP ERS server and update the UNC path maintained in the symbolic link to the SMB on Azure files on DR region (ts3smbdrwest.file.core.windows.net)
    #Execute below command in PowerShell with Administrator New-Item -Type SymbolicLink -Path "F:\usr\sap\TS3\SYS" -Target "\\ts3smbdrwest.file.core.windows.net\sapmnt\TS3\SYS" -Force New-Item -Type SymbolicLink -Path "F:\usr\sap\TS3\ASCS00\data" -Target "\\ts3smbdrwest.file.core.windows.net\sapmnt\TS3\ASCS00\data" -Force New-Item -Type SymbolicLink -Path "F:\usr\sap\TS3\ASCS00\log" -Target "\\ts3smbdrwest.file.core.windows.net\sapmnt\TS3\ASCS00\log" -Force New-Item -Type SymbolicLink -Path "F:\usr\sap\TS3\ASCS00\sec" -Target "\\ts3smbdrwest.file.core.windows.net\sapmnt\TS3\ASCS00\sec" -Force
  5. Change the value of SAPGLOBALHOST and SAPTRANSHOST to DR storage account endpoint (ts3smbdrwest.file.core.windows.net) on DEFAULT, ASCS, ERS and all APP server profiles.
    # SAPGLOBALHOST and SAPTRANSHOST value updated to DR storage endpoint in DEFAULT and instances profile PS F:\usr\sap\TS3\SYS\profile> findstr /i /r /c:"SAPGLOBALHOST" /c:"SAPTRANSHOST" TS3* DEFAULT.PFL TS3_ASCS00_ts3ascseast:SAPGLOBALHOST = ts3smbdrwest.file.core.windows.net TS3_D00_mswinapp55:SAPGLOBALHOST = ts3smbdrwest.file.core.windows.net TS3_ERS10_ts3erseast:SAPGLOBALHOST = ts3smbdrwest.file.core.windows.net DEFAULT.PFL:SAPGLOBALHOST = ts3smbdrwest.file.core.windows.net DEFAULT.PFL:SAPTRANSHOST = ts3smbdrwest.file.core.windows.net DEFAULT.PFL:DIR_TRANS = \\$(SAPTRANSHOST)\sapmnt\trans
  6. [A] [B] [C] SAP services (SAPSID_XX) configured in services.msc has executable path defined with profile value set as UNC path. It needs to be updated with the DR storage endpoint UNC path.
    # Example to edit BINARY_PATH_NAME of SAPTS3_00 maintained in services.msc. # Primary region SMB on Azure files endpoint: ts3smbeast.file.core.windows.net. It need to be changed to secondary region SMB on Azure files endpoint: ts3smbdrwest.file.core.windows.net C:\Windows\system32>sc qc SAPTS3_00 SERVICE_NAME: SAPTS3_00 TYPE : 10 WIN32_OWN_PROCESS START_TYPE : 3 DEMAND_START ERROR_CONTROL : 1 NORMAL BINARY_PATH_NAME : "F:\usr\sap\TS3\ASCS00\exe\sapstartsrv.exe" pf="\\ts3smbeast.file.core.windows.net\sapmnt\TS3\SYS\profile\TS3_ASCS00_ts3ascseast" LOAD_ORDER_GROUP : TAG : 0 DISPLAY_NAME : SAPTS3_00 DEPENDENCIES : RPCSS : LanmanServer SERVICE_START_NAME : SAPCONTOSO\SAPServiceTS3 # Command to change BINARY_PATH_NAME of ASCS, ERS, PAS service in respective server where it is created sc config SAPTS3_00 binPath= "\"F:\usr\sap\TS3\ASCS00\exe\sapstartsrv.exe\" pf=\"\\ts3smbdrwest.file.core.windows.net\sapmnt\TS3\SYS\profile\TS3_ASCS00_ts3ascseast\" sc config SAPTS3_10 binPath= "\"F:\usr\sap\TS3\ERS10\exe\sapstartsrv.exe\" pf=\"\\ts3smbdrwest.file.core.windows.net\sapmnt\TS3\SYS\profile\TS3_ERS10_ts3erseast\" sc config SAPTS3_00 binPath= "\"F:\usr\sap\TS3\D00\exe\sapstartsrv.exe\" pf=\"\\ts3smbdrwest.file.core.windows.net\sapmnt\TS3\SYS\profile\TS3_D00_mswinapp55\" # Example: After changing BINARY_PATH_NAME C:\Windows\system32>sc qc SAPTS3_00 SERVICE_NAME: SAPTS3_00 TYPE : 10 WIN32_OWN_PROCESS START_TYPE : 3 DEMAND_START ERROR_CONTROL : 1 NORMAL BINARY_PATH_NAME : "F:\usr\sap\TS3\ASCS00\exe\sapstartsrv.exe" pf="\\ts3smbdrwest.file.core.windows.net\sapmnt\TS3\SYS\profile\TS3_ASCS00_ts3ascseast" LOAD_ORDER_GROUP : TAG : 0 DISPLAY_NAME : SAPTS3_00 DEPENDENCIES : RPCSS : LanmanServer SERVICE_START_NAME : SAPCONTOSO\SAPServiceTS3​
  7. [A] [B] [C] Change the UNC path value maintained in Path, RSEC_SSFS_DATAPATH, RSEC_SSFS_KEYPATH and SAPEXE environment variable to DR storage endpoint.
    # Example showing the value of Path, RSEC* and SAPEXE env variable updated with DR storage endpoint PS C:\Windows\system32> Get-ChildItem -Path Env: | Where-Object -Property Value -CLike "*ts3smbdrwest*" Name Value ---- ----- Path C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\dotnet\;C:\Program Files (x86)\dotnet\;C:\Users\ts3adm\AppData\Local\Microsoft\WindowsApps;\\ts3smbdrwest.file.core.windows.net\sapmnt\T... RSEC_SSFS_DATAPATH \\ts3smbdrwest.file.core.windows.net\sapmnt\TS3\SYS\global\security\rsecssfs\data RSEC_SSFS_KEYPATH \\ts3smbdrwest.file.core.windows.net\sapmnt\TS3\SYS\global\security\rsecssfs\key SAPEXE \\ts3smbdrwest.file.core.windows.net\sapmnt\TS3\SYS\exe\uc\NTAMD64​
  8. If you have created standard load balancer on the DR region beforehand with empty backend pool. Add ASCS/ERS VMs in the backend pool.
    Backend pool configuration in DR load balancerBackend pool configuration in DR load balancer
  9. [A] Update the IP address for ASCS and ERS server name resource configured in cluster to the Frontend IP configured in the load balancer (the one provisioned in disaster recovery region).
    ASCS server name - IP change in clusterASCS server name - IP change in clusterERS server name - IP change in clusterERS server name - IP change in cluster
  10. [A] Change quorum to the cloud witness storage account created on disaster recovery region.
    Update cloud witness by changing storage name in quorum settingsUpdate cloud witness by changing storage name in quorum settingsUpdate cloud witness with DR storage account name and secret keyUpdate cloud witness with DR storage account name and secret key
  11. [A] Start ASCS and ERS cluster role in failover cluster manager.
    Start ASCS and ERS cluster roleStart ASCS and ERS cluster role
  12. [C] Update the user store in all dialog instance servers with correct database hostname that is running on DR region. Check SAP Note 1852017 to get more insight on how to update hdbuserstore on Windows.
  13. [C] Start all dialog instances.
    Status of all instancesStatus of all instances

Failback to former primary region

Once the service on primary region is available, and you have scheduled failback of your production landscape back to the primary region.

 

  1. Re-protect failed over Azure VMs to the primary region. Refer to the document for more details.
  2. Schedule or run Robocopy command to copy data from SMB on Azure files on Disaster Region to Primary Region. 
  3. On the event of failure, follow the same post steps described above. 

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.