SAP on Azure General Update – November 2020

This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Tech Community.

1. Update to SAP on Azure Documentation

Several important updates to Azure documentation have been made recently.

Customers and Partners are recommended to regularly review the SAP on Azure documentation pages as new features and configurations are continuously improved.

The main SAP on Azure site https://azure.microsoft.com/en-us/solutions/sap/

SAP on Azure Resources https://azure.microsoft.com/en-us/solutions/sap/resources/

SAP on Azure Updates on the main Azure site https://azure.microsoft.com/en-us/updates/?query=sap

SAP on Azure Documentation “Getting Started”  https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/get-started

 

Important Update for Linux High Availability customers – set the Linux kernel parameter net.ipv4.tcp_keepalive_time=300.  It is recommended to set this parameter on DB, Application Servers and Central Services VMs. This now aligns with SAP Note: 1410736 - TCP/IP: setting keepalive interval

https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/high-availability-guide-suse

RHEL 8.1 is now supported on Azure VMs https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/high-availability-guide-rhel-pacemaker

 

The Microsoft documentation has been changed to recommend using the azure-lb resource agent. This is further explained in this SAP Note

2922194 - Linux Utility NetCat Running SUSE Pacemaker Stops Responding https://launchpad.support.sap.com/#/notes/0002922194

HA Scenarios are documented here https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/sap-high-availability-architecture-scenarios

 

Updates to Storage Configurations for Hana and AnyDB databases.  Note some updates to blocksizes, host cache settings and new UltraDisk configurations

Considerations for Azure Virtual Machines DBMS deployment for SAP workload - Azure Virtual Machines | Microsoft Docs  - https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/dbms_guide_general

 

SQL Server Azure Virtual Machines DBMS deployment for SAP workload - Azure Virtual Machines | Microsoft Docs https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/dbms_guide_sqlserver

Oracle Azure Virtual Machines DBMS deployment for SAP workload - Azure Virtual Machines | Microsoft Docs https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/dbms_guide_oracle

IBM Db2 Azure Virtual Machines DBMS deployment for SAP workload - Azure Virtual Machines | Microsoft Docs https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/dbms_guide_ibm

SAP HANA Azure virtual machine storage configurations https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/hana-vm-operations-storage

 

SAP documentation on storage configuration can be found here:

HANA - https://help.sap.com/viewer/2c1988d620e04368aa4103bf26f17727/2.0.05/en-US/4c24d332a37b4a3caad3e634f9900a45.html

SAP ASE (Section 3.7) - https://help.sap.com/doc/4f95c9e3741a1014955595407d8604de/CURRENT_VERSION/en-US/Inst_nw7x_unix_ase_abap.pdf

Oracle - https://help.sap.com/viewer/4b99f675d74f4990b75a8630869a0cd2/CURRENT_VERSION/en-US/9c578484622840bea589ff0eaf1ffa7a.html

DB2 - https://help.sap.com/viewer/4fbd902c7c76410bb82c6311dd4dc94b/CURRENT_VERSION/en-US/713eb64f45c6448c8dbe8a51b85680ee.html

 

2. Azure Linux OMS Agent Now Supports Python3

Older versions of the Azure Linux OMS Agent did not run or deploy after the Python release was updated to Python3 using the package python3-azure-mgmt-compute. 

This issue is now resolved and the Linux OMS Agent now supports modern Python releases on Suse 12.x and Redhat.

Customers who are running Suse 15.1 will still be unable to use the full functionality as the Dependency Agent is not released for Suse 15.1 or Oracle Linux.

https://docs.microsoft.com/en-us/azure/azure-monitor/platform/agents-overview#supported-operating-systems

The Dependency Agent will stay in “Transitioning” status and the only data populated into Log Analytics will the in the “Heartbeat” and “InsightsMetrics” tables.  Check this blog site regularly for updates on the Dependency Agent for Suse 15.1

Cameron_MSFT_SAP_PM_0-1605218259776.png

 

3. High LOGWRITE Waitstat on SQL Server TDE Databases Due to Certificate Revocation List Not Accessible    

The majority of customers running SAP on SQL Server on Azure are encrypting database using SQL Server TDE AES-256.  An important feature of the SQL TDE mechanism is the Certificate Revocation List https://docs.microsoft.com/en-us/sql/relational-databases/security/encryption/encryption-hierarchy?view=sql-server-ver15

 

Several support cases have been analyzed recently where Firewall policies have blocked access to network resources that are checked by the Certificate Revocation mechanism.

The problem manifests as intermittent slow LOGWRITE and on AlwaysOn systems a the command TM REQUEST may be waiting on HADR_SYNC_COMMIT.  The problem occurs on a regular periodic basis.  SQL Server will run very slowly for approximately 180 seconds and them return to normal

 

There are two solutions to this problem:

  1. Turn off Certification Revocation (not recommended)
  2. Ensure that the Firewall is configured to allow access to the addresses in the whitelist in this link

https://docs.microsoft.com/en-us/azure/app-service/environment/firewall-integration

4. Recommended Blogs for SAP on Azure Customers & Consultants  

Many new useful blogs have been created by Microsoft for SAP customers

SAP on Azure: Load Balancing Web Application Servers for SAP BOBI using Azure Application Gateway

https://blogs.sap.com/2020/09/17/sap-on-azure-load-balancing-web-application-servers-for-sap-bobi-using-azure-application-gateway/

SAP on Azure: Tomcat Clustering using Static Membership for SAP BusinessObjects BI Platform

https://blogs.sap.com/2020/09/04/sap-on-azure-tomcat-clustering-using-static-membership-for-sap-businessobjects-bi-platform/

Four Node AlwaysOn Cluster Across Azure Regions : https://blogs.sap.com/2020/10/20/sap-on-azure-sap-netweaver-7.5-on-ms-sql-server-2019-high-availability-and-disaster-recovery-with-4-nodes-alwayson-cluster/

SAP On Azure : HIGH AVAILIABILITY setup for SAP NETWEAVER with SAP ASE 16 DB on WINDOWS SERVER https://blogs.sap.com/2020/04/27/sap-on-azure-high-availiability-setup-for-sap-netweaver-with-sap-ase-16-db-on-windows-server/

SAP HANA HSR Multi-tier Longer Chains on Azure Geographic Clusters.  Tertiary solutions are explained by Apparao in this blog

https://www.linkedin.com/pulse/sap-hana-hsr-multi-tier-longer-chains-azure-geographic-apparao-sanam/

Oracle customers can use this new upcoming blog series to setup and configure Oracle Linux 8.2, Oracle DB 19.8 with ASM and Dataguard.

https://techcommunity.microsoft.com/t5/running-sap-applications-on-the/sap-on-oracle-setup-on-azure-part1/ba-p/1865024

End to End Monitoring on the internal Microsoft SAP system.  This includes the ability to read events such as ST22 shortdumps and correlate these events with infrastructure availability

https://www.microsoft.com/en-us/itshowcase/monitoring-sap-end-to-end-on-azure

5. Oracle Linux & PTP Timer – Especially for Oracle ASM Systems  

Oracle Linux customers may receive errors in Oracle error logs similar to the text below.  If this is seen it is recommended to edit the chrony.conf file and add the PTP timer

 

Warning: VKTM detected a forward time drift.

Time drifts can result in unexpected behavior such as time-outs.

Please see the VKTM trace file for more details

 

Check chrony configuration with the following command

chronyc sources -v

In chrony.conf, add line below,

refclock PHC /dev/ptp0 poll 3 dpoll -2 offset 0

 And restart the chronyd, systemctl restart chronyd.service

 [root@oracle77 ~]#  chronyc sources -v

210 Number of sources = 5

 

  .-- Source mode  '^' = server, '=' = peer, '#' = local clock.

/ .- Source state '*' = current synced, '+' = combined , '-' = not combined,

| /   '?' = unreachable, 'x' = time may be in error, '~' = time too variable.

||                                                 .- xxxx [ yyyy ] +/- zzzz

||      Reachability register (octal) -.           |  xxxx = adjusted offset,

||      Log2(Polling interval) --.      |          |  yyyy = measured offset,

||                                \     |          |  zzzz = estimated error.

||                                 |    |           \

MS Name/IP address         Stratum Poll Reach LastRx Last sample

===============================================================================

#* PHC0                          0   3   377    12  +5187ns[+8446ns] +/- 1124ns ßLocal PTP clock source is added

^- undefined.hostname.local>     2   6    77    57   -251us[ -236us] +/-   81ms

^- tick.hk.china.logiplex.n>     2   6    77    55   -884us[ -887us] +/-   63ms

^? stratum2-01.hkg01.public>     2   7     1    46   -822us[ -827us] +/-   54ms

^- time.cloudflare.com           3   6    77    55  -2205us[-2208us] +/-   43ms

***************************************************************************************

6. Azure CLI Commands After Python3 Update on Suse

After updating to the latest Python3 release Azure CLI may stop working.  The error message will be similar to “No module named azure.cli.__mail___; ‘azure.cli’ is a package and cannot be directly executed”

The message appears after updating to the latest Python/Python3 libraries and the package for the Azure SDK (python-azure-mgmt-compute on SLES12 ; python3-azure-mgmt-compute on SLES15)

 

Follow this procedure to resolve the problem on Suse 15.1:

  1. sudo zypper install --oldpackage azure-cli-2.0.45-4.22.noarch
  2. sudo zypper rm -y --clean-deps azure-cli
  3. Follow the standard installation procedure as detailed here https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-zypper?view=azure-cli-latest

For more information review  https://github.com/Azure/azure-cli/issues/13209#issuecomment-652164784

 

On Suse 12.x the problem presents differently

# az login
Traceback (most recent call last):
File "/usr/lib64/python3.4/runpy.py", line 170, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib64/python3.4/runpy.py", line 85, in _run_code
exec(code, run_globals)

 

Follow this procedure to resolve the problem on Suse 12.x:

  1. Run this command to check the Python3 path

which python3
/usr/local/bin/python3

  1. If the python3 path is /usr/local/bin/python3 then run this command

sudo ln -sf /usr/local/bin/python3 /usr/bin/python3

  1. Run az self-test  

7. SAP License Key May Become Invalid on Azure VMs

Some customers may notice that the SAP License Key on their systems may become invalid and SAP starts running on a temporary license key.  This issue is explained in these SAP Notes. 

https://launchpad.support.sap.com/#/notes/2937144 2937144 - Linux on Microsoft Azure: Upgrade to RHEL 8, SLES 12 SP5 or SLES 15 SP1 results in invalid SAP license

2975682 - Azure – SAP license invalid after reboot of VM due to changed hardware key with Linux guest OS https://launchpad.support.sap.com/#/notes/0002975682

https://launchpad.support.sap.com/#/notes/2243692    2243692 - Linux on Microsoft Azure (IaaS) VM: SAP license issues

8. SAP Note for Proximity Placement Groups      

A new SAP Note discusses how to reduce latency between Database and SAP Application servers.  Note 2931465 includes instructions on how to run /SSA/CAT.  This utility is the most reliable and best way to measure latency between the database and application server.  Ideally the result in the Acc DB and E. Acc DB columns should be below 100. 

Only the Database and Applications servers for a single SID should be in PPG.  The ASCS does not need to be in PPG.  PPG should be kept as small and compact as possible.  The communication between the SAP Application and the ASCS is not highly latency sensitive

2931465 - Reduce network latency (RTT) using Proximity placement groups on Microsoft Azure https://launchpad.support.sap.com/#/notes/2931465

https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/sap-proximity-placement-scenarios

9. Update on Support Matrix for SAP on Azure  

In recent months many new features have become available for SAP customers.  The list below is a very brief overview of recommended features and updated documentation

  1. RHEL 8.1 – is released and Generally Available for Azure  
  2. Suse 15.2 is now certified for Netweaver and Hana
  3. Recommended stack for Oracle Customers – Oracle Linux OL 8.2 + Oracle 19.8c + Grid + ASM.  ASM is highly recommended for all new Oracle systems.  Oracle 18 is not recommended and will be EOL soon
  4. Windows 2019 – fully supported for NetWeaver and most standalone SAP components.  Hyper-V support matrix can be found here
  5. Azure Backup for Hana now supports RHEL https://docs.microsoft.com/en-us/azure/backup/sap-hana-backup-support-matrix  
  6. Azure now supports shared disks https://docs.microsoft.com/en-us/azure/virtual-machines/disks-shared-enable?tabs=azure-cli
  7. A new page documents HA scenarios including using Azure Shared Disks https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/sap-high-availability-architecture-scenarios

10. Saptune Version 2 – Update to Saptune 2 Recommended

Suse has updated Saptune for both Suse 12.x and 15.x

Run the command rpm -qa | grep -i saptune to check the version of Saptune

The blog below explains how to update and migrate to Saptune 2.  Saptune 2 contains important updates to improve supportabilty and stability.

https://blogs.sap.com/2019/12/16/suse-lets-migrate-saptune-to-version-2/

https://blogs.sap.com/2019/05/03/a-new-saptune-is-knocking-on-your-door/

11. Azure Monitor for SAP

Azure Monitor for SAP is now in preview and supports VMs and HLI.  Hana and SQL Server Databases are supported currently.  Telemtry from Pacemaker Cluster is also collected and displayed.  Azure Monitor for SAP is a free of charge service

A very small collector VM is required.  Further documentation can be found here

https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/azure-monitor-overview

12. SAP on Azure – Customer Success Stories   

Cathay Pacific has recently completed moving their SAP S4HANA system to Azure M-series.  SAP application servers are running on E-series.  As part of the project S4HANA was upgraded to 1809.

 

Orica has completed their global S4HANA global transformation with more than 50 countries on a single instance

Azure Load Balancer - switch from Netcat to more reliable Socat was completed this in the Q1’20.

SAP HANA upgrade to SPS 5 in progress 

SuSE 12 Support Pack 5 upgrade in progress 

Gen2 VM Migration – S/4HANA Database migrated to Gen 2 VM [M208ms_v2 (208 vcpus, 5700 GiB memory)] for both nodes in Cluster and in DR VM [E32s_v4 (32 vcpus, 256 GiB memory)].

Azure Application Proxy is used for Fiori launchpad and 1,400+ applications deployed.   Azure Backup is used for SQL Server and HANA,  Azure Site Recovery for DR and Azure Monitor for SAP Solutions for Production systems

More information can be found here: https://customers.microsoft.com/en-us/story/orica-mining-oil-gas-azure

13. Troubleshooting Checklist for Hana Performance Issues     

Customers experiencing performance or reliability issues with SAP Hana installations are highly recommended to follow the troubleshooting process.  The list below is a basic checklist that should be followed before raising a support case to Microsoft and SAP:

 

  1. Check the VM type used and verify Hana Certification https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN/iaas.html#categories=Microsoft%20Azure
  2. Check the Storage Configuration Guide has been followed https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/hana-vm-operations-storage
  3. Ensure the root disk is large enough.  Very small root disks can cause performance problems
  4. Check Write Accelerator is enabled on Log Disks
  5. Check host cache settings is as per the Storage Configuration Guide
  6. Check Accelerated Networking enabled https://docs.microsoft.com/en-us/azure/virtual-network/create-vm-accelerated-networking-cli
  7. Check PPG is configured and run /SSA/CAT as per 2931465 - Reduce network latency (RTT) using Proximity placement groups on Microsoft Azure https://launchpad.support.sap.com/#/notes/2931465
  8. Install and run the latest Minicheck – check the “C” column.  Any “X” in the C column should be investigated.

1999993 - How-To: Interpreting SAP HANA Mini Check Results https://launchpad.support.sap.com/#/notes/1999993

2600030 - Parameter Recommendations in SAP HANA Environments https://launchpad.support.sap.com/#/notes/2600030

14. R3load Based Hana System Copy – Tables >2 Billion Rows

SAP do not generally recommend copying Hana based systems with R3load and there are multiple issues that can occur when performing Homogeneous or Heterogenous system copies using R3load.

One issue is how to handle tables with more than 2 Billion rows.  Hana supports 2 Billion rows per table partition. 

2369936 - Overview of system copy options for systems based on SAP HANA

2784715 - SWPM: Using R3load-based migration for SAP System on HANA when tables contain more than 2 billion rows

15. Very Large Busy Hana VMs Freeze Due to Memory Exhaustion

Very large busy Hana VMs may sometimes freeze due to Linux kernel memory exhaustion.

The procedure below should be reviewed and discussed with the Linux vendor and implemented if appropriate

 

Steps to implement solution:

Set these kernel parametners

vm.min_free_kbytes = 65536

vm.zone_reclaim_mode = 0

 

(a) Get the current value of parameter "vm.min_free_kbytes"

[ root ] # sysctl vm.min_free_kbytes

(b) Increase minimal free memory and monitor for next 5 working days.

[ root ] # sysctl -w vm.zone_reclaim_mode=1

# sysctl -w vm.min_free_kbytes=Y

Example : # sysctl -w vm.min_free_kbytes=98304

(c) monitor in next 5 working days to see if message "Error 11 Resource temporarily unavailable" and/or

"Freeing unused kernel memory" still being reported.

[ root ] # egrep -i "Error 11|Resource temporarily unavailable|Freeing unused kernel memory" /var/log/messages

(d) If messages "Error 11 Resource temporarily unavailable" and/or

"Freeing unused kernel memory" are still reported, continue to increase another 50% ( value Z )

the value vm.min_free_kbytes using command "sysctl -w vm.min_free_kbytes"

[ root ] # sysctl -w vm.min_free_kbytes = Z

(e) If message "Error 11 Resource temporarily unavailable" and/or

"Freeing unused kernel memory" are/is no longer reported, make the

change settings permanently, add an appropriate line to the /etc/sysctl.conf

Refer: https://www.cyberciti.biz/faq/howto-set-sysctl-variables/

 

Information for Redhat 8.1 can be found here https://www.redhat.com/cms/managed-files/Handout%20Performance%20Analysis%20and%20Tuning%20Red%20Hat%20Enterprise%20Linux%202019.pdf

An additional note that should be reviewed on large Hana systems is listed below

1980196 - Setting Linux Kernel Parameter /proc/sys/vm/max_map_count on SAP HANA Systems

https://launchpad.support.sap.com/#/notes/1980196

Additional Links & Notes

Azure Files NFS 4.1 is now in Preview https://azure.microsoft.com/en-us/updates/azure-files-support-for-nfs-v41-is-now-in-preview/   Azure Files NFS removes the need for a highly available NFS VM infrastructure

 

SAP on Azure Free Online Training Course.  Exam AZ-120: Planning and Administering Microsoft Azure for SAP Workloads

https://docs.microsoft.com/en-us/learn/certifications/exams/az-120

A free Certification Exam offer is here https://docs.microsoft.com/en-us/learn/certifications/microsoft-build-cloud-skills-challenge-2020-free-certification-exam-offer

 

Older JVM have been desupported and are not Customer Specific Maintenance only.   2981029 - Desupport of platform combinations for SAP JVM 5 and 6 https://launchpad.support.sap.com/#/notes/2981029

 

When testing Suse Pacemaker cluster the following procedure is useful

Simulating a Cluster Network Failure  https://www.suse.com/support/kb/doc/?id=000018699 

 

New Azure Hana Large Instances are available https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/hana-available-skus

S224OM [4 socket, 9TB (3TB DRAM + 6TB Optane)] – OLTP Scale Up

S896 [16 socket, 12TB] – OLTP with Scale Up and Scale Out

S672 [12 socket, 9TB] – OLTP with Scale Up and Scale Out

S448 [8 socket, 6TB] – OLTP with Scale Up and Scale Out

S672M [12 socket, 18TB] – OLTP Scale Up

S448M [8 socket, 12TB] – OLTP Scale Up

 

Important News About SAP Kernels can be found here

https://wiki.scn.sap.com/wiki/display/SI/SAP+Kernel%3A+Important+News

2880246 - SAP Kernel 722 EX2: General Information and Usage  https://launchpad.support.sap.com/#/notes/0002880246

 

Important or Interesting SAP Notes

2950585 - SAP ASCS/ERS fails in Windows Failover Cluster  https://launchpad.support.sap.com/#/notes/0002950585

2944287 - How to harden SAP systems regarding NTLM relay exploits? https://launchpad.support.sap.com/#/notes/0002944287

2931465 - Reduce network latency (RTT) using Proximity placement groups on Microsoft Azure https://launchpad.support.sap.com/#/notes/0002931465

2890138 - AL11 Alias Mapping for Windows File share on SAP running on Linux https://launchpad.support.sap.com/#/notes/0002890138

2887797 - Permissions problems to access SAPMNT share on Windows  https://launchpad.support.sap.com/#/notes/0002887797

2937583 - SAP NW JAVA is not coming up after disabling TLS 1.0 and enabling TLS 1.2

https://launchpad.support.sap.com/#/notes/0002937583

2922820 - DBSL Support for SQL Server 2019 https://launchpad.support.sap.com/#/notes/0002922820

2917949 - Apply JDBC driver 8.2 https://launchpad.support.sap.com/#/notes/0002917949

2906652 - Deliver Microsoft JDBC Driver 8.2 https://launchpad.support.sap.com/#/notes/0002906652

 

A good presentation on Suse deployment for SAP

https://www.suse.com/c/trust-suse-for-maximizing-sap-system-availability-part-1-of-2/

 

Thanks to Sarah Young for providing these links on Azure Security

Top 10 Best Practices for Azure Security (documentation) – https://aka.ms/azuresecuritytop10

Top 10 Best Practices for Azure Security (video) - https://youtu.be/g0hgtxBDZVE

Microsoft Cloud Adoption Framework for Azure - https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/?WT.mc_id=modinfra-9720-socuff

Microsoft Security Documentation site - https://docs.microsoft.com/en-us/security/

Microsoft Cybersecurity Reference Architecture (MCRA) Slides - http://aka.ms/mcra

Microsoft’s lead security architect’s useful documents list – https://aka.ms/markslist

Azure Security Podcast – https://aka.ms/azsecpod

Azure Security Community Webinars - https://aka.ms/SecurityWebinars

 

REMEMBER: these articles are REPUBLISHED. Your best bet to get a reply is to follow the link at the top of the post to the ORIGINAL post! BUT you're more than welcome to start discussions here:

This site uses Akismet to reduce spam. Learn how your comment data is processed.