This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Tech Community.
1. Update to SAP on Azure Documentation
Several important updates to Azure documentation have been made recently.
Customers and Partners are recommended to regularly review the SAP on Azure documentation pages as new features and configurations are continuously improved.
The main SAP on Azure site https://azure.microsoft.com/en-us/solutions/sap/
SAP on Azure Resources https://azure.microsoft.com/en-us/solutions/sap/resources/
SAP on Azure Updates on the main Azure site https://azure.microsoft.com/en-us/updates/?query=sap
SAP on Azure Documentation “Getting Started” https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/get-started
Important Update for Linux High Availability customers – set the Linux kernel parameter net.ipv4.tcp_keepalive_time=300. It is recommended to set this parameter on DB, Application Servers and Central Services VMs. This now aligns with SAP Note: 1410736 - TCP/IP: setting keepalive interval
RHEL 8.1 is now supported on Azure VMs https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/high-availability-guide-rhel-pacemaker
The Microsoft documentation has been changed to recommend using the azure-lb resource agent. This is further explained in this SAP Note
2922194 - Linux Utility NetCat Running SUSE Pacemaker Stops Responding https://launchpad.support.sap.com/#/notes/0002922194
HA Scenarios are documented here https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/sap-high-availability-architecture-scenarios
Updates to Storage Configurations for Hana and AnyDB databases. Note some updates to blocksizes, host cache settings and new UltraDisk configurations
Considerations for Azure Virtual Machines DBMS deployment for SAP workload - Azure Virtual Machines | Microsoft Docs - https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/dbms_guide_general
SQL Server Azure Virtual Machines DBMS deployment for SAP workload - Azure Virtual Machines | Microsoft Docs https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/dbms_guide_sqlserver
IBM Db2 Azure Virtual Machines DBMS deployment for SAP workload - Azure Virtual Machines | Microsoft Docs https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/dbms_guide_ibm
SAP documentation on storage configuration can be found here:
2. Azure Linux OMS Agent Now Supports Python3
Older versions of the Azure Linux OMS Agent did not run or deploy after the Python release was updated to Python3 using the package python3-azure-mgmt-compute.
This issue is now resolved and the Linux OMS Agent now supports modern Python releases on Suse 12.x and Redhat.
Customers who are running Suse 15.1 will still be unable to use the full functionality as the Dependency Agent is not released for Suse 15.1 or Oracle Linux.
The Dependency Agent will stay in “Transitioning” status and the only data populated into Log Analytics will the in the “Heartbeat” and “InsightsMetrics” tables. Check this blog site regularly for updates on the Dependency Agent for Suse 15.1
3. High LOGWRITE Waitstat on SQL Server TDE Databases Due to Certificate Revocation List Not Accessible
The majority of customers running SAP on SQL Server on Azure are encrypting database using SQL Server TDE AES-256. An important feature of the SQL TDE mechanism is the Certificate Revocation List https://docs.microsoft.com/en-us/sql/relational-databases/security/encryption/encryption-hierarchy?view=sql-server-ver15
Several support cases have been analyzed recently where Firewall policies have blocked access to network resources that are checked by the Certificate Revocation mechanism.
The problem manifests as intermittent slow LOGWRITE and on AlwaysOn systems a the command TM REQUEST may be waiting on HADR_SYNC_COMMIT. The problem occurs on a regular periodic basis. SQL Server will run very slowly for approximately 180 seconds and them return to normal
There are two solutions to this problem:
- Turn off Certification Revocation (not recommended)
- Ensure that the Firewall is configured to allow access to the addresses in the whitelist in this link
4. Recommended Blogs for SAP on Azure Customers & Consultants
Many new useful blogs have been created by Microsoft for SAP customers
SAP on Azure: Load Balancing Web Application Servers for SAP BOBI using Azure Application Gateway
SAP on Azure: Tomcat Clustering using Static Membership for SAP BusinessObjects BI Platform
Four Node AlwaysOn Cluster Across Azure Regions : https://blogs.sap.com/2020/10/20/sap-on-azure-sap-netweaver-7.5-on-ms-sql-server-2019-high-availability-and-disaster-recovery-with-4-nodes-alwayson-cluster/
SAP On Azure : HIGH AVAILIABILITY setup for SAP NETWEAVER with SAP ASE 16 DB on WINDOWS SERVER https://blogs.sap.com/2020/04/27/sap-on-azure-high-availiability-setup-for-sap-netweaver-with-sap-ase-16-db-on-windows-server/
SAP HANA HSR Multi-tier Longer Chains on Azure Geographic Clusters. Tertiary solutions are explained by Apparao in this blog
Oracle customers can use this new upcoming blog series to setup and configure Oracle Linux 8.2, Oracle DB 19.8 with ASM and Dataguard.
End to End Monitoring on the internal Microsoft SAP system. This includes the ability to read events such as ST22 shortdumps and correlate these events with infrastructure availability
5. Oracle Linux & PTP Timer – Especially for Oracle ASM Systems
Oracle Linux customers may receive errors in Oracle error logs similar to the text below. If this is seen it is recommended to edit the chrony.conf file and add the PTP timer
Warning: VKTM detected a forward time drift.
Time drifts can result in unexpected behavior such as time-outs.
Please see the VKTM trace file for more details
Check chrony configuration with the following command
chronyc sources -v
In chrony.conf, add line below,
refclock PHC /dev/ptp0 poll 3 dpoll -2 offset 0
And restart the chronyd, systemctl restart chronyd.service
[root@oracle77 ~]# chronyc sources -v
210 Number of sources = 5
.-- Source mode '^' = server, '=' = peer, '#' = local clock.
/ .- Source state '*' = current synced, '+' = combined , '-' = not combined,
| / '?' = unreachable, 'x' = time may be in error, '~' = time too variable.
|| .- xxxx [ yyyy ] +/- zzzz
|| Reachability register (octal) -. | xxxx = adjusted offset,
|| Log2(Polling interval) --. | | yyyy = measured offset,
|| \ | | zzzz = estimated error.
|| | | \
MS Name/IP address Stratum Poll Reach LastRx Last sample
#* PHC0 0 3 377 12 +5187ns[+8446ns] +/- 1124ns ßLocal PTP clock source is added
^- undefined.hostname.local> 2 6 77 57 -251us[ -236us] +/- 81ms
^- tick.hk.china.logiplex.n> 2 6 77 55 -884us[ -887us] +/- 63ms
^? stratum2-01.hkg01.public> 2 7 1 46 -822us[ -827us] +/- 54ms
^- time.cloudflare.com 3 6 77 55 -2205us[-2208us] +/- 43ms
6. Azure CLI Commands After Python3 Update on Suse
After updating to the latest Python3 release Azure CLI may stop working. The error message will be similar to “No module named azure.cli.__mail___; ‘azure.cli’ is a package and cannot be directly executed”
The message appears after updating to the latest Python/Python3 libraries and the package for the Azure SDK (python-azure-mgmt-compute on SLES12 ; python3-azure-mgmt-compute on SLES15)
Follow this procedure to resolve the problem on Suse 15.1:
- sudo zypper install --oldpackage azure-cli-2.0.45-4.22.noarch
- sudo zypper rm -y --clean-deps azure-cli
- Follow the standard installation procedure as detailed here https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-zypper?view=azure-cli-latest
For more information review https://github.com/Azure/azure-cli/issues/13209#issuecomment-652164784
On Suse 12.x the problem presents differently
# az login
Traceback (most recent call last):
File "/usr/lib64/python3.4/runpy.py", line 170, in _run_module_as_main
File "/usr/lib64/python3.4/runpy.py", line 85, in _run_code
Follow this procedure to resolve the problem on Suse 12.x:
- Run this command to check the Python3 path
- If the python3 path is /usr/local/bin/python3 then run this command
sudo ln -sf /usr/local/bin/python3 /usr/bin/python3
- Run az self-test
7. SAP License Key May Become Invalid on Azure VMs
Some customers may notice that the SAP License Key on their systems may become invalid and SAP starts running on a temporary license key. This issue is explained in these SAP Notes.
https://launchpad.support.sap.com/#/notes/2937144 2937144 - Linux on Microsoft Azure: Upgrade to RHEL 8, SLES 12 SP5 or SLES 15 SP1 results in invalid SAP license
2975682 - Azure – SAP license invalid after reboot of VM due to changed hardware key with Linux guest OS https://launchpad.support.sap.com/#/notes/0002975682
https://launchpad.support.sap.com/#/notes/2243692 2243692 - Linux on Microsoft Azure (IaaS) VM: SAP license issues
8. SAP Note for Proximity Placement Groups
A new SAP Note discusses how to reduce latency between Database and SAP Application servers. Note 2931465 includes instructions on how to run /SSA/CAT. This utility is the most reliable and best way to measure latency between the database and application server. Ideally the result in the Acc DB and E. Acc DB columns should be below 100.
Only the Database and Applications servers for a single SID should be in PPG. The ASCS does not need to be in PPG. PPG should be kept as small and compact as possible. The communication between the SAP Application and the ASCS is not highly latency sensitive
2931465 - Reduce network latency (RTT) using Proximity placement groups on Microsoft Azure https://launchpad.support.sap.com/#/notes/2931465
9. Update on Support Matrix for SAP on Azure
In recent months many new features have become available for SAP customers. The list below is a very brief overview of recommended features and updated documentation
- RHEL 8.1 – is released and Generally Available for Azure
- Suse 15.2 is now certified for Netweaver and Hana
- Recommended stack for Oracle Customers – Oracle Linux OL 8.2 + Oracle 19.8c + Grid + ASM. ASM is highly recommended for all new Oracle systems. Oracle 18 is not recommended and will be EOL soon
- Windows 2019 – fully supported for NetWeaver and most standalone SAP components. Hyper-V support matrix can be found here
- Azure Backup for Hana now supports RHEL https://docs.microsoft.com/en-us/azure/backup/sap-hana-backup-support-matrix
- Azure now supports shared disks https://docs.microsoft.com/en-us/azure/virtual-machines/disks-shared-enable?tabs=azure-cli
- A new page documents HA scenarios including using Azure Shared Disks https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/sap-high-availability-architecture-scenarios
10. Saptune Version 2 – Update to Saptune 2 Recommended
Suse has updated Saptune for both Suse 12.x and 15.x
Run the command rpm -qa | grep -i saptune to check the version of Saptune
The blog below explains how to update and migrate to Saptune 2. Saptune 2 contains important updates to improve supportabilty and stability.
11. Azure Monitor for SAP
Azure Monitor for SAP is now in preview and supports VMs and HLI. Hana and SQL Server Databases are supported currently. Telemtry from Pacemaker Cluster is also collected and displayed. Azure Monitor for SAP is a free of charge service
A very small collector VM is required. Further documentation can be found here
12. SAP on Azure – Customer Success Stories
Cathay Pacific has recently completed moving their SAP S4HANA system to Azure M-series. SAP application servers are running on E-series. As part of the project S4HANA was upgraded to 1809.
Orica has completed their global S4HANA global transformation with more than 50 countries on a single instance
Azure Load Balancer - switch from Netcat to more reliable Socat was completed this in the Q1’20.
SAP HANA upgrade to SPS 5 in progress
SuSE 12 Support Pack 5 upgrade in progress
Gen2 VM Migration – S/4HANA Database migrated to Gen 2 VM [M208ms_v2 (208 vcpus, 5700 GiB memory)] for both nodes in Cluster and in DR VM [E32s_v4 (32 vcpus, 256 GiB memory)].
Azure Application Proxy is used for Fiori launchpad and 1,400+ applications deployed. Azure Backup is used for SQL Server and HANA, Azure Site Recovery for DR and Azure Monitor for SAP Solutions for Production systems
More information can be found here: https://customers.microsoft.com/en-us/story/orica-mining-oil-gas-azure
13. Troubleshooting Checklist for Hana Performance Issues
Customers experiencing performance or reliability issues with SAP Hana installations are highly recommended to follow the troubleshooting process. The list below is a basic checklist that should be followed before raising a support case to Microsoft and SAP:
- Check the VM type used and verify Hana Certification https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN/iaas.html#categories=Microsoft%20Azure
- Check the Storage Configuration Guide has been followed https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/hana-vm-operations-storage
- Ensure the root disk is large enough. Very small root disks can cause performance problems
- Check Write Accelerator is enabled on Log Disks
- Check host cache settings is as per the Storage Configuration Guide
- Check Accelerated Networking enabled https://docs.microsoft.com/en-us/azure/virtual-network/create-vm-accelerated-networking-cli
- Check PPG is configured and run /SSA/CAT as per 2931465 - Reduce network latency (RTT) using Proximity placement groups on Microsoft Azure https://launchpad.support.sap.com/#/notes/2931465
- Install and run the latest Minicheck – check the “C” column. Any “X” in the C column should be investigated.
1999993 - How-To: Interpreting SAP HANA Mini Check Results https://launchpad.support.sap.com/#/notes/1999993
2600030 - Parameter Recommendations in SAP HANA Environments https://launchpad.support.sap.com/#/notes/2600030
14. R3load Based Hana System Copy – Tables >2 Billion Rows
SAP do not generally recommend copying Hana based systems with R3load and there are multiple issues that can occur when performing Homogeneous or Heterogenous system copies using R3load.
One issue is how to handle tables with more than 2 Billion rows. Hana supports 2 Billion rows per table partition.
2369936 - Overview of system copy options for systems based on SAP HANA
2784715 - SWPM: Using R3load-based migration for SAP System on HANA when tables contain more than 2 billion rows
15. Very Large Busy Hana VMs Freeze Due to Memory Exhaustion
Very large busy Hana VMs may sometimes freeze due to Linux kernel memory exhaustion.
The procedure below should be reviewed and discussed with the Linux vendor and implemented if appropriate
Steps to implement solution:
Set these kernel parametners
vm.min_free_kbytes = 65536
vm.zone_reclaim_mode = 0
(a) Get the current value of parameter "vm.min_free_kbytes"
[ root ] # sysctl vm.min_free_kbytes
(b) Increase minimal free memory and monitor for next 5 working days.
[ root ] # sysctl -w vm.zone_reclaim_mode=1
# sysctl -w vm.min_free_kbytes=Y
Example : # sysctl -w vm.min_free_kbytes=98304
(c) monitor in next 5 working days to see if message "Error 11 Resource temporarily unavailable" and/or
"Freeing unused kernel memory" still being reported.
[ root ] # egrep -i "Error 11|Resource temporarily unavailable|Freeing unused kernel memory" /var/log/messages
(d) If messages "Error 11 Resource temporarily unavailable" and/or
"Freeing unused kernel memory" are still reported, continue to increase another 50% ( value Z )
the value vm.min_free_kbytes using command "sysctl -w vm.min_free_kbytes"
[ root ] # sysctl -w vm.min_free_kbytes = Z
(e) If message "Error 11 Resource temporarily unavailable" and/or
"Freeing unused kernel memory" are/is no longer reported, make the
change settings permanently, add an appropriate line to the /etc/sysctl.conf
Information for Redhat 8.1 can be found here https://www.redhat.com/cms/managed-files/Handout%20Performance%20Analysis%20and%20Tuning%20Red%20Hat%20Enterprise%20Linux%202019.pdf
An additional note that should be reviewed on large Hana systems is listed below
1980196 - Setting Linux Kernel Parameter /proc/sys/vm/max_map_count on SAP HANA Systems
Additional Links & Notes
Azure Files NFS 4.1 is now in Preview https://azure.microsoft.com/en-us/updates/azure-files-support-for-nfs-v41-is-now-in-preview/ Azure Files NFS removes the need for a highly available NFS VM infrastructure
SAP on Azure Free Online Training Course. Exam AZ-120: Planning and Administering Microsoft Azure for SAP Workloads
A free Certification Exam offer is here https://docs.microsoft.com/en-us/learn/certifications/microsoft-build-cloud-skills-challenge-2020-free-certification-exam-offer
Older JVM have been desupported and are not Customer Specific Maintenance only. 2981029 - Desupport of platform combinations for SAP JVM 5 and 6 https://launchpad.support.sap.com/#/notes/2981029
When testing Suse Pacemaker cluster the following procedure is useful
Simulating a Cluster Network Failure https://www.suse.com/support/kb/doc/?id=000018699
New Azure Hana Large Instances are available https://docs.microsoft.com/en-us/azure/virtual-machines/workloads/sap/hana-available-skus
S224OM [4 socket, 9TB (3TB DRAM + 6TB Optane)] – OLTP Scale Up
S896 [16 socket, 12TB] – OLTP with Scale Up and Scale Out
S672 [12 socket, 9TB] – OLTP with Scale Up and Scale Out
S448 [8 socket, 6TB] – OLTP with Scale Up and Scale Out
S672M [12 socket, 18TB] – OLTP Scale Up
S448M [8 socket, 12TB] – OLTP Scale Up
Important News About SAP Kernels can be found here
2880246 - SAP Kernel 722 EX2: General Information and Usage https://launchpad.support.sap.com/#/notes/0002880246
Important or Interesting SAP Notes
2950585 - SAP ASCS/ERS fails in Windows Failover Cluster https://launchpad.support.sap.com/#/notes/0002950585
2944287 - How to harden SAP systems regarding NTLM relay exploits? https://launchpad.support.sap.com/#/notes/0002944287
2931465 - Reduce network latency (RTT) using Proximity placement groups on Microsoft Azure https://launchpad.support.sap.com/#/notes/0002931465
2890138 - AL11 Alias Mapping for Windows File share on SAP running on Linux https://launchpad.support.sap.com/#/notes/0002890138
2887797 - Permissions problems to access SAPMNT share on Windows https://launchpad.support.sap.com/#/notes/0002887797
2937583 - SAP NW JAVA is not coming up after disabling TLS 1.0 and enabling TLS 1.2
2922820 - DBSL Support for SQL Server 2019 https://launchpad.support.sap.com/#/notes/0002922820
2917949 - Apply JDBC driver 8.2 https://launchpad.support.sap.com/#/notes/0002917949
2906652 - Deliver Microsoft JDBC Driver 8.2 https://launchpad.support.sap.com/#/notes/0002906652
A good presentation on Suse deployment for SAP
Thanks to Sarah Young for providing these links on Azure Security
Top 10 Best Practices for Azure Security (documentation) – https://aka.ms/azuresecuritytop10
Top 10 Best Practices for Azure Security (video) - https://youtu.be/g0hgtxBDZVE
Microsoft Cloud Adoption Framework for Azure - https://docs.microsoft.com/en-us/azure/cloud-adoption-framework/?WT.mc_id=modinfra-9720-socuff
Microsoft Security Documentation site - https://docs.microsoft.com/en-us/security/
Microsoft Cybersecurity Reference Architecture (MCRA) Slides - http://aka.ms/mcra
Microsoft’s lead security architect’s useful documents list – https://aka.ms/markslist
Azure Security Podcast – https://aka.ms/azsecpod
Azure Security Community Webinars - https://aka.ms/SecurityWebinars