SAP on Oracle ASM on Azure in 5 Easy Steps

Posted by

This post has been republished via RSS; it originally appeared at: Microsoft Tech Community - Latest Blogs - .

Step 1: Introduction

Customers moving SAP on Oracle systems to Azure are strongly advised to follow clear guidance from Oracle, Microsoft and SAP to move to Automatic Storage Management (ASM).  There are many performance, administration and support benefits with ASM.  Customers that have used a migration from on-prem to Azure as an opportunity to move to ASM have given consistent positive feedback that ASM is the best storage solution for Oracle. 

Oracle DBAs who are not familiar with ASM can follow this procedure to learn more about ASM.

For Oracle 19c and higher the SAP Basis team should ensure the Oracle DBA team move away from LVM and use ASM for any new installation or migration to Azure. 


Jan Klokkers – Oracle Corporate Vice President responsible for SAP:

“ASM is the default storage architecture for Oracle databases for many years. It is being used on Oracle Engineered Systems and for all RAC deployments on UNIX/Linux systems.

ASM is the only storage layer which has been optimized for Oracle databases.   Filesystems such as EXT4 and XFS have never been optimized for Oracle databases as they are designed for general purpose use and not for database workload.

Close to ASM comes only Azure NetApp Files with Direct NFS where Direct NFS has been designed to run an Oracle database more efficiently.

All other solutions are not robust and reliable for large databases and do not provide the throughput to run large databases (especially large SAP systems).”

 

 

Step 2: Installation Details & Inputs

The setup and configuration in this document is detailed below:

 

  • OS = Oracle Linux 8.6 Azure Gallery Image
  • DB = Oracle 19c Patch 16
  • SAP Release = NetWeaver 7.5 Application Server
  • SAP SID = C11
  • *Oracle DBMS ORACLE_SID=C11
  • DB_SID=C11
  • Oracle GRID ORACLE_BASE=/oracle/BASE
  • Oracle GRID ORACLE_HOME=/oracle/GRID/19.0.0
  • ORA_INVENTORY=/oracle/oraInventory
  • Oracle GRID ORACLE_SID = +ASM
  • Oracle DBMS ORACLE_HOME=/oracle/C11/19.0.0
  • Oracle DBMS ORACLE_BASE=/oracle/C11
  • Oracle ORACLE_SID=C11
  • Azure VM = E16v5
  • *the Oracle SID should be the same as the SAP SID, but it is permitted to be a different value

 

Experienced Oracle DBAs will note that the installation is using the same user “oracle” to own both the GRID and the DBMS instance.  The Oracle Home and Oracle Base values may also not conform to previous standards, however the values are correct.

 

Cameron_MSFT_SAP_PM_0-1669069477481.png

 

Disk Layout:

Oracle ASM can be used with Azure Premium SSD and Azure Premium SSDv2.  Oracle ASM is not supported on ANF.  Oracle ASM has not been tested with Azure UltraDisk.  In this example the newly introduced Azure Premium SSDv2 storage has been used. Premium SSDv2 has several advantages compared to Premium SSDv1 and can be very attractive for Oracle customers as well. The new Premium SSDv2 storage is supported by SAP to be used for SAP deployments. Including database deployments.

 

Cameron_MSFT_SAP_PM_1-1669069477485.jpeg

 

 

Step 3: Operating System Preparation for SAP on Oracle ASM on Azure

 

  1. Configure the /etc/hosts.  Enter the IP address and the hostname of the VM in the hosts file

sudo vi /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4

xx.xx.xx.xx   <<hostname>>.localdomain <<hostname>>

  1. Either open ports required for Oracle, SWPM and SAP, or alternatively stop and disable the firewall

sudo systemctl disable firewalld

sudo systemctl stop firewalld

 

  1. set the sysctl parameters

sudo vi /etc/sysctl.conf

 

net.ipv6.conf.all.disable_ipv6 = 1

net.core.somaxconn = 4096

net.core.netdev_max_backlog = 300000

net.ipv4.tcp_rmem = 4096 131072 16777216

net.ipv4.tcp_wmem = 4096 16384 16777216

net.ipv4.tcp_max_syn_backlog = 16348

net.core.rmem_max = 16777216

net.core.wmem_max = 16777216

net.ipv4.tcp_slow_start_after_idle = 0

net.ipv4.conf.all.rp_filter = 0

net.ipv4.tcp_dsack = 1

net.ipv4.tcp_sack = 1

net.ipv4.tcp_no_metrics_save = 1

net.ipv4.tcp_moderate_rcvbuf = 1

net.ipv4.tcp_window_scaling = 1

net.ipv4.tcp_syn_retries = 8

net.ipv4.tcp_tw_reuse = 1

net.ipv4.tcp_timestamps = 1

### added by the SAP installation

fs.file-max = 6815744

kernel.shmmni = 4096

kernel.shmall = 1073741824

kernel.shmmax = 4398046511104

kernel.panic_on_oops = 1

net.core.rmem_default = 262144

net.core.rmem_max = 16777216

net.core.wmem_default = 262144

net.core.wmem_max = 16777216

net.ipv4.conf.all.rp_filter = 2

net.ipv4.conf.default.rp_filter = 2

fs.aio-max-nr = 1048576

net.ipv4.ip_local_port_range = 9000 65500

kernel.sem= 32000 1024000000 500 32000

vm.max_map_count=2147483647

 

Activate the parameter

sudo sysctl -p /etc/sysctl.conf

 

 

  1. Disable selinux

sudo vi /etc/selinux/config

...

#     permissive - SELinux prints warnings instead of enforcing.

#     disabled - No SELinux policy is loaded.

SELINUX=disabled

 

  1. Set Huge Pages

Review the Microsoft DBMS Guide for Oracle and this link 7 easy steps to configure HugePages for your Oracle Database Server | CarajanDB

 

  1. Create a temporary installation user/password for using SWPM in dialog mode.  Most companies do not allow the use of root

As user root/admin run

sudo useradd <username>

sudo passwd <username>

sudo usermod -aG wheel <username>

sudo visudo (add user to sudoers)

 

  1. Increase swapfile to 20GB as per Oracle recommendation.  Follow SAP documentation for SAP application servers.  Server Configuration Checklist for Oracle Database Installation

As user with root access run

sudo vi /etc/waagent.conf

 

ResourceDisk.Format=y

Provisioning.Agent= waagent

 

# Set the property ResourceDisk.EnableSwap to y

# Create and use swapfile on resource disk.

ResourceDisk.EnableSwap=y

 

# Set the size of the SWAP file with property ResourceDisk.SwapSizeMB

#ResourceDisk.SwapSizeMB=2000

ResourceDisk.SwapSizeMB=20480

 

 

  1. Update the system and install required and useful packages. The entire list below can be copied and pasted into a terminal session.  This list will run for 5-10 minutes and then restart the OS

cd /tmp

### update all packages

sudo dnf update -y

 

### tuned packages

sudo dnf install -y tuned-profiles-oracle.noarch

 

### required packages

sudo dnf install -y oracle-database-preinstall-19c

sudo dnf install -y oracleasm

sudo dnf install -y binutils

sudo dnf install -y compat-libstdc++-33

sudo dnf install -y elfutils-libelf

sudo dnf install -y elfutils-libelf-devel

sudo dnf install -y fontconfig-devel

sudo dnf install -y glibc

sudo dnf install -y glibc-devel

sudo dnf install -y ksh

sudo dnf install -y libaio

sudo dnf install -y libaio-devel

sudo dnf install -y libXrender

sudo dnf install -y libXrender-devel

sudo dnf install -y libX11

sudo dnf install -y libXau

sudo dnf install -y libXi

sudo dnf install -y libXtst

sudo dnf install -y libgcc

sudo dnf install -y librdmacm-devel

sudo dnf install -y libstdc++

sudo dnf install -y libstdc++-devel

sudo dnf install -y unixODBC

 

### OL8 packages

sudo dnf install -y net-tools

sudo dnf install -y nfs-utils

sudo dnf install -y libnsl

sudo dnf install -y make

sudo dnf install -y targetcli

sudo dnf install -y smartmontools

sudo dnf install -y sysstat

sudo dnf install -y xorg-x11-server-Xorg  

sudo dnf install -y xorg-x11-xauth  

sudo dnf install -y libXaw.x86_64  

sudo dnf install -y xorg-x11-fonts-misc 

sudo dnf install -y uuidd 

 

###Install the X11 Tools (make it easier to test the X11 environment)

cd /tmp

sudo wget https://vault.centos.org/centos/8/PowerTools/x86_64/os/Packages/xorg-x11-apps-7.7-21.el8.x86_64.rpm 

sudo yum localinstall -y  xorg-x11-apps-7.7-21.el8.x86_64.rpm 

 

### packages required for SAP

sudo dnf install -y csh

sudo dnf install -y motif

sudo dnf install -y libXtst-devel

sudo dnf install -y libXp-devel

sudo dnf install -y libX11-devel

sudo dnf install -y libSM-devel

sudo dnf install -y libICE-devel

 

sudo reboot

 

  1. Start the Tuned Daemon and configure Tuned profile

https://docs.oracle.com/en/operating-systems/oracle-linux/8/monitoring/monitoring-WorkingWithTuned.html#topic_nkn_c2v_m5b

 

sudo tuned-adm list

Available profiles:

- accelerator-performance     - Throughput performance based tuning with disabled higher latency STOP states

- balanced                    - General non-specialized tuned profile

- desktop                     - Optimize for the desktop use-case

- hpc-compute                 - Optimize for HPC compute workloads

- intel-sst                   - Configure for Intel Speed Select Base Frequency

- latency-performance         - Optimize for deterministic performance at the cost of increased power consumption

- network-latency             - Optimize for deterministic performance at the cost of increased power consumption, focused on low latency network performance

- network-throughput          - Optimize for streaming network throughput, generally only necessary on older CPUs or 40G+ networks

- optimize-serial-console     - Optimize for serial console use.

- oracle                      - Optimize for Oracle RDBMS

- powersave                   - Optimize for low power consumption

- throughput-performance      - Broadly applicable tuning that provides excellent performance across a variety of common server workloads

- virtual-guest               - Optimize for running inside a virtual guest

- virtual-host                - Optimize for running KVM guests

 

sudo tuned-adm profile oracle throughput-performance virtual-guest

 

tuned-adm active

Current active profile: oracle throughput-performance virtual-guest

 

  1. Create the mountpoints for Oracle and SAP

Replace “C11” with the SAP SID and Oracle SID

sudo mkdir -p /oracle   /oracle/C11/sapbackup

sudo mkdir -p /usr/sap/C11 /sapmnt/C11 /usr/sap/trans /usr/sap/software

 

Create the LVM Physical Volumes, Logical Volumes and Volume Groups for the ORACLE_HOME and the SAP file systems

Check which disks are used buy the boot disks and identify disk(s) that can be used for the Oracle Home (/oracle), SAP and installation

lsblk

 

sudo pvcreate /dev/sda

sudo vgcreate software_vg /dev/sda

 

Create Logical Volumes.  LV can be checked with command lvscan

sudo lvcreate   -L 10G -n software software_vg

sudo lvcreate     -L 50G -n oracle_home software_vg

sudo lvcreate     -L 20G -n usr_sap software_vg

sudo lvcreate     -L 20G -n sapmnt software_vg

sudo lvcreate     -L 10G -n saptrans software_vg

sudo lvcreate     -l 100%FREE -n sapbackup software_vg

 

Create the xfs file systems

sudo mkfs -t xfs /dev/software_vg/software

sudo mkfs -t xfs /dev/software_vg/oracle_home

sudo mkfs -t xfs /dev/software_vg/usr_sap

sudo mkfs -t xfs /dev/software_vg/sapmnt

sudo mkfs -t xfs /dev/software_vg/saptrans

sudo mkfs -t xfs /dev/software_vg/sapbackup

 

Add these entries to the /etc/fstab

sudo vi /etc/fstab

 

#

/dev/software_vg/software  /usr/sap/software  xfs  relatime,inode64   0  0

/dev/software_vg/usr_sap   /usr/sap/C11       xfs  relatime,inode64   0  0

/dev/software_vg/sapmnt    /sapmnt            xfs  relatime,inode64   0  0

/dev/software_vg/saptrans  /usr/sap/trans     xfs  relatime,inode64   0  0

#

/dev/software_vg/oracle_home /oracle          xfs  relatime,inode64  0  0

/dev/software_vg/sapbackup /oracle/C11/sapbackup xfs  relatime,inode64 0  0

 

Mount the file systems.  The mounted filesystems should look similar to the below

sudo mount -a

df -h

Filesystem                           Size  Used Avail Use% Mounted on

devtmpfs                              63G     0   63G   0% /dev

tmpfs                                 63G     0   63G   0% /dev/shm

tmpfs                                 63G   25M   63G   1% /run

tmpfs                                 63G     0   63G   0% /sys/fs/cgroup

/dev/mapper/rootvg-rootlv             19G  3.3G   16G  18% /

/dev/mapper/rootvg-crashlv            10G  104M  9.9G   2% /var/crash

/dev/sda1                            794M  538M  257M  68% /boot

/dev/sda15                           495M  5.1M  490M   2% /boot/efi

/dev/sdb1                            590G   32K  560G   1% /mnt

tmpfs                                 13G     0   13G   0% /run/user/1000

/dev/mapper/software_vg-software      10G  104M  9.9G   2% /usr/sap/software

/dev/mapper/software_vg-usr_sap       20G  175M   20G   1% /usr/sap/C11

/dev/mapper/software_vg-sapmnt        20G  175M   20G   1% /sapmnt

/dev/mapper/software_vg-saptrans      10G  104M  9.9G   2% /usr/sap/trans

/dev/mapper/software_vg-oracle_home   40G  318M   40G   1% /oracle

/dev/mapper/software_vg-sapbackup     28G  232M   28G   1% /oracle/C11/sapbackup

 

Create directories under the ORACLE_HOME /oracle path

Replace “C11” with the SAP SID and Oracle SID

sudo mkdir -p /oracle/C11/sapreorg /oracle/C11/oraInventory /oracle/C11/saptrace /oracle/C11/19.0.0 /oracle/stage /oracle/client /oracle/C11/saparch /oracle/GRID/19.0.0 /oracle/BASE

 

Step 4: Install Oracle ASM Libraries & Configure ASM Disks

Note: If you are installing on OL lower than 8.6 you need to download the oracleasm-support RPM manually from the oracle website

ASM Lib download link Oracle ASMLib Downloads for Oracle Linux 8

ASM Support download link OL8: How To Install ASMLib (Doc ID 2720215.1)

cd /tmp

sudo wget https://download.oracle.com/otn_software/asmlib/oracleasmlib-2.0.17-1.el8.x86_64.rpm

sudo yum localinstall -y ./oracleasmlib-2.0.17-1.el8.x86_64.rpm

 

sudo dnf install -y oracleasm-support

 

Check if all ASM packages are installed.  Do not proceed until both packages are installed

rpm -qa |grep oracleasm

oracleasmlib-2.0.17-1.el8.x86_64

oracleasm-support-2.1.12-1.el8.x86_64

 

Prepare Oracle ASM

sudo chmod -R 775 /oracle

sudo chown -R oracle:oinstall /oracle

 

Setting up the Oracle ASM library driver involves defining the default user (oracle) and default group (oinstall) as well as configuring the drive to start on boot (choose y) and to scan for disks on boot (choose y). You need to answer the prompts from the following command:

sudo oracleasm configure -i

Default user to own the driver interface []: oracle

Default group to own the driver interface []: oinstall

Start Oracle ASM library driver on boot (y/n) [n]: y

Scan for Oracle ASM disks on boot (y/n) [y]: y

Writing Oracle ASM library driver configuration: done

 

You can also use the below command to configure oracleasm in non-interactive mode:

sudo oracleasm configure -u oracle -g oinstall -e -s y

 

Make sure the following entries are present in /etc/sysconfig/oracleasm

  • ORACLEASM_ENABLED=true
  • ORACLEASM_SCANBOOT=true

If the above parameters are not set, a reboot will undo the oracledisk group initialization.

 

if security/permission errors are seen on this step then the user/group ownership on /oracle and /dev/sd* are likely a problem

sudo oracleasm init

Creating /dev/oracleasm mount point: /dev/oracleasm

Loading module "oracleasm": oracleasm

Configuring "oracleasm" to use device physical block size

Mounting ASMlib driver filesystem: /dev/oracleasm

 

sudo systemctl enable oracleasm

This will ensure oracleasm is loaded as part of reboot.

 

Check the status if ASM is loaded successfully. Do not continue if ASM is not loaded

sudo oracleasm status

Checking if ASM is loaded: yes

Checking if /dev/oracleasm is mounted: yes

 

It is required to create partitions on disks for ASMLib as documented here https://docs.oracle.com/en/database/oracle/oracle-database/19/cwlin/configuring-disk-devices-to-use-oracle-asmlib.html#GUID-5464167D-B8D6-4204-8C43-8F1E56E0ACC0

 

Create new primary partitions on the disks that are to be used for ASM.  Use the command lsblk to determine which disk is used as a boot disk and which disks are already used by LVM

Check which disks are used with ASM with command lsblk. Ensure there is only one large partition on the disk used for creating ASM disk groups.

sudo fdisk /dev/sdd                 # 1024 DATA

sudo fdisk /dev/sde                 # 1024 DATA

sudo fdisk /dev/sdf                 # 1024 DATA

sudo fdisk /dev/sdg                 # 1024 DATA

sudo fdisk /dev/sdh                 # 512  OLOG

sudo fdisk /dev/sdi                 # 512  OLOG

sudo fdisk /dev/sdj                 # 3096 ARCH

sudo fdisk /dev/sdk                 # 2048 RECO

Answer the prompts from the above fdisk commands with:

n for new partition

for primary partition

press 3x enter for the partition number, default first cylinder and last cylinder

press to write the changes to the partition table

 

Change the permission on the disks that are to be used for ASM

Check which disks are used with ASM with command lsblk and adjust the commands below

sudo chmod -R 775 /oracle

sudo chown -R oracle:oinstall /oracle

 

sudo chmod 600 /dev/sdd1

sudo chmod 600 /dev/sde1

sudo chmod 600 /dev/sdf1

sudo chmod 600 /dev/sdg1

sudo chmod 600 /dev/sdh1

sudo chmod 600 /dev/sdi1

sudo chmod 600 /dev/sdj1

sudo chmod 600 /dev/sdk1

 

sudo chown oracle:oinstall /dev/sdd1

sudo chown oracle:oinstall /dev/sde1

sudo chown oracle:oinstall /dev/sdf1

sudo chown oracle:oinstall /dev/sdg1

sudo chown oracle:oinstall /dev/sdh1

sudo chown oracle:oinstall /dev/sdi1

sudo chown oracle:oinstall /dev/sdj1

sudo chown oracle:oinstall /dev/sdk1

 

 

Create ASM disks with the naming convention below.  Example DATA_<SID>_NN

Replace “C11” with the Oracle SID.  This clearly identifies the ASM disks on VMs with multiple instances

 

sudo oracleasm createdisk DATAC1101 /dev/sdd1

sudo oracleasm createdisk DATAC1102 /dev/sde1

sudo oracleasm createdisk DATAC1103 /dev/sdf1

sudo oracleasm createdisk DATAC1104 /dev/sdg1

sudo oracleasm createdisk ARCHC1101 /dev/sdh1

sudo oracleasm createdisk OLOGC1101 /dev/sdi1

sudo oracleasm createdisk OLOGC1102 /dev/sdj1

sudo oracleasm createdisk RECOC1101 /dev/sdk1

 

Verify the status of the ASM disks

sudo oracleasm listdisks

ARCHC1101

DATAC1101

DATAC1102

DATAC1103

DATAC1104

OLOGC1101

OLOGC1102

RECOC1101

 

Step 5: Install SAP ASCS, Database and SAP Application Server(s)

Download the required packages from the SAP marketplace

https://support.sap.com/swdc

The Oracle installation DVD and patches are located under “Databases”

The SAP Kernel required is located under: “Support Packages & Patches” à ”N” à “SAP Netweaver” à “SAP Netweaver 7.5” à “Application Server ABAP” à

 

Copy the SAP and Oracle software over from your desktop or jumpserver to /usr/sap/software.

 

The process flow for installing a NetWeaver 7.5 Distributed System is as follows:

  1. Create Azure VM and disk resources.
  2. Download required software.
  3. Follow the Operating System Preparation (Step #3 in this blog).
  4. Install ASM Libraries (Step #4).
  5. Install Oracle GRID on database server.
    1. Adapt the UID and GUID for oracle user and groups.
    2. Install GRID software with X11 Installation Tool.
  6. Install SAP ASCS, Database and SAP Application Server(s) (Step #5).
    1. Run sapinst and install ASCS. 
    2. Run sapinst and install SAP Database Instance.
      1.       When Database Installation pauses install Oracle DBMS.
      2.       Patch Oracle DBMS and GRID.
    3. Continue Database Installation with sapinst.
    4. Run sapinst and install SAP PAS/Application Server(s).
  7. Perform SAP post installation tasks.

 

1.    Install Oracle Grid with Oracle X11 Based Installer  

Oracle Grid software should be installed prior to starting the SAP Database Instance with SAPInst

 

  1. Unpack the DB client and the DB (as root)

cd /usr/sap/software/Oracle19/

ll

oracle oinstall  711317528 Oct 20 08:34 client.zip

oracle oinstall 2815340105 Oct 20 08:32 GIRU19P_2208-70004508.ZIP

oracle oinstall 2815340105 Oct 24 15:02 GIRU19P_2208-70004550.ZIP

oracle oinstall  124138770 Oct 20 08:29 OPATCH19P_2208-70004508.ZIP

oracle oinstall 9931078768 Oct 20 08:41 oracle19.zip

oracle oinstall     27092 Oct 21 06:08 sudo oracleasmlib-2.0.17-1.el8.x86_64.rpm

oracle oinstall      55698 Oct 20 08:30 README19P_2208-70004508.HTM

oracle oinstall   24265636 Oct 20 08:30 SAP19P_2208-70004508.ZIP

oracle oinstall         58 Oct 24 15:54 SAPBundle

oracle oinstall  128461935 Oct 24 15:56 SBPJDK19P_2208-70004508.ZIP

 

mkdir client DB OPatch SAPBundle GRIDupdate

cd client

unzip ../client.zip

...

 cd ../DB/

unzip ../oracle19.zip

...

 cd ../OPatch/

unzip ../OPATCH19P_2208-70004508.ZIP

...

 cd ../SAPBundle/

unzip ../SAP19P_2208-70004508.ZIP

...

 cd ../GRIDupdate

unzip ../GIRU19P_2208-70004550.ZIP

...

cd ..

sudo chmod -R 775 *

sudo chown -R oracle:oinstall *

 

  1.        X11 Setup & Testing

X11 requires software such as xming or mobaxterm.

Review this article and test X11 before continuing Enabling X11 forwarding on Linux | SAP Blogs

 

Enable the X11 forwarding (as root) – The X11 package xorg-x11-apps-7.7-21.el8.x86_64.rpm should automatically update this file. If X11 does not work check this parameter.

sudo vi /etc/ssh/sshd_config

ForwardX11 yes

 

Restart the sshd – only required if the sshd_config was changed.

sudo systemctl restart sshd

 

Login directly as user oracle without doing su - to keep the xterm settings from putty.

If it is not possible to logon directly using the oracle user, follow the procedure below to keep the X11 settings:

sudo cp /home/<azureuser>/.Xauthority /home/oracle/.Xauthority

sudo chown oracle:oinstall /home/oracle/.Xauthority

 

Test the sample X11 application xclock

ssh oracle@<host>

xauth list          #Test the X11 environment

xclock

Cameron_MSFT_SAP_PM_0-1669087077216.png

 

  1. Unzip the Grid archive in the GRID Oracle Home and run gridSetup.sh.  Grid must be unzipped into the final installation directory i.e

/oracle/GRID/19.0.0

cd /oracle/GRID/19.0.0

unzip /usr/sap/software/Oracle19/DB/LINUX_X86_64/grid_home/LINUX.X64_193000_grid_home.zip

 

Install the latest GRID - Cluster Verification Utility (as user with sudo access)

Installing the cvuqdisk RPM for Linux (oracle.com)

cd /oracle/GRID/19.0.0

sudo rpm -ivh ./cv/rpm/cvuqdisk-1.0.10-1.rpm

Verifying...                          ####################### [100%]

Preparing...                          ####################### [100%]

Using default group oinstall to install package

   1:cvuqdisk-1.0.10-1                ####################### [100%]

 

Create the asm groups asmadmin and asmoper

sudo groupadd -g 54327 asmadmin

sudo groupadd -g 54328 asmoper

sudo groupadd -g 54329 asmdba

sudo usermod -a -G asmdba oracle

sudo usermod -a -G asmadmin oracle

sudo usermod -a -G asmoper oracle

 

Set the user- and group ID for the installation software directory

cd /usr/sap/software/

sudo chown -R oracle:oinstall Oracle19

sudo chmod -R 755 Oracle19

 

Set the owner for the /dev/sudo oracleasm directory

sudo chown -R oracle:oinstall /dev/oracleasm

 

Logon directly as oracle (without doing su -) or prepare the .Xauthority file in /user/home/oracle as detailed above

export ORACLE_BASE=/oracle/BASE

export ORACLE_HOME=/oracle/GRID/19.0.0

export ORA_INVENTORY=/oracle/oraInventory

export CV_ASSUME_DISTID=OEL8.1

export DISPLAY=localhost:10.0

cd /oracle/GRID/19.0.0

./gridSetup.sh

Launching Oracle Grid Infrastructure Setup Wizard...

 

The gridSetup.sh summary screen should look like the screenshot below.  Detailed step by step screenshots are attached to this blog.

Cameron_MSFT_SAP_PM_1-1669087282629.png

 

  1. Run asmca to create OLOG, ARCH and RECO Disk Groups and Change Compatibility of DATA, OLOG, ARCH and RECO to 19.0.0.0.0.  Azure storage is already highly redundant and it is supported to use External redundancy.  Alternatively use ASM redundancy and/or Log Mirroring if required.

cd /oracle/GRID/19.0.0/bin

./asmca

 

Add the disk groups OLOG, RECO and ARCH

Right mouse click on “Disk Groups” and select “Create”

Cameron_MSFT_SAP_PM_2-1669087358342.png

 

Cameron_MSFT_SAP_PM_3-1669087358350.png

 

Edit the Disk and change DBMS release to 19.0.0.0 – Right click on disk group in asmca and “Edit Attributes“ -> “Database Compatibility“ = 19.0.0.0.0 

Cameron_MSFT_SAP_PM_4-1669087389514.png

2.    Install ASCS with SAPInst

Install the ASCS.  The screenshots attached to this blog show step by step installation of a NetWeaver 7.5 Distributed ASCS.

Select the desired installation type.  Step by step screenshots are attached to this blog showing installation of a NetWeaver 7.5 Distributed System.

 

Start the SWPM as root

cd /usr/sap/software/NW7.5/SWPM/

sudo ./sapinst SAPINST_REMOTE_ACCESS_USER= <username created in step 3.5>

[==============================] / extracting...  done!

INFO: Native locale set to en_US.UTF-8

INFO       2022-10-20 13:57:54.180 (mainThread) [sixxcreate.cpp:349]

********************************************************************************

Initial log directory: /root/.sapinst/ralforadb01/8791

********************************************************************************

 

 

SAPinst build information:

--------------------------

Version:         753.0.2

...

Open your browser and paste the following URL address to access the GUI

https://<<hostname.domain_name>>:4237/sapinst/docs/index.html

 

3.    Install SAP Database Instance with SAPInst

Start the SAP installer to install the SAP Database Instance (as root)

Detailed step by step screenshots of the Database Instance installation are attached to this blog

cd /usr/sap/software/NW7.5/SWPM

sudo ./sapinst SAPINST_REMOTE_ACCESS_USER= <username created in step 3.5>

 

Choose the following options during the installation:

  1. Oracle Database Installation for = Single Instance
  2. Filesystem or Oracle ASM = Oracle ASM
  3. Database Server Version = 19
  4. Database Client Version = 19
  5. Advanced DB Configuration – For Oracle Experts Only = tick
  6. Oracle Database Software Owner = oracle
  7. An existing database was detected and can be made available = Remove database and recreate
  8. Stop after step ‘createDB’ for patch update
  9. ORACLE_HOME for GRID = /oracle/GRID/19.0.0
  10. ORACLE_SID for GRID = +ASM
  11. Oracle GRID Software Not Installed = untick
  12. Oracle ASM Disk Group = add OLOG

Cameron_MSFT_SAP_PM_5-1669087516549.png

 

Cameron_MSFT_SAP_PM_6-1669087516569.png

 

Cameron_MSFT_SAP_PM_7-1669087516577.png

 

When SAPInst stops proceed to step 4 below

4.    Install Oracle Database Instance with RUNINSTALLER

 

Prepare the environment for the oracle user. Add the lines at the end of the bashrc.

Then logoff and relogin again.

su – oracle

vi .bashrc

...

export ORACLE_HOME=/oracle/C11/19.0.0

export ORACLE_SID=C11

export DB_SID=C11

export ORACLE_BASE=/oracle/C11

export ORA_INVENTORY=/oracle/C11/oraInventory

export DISPLAY=localhost:10.0

export CV_ASSUME_DISTID=OEL8.1

export TNS_ADMIN=$ORACLE_HOME/network/admin

 

logoff

exit

 

Change the owner of the oracle DB installation media (as root)

sudo chown -R oracle:oinstall /usr/sap/software/Oracle19

sudo chmod -R 755 /usr/sap/software/Oracle19

 

logon as oracle user (X11 enabled)

cd /usr/sap/software/Oracle19/DB/LINUX_X86_64/db_home/SAP

./RUNINSTALLER

 

Follow the inputs as per SAP note 2660017 to install Oracle.  Step by step screenshots are attached to this blog

 

Cameron_MSFT_SAP_PM_9-1669087618391.png

 

When RUNINSTALLER finishes, press continue in SAPInst. Do NOT do Oracle and GRID SBP patching immediately after installation of the Oracle DBMS

 

When SAPInst stops after CREATE DB it is then safe to apply patches or apply patches after SWPM has completely finished.

5.    Run OPatch to Patch GRID and Oracle DBMS

Oracle patching can either be done after the creation of the database in SAPInst or after the SAP Database Installation process has completely finished.   Either option will work.

 

Check the actual OPatch version as user = oracle

$ORACLE_HOME/OPatch/opatch version

OPatch Version: 12.2.0.1.17

 

OPatch update for DBMS Instance

Now update the OPatch executables as user = oracle

cd /usr/sap/software/Oracle19

unzip ../OPATCH19P_2208-70004508.ZIP

 

pwd

/usr/sap/software/Oracle19/Opatch/

 

mv /oracle/C11/19.0.0/OPatch /oracle/C11/19.0.0/OPatch-back

cd /usr/sap/software/Oracle19

 

cp -r OPatch /oracle/C11/19.0.0/

$ORACLE_HOME/OPatch/opatch version

========================================================

GENERIC OPATCH VERSION - FOR USE IN SAP ENVIRONMENT ONLY

========================================================

OPatch Version: 12.2.0.1.30

OPatch succeeded.

 

OPatch update for GRID

Update the OPatch as user = oracle

cd /usr/sap/software/Oracle19

unzip ../OPATCH19P_2208-70004508.ZIP

mv /oracle/GRID/19.0.0/OPatch /oracle/GRID/19.0.0/OPatch.backup

cp -r OPatch /oracle/GRID/19.0.0/

MOpatch update for DBMS instance

We extracted the SAP bundle patch earlier as user = oracle

cd /usr/sap/software/Oracle19/SAPBundle/SBP_191600220719_202208

 

cp -r MOPatch /oracle/C11/19.0.0/

 

MOpatch update for GRID

We extracted the SAP bundle patch earlier as user = oracle

cd /usr/sap/software/Oracle19/SAPBundle/SBP_191600220719_202208

 

cp -r MOPatch /oracle/GRID/19.0.0/

 

Install the SAP Bundle Patch

Both files are required in the Patch directory – as user = oracle

SAP19P_2208-70004508.ZIP

SBPJDK19P_2208-70004508.ZIP

This will take some time

cd /usr/sap/software/Oracle19

$ORACLE_HOME/MOPatch/mopatch.sh -v -s SAP19P_2208-70004508.ZIP

...

..

Log file: $ORACLE_HOME/cfgtoollogs/mopatch/mopatch-2022_10_24-16-00-36.log

MOPatch completed successfully.

Overall Status: COMPLETE, NON-SBP PATCHES LOST

 

 

Before you start the GRID patch verify the ORACLE_LISTS:

Display the ORACLE_HOME names, there must be two. One for the DB and one for GRID.

cat /oracle/oraInventory/ContentsXML/inventory.xml

<HOME_LIST>

<HOME NAME="OraDB19Home1" LOC="/oracle/OR1/19.0.0" TYPE="O" IDX="1"/>

 

if you see only one ORACLE_HOME_NAME create a second one. In my case for GRID (as oracle user)

cd $ORACLE_HOME/oui/bin

./runInstaller -silent -ignoreSysPrereqs -attachHome ORACLE_HOME=/oracle/GRID/19.0.0 ORACLE_HOME_NAME=OraGRID19Home1 CRS=true

 

List now the ORACLE_HOME list – you now have two entries

cat /oracle/oraInventory/ContentsXML/inventory.xml

<HOME_LIST>

<HOME NAME="OraDB19Home1" LOC="/oracle/OR1/19.0.0" TYPE="O" IDX="1"/>

<HOME NAME="OraGRID19Home1" LOC="/oracle/GRID/19.0.0" TYPE="O" IDX="2"

 

GRID update

Run as user with sudo access

export ORACLE_BASE=/oracle/BASE

export ORACLE_HOME=/oracle/GRID/19.0.0

export ORA_INVENTORY=/oracle/oraInventory

export OHGRID=/oracle/GRID/19.0.0

cd /oracle/GRID/19.0.0

sudo /oracle/GRID/19.0.0/crs/install/roothas.sh -prepatch

 

Now apply the GRID Patch as user = oracle

cd /usr/sap/software/Oracle19

 

$OHGRID/MOPatch/mopatch.sh -v -s SGR19P_2208-70004550.ZIP

 

==Following patches were SUCCESSFULLY applied:

 

Patch: /usr/sap/software/Oracle19/GRIDupdate/34130714/34160635

Log: /oracle/OR1/19.0.0/cfgtoollogs/opatchauto/core/opatch/opatch2022-10-24_16-52-31PM_1.log

 

OPatchauto session completed at Mon Oct 24 16:53:00 2022

Time taken to complete the session 1 minute, 10 seconds

 

Post patch activities as a user with sudo access

sudo /oracle/GRID/19.0.0/rdbms/install/rootadd_rdbms.sh

sudo /oracle/GRID/19.0.0/crs/install/roothas.sh -postpatch

. CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'

. CLSRSC-672: Post-patch steps for patching GI home successfully completed.

 

Once the patching is completed, press continue in SAPInst. 

6.    Installation of SAP Application Server(s)

Run SAPInst and install SAP Application Server(s).

Step by step screenshots can be found attached to this document

 

Appendix, Notes, Utilities and Links

 

1.      Azure Storage Types and Oracle DataGuard

With SSDv2 there is a support limitation for Oracle Data Guard. Oracle Data Guard does not support mixing different disk sector sizes. Azure Premium Storage v1 has a 512b sector size and Azure Premium Storage v2 is 4k Native.  It is not supported to have different disk sector sizes within a DataGuard configuration.

 

2.      ASMlib

 Document 2720215.1 (oracle.com) OL8: How To Install ASMLib (Doc ID 2720215.1) 

 

3.      Oracle Database Installation Selection Parameters for RUNINSTALLER

   

Cameron_MSFT_SAP_PM_10-1669088037401.png

 

4.  SAP Notes and Downloads

 

 

5.      Check ASM File Locations with ASMCMD

ASMCMD is a simple too that can be used to browse the ASM file system

export ORACLE_SID=+ASM

export DB_SID=OR1

export ORACLE_BASE=/oracle/BASE

export ORACLE_HOME=/oracle/GRID/19.0.0

 

[oracle@ol86asm18 bin]$ export ORACLE_HOME=/oracle/GRID/19.0.0

[oracle@ol86asm18 bin]$ export ORA_INVENTORY=/oracle/oraInventory

[oracle@ol86asm18 bin]$ export OHGRID=/oracle/GRID/19.0.0

[oracle@ol86asm18 bin]$ export ORACLE_SID=+ASM

[oracle@ol86asm18 bin]$ cd /oracle/GRID/19.0.0/bin

[oracle@ol86asm18 bin]$ ./sqlplus /NOLOG

 

SQL*Plus: Release 19.0.0.0.0 - Production on Sat Nov 19 19:32:13 2022

Version 19.3.0.0.0

 

Copyright (c) 1982, 2019, Oracle.  All rights reserved.

 

SQL> connect / as sysasm;

Connected to an idle instance.

SQL> startup

ASM instance started

 

Total System Global Area 1137173320 bytes

Fixed Size                  8905544 bytes

Variable Size            1103101952 bytes

ASM Cache                  25165824 bytes

ASM diskgroups mounted

SQL> exit

Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.3.0.0.0

 

[oracle@ol86asm18 bin]$ ./asmcmd

ASMCMD> ls -la

State    Type    Rebal  Name

MOUNTED  EXTERN  N      ARCH/

MOUNTED  EXTERN  N      DATA/

MOUNTED  EXTERN  N      OLOG/

MOUNTED  EXTERN  N      RECO/

ASMCMD> exit

 

6.      Oracle IO Calibrate

ORACLE-BASE - Measuring Storage Performance For Oracle Systems

Before we start the IO Calibrate we stop the SAP system

sapcontrol -nr 01 -function StopSystem

sapcontrol -nr 01 -function GetProcessList

 

name, description, dispstatus, textstatus, starttime, elapsedtime, pid

disp+work, Dispatcher, GRAY, Stopped, , , 1872566

igswd_mt, IGS Watchdog, GRAY, Stopped, , , 1872567

 

su – c11adm

sqlplus / as sysdba

 

SQL*Plus: Release 19.0.0.0.0 - Production on Wed Nov 2 10:51:56 2022

Version 19.16.0.0.0

 

Connected to:

Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.16.0.0.0

 

SQL>  SELECT INSTANCE_NAME, STATUS, DATABASE_STATUS FROM V$INSTANCE;

 

INSTANCE_NAME    STATUS       DATABASE_STATUS

---------------- ------------ -----------------

OR1              OPEN         ACTIVE

 

Enable the TIMED_STATISTICS and TIMED_OS_STATISTICS

ALTER SYSTEM SET TIMED_STATISTICS=true SCOPE=SPFILE;

ALTER SYSTEM SET TIMED_OS_STATISTICS=60 SCOPE=SPFILE;

*After the test you may want to set the TIMED_OS_STATISTICS back to 0

 

Check the ASYNC status

SQL> SELECT d.name,i.asynch_io FROM   v$datafile d,v$iostat_file I WHERE  d.file# = i.file_no AND i.filetype_name  = 'Data File'; 

 

NAME                                            ASYNCH_IO

----------------------------------------------------------       

+DATA/OR1/DATAFILE/system.262.1119616513         ASYNC_ON

+DATA/OR1/DATAFILE/sysaux.263.1119616515         ASYNC_ON

+DATA/OR1/DATAFILE/psapundo.264.1119616515       ASYNC_ON

+DATA/OR1/DATAFILE/psapsr3.267.1119617687        ASYNC_ON

+DATA/OR1/DATAFILE/psapsr3.268.1119617689        ASYNC_ON

+DATA/OR1/DATAFILE/psapsr3.269.1119617693        ASYNC_ON

+DATA/OR1/DATAFILE/psapsr3.270.1119617695        ASYNC_ON

+DATA/OR1/DATAFILE/psapsr3.271.1119617699        ASYNC_ON

+DATA/OR1/DATAFILE/psapsr3.272.1119617701        ASYNC_ON

+DATA/OR1/DATAFILE/psapsr3750.273.1119617703     ASYNC_ON

+DATA/OR1/DATAFILE/psapsr3750.274.1119617707     ASYNC_ON

+DATA/OR1/DATAFILE/psapsr3750.275.1119617709     ASYNC_ON

+DATA/OR1/DATAFILE/psapsr3750.276.1119617711     ASYNC_ON

+DATA/OR1/DATAFILE/psapsr3750.277.1119617715     ASYNC_ON

+DATA/OR1/DATAFILE/psapsr3usr.278.1119617715     ASYNC_ON

 

15 rows selected.

 

To turn on asynchronous I/O (if not already done, issue the following command and restart the database.

ALTER SYSTEM SET filesystemio_options=setall SCOPE=SPFILE;

 

Run the IO Calibrate test

SET SERVEROUTPUT ON

DECLARE

  l_latency  PLS_INTEGER;

  l_iops     PLS_INTEGER;

  l_mbps     PLS_INTEGER;

BEGIN

   DBMS_RESOURCE_MANAGER.calibrate_io (num_physical_disks => 1,

                                       max_latency        => 20,

                                       max_iops           => l_iops,

                                       max_mbps           => l_mbps,

                                       actual_latency     => l_latency);

 

  DBMS_OUTPUT.put_line('Max IOPS = ' || l_iops);

  DBMS_OUTPUT.put_line('Max MBPS = ' || l_mbps);

  DBMS_OUTPUT.put_line('Latency  = ' || l_latency);

END;

/

 

Result 3 DATA Disks

max_iops = 61745

latency  = 2.269

max_mbps = 3792

Max IOPS = 61745

Max MBPS = 3792

Latency  = 2

 

7.      Adding a New Azure Disk to a ASM Disk Group – Online Balancing

Oracle ASM supports online addition of storage and online data rebalancing.

This is a key advantage of ASM over LVM.  LVM requires a full backup/restore.

SELECT name, header_status, path FROM V$ASM_DISK;

 

NAME          HEADER_STATU   PATH

--------------------------------------------------------------------

DATA_0001      MEMBER        /dev/sudo oracleasm/disks/DATA_DISK_02

DATA_0000      MEMBER        /dev/sudo oracleasm/disks/DATA_DISK_01

ARCH_0000      MEMBER        /dev/sudo oracleasm/disks/ARCH_DISK_01

RECO_0001      MEMBER        /dev/sudo oracleasm/disks/RECO_DISK_02

RECO_0000      MEMBER        /dev/sudo oracleasm/disks/RECO_DISK_01

DATA_0002      MEMBER        /dev/sudo oracleasm/disks/DATA_DISK_03

 

Add another disk from the Azure portal

lsblk -l

NAME                    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT

sda                       8:0    0    1T  0 disk

sda1                      8:1    0 1024G  0 part

sdb                       8:16   0    1T  0 disk

sdb1                      8:17   0 1024G  0 part

sdc                       8:32   0    1T  0 disk

sdc1                      8:33   0 1024G  0 part

sdd                       8:48   0    1T  0 disk

sdd1                      8:49   0 1024G  0 part

sde                       8:64   0    1T  0 disk

sde1                      8:65   0 1024G  0 part

sdf                       8:80   0    1T  0 disk

sdf1                      8:81   0 1024G  0 part

sdg                       8:96   0  1.5T  0 disk

sdh                       8:112  0  1.5T  0 disk

sdi                       8:128  0   30G  0 disk

sdi1                      8:129  0  800M  0 part /boot

sdi2                      8:130  0 28.7G  0 part

sdi14                     8:142  0    4M  0 part

sdi15                     8:143  0  495M  0 part /boot/efi

sdj                       8:144  0  2.4T  0 disk

sdj1                      8:145  0  2.4T  0 part /mnt

sdk                       8:160  0    1T  0 disk

 

format the new disk

fdisk /dev/sdk

 

Command (m for help): n

Select (default p): p

Partition number (1-4, default 1):

First sector (256-268435455, default 256):

Last sector, +sectors or +size{K,M,G,T,P} (256-26835455, default 26435455):

Created a new partition 1 of type 'Linux' and of size 1024 GiB.

Command (m for help): w

The partition table has been altered.

Calling ioctl() to re-read partition table.

Syncing disks.

 

Oracle ASM Disk Groups can either be extended by adding additional disks or by extending current disks.  It is strongly recommended to add additional disks rather than extending existing disks.  Review these MOS articles and links

MOS Notes 1684112.1 and 2176737.1

 

As root

sudo chown oracle:oinstall /dev/sdk1

cd /oracle/GRID/19.0.0/bin

 

sudo oracleasm listdisks

ARCH_DISK_01

DATA_DISK_01

DATA_DISK_02

DATA_DISK_03

RECO_DISK_01

RECO_DISK_02

 

sudo oracleasm createdisk DATA_DISK_04 /dev/sdk1

Writing disk header: done

Instantiating disk: done

 

As oracle

su – oracle

cd /oracle/GRID/19.0.0/bin

 

./asmca -silent -addDisk -diskGroupName DATA -disk '/dev/sudo oracleasm/disks/DATA_DISK_04'

 

System will automatically redistribute the data

ps -ef | grep rbal

oracle   1524207       1  0 Nov01 ?        00:00:11 asm_rbal_+ASM

oracle   1610444       1  0 Nov01 ?        00:00:03 ora_rbal_OR1

 

Once this process is finished, we start the new IO Calibrate to see if there is a difference

 

Thanks to Ralf Klahr and Vamshi Polasa for contributing and work to get to this blog   

 

 

 

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.