This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Tech Community.
Many Azure customers with Azure SAP HANA Large Instances use the Microsoft Snapshot Tools to create snapshots to safeguard their SAP HANA instances. This blog focusses on common questions we received regarding usage of the snapshot tools. Please note that this blog is for HANA Large Instances only, not for SAP HANA on Azure VMs. This blog is supplement to the documentation in GitHub.
For a generic explanation of snapshots see:
- https://tv.netapp.com/detail/video/2579133646001/snapshot
- https://www.netapp.com/us/media/ds-2477.pdf
Question: How can I check what is included in a HANA snapshot?
Please note that snapshots are stored in the same physical storage as the HANA database. In each HANA related mount point, for example, /hana/logbackups/<SID> a subdirectory called .snapshot is present. When snapshots are taken, they are stored in the .snapshot directory based on the –type flag. So a HANA snapshot with –type=logs gets stored in /hana/logbackups/<SID>/.snapshot. Keep in mind that snapshots are taken on the entire volume, in this example everything in /hana/logbackups/<SID> will be included in the snapshot. Hence it is recommended to keep the HANA mount points cleaned up and follow SAP HANA housekeeping recommendations (link | note).
Performing an ‘ls’ or even an ‘ls -la’ inside /hana/logbackups/<SID>/ does not show the .snapshot directory, but you can ‘cd’ into the directory and see the contents. From the OS you cannot write into the directory, if you try it you will see an error “Read-only file system”. To check the contents you can use command ‘ls -lahR’. Since there can be a lot of files please redirect the output to a text file in a directory where you have sufficient space and permissions. The output of the file shows the contents of the snapshots. If, for example, you use /hana/logbackups/<SID> to store SAP installation DVD’s, these will take up snapshot space and reduce the available size for your running HANA instance.
Question: Can I use OS commands such as ‘df’ and ‘du’ to check snapshot space?
Using OS commands is not as reliable as using the snapshot command azure_hana_snapshot_details. This command will provide an output list of individual size per snapshot as well as the total size per snapshot type.
OS commands such as ‘df’ and ‘du’ can be useful to check OS filespace, but can be misleading with regards to snapshot sizes. For example, the output of for example ‘df -h’ will include the regular files + space taken by snapshots. However, when you then run ‘du -sh .’ in the main mount point (e.g. /hana/logbackups/<SID>), you only see the space used for the regular files and snapshot space is excluded. So there is a discrepancy between the output of ‘df’ and ‘du’ due to the snapshots. So what if you ‘cd’ into the .snapshot directory /hana/logbackups/<SID>/.snapshot and use ‘du’…. Now the output will be incorrect and larger than the actual space taken by the snapshots.
In conclusion: use snapshot command azure_hana_snapshot_details to check the size used by snapshots. Be aware that this command might not complete if a snapshot is being taken, so please run it when no other snapshot commands are running.
Question: how to delete snapshots in case a volume is filling up?
There are two ways to delete snapshots using the snapshot commands. The first way is to use azure_hana_snapshot_delete. This snapshot command will delete a single snapshot. However please be aware that deleting a single snapshot, can actually affect the size of other snapshots. This is because pointers are used to avoid duplication of content. This NetApp link provides a simple explanation.
Hence if you really need to free up space, deleting a single snapshot, or even several single large snapshots, will not actually work well. The recommendation is to delete a range of snapshots instead. An easy way is to run the same azure_hana_backup command with a lower retention.
Let’s say currently you execute azure_hana_backup of type “logs” with a retention of 30 which means you keep 30 snapshots of the /hana/logbackups/<SID>/ volume. You notice in ‘df -h’ that this mount point is getting full and you decide to delete some snapshots to free up space. You execute azure_hana_snapshot_details to get a list of snapshots, their date/time stamps, and sizes. Based on this you decide to delete to oldest 10 snapshots. You simply can execute azure_hana_backup with a retention of 20, either execute it manually from the command line or execute it from cron. After successful execution the oldest 10 will be deleted, and the most recent 20 snapshots remain. Now that space is freed up you have the option to revert back to your old retention setting.
If space fills up frequently, it may be a good idea to review what is in the snapshots (see Question: How can I check what is included in a HANA snapshot), your retention setting, and HANA housekeeping (See SAP Note 2399996).
Please note that the decision how many snapshots to delete (and keep) is first and foremost a business decision based on how much risk the business is willing to accept. Because of this, many organizations have an approval process in place before the deletion can be executed.
Question: what could cause the /hana/logbackups/<SID> mount point to grow consistently?
The HANA backup catalog can grow over time and each HANA log backup can protect the HANA backup catalog. If, for example, your HANA logbackup parameter (log_backup_timeout_s) is set to run every 10 minutes, each logbackup writes a backup of the backup catalog to /hana/logbackups/<SID>. Over time the backup catalog grows and grows if not cleaned up, therefore SAP recommends to clean up the backup catalog on a regular basis. With snapshots, the effect of a growing HANA backup catalog is compounded, since each backup of the HANA backup catalog is also captured in the snapshot. This has led to several TBs of HANA backup catalog backups in the .snapshot directory.
There are several ways to reduce the size of the HANA backup catalog:
The documented SAP HANA procedure to reduce the size of the backup catalog is to follow SAP Note 2096851 - "Backup Catalog Housekeeping within HANA DB” and the SAP HANA Administrator guide – section “Housekeeping for Backup Catalog and Backup Storage”.
If you prefer to automate the cleanup of the HANA backup catalog, the suggested option is to use –trim. With HANA2 the snapshot commands provide an automated method to use the --trim option which works in combination with the --retention parameter to automatically remove old log backups and backup catalog entries. Please consult the documentation in GitHub for –trim options.
Question: What if I want to run the snapshot commands as root user?
The documentation in GitHub recommends to use a non-root user and the installer creates a specific user called ‘shoasnap’ for this purpose. The installer also sets the user’s environment variables correctly to find the HANA executables used by the snapshot commands during execution. Some customers decide to use the root user instead.
If, for example, you try to run the snapshot commands as root and it fails, perform a simple test by executing ‘which hdbsql’. If the command fails with a “not found” related error, then you need to set the environment variables correctly for the root user.
There are several ways to do this, but a quick and simple solution is to assume the environment variables for the <sid>adm user. If, for example, the <sid>adm user’s home directory is /usr/sap/<SID>/home, then run ‘source /usr/sap/<SID>/home/.hdbenv.sh’ and retry the test with ‘which hdbsql’. Now the environment variables should be set correctly and you can continue with your setup or test.
Alternately the Snapshot documentation in GitHub provides manual installation instructions which can be followed to help in setting the root user’s environment.
Question: In a HANA scale-out setup, do I have to take a HANA snapshot from each node in the HANA cluster?
No, the azure_hana_backup snapshot command with --type=hana first communicates with SAP HANA to create the savepoint, then it iterates through the volumes taking a storage snapshot, and completes the HANA data snapshot by releasing the savepoint. For more information on HANA data snapshot please review SAP Note 2039883 - "FAQ: SAP HANA database and data snapshots (storage snapshots)".
If, at any time you get an error such as “HANA Backup failing with "[447]: backup could not be completed: [110122] A data backup cannot be created because another data backup is running or a storage snapshot has been prepared" please follow SAP Note 2452735 to resolve the situation.