Moneo: Distributed GPU System Monitoring for AI Workflows

This post has been republished via RSS; it originally appeared at: Microsoft Tech Community - Latest Blogs - .

Microsoft has introduced a new open-source GPU monitoring framework named Moneo (Latin for monitor). Moneo orchestrates metric collection (DCGMI + Prometheus DB) and visualization (Grafana) across multi-GPU/node systems.  This provides useful insights into workflow and system level characterization.

GPUs are optimized for high throughput, massively parallel, workloads. Efficient use of GPUs is dependent on a few factors such as workload characteristics, system characteristics, and sometimes physical environment.  GPU system level monitoring determines GPU utilization and facilitates workload characterization.  This aids in exploring how to utilize the GPU more efficiently.

Monitoring GPU metrics on a single system over a period may be trivial. However, formatting and analyzing the raw metric data so that it can provide intuitive insights can prove to be a tedious task. Our goal is to pair system level characterization with application metrics to determine the efficiency of our use of the GPUs.

Given that there is already some complexity with collecting and analyzing GPU metrics on a single system, unsurprisingly, scaling the same methodology for multiple systems is difficult.

For certain deep learning models use of distributed multi-GPU systems is the only feasible way to train in a reasonable time frame. Much of the application complexity is abstracted away by high level AI frameworks, but there are still configurations and design choices that users must make that ultimately affect the throughput and behavior of model training. 

Moneo’s usefulness in providing system level insights can help guide design choices to achieve the efficient use of GPU systems.

Moneo Design

Figure 1: Design

Three categories of metrics that Moneo monitors:

1. Device Counters
   1. Compute/Memory Utilization
   2. Streaming multiprocessor (SM) and Memory Clock frequency
   3. Temperature
   4. Power
   5. ECC Counts
2. Profiling Counters
   1. SM Activity
   2. Memory Dram Activity
   3. NVLink Activity
   4. PCIE Rate
3. InfiniBand Network Counters
   1. IB TX/RX rate

Once Moneo has been launched these metrics can be viewed from the Grafana portal. See figures 2,3,4 for snapshots of the different metric views.

Figure 2: Device Counter View

Figure 3: Profiling Counter View

Figure 4: IB Counter View

Getting Started

Starting with Moneo is easy. Just clone the latest release from the Moneo Repo and follow the README for detailed setup instructions or take a look at the quick start guide.  In a short period, you should be able to launch Moneo with a single command and log into the Grafana portal to start seeing results!

Moneo is also available on Azure HPC + AI Ubuntu images.  Just navigate to “/opt/azurehpc/tools/Moneo”. The image has all the required dependencies installed. So, all that’s necessary is configuring and deploying Moneo.

Quick start instructions:

1. Clone Moneo from Github and install ansible.
   1. git clone https://github.com/Azure/Moneo.git
   2. cd Moneo
   3. python3 -m pip install ansible

2. Next create a host.ini config file.

   

   • Note: The master node can also be a worker node as well. The master node will have the Grafana and Prometheus docker containers deployed to it.

   • Note: If you have configured password less SSH already, [all:vars] section can be skipped.

   • Note: The master node must be able to ssh into itself. 

3. Now deploy Moneo
   • ansible-playbook -i host.ini src/ansible/deploy.yaml
4. Log into the portal by navigating to http://master-ip-or-domain:3000 and inputting your credentials
   • 
   • Note: By default, username/password are set to "azure". This can be changed here "src/master/grafana/grafana.env"
5. Navigating Moneo Grafana Portal
   1. The current view is labeled in the top left corner:
      • 

         

   2. VM instance and GPU can be selected from the drop-down menus in the top left corner:
      • 

         

   3. Various actions such as dashboard selection or data source configuration can be achieved using the left screen menu:

       

   4. Metric groups are collapsible: