Use an Azure Event Grid Custom Topic as a Message Broker

This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Community Hub.

Asynchronous messaging and event-driven communication are critical assets when building a distributed application composed of several internal and external services. This sample shows how to use an Azure Event Grid Custom Topic as a message broker in a multitenant scenario to send messages to multiple Azure Service Bus queues in different namespaces, one for each tenant. You can find the companion code on GitHub.

Azure Event Grid is a highly scalable, serverless event broker that you can use to integrate applications using events. Events are delivered by Event Grid to subscriber destinations such as applications, Azure services, or any endpoint to which Event Grid has network access. The source of those events can be other applications, SaaS services, and Azure services.

With Azure Event Grid, you can connect solutions using an event-driven architecture. An event-driven architecture uses the Publisher-Subscriber design pattern to dispatch events to one or more consumer applications and services to notify state changes. You can use filters to route specific events to different endpoints, multicast to multiple endpoints, and make sure your events are reliably delivered.

For guidelines on building a messaging or eventing infrastructure in a multitenant solution, see Architectural approaches for messaging in multitenant solutions.

Scenario

A multitenant application, that serves requests for multiple tenants, needs to route incoming messages to multiple Azure Service Bus namespaces, one for each tenant. Every message is intended for a single tenant, and the solution needs to implement a message broker that routes incoming messages based on user-defined properties or payload. The multitenant application uses a specific Service Bus namespace for each tenant. This deployment approach provides your solution with the maximum level of isolation, with the ability to provide consistent performance per tenant. You can also fine-tune messaging capabilities for each tenant based on their needs, such as by using the following approaches:

Deploy the namespace to a region that's close to the tenant.
Deploy a tenant-specific namespace with a pricing tier appropriate to that tenant. For example, you can provision premium namespaces with a different number of messaging units.
Apply networking restrictions based on the tenant's needs.
Use tenant-specific encryption keys.

The disadvantage to this isolation model is that, as the number of tenants grows within your system over time, the operational complexity of managing your namespaces also increases. If you reach the maximum number of namespaces per Azure subscription, you could deploy namespaces across different subscriptions (see deployment stamp pattern). This approach also increases resource costs, since you pay for each namespace you provision. For more information on tenancy models for the Azure Service Bus, see Multitenancy and Azure Service Bus.

Architecture

The following picture shows the architecture of the sample:

The architecture is composed of the following Azure resources:

Microsoft.EventGrid/topics: an Azure Event Grid Custom Topic used as a message broker that receives messages from one or more publisher applications and dispatches messages in JSON format to tenant-specific queues based on the starting or ending value of the event subject or based on filter expression defined on the properties of the payload.
Microsoft.EventGrid/topics/eventSubscriptions: an Azure Event Grid Subscription is used to send events to a tenant-specific Azure Service Bus namespace queue. The sample uses a simple filter based on the ending value of the subject, which contains the name of the tenant in lowercase.
Microsoft.ServiceBus/namespaces: an Azure Service Bus namespace and a queue for each tenant. The sample deploys three tenants (Fabrikam, Contoso, and Acme), but you can modify the list of tenants.
Microsoft.Storage/storageAccounts: when the Azure Event Grid Custom Topic receives an error for an event delivery attempt, the AZure Event Grid service decides whether it should retry the delivery, dead-letter the event, or drop the event based on the type of the error. When creating an event subscription, you can customize the settings for event delivery and dead-letter. This storage account stores dead-letter events for all the Azure Event Grid Subscriptions. For more information, see Set dead-letter location and retry policy.

For more information, see the following articles:

Requirements

Install the latest version of the Azure CLI.

Deploy the Solution

Change the working directory to the scripts folder which contains the bash scripts. Before deploying the sample, make sure to properly set values for the variables imported and used by all the scripts in the 00-variables.sh file in the scripts folder.

# Variables for the Event Grid demo # Location location="WestEurope" # Resource group name resourceGroupName="SampleEventGridRG" # Event Grid Topic topicName="SampleEventGrid" publicNetworkAccess="enabled" tags="service=EventGrid workload=$topicName" # Subscription endpoint type endpointType="servicebusqueue" # Sku of the service bus namespace serviceBusSku="Standard" # Name of the service bus queue serviceBusQueueName="events" # Name of the deadletter storage account storageAccountName="deadletterstore" # Name of the deadletter container containerName="deadletter" # Tenants tenants=("Fabrikam" "Contoso" "Acme") # Events id=1000 eventType="recordInserted" # SubscriptionId of the current subscription subscriptionId=$(az account show --query id --output tsv) subscriptionName=$(az account show --query name --output tsv)

Run the 01-create-event-grid.sh script under the scripts folder to create the Azure Event Grid Custom Topic used as a message broker.

Then, run the 02-create-service-bus-subscribers.sh script under the scripts folder to create the following resources:

An Azure Service Bus namespace and events queue for each tenant.
An Azure Storage Account for dead-letter events.
An Azure Event Grid Subscription for each tenant to send events to the eventsqueue of the corresponding Azure Service Bus namespace.

#!/bin/bash # Variables source ./00-variables.sh # Check if the event grid topic exists echo "Checking if [$topicName] event grid topic actually exists in the [$subscriptionName] subscription..." resourceId=$(az eventgrid topic show \ --name $topicName \ --resource-group $resourceGroupName \ --query id \ --output tsv 2>/dev/null) # Create event grid topic if it does not exist if [[ -n $resourceId ]]; then echo "[$topicName] event grid topic exists in [$resourceGroupName] resource group" else echo "No [$topicName] event grid topic exists in [$subscriptionName] subscription" exit fi # Check if the resource group already exists echo "Checking if [$resourceGroupName] resource group actually exists in the [$subscriptionName] subscription..." az group show --name $resourceGroupName &>/dev/null if [[ $? != 0 ]]; then echo "No [$resourceGroupName] resource group actually exists in the [$subscriptionName] subscription" echo "Creating [$resourceGroupName] resource group in the [$subscriptionName] subscription..." az group create \ --name $resourceGroupName \ --location $location 1>/dev/null # Create the resource group if [[ $? == 0 ]]; then echo "[$resourceGroupName] resource group successfully created in the [$subscriptionName] subscription" else echo "Failed to create [$resourceGroupName] resource group in the [$subscriptionName] subscription" exit fi else echo "[$resourceGroupName] resource group already exists in the [$subscriptionName] subscription" fi # Check if the storage account for deadletter messages exists echo "Checking if [$storageAccountName] storage account exists for the [$topicName] event grid topic..." storageAccountId=$(az storage account show \ --name $storageAccountName \ --resource-group $resourceGroupName \ --query id \ --output tsv 2>/dev/null) if [[ -z $storageAccountId ]]; then # Create Storage Account echo "No [$storageAccountName] storage account exists in the [$resourceGroupName] resource group" echo "Creating [$storageAccountName] storage account in the [$resourceGroupName] resource group..." az storage account create \ --name $storageAccountName \ --resource-group $resourceGroupName \ --query id \ --output tsv 1>/dev/null if [[ $? == 0 ]]; then echo "[$storageAccountName] storage account successfully created in the [$resourceGroupName] resource group" else echo "Failed to create [$storageAccountName] storage account in the [$resourceGroupName] resource group" exit fi else echo "[$storageAccountName] storage account already exists in the [$resourceGroupName] resource group" fi # Get the storage account key echo "Retrieving the primary key of the [$storageAccountName] storage account..." storageAccountKey=$(az storage account keys list \ --account-name $storageAccountName \ --resource-group $resourceGroupName \ --query [0].value -o tsv) if [[ -n $storageAccountKey ]]; then echo "Primary key of the [$storageAccountName] storage account successfully retrieved" else echo "Failed to retrieve the primary key of the [$storageAccountName] storage account" exit fi # Create the deadletter container echo "Checking if the [$containerName] container already exists in the [$storageAccountName] storage account..." name=$(az storage container show \ --name $containerName \ --account-name $storageAccountName \ --account-key $storageAccountKey \ --query name \ --output tsv 2>/dev/null) if [[ -z $name ]]; then # Create Storage Account echo "No [$containerName] container exists in the [$storageAccountName] storage account" echo "Creating [$containerName] container in the [$storageAccountName] storage account..." az storage container create \ --name $containerName \ --account-name $storageAccountName \ --account-key $storageAccountKey \ --query id \ --output tsv 1>/dev/null if [[ $? == 0 ]]; then echo "[$containerName] container successfully created in the [$storageAccountName] storage account" else echo "Failed to create [$containerName] container in the [$storageAccountName] storage account" exit fi else echo "[$containerName] container already exists in the [$storageAccountName] storage account" fi for tenant in ${tenants[@]}; do # Variables serviceBusNamespace="${tenant}ServiceBusNamespace" eventGridSubscriptionName="${tenant}EventGridSubscription" subjectEndsWith="${tenant,,}" # Check if the service bus namespace already exists echo "Checking if [$serviceBusNamespace] service bus namespace actually exists in the [$subscriptionName] subscription..." az servicebus namespace show \ --name $serviceBusNamespace \ --resource-group $resourceGroupName &>/dev/null if [[ $? != 0 ]]; then echo "No [$serviceBusNamespace] service bus namespace actually exists in the [$subscriptionName] subscription" echo "Creating [$serviceBusNamespace] service bus namespace in the [$subscriptionName] subscription..." az servicebus namespace create \ --name $serviceBusNamespace \ --location $location \ --resource-group $resourceGroupName \ --mi-system-assigned \ --sku $serviceBusSku 1>/dev/null # Create the service bus namespace if [[ $? == 0 ]]; then echo "[$serviceBusNamespace] service bus namespace successfully created in the [$subscriptionName] subscription" else echo "Failed to create [$serviceBusNamespace] service bus namespace in the [$subscriptionName] subscription" exit fi else echo "[$serviceBusNamespace] service bus namespace already exists in the [$subscriptionName] subscription" fi # Check if the service bus queue already exists echo "Checking if [$serviceBusQueueName] service bus queue actually exists in the [$serviceBusNamespace] service bus namespace..." az servicebus queue show \ --name $serviceBusQueueName \ --namespace-name $serviceBusNamespace \ --resource-group $resourceGroupName &>/dev/null if [[ $? != 0 ]]; then echo "No [$serviceBusQueueName] service bus queue actually exists in the [$serviceBusNamespace] service bus namespace" echo "Creating [$serviceBusQueueName] service bus queue in the [$serviceBusNamespace] service bus namespace..." az servicebus queue create \ --name $serviceBusQueueName \ --namespace-name $serviceBusNamespace \ --resource-group $resourceGroupName 1>/dev/null # Create the service bus namespace if [[ $? == 0 ]]; then echo "[$serviceBusQueueName] service bus queue successfully created in the [$serviceBusNamespace] service bus namespace" else echo "Failed to create [$serviceBusQueueName] service bus queue in the [$serviceBusNamespace] service bus namespace" exit fi else echo "[$serviceBusQueueName] service bus queue already exists in the [$serviceBusNamespace] service bus namespace" fi # Get service bus resource id serviceBusId=$(az servicebus queue show \ --name $serviceBusQueueName \ --namespace-name $serviceBusNamespace \ --resource-group $resourceGroupName \ --query id \ --output tsv) if [[ -n $serviceBusId ]]; then echo "Resource id for the [$serviceBusQueueName] service bus queue successfully retrieved" else echo "Failed to retrieve the resource id of the [$serviceBusQueueName] service bus queue." exit fi # Check if the Event Grid subscription exists az eventgrid event-subscription show \ --name $eventGridSubscriptionName \ --source-resource-id $resourceId &>/dev/null if [[ $? != 0 ]]; then echo "No [$eventGridSubscriptionName] Event Grid subscription actually exists for [$subscriptionName] subscription events" echo "Creating [$eventGridSubscriptionName] Event Grid subscription for [$subscriptionName] subscription events..." # Create Event Grid subscription az eventgrid event-subscription create \ --endpoint-type $endpointType \ --endpoint $serviceBusId \ --deadletter-endpoint ${storageAccountId}/blobServices/default/containers/$containerName \ --name $eventGridSubscriptionName \ --subject-ends-with $subjectEndsWith \ --source-resource-id $resourceId 1>/dev/null # Create the Event Grid subscription if [[ $? == 0 ]]; then echo "[$eventGridSubscriptionName] Event Grid subscription successfully created in the [$subscriptionName] subscription" else echo "Failed to create [$eventGridSubscriptionName] Event Grid subscription in the [$subscriptionName] subscription" exit fi else echo "[$eventGridSubscriptionName] Event Grid subscription already exists in the [$subscriptionName] subscription" fi done

Verify the Deployment

If you successfully deployed the sample, you should see the following Azure resource in the target resource group:

If you open the Azure Event Grid Topic and select the Event Subscriptions, you should see the following subscriptions, one for each tenant, and each with a ServiceBusQueue endpoint.

Test the solution

You can use the 03-send-events.sh script under the scripts folder to send a batch of events to the Azure Event Grid Custom Topic, one for each tenant, using a curl command. For more information, see Publish events to Azure Event Grid custom topics using access keys. The scripts performs the following operations:

Creates a JSON array containing an event for each tenant.
Retrieves the URL of the Azure Event Grid Custom Topic endpoint.
Retrieves the key of the Azure Event Grid Custom Topic.
Sends the event array to the Azure Event Grid Custom Topic endpoint using a curl command. The call uses the key retrieved at the previous step in the aeg-sas-key header to authenticate with the Azure Event Grid. For more information, see Authenticate Azure Event Grid publishing clients using access keys or shared access signatures.

#!/bin/bash # Variables source ./00-variables.sh json="[" for ((i=0;i<${#tenants[@]};i++)); do ((id=id+1)) subject="atom/events/${tenants[$i],,}" eventTime=$(date +%FT%T.%3N%:z) data="{\"tenant\":\"${tenants[$i]}\",\"date\":\"$(date -u +"%Y-%m-%dT%H:%M:%SZ")\"}" event="{ \"id\":\"$id\", \"eventType\":\"$eventType\", \"subject\":\"$subject\", \"eventTime\":\"$eventTime\", \"data\": $data }" if [ $i == 0 ] then json=$json$event else json=$json,$event fi done json=$json"]" json=$(echo $json | sed 's/ //g') # Retrieve the endpoint of the event grid topic echo "Retrieving the endpoint of the [$topicName] event grid topic..." endpoint=$(az eventgrid topic show \ --name $topicName \ --resource-group $resourceGroupName \ --query endpoint \ --output tsv 2>/dev/null) if [[ -n $endpoint ]]; then echo "[$endpoint] endpoint of the [$topicName] event grid topic successfully retrieved" else echo "Failed to retrieve the endpoint of the [$topicName] event grid topic" exit fi # Retrieve the key of the event grid topic echo "Retrieving the key of the [$topicName] event grid topic..." key=$(az eventgrid topic key list \ --name $topicName \ --resource-group $resourceGroupName \ --query key1 \ --output tsv 2>/dev/null) if [[ -n $key ]]; then echo "[$key] key of the [$topicName] event grid topic successfully retrieved" else echo "Failed to retrieve the key of the [$topicName] event grid topic" exit fi # Send events to the event grid topic echo "Sending events to the [$topicName] event grid topic..." echo $json | jq -r curl -X POST \ -H "aeg-sas-key: $key" \ -H "Content-Type: application/json" \ -d "$json" \ $endpoint

The events.json file under the scripts folder contains a sample of the events generated and sent by the 03-send-events.sh script.

Conclusions

This sample shows how you can use an Azure Event Grid Custom Topic as a message broker in a multitenant scenario to send messages to multiple Azure Service Bus queues in different namespaces, one for each tenant, without the need to create a custom solution.