Integrate private access to your Azure Open AI Chatbot

This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Community Hub.

Do you want to create your own private (non-Internet exposed) chatbot using Azure Open AI (AOAI) services with your company-specific data?  In this blog post, we will walk you through the steps to integrate the solution into your Azure environment and secure from public exposure.

Figure 1: Reference Architecture for internal chatbot using Azure AI Services

For a high-level reference of the Azure Landing zone integration and resource details see Azure OpenAI Landing Zone reference architecture (microsoft.com)

This post is based on the ChatGPT + Enterprise data with Azure OpenAI and Cognitive Search Demo, which demonstrates how to build a chatbot using Azure AI services. See the link for more details and instructions on how to run the demo. The diagram above details the infrastructure components needed to accomplish this, a hub connectivity network and application spoke that host the AOAI app.

The solution achieves the following goals:  

• Accessing the Azure Open AI chat bot only via the corporate network, without exposing it to the internet. 
• Protecting your enterprise data from public access. 
• Guidance on integrating the Azure Open AI solution following Azure best practices.

Prerequisites

Azure

• Azure Account. If you don't have an Azure subscription, create a free account before you begin.
• Azure subscription with access enabled for the Azure OpenAI service. You can request access with this form. Models are subject to region availability. You can review the list here.
• The deployment must be started by a user who has sufficient permissions to assign roles, such as a User Access Administrator or Owner.
• Your Azure account also needs Microsoft.Resources/deployments/write permissions at the subscription level.
• IP scheme that is not overlapping with on-premises and subnet plan

On-Premises

• Hybrid Connectivity to Azure either through VPN or ExpressRoute
  • VPN device for site-to-site connection
  • Connectivity provider for ExpressRoute
• Public IP address from Azure Virtual Network Gateway
• On-Premises DNS server
• SSL Certificate

Subnet Planning

The choice of CIDR blocks for your subnets depends on various factors including the size of your network, the number of resources you plan to deploy in each subnet and your future scaling needs. The following are the general guidelines for this architecture however your specific needs may require a different approach.

Connectivity between On-Premises and Azure

There are two options for connecting Azure to on-premises networks: Azure ExpressRoute private peering and VPN Gateway. Both solutions allow you to send encrypted traffic between Azure and on-premises locations over a private or public connection. However, there are some key differences that you should consider when choosing the best option for your organization.

1. Azure ExpressRoute private peering extends on-premises networks into the Microsoft cloud over a private connection. This option provides high bandwidth, low latency, and predictable performance. Steps to setup ExpressRoute
   
   Figure 2: ExpressRoute Setup
2. VPN Gateway sends encrypted traffic between an Azure virtual network and an on-premises location over the public Internet. You can also use VPN Gateway to send encrypted traffic between Azure virtual networks over the Microsoft network. Steps to setup S2S VPN

Figure 3: Site-to-site VPN Configuration

Comparison and Recommendations

You can use either ExpressRoute or VPN Gateway to connect Azure to on-premises networks, depending on your requirements and preferences. Here is a side-by-side comparison of the two options based on some common criteria:

For a more comprehensive comparison, see About Azure VPN Gateway.

Generally, you should choose ExpressRoute if you need high performance, reliability, and security for your connectivity. You should choose VPN Gateway if you need lower cost, simplicity, and flexibility for your connectivity. You can also use both options together to create a hybrid solution that meets your specific needs.

Azure Application Gateway with Web Application Firewall

Application Gateway is a layer-7 (application layer) load balancer that manages traffic to web applications. It distributes application traffic to the app services backend pools; provides Web Application Firewall integrated capabilities, SSL/TLS termination, and session affinity; and helps to ensure that all users of the Azure OpenAI APIs get the fastest response and highest throughput for model completions. Steps to setup Application Gateway

Figure 4: Application Gateway functions

Azure Application Gateway works together with Azure Web Application Firewall to help protect web applications from common exploits and vulnerabilities. Web Application Firewall is a service that monitors and filters incoming and outgoing web traffic based on rules and policies. Steps to setup Web Application Firewall (WAF)

Considerations:

• It is recommended to store the TLS Certificate in Key Vault. Note: Specifying Azure Key Vault certificates that are subject to the role-based access control permission model is not supported via the portal. See here for PowerShell example.
• A critical issue with any Network Security Firewall products like WAF is efficiently analyzing WAF block actions and analyzing potential false positives. AI integration with WAF can assist with detecting false positives and mitigating them using predictive rule guidance.
• Use Troubleshoot - Azure Web Application Firewall | Microsoft Learn to understand how to identify false positives and tune WAF policies and leverage the WAF Triage Workbook to more easily tune your WAF ruleset.

Azure Private Endpoint: A Secure and Efficient Way to Connect to PaaS Services

Azure Private Endpoint is a network interface card (NIC) inside a subnet that attaches to an Azure service. Private endpoints are needed in this configuration to limit public access to PaaS services. Private Endpoints offer support for inbound connectivity (and its respective return traffic) only. For each service in this architecture, below are the steps and resources that you need to set up private endpoints.

• Azure App Services: Involves enabling VNet integration and setting up Private endpoint
• Azure Storage Account
• Azure Open AI
  • Note: Azure Open AI Studio does not support private endpoints in bring-your-own data scenarios. You can learn more about this limitation at Using your data with Azure OpenAI Service - Azure OpenAI | Microsoft Learn
• Azure Key Vault
• Azure AI Search (formerly known as Azure Cognitive Search)
• Azure AI Document Intelligence

By using private endpoints, you can:

• Enhance the security of your data and network by preventing unauthorized access to the internet or other networks.
• Reduce data exposure and latency by keeping the traffic within your virtual network.
• Optimize the performance and reliability of your applications by using the closest and fastest Azure service instances.

You can find the full list of Azure services that support private link at What is a private endpoint? - Azure Private Link | Microsoft Learn

DNS Configuration

Azure Private DNS zones are a feature that enables you to use custom domain names for your Azure services within your virtual network. By using Azure Private DNS Zones, you can:

• Improve the security of your network by preventing DNS queries from leaving your virtual network or being exposed to the internet.
• Simplify the management of your DNS records by using the same domain names for your Azure services across different regions and subscriptions.
• Ensure consistent resolution of your Azure services by using the same DNS zone for both public and private endpoints.

Private DNS Zones are global services where best practice is to deploy them in the Hub Virtual Network. For a list of Private DNS Zones, visit here.

The following table shows the private DNS zone for each service domain name. You can also find more details and screenshots for setup steps in this article: Quickstart - Create an Azure private DNS zone using the Azure portal | Microsoft Learn.

DNS for Custom Domain Name

A custom domain is used to access the chat bot app to give the user a friendly URL. Outlined in the following article is the DNS Zone setup for custom domains in Azure: Map existing custom DNS name - Azure App Service | Microsoft Learn  

On Premises DNS Server

DNS Private Resolver simplifies private DNS resolution from on-premises to the Azure private DNS service and vice versa.

1. Setup DNS Private Resolver in Azure
   1. Choose between a Distributed vs. Centralized DNS Private Resolver implementation. Consider Distributed if your spoke virtual networks will require auto-registration for Virtual Machine resources.
2. Configure on-premises Windows DNS conditional forwarder
3. For redundancy add another DNS Private Resolver in the secondary region

Figure 5: Azure DNS Private Resolver architecture

An Azure Private DNS resolver ensures a secure and customizable DNS resolution service within your virtual network, guaranteeing that queries are resolved accurately to the appropriate URL.

NSG Rules

Network security groups (NSGs) are used to filter network traffic to and from Azure resources in an Azure virtual network. A network security group contains security rules that allow or deny inbound network traffic to, or outbound network traffic from, several types of Azure resources. For more information, see Best practices for network security - Microsoft Azure | Microsoft Learn

Application security groups (ASG) allow you to group resources based on their application and define network security policies based on those groups. In this scenario, it is used to separate out the PaaS service endpoints based on their front-end and back-end functionality.

Each subnet has default rules that allow communication within the subnet and with the Internet but block any other traffic. You can learn more about these rules here: Azure network security groups overview | Microsoft Learn.  Further, subnets in Azure are able to honor NSGs rules for private endpoints when the network policy is enabled. Make sure to enable the network policy for each subnet in your environment.

Below is the configuration for the subnets outlined in this scenario:

• Web Tier Egress Subnet | nsg-app: This subnet handles the outbound traffic from the web application.

• APIM Subnet | nsg-apim: This subnet hosts the Azure API Management service.

• Shared Tier Subnet | nsg-shared: This subnet contains the shared resources for the chat app, such as the Azure Key Vault.

• Data Tier Subnet | nsg-data: This subnet connects to the data services, such as Azure Storage

• Web Tier Ingress Subnet | nsg-fe: This subnet handles the inbound traffic to the chat app via HTTPS

• AI Tier Subnet | nsg-ai: This subnet hosts the Azure AI Services that provide capabilities for the chat app.

Authentication and Authorization

• To learn more about how authentication works in the sample chat app, visit here.
• Setup managed identity for Document Intelligence
• Enable access to storage from Document Intelligence  

Monitoring

• Azure Monitor - WAF Workbook 
• In this architecture, APIM is used to provide additional insights, API management, additional security and usage monitoring to chat app. More information on the setup is available here: Implement logging and monitoring for Azure OpenAI large language models - Azure Architecture Center | Microsoft Learn

Resources 

• What is Azure OpenAI Service? - Azure AI services | Microsoft Learn 
• GitHub repository for the baseline app that was used in this scenario: ChatGPT + Enterprise data with Azure OpenAI and Cognitive Search 
• Data, privacy, and security for Azure OpenAI Service - Azure AI services | Microsoft Learn 
• Design and implement private access to Azure Services - Training | Microsoft Learn 

Contributors

Thanks to everyone for their contributions to the development of this reference architecture: Alejandra Palacios, Alexandra Miller-Browne,  Andre Murrell, Charles Shea, Jen Sheerin, Mauricio Rojas Martinez, Sowmya Mahadevaiah, Simona Kovatcheva