Navigating Common VNET Injection Challenges with Azure Spring Apps

Posted on by — No Comments ↓

This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Community Hub.

This article outlines the typical challenges that customers may come across while operating an Azure Spring Apps service instance using the Standard and Enterprise plans within a virtual network environment. Deploying Azure Spring Apps within a customized Virtual Network introduces a multitude of components into the system. This article will guide you to set network components such as Network Security Groups (NSG), route tables, and custom DNS servers correctly to make sure service is functioning as expected.

 

Hanli_Ren_1-1693388661034.png

 

In the following sections, we've compiled a list of common issues our customers encounter and offer recommendations for effectively resolving them.

 

VNET Prerequisites not met Issues
Custom Policy issues

Custom DNS Server Resolution Issues

Outbound connection issues 
Route Table Issues

User Defined Route Issues

 

VNET Prerequisites not met Issues

Symptoms:

  1. Failed to create Azure Spring Apps service.

Error Messages:

"InvalidVNetPermission" error messages reporting ""Invalid Subnet xxx. This may be a customer error if 'Grant service permission to virtual network' step is missing before create Azure Spring Apps in vnets."

 

Common Causes of the issue:

Azure Spring Apps platform need to execute some management operations (e,g, create route tables, add rules into existing route tables) in the customer provided VNET. Without the above permissions, the platform operations will be blocked.

 

How to fix:

Grant the Owner permission on your virtual network to "Azure Spring Cloud Resource Provider" (id: e8de9221-a19c-4c81-b814-fd37c6caf9d2). Or you can grant User Access Administrator and Network Contributor roles to it if you can't grant Owner permission.

 

az role assignment create \ --role "Owner" \ --scope ${VIRTUAL_NETWORK_RESOURCE_ID} \ --assignee e8de9221-a19c-4c81-b814-fd37c6caf9d2

 

 

If you associated your own route tables to the given subnets, also need to make sure to grant Owner or User Access Administrator & Network Contributor permission to "Azure Spring Cloud Resource Provider" on your route tables.

 

az role assignment create \ --role "Owner" \ --scope ${APP_ROUTE_TABLE_RESOURCE_ID} \ --assignee e8de9221-a19c-4c81-b814-fd37c6caf9d2

 

 

Reference docs:

Virtual network requirements

Grant service permission to the virtual network

 

Custom Policy issues

Symptoms:

  1. Failed to create Azure Spring Apps service.
  2. Failed to start/stop Azure Spring Apps service.
  3. Failed to delete Azure Spring Apps service.

Error Messages:

Resource request failed due to RequestDisallowedByPolicy.
Id: /providers/Microsoft.Management/managementGroups/<group-name>/providers/Microsoft.Authorization/policyAssignments/<policy-name>

 

Common Causes of the issue:

One of the most common policies we saw that blocks the platform operation is "Deny Public IP Address creation". The Azure Spring Apps platform need to create a public IP in the load balancer to communicate with those targets mentioned in Customer responsibilities running Azure Spring Apps in a virtual network for management and operational purposes.


How to fix:

Find the policy id provided in the error message, then check if the policy can be deleted or modified to avoid the issue.
For Deny Public IP Address policy, if your company policy does not allow that, you can also choose to use Customize Azure Spring Apps egress with a user-defined route.

 

 

Custom DNS Server Resolution Issues

Symptom：

  • Failed to create Azure Spring Apps service.
  • Failed to start Azure Spring Apps service.
  • App cannot register to Eureka server.
  • App cannot resolve or resolve wrong IP address of a target URL.

 

Error Messages:

  • Could not resolve private dns zone.
  • java.net.UnknownHostException
  • If the DNS server resolved wrong IP address of a host, we may also see connection failure in the log.
    java.net.SocketTimeoutException


Common Causes of the issue:

  1. Custom DNS server is not correctly configured to forward DNS requests to upstream public DNS server.
    In the Troubleshooting Azure Spring Apps in virtual network Doc, it mentioned if your virtual network is configured with custom DNS settings, be sure to add Azure DNS IP 168.63.129.16 as the upstream DNS server in the custom DNS server.
    But sometimes it got misunderstood as adding both custom DNS server IP and 168.63.129.16 into the VNET DNS servers setting. This will introduce unpredictable behavior in name resolving.
    Hanli_Ren_0-1693387034199.png

     

  2. Some companies' policy does not allow forwarding DNS requests to Azure DNS server.
    Customers have the flexibility to direct their DNS requests to any upstream DNS server of their choice, provided that the chosen server is capable of resolving the public targets outlined in the Customer responsibilities running Azure Spring Apps in a virtual network.

    In this scenario, you will also need to add an additional DNS A record for *.svc.private.azuremicroservices.io (Service Runtime cluster ingress domain) -> the IP of your application (Find the IP for your application).
    While the Azure Spring Apps platform does configure this record, it is specifically recognizable by Azure DNS servers. If an alternative DNS server is in use, this record needs to be manually added into your own DNS server.

    Since our Spring boot Configure server and Eureka server are hosting in Service Runtime cluster, if your DNS server setting cannot resolve *.svc.private.azuremicroservices.io domain, your app may fail to load config and register itself to the Eureka server.

  3. Upon modifying your DNS configuration file or adjusting the VNET DNS server settings, it's important to note that these changes will not be instantly propagated across all the cluster nodes that host your application. To effectively implement the modifications, it is necessary to stop then start the Azure Spring Apps instance. This action enforces a reboot of the underlying cluster nodes.
    It's a by design behavior, since any cluster node virtual machine will only load the DNS settings from VNET for one time when it got created, restarted or being manually restart the network daemon.

  4. The network connection to the custom DNS server or Azure DNS server is blocked by NSG rule or firewall.

 

How to investigate:

  1. Use Diagnose and Solve problems -> DNS Resolution detector
    Hanli_Ren_1-1693231219301.png
    Hanli_Ren_2-1693231257081.png
  2.  Use App Connect console to test the DNS resolve result.
     We can connect to the App container, and directly run nslookup command to test the host's name resolve result.
     Please refer to Investigate Azure Spring Apps Networking Issue with the new Connect feature - Microsoft Community Hub.

  3. If the wrong DNS server setting blocked Azure Spring Apps service creation, since there is no resource being created yet, the above 2 troubleshooting method won't be available.
    You will need to create a jump-box (windows or Linux VM) in the same VNET subnet as your Azure Spring App, then use nslookup or dig command to verify the DNS settings. Make sure all the targets mentioned in Customer responsibilities running Azure Spring Apps in a virtual network can be resolved.
    nslookup mcr.microsoft.com​

Outbound connection issues

Symptom:

  • Failed to create Azure Spring Apps service.
  • Failed to start Azure Spring Apps service.
  • App Deployment failed to start.
  • App Deployment cannot mount Azure File Share.
  • App Deployment cannot send metrics/logs to Azure Monitor, Application Insights.

Error Messages:

"code": "InvalidVNetConfiguration",
"message": "Invalid VNet configuration: Required traffic is not whitelisted, refer to https://docs.microsoft.com/en-us/azure/spring-cloud/vnet-customer-responsibilities


Common Causes of the issue:

  1. If the traffic targeting *.azmk8s.io being blocked, the Azure Spring Apps service will lost connection to the underneath Kubernetes Cluster to send management requests. It will cause the service failed to create and start. 
  2. If the traffic targeting mcr.microsoft.com, *.data.mcr.microsoft.com,  packages.microsoft.com being blocked, the platform won't be able to pull images and packages to deploy the cluster node. It will cause the service failed to create and start.
  3. If the traffic targeting *.core.windows.net:443 and *.core.windows.net:445 being blocked, the platform won't be able to use SMB to mount the remote storage file share. It will cause the app deployment failed to start.
  4. If the traffic targeting login.microsoftonline.com being blocked, the platform authentication feature like MSI will be blocked. It will impact service start and app MSI usage.
  5. If the traffic targeting global.prod.microsoftmetrics.com and *.livediagnostics.monitor.azure.com being blocked, the platform won't to able to send data to Azure metrics, so you will lose App Metric data.

 

How to investigate:

  1. Examine the logs of your NSG and firewall to verify whether any traffic directed towards the targets outlined in the Customer responsibilities running Azure Spring Apps in a virtual network is encountering blocks or restrictions.
  2. Review your route table settings to ensure that they are configured to effectively route traffic towards the public network (either directly or via firewall, gateway or other appliances) for the specific targets detailed in the Customer responsibilities running Azure Spring Apps in a virtual network.
  3. Use Diagnose and Solve problems -> Required Outbound Traffic
    Hanli_Ren_0-1693232309834.png

    Hanli_Ren_1-1693232348305.png

     

  4. Use App Connect console to test the outbound connection blocking issue.
    We can connect to the App container, and directly run "nc -zv <ip> <port>"command to test the outbound connection blocking issue.
    nc -zv <ip> <port>
    Please refer to Investigate Azure Spring Apps Networking Issue with the new Connect feature - Microsoft Community Hub

  5. If the outbound connection issue blocked Azure Spring Apps service creation, since there is no resource being created yet, the above 2 troubleshooting method won't be available.
    You will need to create a jump-box (windows or Linux VM) in the same VNET subnet as your Azure Spring App, then use "nc" or "tcpping" command to verify the connection. Make sure all the targets mentioned in Customer responsibilities running Azure Spring Apps in a virtual network are not blocked by your NSG or firewall rules.


How to fix:

Use the above method to check which outbound traffics being blocked. Fix the setting in route table, NSG and firewall in both service runtime and app subnets.

 

 

Route Table Issues

Symptom:

  • Failed to create Azure Spring Apps service.
  • Failed to start Azure Spring Apps service.
  • Failed to delete Azure Spring Apps service.

Error Messages:

  1. Invalid VNet configuration: Invalid RouteTable xxx. This may be a customer error if providing wrong route table information
  2. Invalid VNet configuration: Please make sure to grant Azure Spring Apps service permission to the scope /subscriptions/xxx/resourceGroups/xxx/providers/Microsoft.Network/routeTables/xxx. Refer to https://aka.ms/asc/vnet-permission-help for more details

 

Common Causes of the issue:

  1. You did not grant Owner or User Access Administrator & Network Contributor permission to "Azure Spring Cloud Resource Provider" on your route tables.
    The platform needs to write new route table rules into the given route table. Without qualified permission, the operation will fail.
  2. You provided wrong route table information, so the platform cannot find it.

 

How to fix:

  1. Unless you have distinct prerequisites necessitating the routing of outbound traffic through a designated gateway or firewall, there is no need to establish your own route tables within the subnets allocated to Azure Spring Apps.
    The platform itself will autonomously generate a new route table within the subnets and configure appropriate route rules for the underlying nodes.

  2. If you need routing outbound traffic to your own gateway or firewall:
    • You have the option to employ your personal route table. Prior to initiating the creation of Azure Spring Apps, please make sure to associate your custom route tables with the two specified subnets.
      If your custom subnets contain route tables, Azure Spring Apps acknowledges the existing route tables during instance operations and adds/updates and/or rules accordingly for operations.
    • You can also use the route tables created by the platform. Just add your new routing rules into it.
      We can see rules named like this: aks-nodepoolxx-xxxx-vmssxxxxx_xxxx. These are the rules added by Azure Spring Apps and they must not be updated or removed.
      You can add your own rules to routing other outbound traffic. For example, using 0.0.0.0/0 to route all the other traffic to your gateway or firewall.
      If you created your own routing rule to route internet outbound traffic, make sure the traffic can reach all the targets mentioned in Customer responsibilities running Azure Spring Apps in a virtual network.
      Hanli_Ren_2-1693233059509.png

       

User Defined Route Issues

Symptom

  • Failed to create Azure Spring Apps service.
  • Failed to start Azure Spring Apps service.
  • Failed to use Public Endpoint feature.
  • Failed to use log stream or connect to App console from public network.

 

Error Messages:

  • Invalid VNet configuration: Route tables need to be bound on subnets when you select outbound type as UserDefinedRouting..
  • Invalid VNet configuration: Both route table need default route(0.0.0.0/0).
  • When connecting to Console from public network, it's reporting:
    A network error was encountered when connecting to "xxx.private.azuremicroservices.io". Please open the browser devtools or try with "az spring app connect" to check the detailed error message.
  • Azure CLI log stream command may report:
    requests.exceptions.ConnectionError: HTTPSConnectionPool(host='xxx.private.azuremicroservices.io', port=443): Max retries exceeded with url: /api/logstream/apps/authserver/instances/xxxxx

 

Common Causes of the issue:

When you choose to use User Defined Route, the platform won’t create any public IP address on the load balancer anymore.
So, it's customer's responsibility to make sure that both subnets can still make calls to all the public targets mentioned in Customer responsibilities running Azure Spring Apps in a virtual network.
This is the reason why we require both route tables need default route (0.0.0.0/0) and route the traffic to your firewall. This firewall is responsible for managing Source Network Address Translation (SNAT) and Destination Network Address Translation (DNAT) processes, converting local IP addresses to corresponding public IP addresses. This configuration enables the platform to successfully establish outbound connections to targets located in the public network.

By design, in UDR type of Azure Spring Apps, we cannot use the following features:
- Enable Public Endpoint
- Use public network to access the log stream
- Use public network to access the console
The same limitations also apply to the Azure Spring Apps using Bring Your Own Route Table feature when customer route egress traffics to a firewall. Because they introduce asymmetric routing into the cluster, this is where the problem occurs. Packets arrive on the firewall's public IP address but return to the firewall via the private IP address. So, the firewall must block such traffic.
We can refer to this doc to get more details: Integrate Azure Firewall with Azure Standard Load Balancer

How to fix:

  1. When creating Azure Spring Apps using UDR outbound type, please carefully read this Control egress traffic for an Azure Spring Apps instance doc. It is recommended to diligently adhere to each step outlined in this document while establishing your individual route tables and configuring the associated firewall settings. This approach ensures the proper and effective control of outbound traffic for your Azure Spring Apps instance.

 

Hope the troubleshooting guide is helpful to you!

 

Items.jpg

To help you get started, we have monthly FREE grants on all tiers – 50 vCPU Hours and 100 memory GB Hours per tier. 

 

Additional Resources 

 

To learn more about Spring Cloud Azure, we invite you to visit the following links: 

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.