This post has been republished via RSS; it originally appeared at: New blog articles in Microsoft Tech Community.
Using PaaS resources in Azure brings convenience to developers such as saving times, concentrating on coding, and etc. However, there is a paradox which convenience and secure might never on the same direction.
Your customers might challenging you by exposing the inter-connection among resources under public networks. For example, You builded a Function App for tracking customers' information and storing them in to a Blob container (from a Storage Account). Meanwhile, connections invoked by that function uses public domain name to the container, which means those requests might have a great opportunity routing outside.
For security purpose such as ISO/IEC 27001, you may want your resources to co-op privately, however, detail network settings among PaaS services are extreamly hard to find (at least for me, a pure newbie). So this article is my research for connecting PaaS devices such as Azure Function App to a Blob container under private network (and cross region).
Here is the outline, I will inreoduce 2 methods: Service Endpoints and Private Endpoints, help you to protect your connection.
Outline
- Service Endpoints
- Private Endpoints
- Additional Information
- Conclusion
- References
Service Endpoints
A quick conclusion: We cannot perform a cross-region private connection under a Service Endpoints settings.
Service Endpoints is a combination of settings related to "route for the inter-resource connections by avoiding expose them under public networks". For example, a function app invoking an outbound request to a blob container under a domain named "someblobs.blob.core.windows.net", then, a DNS server resolve that name to an IP address of that container, and you all know the rest of the things.
The point is, "someblobs.blob.core.windows.net" via resolvable in Global DNS server and you will retrieve a public IP address from that, even thought you can directly reach to that resource through private network. In some special cases under the function app and blob container are located in the different regions, you may not have a chance to achieve such a purpose.
If you don't mind, studying from this part is a goot way to start. Let's go to the Azure portal and create a Resource Group.
| Column Name | Value |
|---|---|
| Resource group | <you specified> |
| Region | <you specified> e.g., Japan East |
Create a Log Analytics Workspaces inside that Resource Group.
| Column Name | Value |
|---|---|
| Name | <you specified> |
| Region | <you specified> e.g., Japan East |
Create 2 Storage Accounts, one of them in the region is the same with the Resource Group, the other is different from it.
| Column Name | Value |
|---|---|
| Storage Account Name | <you specified> |
| Region | <you specified> e.g., Japan East / West US |
Inside 2 Storage Accounts, create a Blob container from each of them.
| Column Name | Value |
|---|---|
| Name | <you specified> |
| Public Access Level | Blob |
Inside 2 Blobs, upload a test file to each of them, you can create a file named "test" and write some message in it, and upload through Azure portal.
Last step of the Storage Account, go to the Diagnostic Settings and add a setting on it. We use it to determine the behavior to the blob such as information (source IP) from the requests.
Left side manu > Monitoring > Diagnostic Settings > Blob (under your Storage Account Name) > Add diagnostic setting
| Column Name | Value |
|---|---|
| Diagnostic Settings Name | <you specified> |
| Categories | StorageRead |
| Destination Details | Send to Log Analytics workspace |
| Log Analytics Workspaces | <you specified> |
Now we start creating our Function App. We are using Functions Premium plan because it supports Virtual Network Integration, thus, please ensure to remove all the resources once you're finished testing.
| Column Name | Value |
|---|---|
| Resource Group | <you specified> |
| Function App Name | <you specified> |
| Publish | Code |
| Runtime Stack | Node.js |
| Version | 14 LTS |
| Region | Japan Esat |
| Operating System | Windows |
| Plan Type | Functions Premium |
| Windows Plan | <you specified> |
| Sku and Size | EP1 |
| Storage Account | <you specified> please do not use your previous created ones |
| Enable network injection | Off |
| Enable Application Insights | No |
Please download the sample codes from my Github, consider the readme file and make some changes to fit your experiment. Finally deploy it on your Function App.
Now by accessing the function triggers (there are 2) via the following URLs, they may invoke a GET request to the blob file from the container:
https://0502lsecpriendplfuncapp.azurewebsites.net/track_a (Function App and Blob are in the same region)
https://0502lsecpriendplfuncapp.azurewebsites.net/track_b (Function App and Blob are in different regions)
You will get an OK message once you click it, and can close it after that.
Now we're going to the Log Analytics workspace.
Left side manu > General > Logs
Please input the following KQL in the Query Zone
You can see the line with AccountName 0502lsecpriendplstoragea which Function App and Blob are in the same region, the Caller IP Address is 100.79.52.132, which is an interal IP address[5].
And the line with AccountName 0502lsecpriendplstorageb which Function App and Blob are in different regions, the Caller IP Address is 20.210.7.31, whioh can be found on outbound IP section from your Function App.
It means you are exposing the connection in public when a cross-region inter-connection be invoked, and might get the potencial risk from your system.
Now we're going to create a Virtual Network for our Function App, to verify whether a Function App inside a VNet can reach the containers under private network or not.
| Column Name | Value |
|---|---|
| Name | <you specified> |
| Region | <you specified> e.g., Japan East |
And then we immediatly specify some subnet range for our Function APP
Left side manu > Settings > Subnets > Add subnet
| Column Name | Value |
|---|---|
| Name | <you specified> |
| Subnet Access Range | 10.1.1.0/24 (i.e., it could be different on yours based on your default suubnet ranges) |
| All the rest | use the default value |
Now we're going back to our Function App and bind it on the VNet
Left side manu > Settings > Networking > Outbound Traffic > VNet integration > Add VNet > Provice the information you've just created
In the rest of this part, we gonna make a service endpoint by specifying a VNet from both of your Storage Accounts.
Left side manu > Security + Networking > Networking > Firewalls and Virtual Networks
| Column Name | Value |
|---|---|
| Public network access | Enabled from selected virtual networks and IP addresses |
| Virtual networks | Add Existing Virtual Network |
| Virtual Netework | <you specified> |
| subnet | default / <you specified> (i.e., yes, add them separately) |
| All the rest | use the default value |
And please wait for nearly 15 minutes for populating network settings inside Azure.
Maybe you are realizing what's going on, you cannot specify the VNet from a Storage Account if they are on different regions, so we temporally stop here.
Since you've just specify the Function App (Japan East) inside a VNet (Japan East) corresponding to a Storage Account (Japan East), we're going to check the result by re-invoke a connection
PS: You can restart the Function App before invoking, and then click this link:
https://0502lsecpriendplfuncapp.azurewebsites.net/track_a
And see the result
You can see the line with AccountName 0502lsecpriendplstoragea which Function App and Blob are in the same region, the Caller IP Address is change to the VNet internal IP.
Maybe you're thinking: why don't we put a Function App (Japan East) inside a VNet (West US) corresponding to a Storage Account (West US)?
Well, the "Firewalls and Virtual Networks" setting is a passive restriction, use to detect the source information and apply some extra work (e.g., accept, drop) due to rules. So it has no idea how to interrupt the resolving rule from the Function App of a VNet. You still get the public IP address from the Logs when changing to new VNet (West US) and accessing Storage Account (West US). Meanwhile, you cannot even choose the VNet on the binding list from Function App under Azure portal if they are not in the same region[4].
PS: Although there do have some extra setting would be launch once the "Firewalls and Virtual Networks" are finished setting, we cannot apply this on our situation because we're on the PaaS service. The extra setting is: Adding a "Effective security rules" to the NIC if there are VMs inside the target VNet. Which used to tell the VM taking internal resolving rules to the "someblobs.blob.core.windows.net " domain.
Finally, the bad news is we now understand why using Service Endpoints between PaaS resources and Storage Accounts under cross-region connection is currently unhidable. And the good news is there are some ways to compromise this kind of issue.
Thus, I am going to introduce the next tool.
Private Endpoints
Private Endpoints shares part of the same purpose from Service Endpoints but more flexible (i.e., which means there are more details settings behind each resources). Now we use the previous resources created from last chapter and keep going.
Firstly we clean up some of the previous resources:
- In the first Storage Account, in the "Firewalls and virtual networks", remove all the VNet, and switch back to the "Enabled from all networks".
- In the Function App, disconnect the (previously binded) VNet from it.
- In the Virtual Network, delete the VNet you've created on last chapter.
Then, we create a VNet in same regions matching your Storage Accounts:
We skip the similar snapshots.
| Column Name | Value |
|---|---|
| Name | <you specified> |
| Region | <you specified> e.g., Japan East |
Inside this VNet, we keep creating a subnet form in it
| Column Name | Value |
|---|---|
| Name | <you specified> |
| Subnet Access Range | 10.1.1.0/24 |
| All the rest | use the default value |
We have to link VNet which under the different region from Function App to the related Storage Account, so go to Storage Account:
Left side manu > Security + Networking > Networking > Firewalls and Virtual Networks > Disable "Public network access"
Left side manu > Security + Networking > Networking > Provate Endpoint Connections > Add Private Endpoint
| Column Name | Value |
|---|---|
| Name | <you specified> |
| Region | <you specified> select the VNet under the same region from Function App |
| Target sub-resource | Blob |
| Virtual Network | <you specified> |
| subnet | default |
You might find the tricky part: you can specify the region of your VNet different from the private endpoint of your Storage Account, that means we could logically invoke an internal connection from our Function App to the blob.
Now go to the Private DNS zones:
Left side manu > Settings > Virtual network links > Add
| Column Name | Value |
|---|---|
| Link Name | <you specified> |
| Virtual Network | <you specified> |
| Auto-Registration | check |
PS: You may already found one on the list, if so, please delete it first
This step grants the ability to the target VNet which could use the internal name resolving rules form my Private DNS zones, and is the key step to override the VNet's resolving rule. You have to add each of the VNets from that interface when having a complex VNet chaining topology.
Back to the Function App, and bind it on the VNet
Left side manu > Settings > Networking > Outbound Traffic > VNet integration > Add VNet > Provice the information you've just created
Since you've just specify the Function App (Japan East) inside a VNet (Japan East) corresponding to a Storage Account (West US) using Private Endpoints (Japan East), we're going to check the result by re-invoke a connection
PS: You can restart the Function App before invoking, and then click this link:
https://0502lsecpriendplfuncapp.azurewebsites.net/track_b
And see the result
Voila! You can see the line with AccountName 0502lsecpriendplstorageb which Function App and Blob are in the different regions, and the Caller IP Address is change to the VNet internal IP.
Finally, we secure the inter-connectionn among our cross-region PaaS resources with some additional settings and get the information about how the Vnet works. It's cool isn't it?
Additional Information
- You cannot create a new Function App using an existing Blob you've create for test, or you will no longer acccessing the file system from the Function App once you disable the Public network access from that Storage Account.
- Although you disable the Public network access from that Storage Account, the accessing log still occurs on the Log Analytics workspace when there is an outside invoking attempt.
- For some Function Apps upgraded from consumption plan, you might add a confuguration "WEBSITE_CONTENTOVERVNET=1" to resolve the Storage Accounts' restriction[1][2].
Conclusion
To achieve cross-region inter-connection, we have to choose a right network resources equipping a "region choosable" feature, such as Private Endpoints in this case.
To resolve the public domain name correctly, we have to add a Virtual Network Link[3] on each of the related VNets.
References
- Tutorial: Integrate Azure Functions with an Azure virtual network by using private endpoints
- App settings reference for Azure Functions
- What is a virtual network link?
- (MS internal) App Service to Storage Account Connection Condition Summary
- (MS internal) Pseudo IP does not SNAT ICMP requests
Special Thanks (alphabetical order)
- boqian.wang
- dylan.ho
- po.chen
- wanjing
- yawei.hu