This week virtual network integration and with that a bring-your-own virtual network capability was made available for Container App Environments: https://techcommunity.microsoft.com/t5/apps-on-azure-blog/azure-container-apps-virtual-network-integration/ba-p/3096932

As mentioned in my Container Apps post last week this virtual network integration is elementary in my scenario to migrate workloads from currently VNET integrated Service Fabric clusters to Container Apps.

Unfortunately - I guess for the moment until scaling behavior is more nuanced - subnets with a CIDR size of /21 or larger are required: https://docs.microsoft.com/en-us/azure/container-apps/vnet-custom?tabs=bash&pivots=azure-portal#restrictions

Still I want to continue exploring Container Apps for our scenario and so I mixed in some private link and private DNS magic I already used a while back.

solution elements

To simplify terms for this post I assume the corporate network with the limited address space would be the hub network and the virtual network containing Container Apps (with the "huge" address spaces) would be spoke network.

This is the suggested configuration:

private link service with private endpoint hub/spoke setup

a private link service within spoke network linked to the kubernetes-internal Load balancer
a private endpoint in the hub network linked to private link service above
a private DNS zone with the Container Apps domain name and a * A record pointing to the private endpoint's IP address
a jump VM in the hub network to test service invocation

disclaimer: as the title states, this is a preliminary configuration, some elements may change or go away in the coming weeks or the whole setup could be obsolete!

Below I will refer to shell scripts and Bicep templates I keep in this repository path: https://github.com/KaiWalter/container-apps-experimental/tree/ca-private-link/ca-bicep.

prerequisites

Azure CLI
Bicep
.NET Core 3.1 SDK for application build and deployment

Why mixing Bicep and Azure CLI?

I handled deployment for my previous post with Pulumi, yet had to turn to Bicep for this deployment as the InternalLoadBalancerEnabled = true setting was not processed correctly (tested with Pulumi.Native 1.56.0-alpha.1643832293). Although I personally prefer the descriptive style of Bicep over a long string of Azure CLI commands and have it processed in a linked set of templates, I had to fall back to the CLI as not yet all Container App properties like staticIp and defaultDomain could be processed as Bicep outputs.

stage 1 - deploy network and Container Apps environment

The first section of deploy.sh together with main.bicep deploys

hub and spoke network with network.bicep
logging workspace and Application Insights with logging.bicep
Container App Environment connected to spoke network and an internal load balancer with environment.bicep
a jump VM with vm.bicep
a Container Registry with cr.bicep for later application deployments

RESOURCE_GROUP="ca-kw"
LOCATION="northeurope"
ENVIRONMENTNAME="ca-kw"
if [ $(az group exists --name $RESOURCE_GROUP) = false ]; then
    az group create --name $RESOURCE_GROUP --location $LOCATION
fi
SSHPUBKEY=$(cat ~/.ssh/id_rsa.pub) # create with ssh-keygen first
az deployment group create --resource-group $RESOURCE_GROUP \
    --template-file main.bicep \
    --parameters "{\"environmentName\": {\"value\": \"$ENVIRONMENTNAME\"},\"adminPasswordOrKey\": {\"value\": \"$SSHPUBKEY\"}}"

for the VM deployment it is assumed that a id_rsa and id_rsa.pub key pair is generated or available on ~/.ssh

stage 2 - private link service and private endpoint

Here the static IP of the Container App Environment is used to find the corresponding Internal loadbalancer's Frontend IP configuration. This is not the most elegant and reliable way, but should do it until I find a better reference from Container App Environment to Frontend IP configuration.

With privatelink.bicep these elements are created:

a Private link service in the spoke network connected to Internal loadbalancer's Frontend IP configuration
a Private endpoint in the hub network connected to the Private link service

ENVIRONMENT_STATIC_IP=`az containerapp env show --name ${ENVIRONMENTNAME} --resource-group ${RESOURCE_GROUP} --query staticIp -o tsv --only-show-errors`
ILB_FIP_ID=`az network lb list --query "[?frontendIpConfigurations[0].privateIpAddress=='${ENVIRONMENT_STATIC_IP}'].frontendIpConfigurations[0].id" -o tsv`
VNET_SPOKE_ID=`az network vnet list --resource-group ${RESOURCE_GROUP} --query "[?contains(name,'spoke')].id" -o tsv`
SUBNET_SPOKE_JUMP_ID=`az network vnet show --ids $VNET_SPOKE_ID --query "subnets[?name=='jump'].id" -o tsv`
VNET_HUB_ID=`az network vnet list --resource-group ${RESOURCE_GROUP} --query "[?contains(name,'hub')].id" -o tsv`
SUBNET_HUB_JUMP_ID=`az network vnet show --ids $VNET_HUB_ID --query "subnets[?name=='jump'].id" -o tsv`
az deployment group create --resource-group $RESOURCE_GROUP \
    --template-file privatelink.bicep \
    --parameters "{\"subnetSpokeId\": {\"value\": \"$SUBNET_SPOKE_JUMP_ID\"},\"subnetHubId\": {\"value\": \"$SUBNET_HUB_JUMP_ID\"},\"loadBalancerFipId\": {\"value\": \"$ILB_FIP_ID\"}}"

stage 3 - private DNS

Finally in privatedns.bicep

a private DNS zone with the domain name of the Container App Environment
a A record pointing to the Private endpoint
virtual network link of the private DNS zone to the hub network

is created.

ENVIRONMENT_DEFAULT_DOMAIN=`az containerapp env show --name ${ENVIRONMENTNAME} --resource-group ${RESOURCE_GROUP} --query defaultDomain -o tsv --only-show-errors`
PEP_NIC_ID=`az network private-endpoint list -g $RESOURCE_GROUP --query [0].networkInterfaces[0].id -o tsv`
PEP_IP=`az network nic show --ids $PEP_NIC_ID --query ipConfigurations[0].privateIpAddress -o tsv`
az deployment group create --resource-group $RESOURCE_GROUP \
    --template-file privatedns.bicep \
    --parameters "{\"pepIp\": {\"value\": \"$PEP_IP\"},\"vnetHubId\": {\"value\": \"$VNET_HUB_ID\"},\"domain\": {\"value\": \"$ENVIRONMENT_DEFAULT_DOMAIN\"}}"

add some apps

build.sh is used to deploy 2 ASP.NET Core apps which provide some basic Dapr service-to-service invocation calls.

testing the approach

test.sh uses the jump VM to test the direct invocation and the service-to-service invocation of both apps:

app1
{"status":"OK","assembly":"app1, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null","instrumentationKey":null} <<-- check app1 own health
{"status":"OK","assembly":"app2, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null","instrumentationKey":null} <<-- check app1 remote health
app2
{"status":"OK","assembly":"app2, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null","instrumentationKey":null} <<-- check app2 own health
{"status":"OK","assembly":"app1, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null","instrumentationKey":null} <<-- check app2 remote health

conclusion

I hope that the BYO virtual network footprint of Container Apps will be reduced before going into GA - one of the main reasons Function / App service deployment do not really work for our ~200 (~40 apps, 5 deployment stages) Function App microservices scenario.

Nevertheless with the private link approach - separating corporate address space from some virtual address space without bringing in one of the more heavier resources like Azure Firewall or VPN gateway - would be a valid option for me.