Azure File Share Replication Part 1

Introduction Link to heading

Azure File Share Replication is a script that will allow you to replicate your Azure file shares to a secondary region for disaster recovery purposes. In this first blog post, we will focus on the infrastructure deployment for the VMSS that will be use as the self-hosted agent for the pipeline. We’ll also deploy two storage accounts with file shares, private DNS zone and private endpoints for secure communication. The second blog post, I will explain and show the script behind the replication file sync pipeline that will sync azure file shares from source and destination storage account.

Prerequisites Link to heading

List the necessary knowledge, tools, and setup required to follow along.

An Azure subscription
Azure File Share in the primary region
Azure File Share in the secondary region
Azure Storage Account in the primary region
Azure Storage Account in the secondary region
Private endpoints for both storage accounts
Azure DevOps Project
Azure DevOps service connection
Azure PowerShell module installed

Architecture Overview Link to heading

Infrastructure Diagram

High level steps Link to heading

Deploy the VMSS infrastructure for the self-hosted build agent
Create Azure DevOps service connection
Update Bicep parameters to your desired value
Provision Azure File Shares in both primary and secondary regions
Configure replication between the Azure File Shares
Set up Private Endpoints for secure access
Install and configure the Azure PowerShell module
Validate the setup
Monitor and maintain the infrastructure

Step-by-Step Guide Link to heading

Step 1: Create the file shares in the primary and secondary region Link to heading

In this step, we will use Bicep to deploy two azure file shares in the primary region (eastus) and secondary regions (westus)

// fileshares.bicep
// Define File Share within the primary Storage Account
resource saPrimary 'Microsoft.Storage/storageAccounts@2021-04-01' = {
  name: 'stprimaryeastus'
  location: 'eastus'
  kind: 'StorageV2'
  sku: {
    name: 'Standard_LRS'
  }
  properties: {
    accessTier: 'Hot'
  }
}

resource fileSharePrimary 'Microsoft.Storage/storageAccounts/fileServices/shares@2021-04-01' = {
  name: '${saPrimary.name}/default/eastshare1'
}


// Define File Share within the secondary Storage Account
resource saSecondary 'Microsoft.Storage/storageAccounts@2021-04-01' = {
  name: 'stprimarywestus'
  location: 'westus'
  kind: 'StorageV2'
  sku: {
    name: 'Standard_LRS'
  }
  properties: {
    accessTier: 'Hot'
  }
}

resource fileShareSecondary 'Microsoft.Storage/storageAccounts/fileServices/shares@2021-04-01' = {
  name: '${saSecondary.name}/default/westshare1'
}

Use Azure powershell to deploy the resources in your Azure subscription

New-AzResourceGroupDeployment -name '<DeploymentName>' -ResourceGroupName 'myResourceGroupName' -TemplateFile './FileShares.bicep

Step 2: Provision the Azure DevOps self-hosted agent VMSS in the primary region using Bicep Link to heading

In this step, we will use Bicep to deploy an Azure Virtual Machine Scale Set (VMSS) that will host the Azure DevOps self-hosted agent in the primary region.

// main.bicep
param username string = 'azadmin'
param vnetName string = 'vnet-afs'
param vmssName string = 'vmss-afs'
param location string = resourceGroup().location
param sshPublicKeys_azadmin_name string = 'azadmin'
param storageAccountName string = 'afssa${uniqueString(resourceGroup().id)}'

// SPN ObjectId for RBAC over ADO-Connector
param adospnId string
param adospn string

// list of containers to create in storage account
param containerNames array = [
  'customscriptext'
  'afslogs'
]

@description('The subscription the resources will be deployed into')
param subscriptionName string = subscription().displayName

@description('SSH public key to allow auth to the VMSS cluster')
param rsaPub string

@description('SSH public key to allow auth to the VMSS cluster')
@secure()
param rsaPrivate string 


@description('Configuration for Keyvault unique name per subscription')
var ConfigurationMap = {
  sub1: {
    keyVault: {
      name: 'kv-afs-sub1'
    }
  }
  sub2: {
    keyVault: {
      name: 'kv-afs-sub2'
    }
  }
  sub3: {
    keyVault: {
      name: 'kv-afs-sub3'
    }
  }
  sub4: {
    keyVault: {
      name: 'kv-afs-sub4'
    }
  }
}

@description('Specifies the role the user will get with the secret in the vault. Valid values are: Key Vault Administrator, Key Vault Certificates Officer, Key Vault Crypto Officer, Key Vault Crypto Service Encryption User, Key Vault Crypto User, Key Vault Reader, Key Vault Secrets Officer, Key Vault Secrets User.')
@allowed([
  'Key Vault Administrator'
  'Key Vault Certificates Officer'
  'Key Vault Crypto Officer'
  'Key Vault Crypto Service Encryption User'
  'Key Vault Crypto User'
  'Key Vault Reader'
  'Key Vault Secrets Officer'
  'Key Vault Secrets User'
])
param roleName string = 'Key Vault Secrets Officer'

var roleIdMapping = {
  'Key Vault Administrator': '00482a5a-887f-4fb3-b363-3b7fe8e74483'
  'Key Vault Certificates Officer': 'a4417e6f-fecd-4de8-b567-7b0420556985'
  'Key Vault Crypto Officer': '14b46e9e-c2b7-41b4-b07b-48a6ebf60603'
  'Key Vault Crypto Service Encryption User': 'e147488a-f6f5-4113-8e2d-b22465e65bf6'
  'Key Vault Crypto User': '12338af0-0e69-4776-bea7-57ae8d297424'
  'Key Vault Reader': '21090545-7ca7-4776-b22c-e363652d74d2'
  'Key Vault Secrets Officer': 'b86a8fe4-44ce-4948-aee5-eccb2c155cd7'
  'Key Vault Secrets User': '4633458b-17de-408a-b874-0445c86b69e6'
}

var customData = loadFileAsBase64('cloud-init.yml')

// Create the user assigned identity
resource userAssignedID_afs 'Microsoft.ManagedIdentity/userAssignedIdentities@2022-01-31-preview' = {
  name: 'afsUAID'
  location: location
}

// Create the ssh public key
resource sshPublicKeys_azadmin_name_resource 'Microsoft.Compute/sshPublicKeys@2022-03-01' = {
  name: sshPublicKeys_azadmin_name
  location: location
  properties: {
    publicKey: rsaPub
  }
}

// Create the vnet
resource vnetName_resource 'Microsoft.Network/virtualNetworks@2020-11-01' = {
  name: vnetName
  location: location
  properties: {
    addressSpace: {
      addressPrefixes: [
        '10.7.0.0/22'
      ]
    }
    subnets: [
      {
        name: 'default'
        properties: {
          addressPrefix: '10.7.0.0/24'
          delegations: []
          privateEndpointNetworkPolicies: 'Enabled'
          privateLinkServiceNetworkPolicies: 'Enabled'
        }
      }
      {
        name: 'mgmt'
        properties: {
          addressPrefix: '10.7.1.0/24'
          serviceEndpoints: []
          delegations: []
          privateEndpointNetworkPolicies: 'Enabled'
          privateLinkServiceNetworkPolicies: 'Enabled'
        }
      }
    ]
    virtualNetworkPeerings: []
    enableDdosProtection: false
  }
}

// Create the VMSS
resource vmssName_resource 'Microsoft.Compute/virtualMachineScaleSets@2022-03-01' = {
  name: vmssName
  location: location
  sku: {
    name: 'Standard_D2ads_v5'
    tier: 'Standard'
    capacity: 0
  }
  identity: {
    type: 'UserAssigned'
    userAssignedIdentities: {
      '${userAssignedID_afs.id}': {}
    }
  }
  properties: {
    singlePlacementGroup: false
    upgradePolicy: {
      automaticOSUpgradePolicy: {
        enableAutomaticOSUpgrade: false
        disableAutomaticRollback: false
      }
      mode: 'Automatic'
      rollingUpgradePolicy: {
        maxBatchInstancePercent: 20
        maxUnhealthyInstancePercent: 20
        maxUnhealthyUpgradedInstancePercent: 20
        pauseTimeBetweenBatches: 'PT0S'
      }
    }
    virtualMachineProfile: {
      osProfile: {
        computerNamePrefix: 'afs0224'
        customData: customData
        adminUsername: username
        linuxConfiguration: {
          disablePasswordAuthentication: true
          ssh: {
            publicKeys: [
              {
                path: '/home/azadmin/.ssh/authorized_keys'
                keyData: rsaPub
              }
            ]
          }
          provisionVMAgent: true
        }
        secrets: []
        allowExtensionOperations: true
      }
      storageProfile: {
        osDisk: {
          osType: 'Linux'
          diffDiskSettings: {
            option: 'Local'
            placement: 'ResourceDisk'
          }
          createOption: 'FromImage'
          caching: 'ReadOnly'
          managedDisk: {
            storageAccountType: 'Standard_LRS'
          }
          diskSizeGB: 60
        }
        imageReference: {
          publisher: 'Canonical'
          offer: '0001-com-ubuntu-server-focal'
          sku: '20_04-lts'
          version: 'latest'
        }
      }
      networkProfile: {
        networkInterfaceConfigurations: [
          {
            name: 'afs0224Nic'
            properties: {
              primary: true
              dnsSettings: {
                dnsServers: []
              }
              enableIPForwarding: false
              ipConfigurations: [
                {
                  name: 'afs0224IPConfig'
                  properties: {
                    subnet: {
                      id: vnetName_default.id
                    }
                    privateIPAddressVersion: 'IPv4'
                  }
                }
              ]
            }
          }
        ]
      }
      diagnosticsProfile: {
        bootDiagnostics: {
          enabled: true
        }
      }
    }
    overprovision: false
    doNotRunExtensionsOnOverprovisionedVMs: false
    platformFaultDomainCount: 1
  }
}

// Configure the autoscaling settings
resource vmss_autoscale 'Microsoft.Insights/autoscaleSettings@2015-04-01' = {
  name: 'vmssAutoscaleSetting'
  location: location
  properties: {
    profiles: [
      {
        name: 'Auto created scale condition'
        capacity: {
          minimum: '1'
          maximum: '3'
          default: '1'
        }
        rules: [
          {
            metricTrigger: {
              metricName: 'Percentage CPU'
              metricResourceUri: vmssName_resource.id
              timeGrain: 'PT1M'
              statistic: 'Average'
              timeWindow: 'PT5M'
              timeAggregation: 'Average'
              operator: 'GreaterThan'
              threshold: 75
            }
            scaleAction: {
              direction: 'Increase'
              type: 'ChangeCount'
              value: '1'
              cooldown: 'PT5M'
            }
          }
          {
            metricTrigger: {
              metricName: 'Percentage CPU'
              metricResourceUri: vmssName_resource.id
              timeGrain: 'PT1M'
              statistic: 'Average'
              timeWindow: 'PT5M'
              timeAggregation: 'Average'
              operator: 'LessThan'
              threshold: 25
            }
            scaleAction: {
              direction: 'Decrease'
              type: 'ChangeCount'
              value: '1'
              cooldown: 'PT5M'
            }
          }
        ]
      }
    ]
    enabled: true
    name: 'vmssAutoscaleSetting'
    targetResourceUri: vmssName_resource.id
  }
}

// Create the default subnet
resource vnetName_default 'Microsoft.Network/virtualNetworks/subnets@2020-11-01' = {
  parent: vnetName_resource
  name: 'default'
  properties: {
    addressPrefix: '10.7.0.0/24'
    delegations: []
    privateEndpointNetworkPolicies: 'Enabled'
    privateLinkServiceNetworkPolicies: 'Enabled'
  }
}

// Create the mgmt subnet
resource vnetName_mgmt 'Microsoft.Network/virtualNetworks/subnets@2020-11-01' = {
  parent: vnetName_resource
  name: 'mgmt'
  properties: {
    addressPrefix: '10.7.1.0/24'
    serviceEndpoints: []
    delegations: []
    privateEndpointNetworkPolicies: 'Enabled'
    privateLinkServiceNetworkPolicies: 'Enabled'
  }
}

// Create the keyvault
resource keyvault_afs 'Microsoft.KeyVault/vaults@2022-07-01' = {
  name: ConfigurationMap[subscriptionName].keyVault.name
  tags:{
    Ansible: subscriptionName
  }
  location: location
  properties: {
    enabledForDeployment: false
    enabledForDiskEncryption: false
    enabledForTemplateDeployment: false
    enableRbacAuthorization: true 
    enableSoftDelete: true
    provisioningState: 'Succeeded'
    publicNetworkAccess: 'Enabled'
    sku: {
      family: 'A'
      name: 'standard'
    }
    softDeleteRetentionInDays: 90
    tenantId: subscription().tenantId
  }
}

// Create the role assignment for the user assigned identity
resource keyvault_afs_role_assignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  name: guid(roleIdMapping[roleName],userAssignedID_afs.id,keyvault_afs.id)
  scope: keyvault_afs
  properties: {
    roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', roleIdMapping[roleName])
    principalId: userAssignedID_afs.properties.principalId
    principalType: 'ServicePrincipal'
  }
}

// Create the role assignment for the ADO-Connector
resource keyvault_afs_role_assignment_ado_connectors 'Microsoft.Authorization/roleAssignments@2022-04-01' = if (adospn == 'True') {
  name: guid(roleIdMapping[roleName],keyvault_afs.id)
  scope: keyvault_afs
  properties: {
    roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', roleIdMapping[roleName])
    principalId: adospnId
    principalType: 'ServicePrincipal'
  }
}


// Create the afs host private ssh key secret
resource secret 'Microsoft.KeyVault/vaults/secrets@2019-09-01' = {
  name: 'afs-vmss'
  parent: keyvault_afs
  properties: {
    value: rsaPrivate
    contentType: 'SSH Private Key'
  }
  tags:{
    afsSSH : vmssName_resource.id
  }
}

// Create the afs storage account
resource storageAccount_afs 'Microsoft.Storage/storageAccounts@2022-05-01' = {
  name: storageAccountName
  tags: {
    afs:'CustomScriptExtension'
  }
  location: location
  kind: 'StorageV2'
  sku: {
    name: 'Standard_LRS'
  }
}

// Create the afs storage account blob service
resource storageAccount_blob 'Microsoft.Storage/storageAccounts/blobServices@2021-04-01' = {
  name: 'default'
  parent: storageAccount_afs
  properties: {
    deleteRetentionPolicy: {
      enabled: false
    }
    containerDeleteRetentionPolicy: {
      enabled: false
    }
  }
}

// Create the afs storage account containers
resource afs_containers 'Microsoft.Storage/storageAccounts/blobServices/containers@2021-06-01' = [for name in containerNames: {
  name: '${name}'
  parent: storageAccount_blob
  properties: {
    immutableStorageWithVersioning: {
      enabled: false
    }
    defaultEncryptionScope: '$account-encryption-key'
    denyEncryptionScopeOverride: false
    publicAccess: 'None'
  }
}]

// Create the azure file share private endpoint in primary region
module afsPE 'storageEndpoint.bicep' = {
  name: 'afsPE'
  params: {
    appName: 'afs'
    location: location
    storageEndpointType: 'file'
    storageAccountName: '<primary storage account>'
    subscriptionID: subscription().subscriptionId
    storageResourceGroup: '<primary storage account>'
    vnetName: vnetName
    subnetName: 'default'
  }
}

// Create the azure file share private endpoint in secondary region
module afsPEdr 'storageEndpoint.bicep' = {
  name: 'afsPEdr'
  params: {
    appName: 'afsdrpe'
    location: location
    storageEndpointType: 'file'
    storageAccountName: '<secondary storage account>'
    subscriptionID: '00000000-0000-0000-0000-000000000000'
    storageResourceGroup: '<secondary storage account>'
    vnetName: vnetName
    subnetName: 'default'
  }
  dependsOn: [
    afsPE
  ]
}

output containers array = containerNames

Use Azure powershell to deploy the resources in your Azure subscription

New-AzResourceGroupDeployment -name 'DeploymentName' -ResourceGroupName 'myResourceGroupName' -TemplateFile './main.bicep

Step 3: Update Bicep parameters to your desired value Link to heading

In this step, Update Bicep parameters to your desired value: Customize the Bicep templates with parameters specific to your environment, such as region, size of the VM instances, and the number of instances in the scale set.

// Example parameters
param username string = 'azadmin'
param vnetName string = 'vnet-afs'
param vmssName string = 'vmss-afs'
param location string = resourceGroup().location
param sshPublicKeys_azadmin_name string = 'azadmin'
param storageAccountName string = 'afssa${uniqueString(resourceGroup().id)}'

Step 4: Create Private Endpoints for the storage accounts Link to heading

In this step, We create the private endpoints for the two storage accounts using bicep.

// storageEndpoint.bicep
@description('Provide name of region where resources will be deployed')
param location string = resourceGroup().location

@description('Application Name')
param appName string

@description('Storage endpoint type')
@allowed([
  'file'
  'blob'
  'table'
  'queue'
  'web'
])
param storageEndpointType string

@description('Name of the storage account')
param storageAccountName string

@description('Subscription ID of the storage account')
param subscriptionID string

@description('Resource group of the storage account')
param storageResourceGroup string

@description('Virtual Network Name to be used for PrivateLink')
param vnetName string

@description('Subnet Name to be used for Private Endpoint')
param subnetName string

var privateDNSZoneName = 'privatelink.${storageEndpointType}.${environment().suffixes.storage}' // Generate zone name
var privateEndpointName = 'pe-stg-${storageEndpointType}-${appName}'

// Refrence existing resources
resource storageAccount 'Microsoft.Storage/storageAccounts@2021-02-01' existing = {
  name: storageAccountName
  scope: resourceGroup(subscriptionID, storageResourceGroup)
}

resource vnet 'Microsoft.Network/virtualNetworks@2021-05-01' existing = {
  name: vnetName
}

resource privateEndpointSubnet 'Microsoft.Network/virtualNetworks/subnets@2021-05-01' existing = {
  name: subnetName
  parent: vnet
}

// create private endpoint for storage account
resource privateEndpoint 'Microsoft.Network/privateEndpoints@2020-06-01' = {
  name: privateEndpointName
  location: location
  properties: {
    subnet: {
      id: privateEndpointSubnet.id
    }
    privateLinkServiceConnections: [
      {
        name: privateEndpointName
        properties: {
          privateLinkServiceId: storageAccount.id
          groupIds: [
            storageEndpointType
          ]
        }
      }
    ]
  }
}

resource PrivateDNSZone 'Microsoft.Network/privateDnsZones@2020-01-01' = {
  name: privateDNSZoneName
  location: 'global'
}

resource filePrivateEndpointDns 'Microsoft.Network/privateEndpoints/privateDnsZoneGroups@2020-06-01' = {
  parent: privateEndpoint
  name: 'file-PrivateDnsZoneGroup'
  properties: {
    privateDnsZoneConfigs: [
      {
        name: privateDNSZoneName
        properties: {
          privateDnsZoneId: PrivateDNSZone.id
        }
      }
    ]
  }
}

resource PrivateDNSZoneVnetLink 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2020-01-01' = {
  parent: PrivateDNSZone
  name: uniqueString(PrivateDNSZone.name)
  location: 'global'
  properties: {
    registrationEnabled: true
    virtualNetwork: {
      id: vnet.id
    }
  }
}

Use Azure powershell to deploy the resources in your subscription

New-azResourceGroupDeployment -name 'vmss' -ResourceGroupName 'afs' -TemplateFile './storageEndpoint.bicep'

Explanation of Key Concepts Link to heading

Azure File Share: A fully managed file share in the cloud that is accessible via the SMB protocol. It allows for the storage and access of data in the Azure cloud, similar to how you would use a local file share.

Replication: The process of copying data from one location to another to ensure data redundancy and availability. In the context of Azure File Share, replication can be used to synchronize data across different regions or accounts for disaster recovery and high availability.

Azure Storage Account: A Microsoft Azure service that provides cloud storage that is highly available, secure, durable, scalable, and redundant. Azure File Shares are created within a storage account.

SMB Protocol: Stands for Server Message Block, a protocol for sharing files, printers, serial ports, and communications abstractions such as named pipes and mail slots between computers.

Azure File Sync: A service that allows you to centralize your file shares in Azure Files while keeping the flexibility, performance, and compatibility of an on-premises file server. Azure File Sync transforms Windows Server into a quick cache of your Azure file share.

Data Redundancy: The duplication of data in the cloud to ensure its availability in case of hardware failure or other issues. Azure offers several types of data redundancy, including locally redundant storage (LRS), zone-redundant storage (ZRS), and geo-redundant storage (GRS).

Disaster Recovery: Strategies and processes put in place to ensure the continuation of vital technology infrastructure and systems following a natural or human-induced disaster. In Azure File Share, replication plays a key role in disaster recovery planning.

High Availability: The ability of a system or component to be continuously operational for a desirably long length of time. Replicating Azure File Shares across different regions can help achieve high availability by ensuring that if one region goes down, another can take over.

Geo-Redundant Storage (GRS): A storage option within Azure that automatically replicates your data to a secondary region, far away from the primary region, to protect against regional outages.

RPO (Recovery Point Objective) and RTO (Recovery Time Objective): Key metrics in disaster recovery and business continuity planning. RPO refers to the maximum age of files that an organization must recover from backup storage for normal operations to resume after a disaster. RTO refers to the maximum amount of time, after a disaster, that an organization’s IT system can remain down before causing significant harm to the business.

Common Issues and Troubleshooting Tips Link to heading

Common Pitfalls Insufficient Permissions: A common issue is not having the necessary permissions to create or manage replication on Azure File Shares.

Network Connectivity Issues: Replication might fail or be slow due to poor network connectivity between the primary and secondary regions.

Incorrect Configuration: Misconfiguration of replication settings can lead to data not being replicated correctly or efficiently.

Storage Account Limitations: Hitting the limits of the storage account, such as IOPS or storage capacity, can hinder replication processes.

Incompatible Features: Using features or settings that are not compatible with Azure File Share replication, such as certain types of encryption or data tiering.

Troubleshooting Tips Verify Permissions:

Ensure that you have the necessary role assignments, such as Storage Account Contributor or a custom role with the appropriate permissions for managing file shares and replication. Check Network Connectivity:

Use Azure Network Watcher’s Connectivity Check to verify network connectivity between the regions involved in replication. Consider using Azure ExpressRoute for more reliable and faster connectivity if replication is critical. Review Configuration Settings:

Double-check the replication settings in the Azure portal or via Azure CLI/PowerShell commands to ensure they are correctly configured. Pay attention to the replication interval, bandwidth settings, and the file share’s tier. Monitor Storage Account Limits:

Regularly monitor the usage against the limits of your storage account through the Azure portal or Azure Monitor. Consider upgrading your storage account or adjusting the workload if you’re close to hitting the limits. Ensure Feature Compatibility:

Review the Azure documentation to ensure that all features you’re using are compatible with Azure File Share replication. If you’re using an incompatible feature, look for alternative approaches or consider not using that feature for replicated file shares. Use Azure Support and Documentation:

If you’re unable to resolve the issue, use the Azure support channels for assistance. The Azure documentation is regularly updated with troubleshooting guides and best practices for managing Azure File Shares and replication.

Conclusion Link to heading

In the post, we explored how to provision the ADO build agent that will be used to run our replication script for the Azure File Share replication, covering key concepts such as Azure File Share, replication types (LRS, ZRS, GRS, and GZRS), and the importance of data redundancy for disaster recovery and high availability.

For more information and to deepen your understanding, here are some useful links to Azure documentation and resources:

These resources provide a comprehensive guide to getting started with Azure File Shares and mastering replication and redundancy options.

Call to Action Link to heading

Your feedback and questions are invaluable to us. Whether you’re experimenting with Azure File Share replication, facing challenges, or have insights to share, we’re all ears. Here are a few ways you can engage:

Leave a Comment: Got thoughts or questions? Don’t hesitate to leave a comment below. We’re looking forward to hearing your experiences and insights.
Ask Questions: If you have any questions or need clarification on any of the topics covered, feel free to ask. We’re here to help you navigate your Azure File Share replication journey.
Subscribe for Updates: Stay in the loop! Subscribe to our newsletter for the latest updates, tips, and best practices. Subscribe here.

Your participation helps us create better content and fosters a community of learners and experts. Let’s learn and grow together!