Backing Up Azure Stack Infrastructure

So I pretty much wasted my Friday driving 2 hours to Dallas and watching a movie for over 2 hours then driving back for 2 hours.  6 hours that I can’t get back watching a movie that wasn’t really worth driving to go watch.  I really don’t know what I was thinking.  What does this have to do with Azure Stack Infrastructure Backup?  Pretty much nothing except I wish I could restore those 6 plus hours of my life and actually get real work done.  So this is why I am blogging on Friday night, to make myself feel slightly better for wasting that time and hopefully helping someone else out who may be trying to configure the Infrastructure Backup Service.  Most of the information below will come from reading things I learned from Microsoft Documentation on this subject and of course my experience implementing on my multi-node Azure Stack.

So to start with always read through Microsoft’s documentation first.  As I keep stating, their documentation team does a very good job and it is night and day from Technet.

The first thing I need to stress is when creating your Encryption Key, make sure you copy it and put it in a safe spot!  For me, my first backup I forgot to do that so now that data is pretty much worthless if my Stack crashes.

Basics of the Infrastructure Backup

What Infrastructure backup is vs what it doesn’t do.  The sole purpose of the Infrastructure backup is for backing up and restoring data for the Azure Stack infrastructure.  Which includes data from AD, CA, ARM, CRP, SRP, NRP, KeyVault, and RBAC.

This does not include the physical infrastructure of your stack supported by your OEM.  So your network switch configurations, backups of your HLH and OEM Tools, and various third-party resource providers.

It also doesn’t include backups for App Services, SQL, and MySQL Resource providers at this time.

Get It All Out

Let’s get everything out on the table here.  I love Azure Stack and I think everything about this product is great.  However, now that we have multi-node stacks I am finding that the Infrastructure Backup has limitations that need some serious updating and soon.

So here are some of my honest thoughts, the Infrastructure backup which consists of two components basically; the Infrastructure Backup Controller and the Backup Resource Provider need some changes and updates and soon.  Here is my list that should be addressed and I hope it is:

  • Troubleshooting, currently it isn’t very easy to troubleshoot why the configuration of the Backup Infrastructure Controller as well as failed or incomplete backups.
  • Currently, you can only do Full Backups.  Not a bad thing since a full backup is only about 10 GB.
  • All backups are currently manual backups.  There is no integrated way to schedule backups at this time.
  • The manual backups need to be kicked off from within a PowerShell session.  There should be a way to kick off a backup session from within the Admin Portal.


Enable Backup for Azure Stack

Now on to the fun stuff that this blog is supposed to be about.

File Share

The first thing that we will need is a location for the Infrastructure Backup Controller to store the data that it protects.  This is pretty basic right now and nothing fancy.  The storage location must be an SMB v3.0 file share hosted on a storage device that the backup controller has access to.   You will need about 10 GB of space per backup.  Microsoft recommends backing up at least 2 times a day.  For my Stack, we will backup 4 times a day and keep that data for 7 days.  If my math is correct that is about 280 GB of data per scale unit that we are backing up.

For networking, what I had our network guys do is anything from the Azure Stack Infrastructure network, the same one your PEP’s exist on, has outbound rules set for SMB traffic to our file share that was created.

So, what I did was create a basic Windows Server 2016 VM with an attached volume of about 2 TB’s.  This VM resides on a network that the Azure Stack Infrastructure network has access to.  Now, I guess if we wanted to be truly highly available on all our Azure Stack components I could have set up a cluster file share for this but I went with just a single VM with a single share.  The share I took from the Microsoft Documents which was \\AzSBackupStore and setup paths for our current region and stack as well as allowing us to grow as we add more stacks in the future.


We have a service account with full rights to this share, the same service account we will use later during the configuration.

Setting Up Environment

By now you should have your PowerShell and Azure Stack tools installed.  There are other tools that will help along the way as an Azure Stack Operator as well.  Thomas Maurer has a very good blog SETUP AN AZURE STACK CLOUD OPERATOR AND DEVELOPER WORKSTATION ENVIRONMENT on how to set up a good management server.  You can use it as a guild line if needed.

However, for the next section if you already haven’t you can follow the Microsoft Documentation Install PowerShell for Azure Stack to get you through the things you need for backing up your Azure Stack Infrastructure.

For a high-level overview, the following will work as well:

Setup PowerShell for Azure Stack

Get-PSRepository -Name “PSGallery”
Set-PSRepository -Name “PSGallery” -InstallationPolicy Trusted

# Install the AzureRM.Bootstrapper module. Select Yes when prompted to install NuGet
Install-Module -Name AzureRm.BootStrapper

# Install and import the API Version Profile required by Azure Stack into the current PowerShell session.
Use-AzureRmProfile -Profile 2017-03-09-profile -Force

# Install Module Version 1.3.0 if Azure Stack is running 1804 at a minimum
Install-Module -Name AzureStack -RequiredVersion 1.3.0

Get-Module -ListAvailable | where-Object {$_.Name -like “Azs*”}

Download Azure Stack Tools

# Change directory to the root directory.
cd \

# Download the tools archive.
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
invoke-webrequest ` `

# Expand the downloaded files.
expand-archive `
-DestinationPath . `

# Change to the tools directory.
cd AzureStack-Tools-master

Configure Operator Environment to sign in to Azure Stack

# Configure the Azure Stack operator’s PowerShell environment.
$ArmEndpoint = “https://adminmanagement.%5BEnter Your External Address Here]”

# Register an AzureRM environment that targets your Azure Stack instance
Add-AzureRMEnvironment `
-Name “AzureStackAdmin” -ArmEndpoint $ArmEndpoint

Add-AzureRmAccount -EnvironmentName “AzureStackAdmin”


Encryption Key

Take note, I already mentioned above, make sure you copy and save this key.  If you lose this key, you can’t restore the data in case of a catastrophic event.

Now we need to create that encryption key.  From a PowerShell window that is running as Administrator change the director to the location of your Azure Stack Tools.  Then import the AzureStack.Infra Module.  For example:

CD C:\AzureStack-Tools-master\Infrastructure

Import-Module .\AzureStack.Infra.psm1

In the same PowerShell window run the following command to generate the key:

$encryptionkey = New-EncryptionKeyBase64

Now, not that we put the results into a variable.  This will be used later when or if you decide to enable the backups using PowerShell versus the Admin Portal.  If you need to see the results just type the variable and it will display on the screen.


Remember, copy this key and save it for later.


Enable Backup for Azure Stack

You have a choice to enable backups for Azure Stack using the Admin Portal or doing it via PowerShell.  There isn’t a better way but using PowerShell wins your bonus points.  Well, maybe.  I personally did it via the Admin Portal since most likely most of our operators will do the same.  Easier to access than to getting on VPN, jumping to a jump server, then jumping to our Tools server we have set up for managing Azure Stack, and then running the PowerShell command.

Using The Admin Portal

The Admin Portal is good to use if you don’t like PowerShell (not sure why not) or if you just don’t have access to a management/tools server to run Azure Stack PowerShell from.

  1. Open up the Admin Portal
  2. Click on More Services
  3. Click on Infrastructure backup
  4. Click on Configuration in the Infrastructure backup blade.
  5. Fill in the following fields:
    Backup Storage Location
    Confirm Password
    Encryption Key
  6. Click OK.

It will take a few minutes for the Infrastructure backup to get configured.  After a while, the configuration job should show successful.

Now time to move on to running backups.

Using PowerShell

PowerShell is pretty simple, to be honest, and a lot easier in my opinion because if you type something wrong it is fairly easier to just rerun the script with the fix then to have to fill out all the fields again like you would in the Admin Portal.

From your PowerShell session that you have had configured for the operator environment earlier run the following script with the parameters changed to match your environment.


$username = "domain\backupoadmin"
 $password = "password"
 $credential = New-Object System.Management.Automation.PSCredential($username, ($password| ConvertTo-SecureString -asPlainText -Force)) 
 $location = Get-AzsLocation
 $sharepath = "\\serverIP\AzSBackupStore\\seattle"

Set-AzSBackupShare -Location $location.Name -Path $sharepath -UserName $credential.UserName -Password $credential.GetNetworkCredential().password -EncryptionKey $encryptionkey

If you would like to confirm the location and username run the following PowerShell command:

Get-AzsBackupLocation | Select-Object -ExpandProperty externalStoreDefault | Select-Object -Property Path, UserName, Password | ConvertTo-Json

Now time to move on to running backups.


Backing Up Azure Stack

Now it is time to kick off your backup.  Now, currently, there is no way to schedule backups as well as kicking off manual backups from within the Admin Portal.  I am so hoping this changes soon but nothing is currently on the published roadmap for the backup resource provider.

So from a PowerShell Session that has been configured previously, you run a short and simple command.

$location = Get-AzsLocation
Start-AzSBackup -Location $location.Name

You should see this for about 15 – 20 minutes.  If using the PowerShell ISE you will see a status bar pop up but it never shows anything more than 0%.  The job will show completed or failed when done.  Sometimes it will show it was completed with Partial Succeeded status.


Once completed you can see the available backups from the Admin Portal, the last backup time, the location of the backup, and the size.  There is a Next Backup but that is always blank.  Maybe a placeholder for some improvements coming?



At this time I really don’t have any advice on how to troubleshoot failed or partial backups.  The only way right now I can think of is opening a ticket with Microsoft and grab logs for them.


Final Thoughts

Now, this blog only covered backing up the Azure Stack Infrastructure.  I didn’t go into how to recover from a catastrophic loss.  Hopefully I will never need to write a blog about that but for now, you can just read the Microsoft Documentation.  If you are recovering from a catastrophic loss, more than likely you have your OEM and Microsoft involved anyway.

My final thoughts, the backup configuration was pretty simple to configure.  However, it is one area that needs some major updates and hopefully, those updates are coming.  I am going to work on a way to automate running that PowerShell command in order to schedule backups 4 times a day without myself or an operator having to manually run it every time.  I will blog about my solution at a later day.

As for backing up the rest of your Azure Stack, you can talk to your OEM for solutions.  I know Dell EMC has a backup solution that will back up their hardware configurations and it may also extend to backing up the App services, MySQL, and SQL RP’s.  I will check on that and blog about that as well.  As for the Tenant level backups, that can be done with most anything that is supported in Azure.  A lot of vendors have solutions that will work for tenant level backups.



Leave a Reply

Please log in using one of these methods to post your comment: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s