There are instances where we would need to backup blobs. As Azure does not offer automated backups of Blob storage’s, we need to think this out. Here, I will attempt to outline what we need to consider to come up with a backup strategy and what we need to do, to backup a blob storage.
The app that I worked on, had a blob storage that would have a container with PDFs in them. The PDFs uploaded into the system would be stored inside this container. Once stored, they would not be updated. We needed to maintain a backup of these documents.
Backup Strategy
When we backup solutions, we need to consider if the application is a “Mission critical” application. For example, does the application have a binding SLA stating that a downtime of 24 hours or longer would result in a Financial liability? We also need to see if it has a low rate of data change. For example, losing up-to an hour of change is acceptable or something in those lines according to the needs of the application. Finally, the solution should also be cost effective.
Decide the Recovery Point Objective (RPO), maximum time for which data loss can be tolerated. Example: 5 minutes
Decide the Recovery Time Objective (RTO), maximum time that can be afforded to fix or restore the service. Example: 6 hours
Decide the Backup retention policy, how long the backups need to be maintained. Example: 5 years
The backup strategies that I have proposed here, uses Azure storage as against copying down the Backups into on-prem Server. This is done so that, apart from avoiding use of additional storage on-prem, using Azure allows us to leverage features in Azure that maintains blobs, allows us to create alerts, allows easy restore etc. Also, Azure has tiers with very competitive rates.
Blob Snapshots did not suit our requirement since snapshots would be created for individual blobs. In times of restore, a copy (like AZCopy) of storage with Snapshots, would create Blobs equal to number of Snapshots. Also, the snapshots are created in the same URL as the blob.
Backup solution
Soft-Delete feature is to be set ON in the Storage Account used by the Blob. This would allow accidentally deleted files to be restored easily (instead of looking at the backup to recover the same). This also protects against data erase that can happen by overwrite. This would allow to meet maximum RPO and RTO, no data would be lost for any time within the recoverable time. Example: 30 days.
The following steps were done to backup blobs
- Create Storage account for back ups
- Create an Automation Account
- Create a Runbook of type PowerShell within the Automation Account
- Add a schedule to schedule run of the powershell Runbook
Except point 3, all the above steps are fairly simple to look up in Azure, I will leave you to find them.
Below is the Powershell Code to create containers with the date on which the backup runs, and copy blobs into them as a backup. The first run would copy all the blobs into the backup container (named with date as suffix). The subsequent runs would create containers for the day and only copy in blobs that were created after the last run date of backup.
<# Parameters #>
Param (
[Parameter(Mandatory=$false,Position=0)][String]$azureStorageAccountNameSource = “azureStorageAccountNameSource”,
[Parameter(Mandatory=$false,Position=2)][String]$azureStorageAccountSourceKey = “999sjisjlhkjhckjhskhshhsomerelevantkeyhere”,
[Parameter(Mandatory=$false,Position=3)][String]$azureStorageAccountSourceContainer = “someazureStorageAccountSourceContainer”,
[Parameter(Mandatory=$false,Position=4)][String]$azureStorageAccountNameDest = “someazureStorageAccountNameDest”,
[Parameter(Mandatory=$false,Position=5)][String]$azureStorageAccountDestKey= “jdiqifoiuqwoiufoiwusomerelevantkeyhere”,
[Parameter(Mandatory=$false,Position=6)][String]$azureStorageAccountDestContainerPreFx=”someazureStorageAccountDestContainerPreFx”
)# Build backup container Name
$date = ("{0:yyyy-MM-dd}" -f (Get-Date))
$bkUpContainerName = $azureStorageAccountDestContainerPreFx + $date# Initiate the Source Azure Storage Context
$ctxSource = New-AzureStorageContext -StorageAccountName $azureStorageAccountNameSource -StorageAccountKey $azureStorageAccountSourceKey# Initiate the Destination Azure Storage Context
$ctxDest = New-AzureStorageContext -StorageAccountName $azureStorageAccountNameDest -StorageAccountKey $azureStorageAccountDestKeytry{
#get containers within the storage
$azContainersDest = Get-AzureStorageContainer -Context $ctxDest | sort @{expression="LastModified";Descending=$true}
#is this is the first container created for Backup? if yes, create new container and copy all contents into the new destination container
if($azContainersDest -eq $null) {
# Create dest Container
New-AzureStorageContainer -Name $bkUpContainerName -Context $ctxDest -Permission blobGet-AzureStorageBlob -Container $azureStorageAccountSourceContainer -Context $ctxSource |
Start-AzureStorageBlobCopy -DestContainer $bkUpContainerName -Context $ctxDest
}
else {$azContainer = Get-AzureStorageContainer -Container $bkUpContainerName -Context $ctxDest -ErrorAction SilentlyContinue# there are other containers but a Container with todays date does not exist
if($azContainer -eq $null) {$lastCreatedContainerName = $azContainersDest[0].Name;
Write-Output "The lastCreatedContainerName ... " $lastCreatedContainerName$lastCopiedDateStr = $lastCreatedContainerName.Substring($lastCreatedContainerName.Length-10, 10)
Write-Output "The last copied date... " + $lastCopiedDateStr
$lastCopiedDate = get-date $lastCopiedDateStrWrite-Output "Creating new container...." + $bkUpContainerName
# Create dest Container
New-AzureStorageContainer -Name $bkUpContainerName -Context $ctxDest -Permission blob
} # a container with todays date exists
else {
# Find the last copied date time of Backup with todays date
$blobsDest = Get-AzureStorageBlob -Container $bkUpContainerName -Context $ctxDest | sort @{expression="LastModified";Descending=$true}
# First blob in that list would be the last modified.
$latestBlobDest = $blobsDest[0]
$lastCopiedDate = $latestBlobDest.LastModified
}$blobsSrc = Get-AzureStorageBlob -Container $azureStorageAccountSourceContainer -Context $ctxSource | sort @{expression="LastModified";Descending=$true}
Write-Output "Are there any files uploaded since the last copy? " $lastCopiedDate
foreach ($blobSrc in $blobsSrc) {
if($blobSrc.LastModified -gt $lastCopiedDate) {
Write-Output "Copying blob into backup container. Blob last modified on ... " $blobSrc.LastModified.
$blobSrc | Start-AzureStorageBlobCopy -DestContainer $bkUpContainerName -Context $ctxDest
}
else {
Write-Output "No more files to copy"
return
}
}
}
}
catch [Microsoft.WindowsAzure.Commands.Storage.Common.ResourceNotFoundException] {
Write-Output "Container Not Found..."
# Report any other error
Write-Error $Error[0].Exception;
}
catch {Write-Output "An Error has occured..."
# Report any other error
Write-Error $Error[0].Exception;
}
Hope that was of help!
