Virtualize: Backup Virtual Machines to Multiple iSCSI Targets

If you have a tight budget and need a backup solution for multiple virtual machines, you can create your own using PowerShell. While there are a few completely free options available such as Veeam or Altaro, you are either locked into what may be proprietary archive formats or limits on the total number of virtual machines you can backup. In my case, I choose to keep my costs to a minimum and leverage the integrated ability to provide a customizable solution with PowerShell and Windows Server Backup.

More often than not, each environment has its’ own requirements. This solution is not use case dependent, meaning it has been design generically and can be used for nearly any standard Hyper-V virtual machine backup to a standard iSCSI Target. I will explain what this use case looks like, what the dependencies are, and how to adapt the PowerShell script to fit your needs. While each use case is unique, if the underlying platforms are similar then common frameworks can be mashed up to create a solution that will work.

Infrastructure Overview:

For this environment, I have two systems in place the first being a Hyper-V Server and the second being a basic file storage server with the iSCSI Target. This is pretty common, and there are a myriad of ways to setup the infrastructure between the two systems. The hardware makes no difference; however, the methods with which we provision our storage for use is important and can be varied. I have chosen to utilize a SAN storage solution with an iSCSI Initiator & Target design. This may be a little less common in a home lab, but for small business solutions and large enterprises it is a common standard. While it does require a few more steps to setup versus using SMB 3.0 Multi-Channel, I find that it is reliable and stable.

 

Figure 1: Generic Infrastructure for iSCSI Target

 

Connectivity:

In general, it is recommended to separate iSCSI traffic from your LAN traffic to create what I like to think of as an MPIO network. I have found that this is not absolutely necessary if you have a switch that can leverage VLAN segmentation, but it becomes a bit expensive and in my opinion unnecessary when you can go and buy a cheap unmanaged switch and a few CAT 6 cables for $50USD. In my case, this is exactly what I have done to separate the segments and it works really well.

One nice thing that I have found is that you can still utilize SMB 3.0 Multichannel. In environments for clustering, this may be nice as you can mixed straight iSCSI services with SMB and in Server 2012 R2 it just seems to work. That is until you decide to tweak your MPIO configuration and eliminate unnecessary adapter bindings. While Multichannel is great and appears to work out of the box, if you do not have RDMA and RSS supported adapters the performance just did not seem to be as good as a dedicated MPIO segment in my testing. The only binding required in an MPIO segment is TCPv4, all other bindings can be disabled. You cannot get much simpler than that if you really want to minimize an attack surface and streamline throughput.

 

Adapters Organized:

For each system, I also renamed the adapters to make things easier to script and manage. This isn’t necessary, but if you are like me I will often forget which adapter is what while looking at names like “Ethernet 2” or “Ethernet 3.” Since I’ve designed the segment for something specific, I’ll rename the adapters appropriately.

 

Figure 2: Rename Adapter by Segment

 

If you are using Hyper-V Server, you’ll have to use the PowerShell to configure the adapters, but it is really simple:

Rename-NetAdapter –Name “Ethernet” –NewName “MPIO1”

I will do this on both the Initiator (Hyper-V Server) and the Target (iSCSI) servers, just to keep things straight. If you have some adapters that are also setup to be used as virtual switches, it is ok to rename the adapters on the host as the Virtual Switch Manager utilizes the Interface Description property of the adapter when listing available hardware.

Figure 3: PowerShell Get-NetAdapter

 

Figure 4: Virtual Switch Manager

 

 

Tweak Adapter Performance:

This topic is covered a gazillion times over, but needs to be mentioned. There are only two things that truly need to be adjusted as far as the MPIO segment is concerned, enabling Jumbo Frames and tapping down interrupt moderation. Even that second part is an option; however, based on Intel guidelines it is a recommendation that I follow.

While these adapter features are easily adjusted in the standard GUI, PowerShell comes to our rescue here if you are using server core or Hyper-V Server.

List bindings for adapter MPIO1:

Get-NetAdapterAdvancedProperty | Where-Object {$_.Name -like ‘MPIO1’} | Format-Table

Set binding property for Jumbo Frames:

Set-NetAdapterAdvancedProperty -Name “MPIO1” -RegistryKeyword “*JumboPacket” -RegistryValue 9216

Just note that depending on the hardware, the values for “RegistryValue” may be different. If you enter in a bogus number like 1111111, PowerShell will error and provide you with a list of acceptable values for the property, how nice is that?

Figure 5: Binding Property Error

 

Optional Tweaks:

If you are interested in retaining some SMB 3.0 Multichannel capability, the following adjustments are probably going to spoil things for you. However for MPIO, any bindings other than TCPv4 are not needed and should be disabled. For example, disable IPv6, Client, and File and Print Services.

Disable IPv6:

Set-NetAdapterBinding -Name “MPIO1” -ComponentID ms_tcpip6 -Enabled $false

Disable Client Services:

Set-NetAdapterBinding -Name “MPIO1” -ComponentID ms_msclient -Enabled $false

Disable File and Print Services:

Set-NetAdapterBinding -Name “MPIO1” -ComponentID ms_server -Enabled $false

 

Add Roles and Features:

For both systems, you will need to enable the Windows Server Backup feature… which would seem obvious. The other role that is required is to add the iSCSI Target and VSS providers to the file server. You will also need to add the MPIO feature to the Hyper-V Server or Initiator. These steps are simple, and I really like being able to manage these systems from my workstation via Server Manager. Microsoft has done a bang-up job so far with these tools.

Figure 6: Server Manager Roles and Features, File Server

 

Update PowerShell:

One issue I ran into after installation and applying the latest updates was related to PowerShell. You may need to manually update it after adding some of the features. So just make certain that after all updates are applied and the roles and services have been added to execute the command to update PowerShell. Just open an elevated terminal of PowerShell by right-clicking its icon (I have added it to my taskbar), and select “run as administrator” then execute the Update-Help cmdlet. It will automagically go out and grab the latest stuff from Microsoft.

 

Configure iSCSI Targets:

On your file server, create a location in which you would like to store your iSCSI Target virtual disks. By default in Server Manager under File and Storage Services in the iSCSI section, when you choose to create a new iSCSI Virtual Disk via the wizard it will default to a volume and create a directory called “iSCSIVirtualDisk” which is fine and dandy. However, I like to be different and choose the custom path option and browse to a volume and path that I’ve create.

Creating the iSCSI Targets, and getting the Initiator connected to them takes a few extra steps and while it is not hard to do, I will not break down all the steps necessary for it here. There are dozens of tutorials for this out there, and here are a few links to some I’ve taken a look at and followed:

CSCM IT Solutions guide to configure MPIO

WindowsNetworking.com “Configuring iSCSI Storage”

Windows IT Pro: “Microsoft Multipath I/O for iSCSI”

 

Once you’ve got a handle on how to setup the Initiator and Target, you need to make a decision on how many targets you want to create. For the purpose of this exercise and for the use of the sample backup script, I create one target for each virtual machine. You could just as easily create a single target with a large dynamically expanding virtual disk for all of the virtual machines. However, in my mind it is far more flexible to keep things separate and use a one to one scenario and have one virtual disk on the file server represent one virtual machine from the host. If I need to expand the virtual disk or move it to another storage server later, I have the ability to do so without putting all of my eggs into one basket.

 

Prepare the Volumes:

After creating all of the iSCSI Targets on the file server via Server Manager, the next step from the Initiator (besides mapping them via MPIO) is to open Computer Management on the host server and use Disk Management to bring the disks online, initialize them, and format the volumes. Make note of the volume names you create when formatting as you will need this in the backup script. I choose to follow a naming convention for the volume names that would match the virtual machine I planned to back up to the target.

The following image shows a sample of my configuration with one of the volumes manually brought online (Disk 9). You will want to take the disks offline after formatting as it is not necessary to have the host system accessing them until it is time to make the backups. While I did experiment with leaving the iSCSI Initiators Inactive, I found it much easier to just leave them in a connected state which also makes them persistent. I will provide a sample script that allows you to take the Initiators to an inactive state, but there are some issues with doing this as there is a delay between the time the Initiator connects to the target and the host making the disk state accessible. I am not certain how to handle that delay via a script yet, and it would at times result in a NULL object reference to the disk. This would result in the virtual machine not being backed up. So, the script I recommend and created assumes that the iSCSI portals are connected, and the disks are waiting to be brought online for use.

Figure 7: Initiator and DM

 

Getting down to Backups:

At this point, I’m going to guess that you now have both the Initiator and the iSCSI Target working well… they see and love each other and it is a great romantic comedy. Your targets should be persistent, meaning that even after the host server reboots they will be automatically connected via MPIO. Life is grand. The next thing to ensure, is that your Hyper-V host (Initiator) has virtual machines running, I mean that is the point of all this right… to get backups of those things?

Here are the required pieces of data you need to note for your version of the PowerShell backup script:

  1. The Names of the virtual machines you wish to backup
  2. The Names of the volumes that you created for each virtual machine
  3. The IQN values that were used to create the iSCSI Targets for the virtual machine backups

You will need to create a directory on your Hyper-V Server to store the backup script and for the log files it will generate. I created a directory off the root of C: called “BackupScripts” with a subdirectory called “BackupLogs.” Makes sense, I’m a simple guy.

You will then edit the script to reflect the information you have gathered and completed so far (the list I just mentioned). I placed my script into the “C:\BackupScripts” directory, but you can do whatever you want.

 

Edit the Script:

Step 1:

Replace the value property for the variable $LogDir with the directory path to the location to where you want to keep the log files. It can be anything your heart desires!

$LogDir = “C:\BackupScripts\BackupLogs\”

Step 2:

Edit the array of virtual machines to use the names of the ones you want to backup. I’m going to guess that you are limited to a certain number of array elements, but if you have ten thousand virtual machines, go for it.

# List of VMs to backup
$VMList = @(“VM00″,”VM01″,”VM02″,”VM03”)

You can get a quick list of the virtual machines on the host by executing the Get-VM PowerShell command and make note of the “Name” property. If you happen to be more of an automated person, you may considering using that property field in a loop and have the script generate a fresh list for every execution and then not worry so much about maintaining a static array. For me, this isn’t an issue as it is one host and a few virtual machines that will not be changing so frequently. Plus, you would need to figure out how to get the IQN values to match your virtual machines and target volumes. Perhaps having a standard naming convention would help you out there. J

 

Step 3:

Edit the variable $IQNTable to reflect the iSCSI portal mapping path for each virtual machine and their associated target volume for backup.

# Hash table of IQN per VM
$IQNtable = @{
“VM00″=”iqn.2014-05.net.whatever:myserver-vmbackup-vm00-target”;
“VM01″=”iqn.2014-05.net.whatever:myserver-vmbackup-vm01-target”;
“VM02″=”iqn.2014-05.net.whatever:myserver-vmbackup-vm02-target”;
“VM03″=”iqn.2014-05.net.whatever:myserver-vmbackup-vm03-target”
}

 

Step 4:

Save it and open up Computer Management or Task Scheduler from the administration tools. You will now add your own task to run the backups as frequently as you would like. As long as the schedule does not conflict with the timing of a Windows Server Backup schedule (you know, the one generated by the WSB GUI tool), you should be fine. In my case, I run a WSB GUI backup of the host system state and some other volumes at 1am, and I configured my backup script to execute from Task Scheduler every 6am. The initial backup always does a full from WSB, and can take several hours sometimes. So I just created a schedule that I thought may avoid any overlaps. Just monitor the event logs and make adjustments accordingly.

To specify the “Action,” apply the following properties (with your specific path entries) to the Task:

Program/Script
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe

Add arguments (optional [but not really])
-Mta -NonInteractive -NoLogo -File .\vmbackupscript.ps1

Start in (optional [but not really])
C:\BackupScripts

If you use the basic task wizard, you will want to open the task after it has been created and edit a few additional properties. Set the task to “Run whether user is logged on or not” with “Highest Privileges” configured for “Windows Server 2012 R2.” If you are doing this remotely via Server Manager, you may only have the option to select “Windows 8.1,”… I don’t believe it matters, and you can also RDP to the desktop to change it later if you like.

Modify the Settings properties to “Allow task to be run on demand” and “Do not start a new instance.” I believe these are default, but check them anyway.

Save the task and apply your credentials if prompted. You should be good to go. If you like, run the script and check for any errors in the logs generated by the script in the “BackupLogs” directory. You will also see events related to Backup if there is a problem, and you can always check using the “Windows Server Backup” tool as logs are also accessible there with some details on what and how much was backed up.

Figure 8: WSB GUI Logs

 

 

Final Notes:

You may notice that all of these backups are FULL backups. They are not incremental. This means you will get one, and only one backup maintained to your iSCSI Target volume. If you want to be able to do both full and incremental backups, you will need to change to using a different method of backing up the virtual machines. The method I provide in this script is what is considered to be “hot” backups, or what Microsoft calls “Child VM Snapshots.” The virtual machines are required to have guest services enabled (which are integrated by default with 2012 R2), and supposedly any Linux VM that requires “Offline” maintenance will be put into a saved state for VSS to coordinate its’ snapshots.

I have a virtual machine guest that is running Sophos, which appears to be a SUSE kernel (my guess from some digging), and while Hyper-V shows that it must be taken offline for the VSS snapshot, it doesn’t and seems to be working quite well with the script I provide.

For my purpose, having a single full backup is fine, as I run nightly backups of the iSCSI Target file server anyway, and can roll back a complete virtual machine for any day I choose to the target.

However, in order to do incremental and full within the same volume you will not be able to do “hot” backups as Microsoft explains things. But again, this was not my experience and I was able to do incremental by changing the script to utilize the PowerShell “New-WBFileSpec” cmdlet within the backup policy. I’ll provide a sample script of this along with the others for you to browse. It still requires all the volume designations and IQN properties, but instead of using Get-WBVirtualMachine to add the VM to the backup policy, you specify the virtual disk of the VM with New-WBFileSpec. Hopefully with a little reading and digging through the script it will make sense to you, but feel free to post any questions or comments you may have.

 

Download the Scripts:

OneDrive ZIP File

(zip contains hot backup script called “vmbackupscript.ps1” and a sample of full and incremental using New-WBFileSpec)

 

Conclusion:

I hope that this simple setup and script saves you a bit of cost and time in the long run and gives you a bit of exposure to using PowerShell for doing your own backup solutions.

Good Luck out there!

Advertisements
Tagged with: , ,
Posted in Virtualization

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: