Sean Wallbridge
As promised, this is the next instalment in my Veeam posts, specifically on Remote Backup (backup to a remote site across the WAN).
This has been a learning experience even though I already had one site doing remote backup for almost a year. The goal with this particular project is to leverage as much goodness as possible out of Veeam v7 for our customer. This meant learning how to make best use of some of the features of Veeam that are either new to v7 or are things I hadn’t really looked at in the past. This particular customer has Veeam Enterprise Essentials, licensing that lines up with their VMware vSphere Essentials licensing. They do NOT have the WAN Accelerator goodness that comes with the PLUS versions so it will be interesting to see how we do with traffic over the WAN.
The very first thing that I have learned from this project is to ensure the underlying VMware ESXi environment is healthy (or Hyper-V if that is your platform). An obvious statement but something that can bite you in the ass if you are not careful. And I got bit in the ass much to my chagrin. I had all sorts of weird Veeam issues happening until I sorted out the underlying ESXi issues which mostly revolved around problems with VMware Tools in the guests. All problems were solved by upping the patch level on ESXi and refreshing VMware Tools in the guests.
The next thing I learned was the Veeam documentation can sometimes be a bit “thin”. I needed to use the seeding function for backups which is NOT the same as the seeding function for replicas, at least from a process point of view. My perusal of the docs seemed to indicate that to seed a backup you need to copy a FULL backup (a Veeam .vbk file) into the remote repository (created when you create the remote Veeam agent). You then scan the remote repository in the Veeam console which should “import” the backup into the overall Veeam config, then you can use the “map backup” function to point at that backup as the seed data and you are good to go. Well, yes and no …
I tried all of that and sneaker-netted my VBK file over to the remote site, copied it into the repository then went around in 27 circles trying to get the blasted thing to scan and import. Epic fail. Nothing. I finally gave up and contacted my good buddy at Veeam, Matt Price, for some advice. It seems I was doing about 75% of what I needed to do. Matt directed me to create a net-new backup for each of the VM’s I want to have backed up to the remote site because once data is seeded at the remote site the backup job would be repointed (remapped) to the remote site. That way there would be no conflicts with the current backup job that will continue to run at the main site. He also advised me to set the remote job as a “forward incremental” with a weekly synthetic backup so that we end up with a full backup at the remote site and keep the repository somewhat pruned. So that was “ah ha!” moment number 1. “Ah ha!” moment number two was when Matt pointed out that you have to copy both the .VBK file AND the metadata file to the remote repository in order for the rescan and import to work.
So, I followed Matt’s instructions to a “T” and created my two new jobs. After they ran I copied the data via sneaker-net out to the remote repository and rescanned the repository. Voila! Things rescanned properly and the backups imported properly. I then modified each of the two jobs and used the “map backup” function to point to the remote repository backup. This effectively “seeded” the backup and I set the jobs to run. I ended up with a 50/50 status … one job ran perfectly and the other messed up.
The job that ran perfectly was for the SBS 2011 VM and it copied about 8GB of changed data out over a roughly 4 hour period (about par for the 5 megabit uplink provided by Shaw). I have to play with this backup some more in order to decide if it is going to be a daily run vs a weekly run but I am happy with the results so far.
The job that messed up seemed to mess up, once again, due to issues with the underlying VM. The Veeam job seemed to cause issues with the VM’s network stack although I cannot say why as the VM backs up properly when running the local Veeam job. I killed the running job as it had slowed to a crawl, fixed the network issues then tried re-running the job, a couple of times, as it turns out. When I finally got networking sorted out it seemed like I had totally messed up the remote “seed” backup as Veeam appeared to be trying to send ALL data back out to the remote site rather than just the changed data that I would have expected since the seed backup was taken. I have killed the whole job, deleted the job data (in local repository as well as remote repository) and am trying again form scratch. I’m hoping that I have fixed all of the underlying problems with the VM and that this job will now work as well as the job for the other VM.
So, there you have it. I’ll blog the results of my (hopefully) fixed second VM job as things progress.