Brainlitter ยป Learned a few cool things about VMware this weekend (recovering from VMDK Corruption)

Learned a few cool things about VMware this weekend (recovering from VMDK Corruption)

Ok, none of this surprised me but it was interesting to discover finally (I always wondered what I’d do if I had a corrupted VM and last night until 4am I got to find out)…

  • So, our old system was on VMware Server v1.0.6. It worked okay, sucked in regards to performance though and didn’t support our Windows Server 2008 guest very well. So, when ESXi became available for free, it became time to switch.
  • Of course, if you are going to migrate virtual machines from a piece of hardware to the *SAME* piece of hardware, things get a little trickier, so here’s what ended up happening through Friday/Saturday night.
  • VMware server was stopped on the Windows Host
  • VM’s were copied to other storage (in our case, external USB drive)
  • The old Windows Server host was retired (aka ESXi formatted it)

ESXi was installed on the host and VI Client was installed on a workstation and pointed to the server to complete configuration

The original plan was to simply open the ESXi Storage via VI client and copy the VMDK’s/VMX’s across the network to local storage. The problem is, we had two problems

  • ESXi wouldn’t let us add these VM’s to its inventory (bummer and cause for short term panic) – see note on VMconverter Enterprise License (the other option would have been to mount these in another VMware Server instance and VMconvert from within there, across the network
  • Our Exchange Server (Exchange 2007 on Windows Server 2008/64bit) D: image was somehow corrupted (thanks Windows, copy a file and not tell us you pooched it!) – THAT was an interesting fix (and time consuming), see that below

So, Solutions (ya, we’re back and operational thanks!)

Getting those VMWare Server VM’s onto ESXi when the host is gone

  1. We had a copy of VMWare Converter Enterprise which opens up a number of extra options, including the ability to choose a source of ‘Other’ instead of your physical disk. By choosing ‘Other’, we simply selected the VMX file and VMconverter was quite happy to start sending those guests across the network and mount them in ESXi – beautiful … except, trying to do our Exchange Server VM, it died around 25% (this is where it became clear the D: VMDK was gassed!)

Fixing the Corrupted VMWare Server VMDK (before being able to mount in ESXi)

Ok, this was the cool part. I learned about a couple of tools that I couldn’t use, because the D: VMDK was so bad, but they are cool anyways. I was getting CRC (Cyclic Redundancy Check) failures no matter what I tried to do with it (attach, copy, move, whatever)…

  1. The first one is VMWARE-DISKMANAGER which *would* have allowed me to extend the size of the VMDK, shrink it, etc. I tried that first, as I figured if I could write some extra info to the file, it might fix itself. Okay, that didn’t work but it was still cool to know that tool existed as it might help next time
  2. The next one was VMWARE-MOUNT. This tool can be downloaded and installed so that you can mount your VMDK as a local disk drive (ya, so that VMDK could appear as a local “P:” on my desktop. Cool, I thought. I’ll copy the contents out of the file and make a new VMDK. Well, still a cool tool for other reasons but alas, with a corrupted VMDK, that didn’t help either
  3. What DID fix the problem was not using CHKDSK /F but rather, using CHKDSK /R. I couldn’t move the file elsewhere but I could run this, on the local disk where the VMDK was sitting. This ran for about 90 minutes (500GB SATA Disk) and indicated that it had corrected a bunch of stuff. Sure enough, when it was done, I was able to run the Vconvert process and bring Exchange 2007/Windows Server 2008 64bit online on my new ESXi server … or so I thought… I had one step left…

So, my Exchange 2007/Windows Server 2008 64bit VM is editable (VMX) on ESXi. It starts to boot and now I get a WINLOAD.EXE not found error. ARGH! While it is unclear whether this issue was due to the repairs of the corruption or due to the fact that others are seeing this too (scanning VMWare forums presented lots of people with the same problem, and as usual, very few actually presenting a solution *SIGH*.), I came across one post that made my day. I blogged it last night (4am) here: http://brainlitter.com/archives/2008/09/windows-server-2.htm but the solution to that was to do the following:

I had the ISO for the Windows Server 2008 DVD, mounted that, booted the VM off of the ISO and performed the following using the command line repair tools.

  • bootrec /RebuildBcd

  • bootrec /FixMbr