|

My failed migration of my Homelab from VMware to XCP-NG with XO

The Backstory

It is coming up on time to renew my VMug advantage, but with VMware dropping support for a lot of older hardware, plus Broadcom’s statement about not really caring about any one but their 5 biggest customers. I figure I should migrate to a more open source solution, plus my wallet will feel a little reprieve.

In order to do this I did some research on a few solutions but XCP-NG with XO seemed like the most fully featured option that has what I want for now, I might change my mind in the future for features like container support, and move to Prox-mox. Some videos that helped break down the difference and helped me understand the features was a youtube video by Lawrence System talking about the comparisons. I also used his guide on compiling XO which was super helpful and detailed.

Once I got XCP-NG running on one of my servers I setup XO as a VM on the ESXi host (more on this later) and started playing around with it for a bit, one thing I don’t like is how the virtual switches and physical NICs are tied together, but I might just need to learn more about it.

Setup for Migration

Next I needed to figure out how to migrate all my systems from ESXi to XCP-NG, as I really don’t want to rebuild 50 VMs, plus I need some of that data. I found that XCP-NG uses OVA format to import VMs. Makes sense as it is an open standard and I know ESXi can export to that so I found my migration path.

At first I started using the OVFtool from VMware, but that was annoying and had to be done one system at a time without a good way to script it, that I found in my 5 minutes of googling. So I decided to use VMware’s PowerCLI, which is a Powershell module that lets you do basically everything you can do with the VMware Web interface and even more. I have experience with PowerCLI as that is what I used to write my script to backup my ESXi servers and send me discord notifications on current status.

But I ran into some issues, some things like ISO that were mounted to the VM and no longer accessible, due to shutting down the ISO storage system. So I had to unmount all of the ISOs across 50 VMs that’s a lot of clicking. But much shorter to script. I used two methods across two different servers, one was the powershell one liner the other was a script that has error checking.

Cleaning up the VMs

I have tested these scripts and wrote some basic error catching into them. But I will also give you the one liners, which assume that you have already used powerCLI to connect to your server.
Connect-VIServer -Server <hostname/IP>

  1. Remove all mount ISOs using the script “remove_isomount_from_all_VMs.ps1” or this one liner:
    Get-VM | Get-CDDrive | where {$_.IsoPath -ne $null} | Set-CDDrive -NoMedia -Confirm:$false
  2. Remove all snapshots from all VMs using the script “remove_snapshots_from_all_VMs.ps1” or this one liner:
    Get-VM | Get-Snapshot | % { Remove-Snapshot $_ -Confirm:$false }
  3. Uninstall VMware tools or open-vm-tools, and for linux systems you need to run the following on all systems:
    dracut --add-drivers "xen-blkfront xen-netfront" --force
    NOTE! — I had to install dracut on my debian based VMs, RHEL based already had it.
  4. Shutdown all your VMs, which my export script will do for you.

Export to OVA

Finally run the script “export_all_VMs_to_OVA.ps1” or the one liner:
Get-VM | Export-VApp -Destination ‘<Location to save all OVA to>‘ -Format OVA
NOTE! — All the VMs must be powered off in order to export them the script makes sure they are shutdown before exporting.

The Tragedy of Importing

So it was more difficult to get a hold of the Powershell module for this migration then I think it should have been. That is to say I had to create a citrix account (Boo!) and then I was able to access the download I needed for free. But that wasn’t the only hurdle, once I had extracted the SDK windows blocked all the files from executing, which while that can be a good security feature, I forgot about it and it caused a headache for a few minutes, while I tried to figure out why importing the module was throwing an error that a dll wasn’t there when it was.

Solution extract the Powershell module folder to somewhere in your user account folder, I used the desktop. Then I had to right click, go to properties and unblock every single file, including the txt files. Then copy it to C:\windows\system32\WindowsPowershell\v1.0\Modules

Powershell – Attempt #1

So I connected to my XCP-NG server using `Connect-XenServer` connected no problem, I was able to query things it seemed to be working. Then I tried to import my first VM. It failed immediately. I tried another VM, same thing, I verified my path, looked good, the error seemed to be coming from the server side of things. I would open the connection on for it to be forcibly closed on the other end. See screenshot below:

xo-upload-ova – Attempt #2

After some research and not finding anything but use the “xo-upload-ova” tool I decided to try that. It took me a long time to get it setup correctly as the documentation that I found was lackluster at best. But even after getting the tool installed and configured it didn’t work as I was never able to connect to the XO server.

I did use xo-upload-ova to verify the image was recognized as well. All seemed to be good.

XCP-ng Center – Attempt #3

Having exhausted the scriptable options I turned to the next solution on the list using the XCP-ng Center application. I connected my server (bypassing XOA) and then attempted an import, first thing I ran into was a warning from XCP-ng stating the OVF version was not set and there was unsupported hardware. I ignored it and continued to try and import it and like the other attempts it failed, though it took a little longer to fail.

Okay, maybe it had something to do with that error.

So I do some more research and figure out that VMware doesn’t output in a standardized format, maybe that is the issue, with powershell and XCP-ng Center. So I use vBox to convert the ovas to ova type 1 I also tried type 2

Repeated the steps above same results, minus the warning about the OVA format.

Conclusion

In the end I spent about 2 weeks going through all these steps and trying to learn XCP-ng without luck. I really wanted this to work because I had heard great things about XCP-ng and while creating new VMs is great and doesn’t cause issues, not having a viable migration path makes it a hard enterprise sell.