The other night was one of those nights from hell that inevitably come up once in a while. This occasion was in one of my day job's lower tier environments where we run everything in a XenServer cluster. Well, this particular night was a power maintenance night for our data center, and two of our XenServer only had single power supplies.
Well, if you guessed that the two with single power supplies shut down, you guessed right. There were some other issues as well, that lead to this next problem, but I won't bore you with the details. You're probably here because of what happened after I finally got the other nodes back online.
What happened once I got some of my other equipment back online, and with only three out of five XenServers still running, I couldn't see any of the VMs that were originally running on the two offline nodes. Those VMs were just gone!
It turns out, they weren't gone, the cluster just thought they were still running on the two other nodes. When that happens, you have to tell the cluster to forcefully shut them down. To do that, SSH into your master node and run the following to get a list of UUID's of all the hosts in the cluster:
- xe host-list params=uuid,name-label,host-metrics-live
Note the long UUID string of the host or hosts that are offline. Then run this command to tell the cluster that any VMs running on the offline nodes are powered off:
- xe vm-reset-powerstate resident-on=<UUID OF OFFLINE XENSERVER> --force --multiple
Obviously replace the item in red above with the UUID of your offline XenServer, and run that command for any other offline servers as well.
Now the VMs should show up as powered off in XenCenter. You should now be able to power them on and have them run on one of the live nodes.
Please note, when powering on one of these VMs you may get an error that says:
the vdi is not available
I wrote about how to fix that error yesterday. To read that article, please click here: (How to fix "The VDI is not available" error in Xenserver)
Did this help you out? Did you do something different? Let us know in the comments!