Notes from the field (SDDC) – VMware Cloud Director cannot Power On a VM

Not long ago, my team received an incident report from one of our customers about a VMware virtual machine in Cloud Director that could not be Powered On.

From the Cloud Director side the customer sees a “Starting Virtual Machine” task which keeps in running state (after a longer period it will run in a time-out):

Endless running task at 0%.

Eventually, it runs in a time-out.

From the vCenter side, we see the task is processing the power-on request successfully:

, but the status of the virtual machine is still powered off:

Troubleshooting the virtual machine yielded no results. After some time, we found out that the issue only resided on this particular vSphere Cluster where the VM was running including other VMs, we had the same symptoms. Other vSphere Clusters didn’t have these symptoms.

In the end, the issue was found in the vSphere Cluster settings at the vSphere DRS. The “Automation Level” of vSphere DRS was set to “Manual”. Probably another co-worker had, during another troubleshooting issue or patching/upgrade, forgotten to set the vSphere DRS setting back to “Fully Automated” (or “Partially Automated”).

If you run into this kind of behaviour, probably check the “Automation Level” of vSphere DRS setting first. VMware Cloud Director expects vSphere DRS to be set to “Fully Automated” (or “Partially Automated”). It cannot handle the “Manual” value in vSphere DRS. A customer cannot decide on which vSphere Host the VM must start on, they don’t want to get bothered by this kind of message in Cloud Director.