I can not vMotion any VM’s off a ESXi 5.5 U2 (build 2143827) host. Windows 2012 vCenter Server 5.5 Update 2d
#1 - VC Errors
From the popup window in the C# client when trying to vMotion a VM by hand:
Description: Migrate virtual machine
Migration to host <xxxxx> failed with error msg.vmk.status.VMK_IS_DISCONNECTED (195887150).
vMotion migration [175583809:1431386589121868] failed writing stream completion: Already disconnected
vMotion migration [175583809:1431386589121868] failed to flush stream buffer: Already disconnected
vMotion migration [175583809:1431386589121868] socket connected returned: Already disconnected
Timed out waiting for migration data.
Additional Event Details:
Type ID: Info
Target Object Type: ClusterComputeResource
DataCenter Object Type: Datacenter
Host Object Type: HostSystem
Host Build: 2143827
VirtualMachine Object Type: VirtualMachine
Additional Task Details:
VC Build: 2442329
Task Id: Task
Bottom vCenter task pane error:
Migrate virtual machine
A general system error occurred: Migration to host <xxxx> failed with error Connection closed by remote host, possibly due to timeout (0xbad003f).
#2 - ESXi Errors
From the ESXi hostd logs.
[29106B70 verbose ‘Vmsvc.vm:/vmfs/volumes/xxxx/xxxxx/xxxvmx’] Handling message _vmx9: vMotion migration [a773241:1431386014948294] The remote host closed the connection
unexpectedly and migration has stopped. The probable clause of the closed connection is a migration failure detected on the remote host.
–> Migration to host <xxxxx> failed with error Connection closed by remote host, possibly due to timeout (0xbad003f).
message = “Failed to initialize migration at source. Error 0xbad00a4. VMotion failed to start due to lack of cpu or memory resources.”,
reason = “Failed to initialize migration at source. Error 0xbad00a4. VMotion failed to start due to lack of cpu or memory resources.
#3 - ESXi NIC CRC's
Ran the following command on the ESXi host. Noted the bad checksum aka Receive CRC errors. This count increased after each vMotion attempt.
# esxcli network nic stats
get -n vmnic3
NIC statistics for vmnic3
Packets received: 208603466
Packets sent: 1959043
Bytes received: 67322218265
Bytes sent: 16126893211
Receive packets dropped: 30
Transmit packets dropped: 0
Total receive errors: 217710
Receive length errors: 0
Receive over errors: 0
Receive CRC errors: 112805
Receive frame errors: 0
Receive FIFO errors: 0
Receive missed errors: 0
Total transmit errors: 0
Transmit aborted errors: 0
Transmit carrier errors: 0
Transmit FIFO errors: 0
Transmit heartbeat errors: 0
Transmit window errors: 0
#4 - Down VMnic
Once the “bad” vmnic3 was administratively marked as down, the vMotion traffic was switched to vmnic2 (a different physical adapter) and all of the vMotions worked! The “Receive CRC errors” count did NOT increase.
Once the host was in maintenance mode we tried to
- Replace the cable
- Check the switch port for vmnic3
- Ended up having the physical adapter be faulty and replaced.