Failed vMotion – Migration to host failed with error msg.vmk.status.VMK_IS_DISCONNECTED (195887150)

From the popup window in the C# client when trying to vMotion a VM by hand:

Task Details:
Name: MigrateVM_Task
Description: Migrate virtual machine
Error Stack:
Migration to host <xxxxx> failed with error msg.vmk.status.VMK_IS_DISCONNECTED (195887150).
vMotion migration [175583809:1431386589121868] failed writing stream completion: Already disconnected
vMotion migration [175583809:1431386589121868] failed to flush stream buffer: Already disconnected
vMotion migration [175583809:1431386589121868] socket connected returned: Already disconnected
Timed out waiting for migration data.
Additional Event Details:
Type ID: Info
Target Object Type: ClusterComputeResource
DataCenter Object Type: Datacenter
Host Object Type: HostSystem
Host Build: 2143827
VirtualMachine Object Type: VirtualMachine
ChainId: xxxxx
Additional Task Details:
VC Build: 2442329
Task Id: Task
Cancelable: False
Cancelled: False

Bottom vCenter task pane error:

Migrate virtual machine
A general system error occurred: Migration to host <xxxx> failed with error Connection closed by remote host, possibly due to timeout (0xbad003f).
System

From the ESXi hostd logs.

[29106B70 verbose ‘Vmsvc.vm:/vmfs/volumes/xxxx/xxxxx/xxxvmx’] Handling message _vmx9: vMotion migration [a773241:1431386014948294] The remote host closed the connection
unexpectedly and migration has stopped. The probable clause of the closed connection is a migration failure detected on the remote host.
–> Migration to host <xxxxx> failed with error Connection closed by remote host, possibly due to timeout (0xbad003f).
message = “Failed to initialize migration at source. Error 0xbad00a4. VMotion failed to start due to lack of cpu or memory resources.”,
reason = “Failed to initialize migration at source. Error 0xbad00a4. VMotion failed to start due to lack of cpu or memory resources.

Ran the following command on the ESXi host.  Noted the bad checksum aka Receive CRC errors.  This count increased after each vMotion attempt.

# esxcli network nic stats
get -n vmnic3
NIC statistics for vmnic3
Packets received: 208603466
Packets sent: 1959043
Bytes received: 67322218265
Bytes sent: 16126893211
Receive packets dropped: 30
Transmit packets dropped: 0
Total receive errors: 217710
Receive length errors: 0
Receive over errors: 0
Receive CRC errors: 112805
Receive frame errors: 0
Receive FIFO errors: 0
Receive missed errors: 0
Total transmit errors: 0
Transmit aborted errors: 0
Transmit carrier errors: 0
Transmit FIFO errors: 0
Transmit heartbeat errors: 0
Transmit window errors: 0

Run the following command to down vmnic3

esxcli network nic down -n vmnic3

Then run esxtop and hit “n” .  Under the TEAM-PNIC column,  vmk1 (my vmotion network) is using the other vmnic number 2.

Once the “bad” vmnic3 was administratively marked as down, the vMotion traffic was switched to vmnic2 (a different physical adapter) and all of the vMotions worked!    The “Receive CRC errors” count did NOT increase.

Once the host was in maintenance mode we tried to

1. Replace the cable
2. Check the switch port for vmnic3
3. Ended up having the physical adapter be faulty and replaced.