Today my colleague, who handles our Veeam installation was diagnosing an error we were sporadically seeing. The error was this (removed names):
Error: Client error: NFC storage connection is unavailable. Storage: [stg:datastore-xxxxx,nfchost:host-xxxx,conn:vcenter.server.fqdn]. Storage display name: [DatastoreName].
Failed to create NFC download stream. NFC path: [nfc://conn:vcenter.server.fqdn,nfchost:host-xxx,stg:datastore-xxxxx@VMNAME/VMNAME.vmx].
Now this error indicates that it failed to get a connection to the host via NFC stream (port 902). Or so I thought. We have seen sporadic problems for vCenter heartbeats over the same port so that was what we expected. Turns out that some of the hosts in the cluster were missing the “datastore symlink” in /vmfs/volumes.
When running “ls -1a /vmfs/volumes” the result was not the same on each host. 4 of 8 hosts were missing a symlink and two others had a wrongly named symlink. I recalled that when I was creating the datastores I used PowerCLI to change the names of the datastores several times in rapidily after each others as my script has slight errors when constructing the correct datastore names. It seems that this left some of the datastores on some hosts either with no symlink or a wrongly named symlink.
Fortunately the fix is easy:
- Enter Maintenance Mode
- Reboot host
- ?????
- Profit
That is it! 🙂