This fall I had a task to upgrade some old ESXi 6.0 hosts in a tightly controlled environment without internet access for vCenter and other conveniences. So I resorted to doing the old classic:
esxcli software vib install -d /vmfs/volumes/datastore/esxi-depot.zipThis worked from one of the sites no issue and updates completed quickly. However once I got to the second site the first host of 3 updated no problem but second failed with the error:
[Errno 32] Broken pipe vibs = VMware_locker_tools-light_6.0.0-3.76.6856897.vibThis got me a bit confused so started looking into the filesystem of the ESXi host and discovered that the symlink /productLocker was pointing to a folder in red which usually means a broken link.
ProductLocker contains the files from VMware tools.
/productLocker is a symlink to a folder inside the /locker folder which inturn is a symlink to /store which symlinks to a partition on the boot volume. I tried changing into the /store and doing an ls which corrupted my terminal output.
Turns out the filesystem on the /store partition was corrupted. I check the two bootbank partitions which were okay and then realized that this host and it’s partner were both booting from SD cards which I hate working with!
As this was old 6.0 hosts – support was out of the question so I started looking around the internet for a possible fix that didn’t involve reinstalling, and to my luck I found a blog post from VirtualHackey from 2020 which detailed almost exactly the same situation as I was seeing.
The fix was simple – find and copy the actual content from the corrupted filesystem, format the partition and copy content back. He even describes with a link to another article how to locate the content. Unfortunately I could not find the files inside the filesystem.
So I had two options: format the partition and hope I could upgrade without the packages present or reinstall. That is until I realized that I had another site with identical hosts and ESXi 6.0 build. I checked those and found the /store partition in perfect state.
So I reached out to my TAM to ask if the method of formatting the partition and copying the content from a working host was viable. This was of course best effort support but reached out to see if the procedure perhaps was done in an old support case.
I got response that it ought to work otherwise a reinstall was necessary.
So – I tried it and it worked like a charm. I completed the upgrades without further hitches.
So shout out to VirtualHackey for providing the method to fix this problem – much appreciated!