Month: December 2022

This fall I had a task to upgrade some old ESXi 6.0 hosts in a tightly controlled environment without internet access for vCenter and other conveniences. So I resorted to doing the old classic:

esxcli software vib install -d /vmfs/volumes/datastore/esxi-depot.zip

This worked from one of the sites no issue and updates completed quickly. However once I got to the second site the first host of 3 updated no problem but second failed with the error:

[Errno 32] Broken pipe vibs = VMware_locker_tools-light_6.0.0-3.76.6856897.vib

This got me a bit confused so started looking into the filesystem of the ESXi host and discovered that the symlink /productLocker was pointing to a folder in red which usually means a broken link.

ProductLocker contains the files from VMware tools.

/productLocker is a symlink to a folder inside the /locker folder which inturn is a symlink to /store which symlinks to a partition on the boot volume. I tried changing into the /store and doing an ls which corrupted my terminal output.

Turns out the filesystem on the /store partition was corrupted. I check the two bootbank partitions which were okay and then realized that this host and it’s partner were both booting from SD cards which I hate working with!

As this was old 6.0 hosts – support was out of the question so I started looking around the internet for a possible fix that didn’t involve reinstalling, and to my luck I found a blog post from VirtualHackey from 2020 which detailed almost exactly the same situation as I was seeing.

The fix was simple – find and copy the actual content from the corrupted filesystem, format the partition and copy content back. He even describes with a link to another article how to locate the content. Unfortunately I could not find the files inside the filesystem.

So I had two options: format the partition and hope I could upgrade without the packages present or reinstall. That is until I realized that I had another site with identical hosts and ESXi 6.0 build. I checked those and found the /store partition in perfect state.

So I reached out to my TAM to ask if the method of formatting the partition and copying the content from a working host was viable. This was of course best effort support but reached out to see if the procedure perhaps was done in an old support case.

I got response that it ought to work otherwise a reinstall was necessary.

So – I tried it and it worked like a charm. I completed the upgrades without further hitches.

So shout out to VirtualHackey for providing the method to fix this problem – much appreciated!

Today VMware is releasing the next installment in their series on Multi-Cloud Briefings named “The Future of Cloud Management” which will cover new additions to VMware’s vision of multi-cloud management at new features added to VMware Aria.

If you haven’t seen any of the previous multi-cloud briefings I recommend checking out the Youtube channel where all the previous briefings can be watched.

VMware Aria

Much of the portfolio of VMware Aria I have some experience with from the older on-prem solutions but you may, as I, not know what VMware Aria actually covers.

VMware Aria is VMware’s portfolio of tools for multi-cloud management (formerly known as vRealize Cloud Management) which was announced in August – check out this blog post for more information.

Many of the products and tools of Aria are rebranded from old products as well as addition of new features built upon that foundation.

First of you may have used products like vRealize Automation, Operations, Log Insight or Network Insight before. All of these products are part of Aria now and have been rebranded to Aria Automation, Aria Operations, Aria Operations for Logs and Aria Operations for Networks respectively. As well as now including Skyline and CloudHelath and CloudHelath Secure State, that latter two being rebranded to Aria Cost powered by CloudHealth and Aria Automation for Secure Clouds.

These tools have been integrated into a new product named Aria Hub (formerly Project Ensemble). Underneath Aria Hub sits Aria Graph which is the data source that powers the new features of the Aria portfolio.

Data from the above tools and solutions along with data that is pulled from the cloud providers that you want like Azure, AWS, VMConAWS etc is then collected (cool detail is that data is not duplicated but rather referenced via pointers) into an inventory in Aria Graph.

From Aria Graph, Aria Hub is then able to show you your cost, usage, problems, performance and possible security compliance issues based on the data across all cloud endpoints.

Via the Aria Hub UI you can look at different perspectives based on whether you are a business manager, application owner or operations engineer. You can customize the home page for your different groups of people so that they get the info they need, be it cost, performance or security.

You can select your SDDCs or your applications and drill down into the elements that make them up like VM instances, Kubernetes pods, networks, storage etc and look at consumption, performance, cost etc for the entire SDDC or application on all the way down the stack to the individual components that it is made of.

Aria Guardrails and Business Insights

VMware Aria is not just a rebranding of existing products but also introduces new features built on-top of those products like Guardrails and Business Insights.

Guardrails allows you to setup automatable policies for things like security, cost, performance etc that can be enforced on the different applications and SDDCs attached to your Aria hub.Being powered by and “everything-as-code” approach Guardrails includes a library of policy templates that can be imported and customized to your environments and allow for automatic remediation of things like making sure all your Kubernetes pods are attached to Tanzu Mission Control so that security policies inside TMC can be enforced and monitored.

Business Insights integrates with Guardrails and the other products inside Aria Graph to allow for AI/ML powered analytics to inform you of compliance issues and optimizations that can be useful – all available via Aria Hub and tailored for the specific class of user logged in.

App Migrations

One of the very exciting new features coming is Aria Migrations which will assist you in analyzing and planning migrations of applications running in your on-prem infrastructure to VMC. Currently it is the only migration type supported but the types of migrations will be expanded in the future.

Via Aria Hub you can plan the migrations by selecting the subset of resources you want to migrate and then Migrations will assist you in identifying any dependencies that are not in your scope that might impact your performance or security if migrated. You can then add these dependencies if needed.

You can then compare the expected TCO of keeping the application on-prem versus in the cloud and make decisions based on this – all powered by Aria Cost.

If you want to perform the migration you can continue to the planning of the migration where App Migrations will assist you in planning bundles of workloads to be moved and in what order.

After splitting up the migration each bundle of workloads can be planned across multiple migration steps.

All of the migration being powered by HCX and CodeStream allowing for testing of the steps, rerunning failed steps and monitoring the process.

But how does Aria know what entities are linked to each other you may ask? Well there are multiple ways like using flow information from Aria Operations for Networks to see who is communicating with who or what deployment the entity is a part of in Aria Automation.

When a link is detected users will be presented with the discovery of the entity and the link to an application and can then either confirm the link is correct or let it automatically be accepted. Very neat!

All about the APIs

All of this sounds excellent but how does it fit into your existing business? Well here’s the cool part. With Aria Graph at the base of everything you get a full GraphQL based API. Everything you can do in the UI can be done via GraphQL against Aria Graph so if your can write a piece code that can read or write to a GraphQL based data source you can integrate it to your existing tooling like Service Now.

The approach of allowing focusing on API to powered the UI is not a new but it is very nice to see VMware take this approach so seriously with Aria Hub and Aria Graph.

Final words and the freebie

If this sounds interesting to you I highly recommend heading over to the Aria Hub landing page and sigining up for the free trial and trying out the product.

Personally I am very excited to see where this is going – being a on-prem data center operations engineer this a big world for me to step into and a lot of information to digest.

One note for the security minded people. VMware Aria Hub is currently a SaaS offering which means your data will be located in a cloud. This might prohibit you from using the offerings. This is of course a shame but VMware have informed that they might on a +1 year timeframe look into an on-prem version of the product.