vSphere HA virtual machine monitoring error on VM

Hi all

Today I found a couple of VMs in a cluster, that had HA with VM monitoring enabled, that were showing a “vSphere HA virtual machine monitoring error” with a couple of different dates.

Looking into the event log of the VM via vCenter I could see the following events about every 20 seconds:

This indicated that HA VM monitoring wanted to reset the VM but failed. I tried searching for answers on first Google but with no luck. I remembered that the setup had vRealize Log Insight installed and collecting data so perhaps Log Insight had more logs to look at.

Made a simple filter on the VM name and found repeating logs starting with this error from “fdm” which is the HA component on the ESXi host.

This error looks bad to me. Not being able to find the MoRef of a running VM? Hmm. I asked out on Twitter if anyone had seen this before with out much luck. Think about it over lunch I figured that maybe it was the host that was running the VM that had gotten into some silly state about the VM and not knowing it’s MoRef. So the quick fix I tried was to simply do a compute and storage migration of the VM to a different host and a different datastore to clean up any stale references to files or the VM world running. The event log of the VM immediately stopped spamming the HA message and returned to normal after migration.

I cannot say for sure what caused the issue and what the root cause is but that at least solved it.

Simple network debugging – ESXi and VCSA

I ran into an old issue yesterday with ESXi 5.5 and vCenter disconnecting after 60 seconds. The issues is described in KB1029919 and is due to heartbeat UDP packets from ESXi not reaching vCenter on port 902 within 60 seconds or at all.

Now how to test this! Luckily the KB has a simple Python script that allows you to send UDP packets. Just edit the script and insert the IP of your vCenter server and run it. It will default send 10 packets of 100 bytes.

Now the KB mentions using Wireshark as an option on the vCenter side but when using VCSA that is not really an option. Luckily VCSA 6.5 comes with tcpdump installed! 5.5 and 6.0 don’t but worry not, VMware to the rescue. You can install tcpdump on 5.5 and 6.0 by following KB2084896. This gives you access to a client on ESXi to send UDP packets and a monitoring option on VCSA to see if the packets arrive.

All that needs to be done then is run the following:

#on the VCSA
tcpdump -i eth0 udp port 902 -vv | grep <srcIP>

#on ESXi
python udp_client.py

And then look at the results. You can lose the “| grep <srcIP>” if you want to see all packets but depending on your setup there may be many ESXi hosts sending heartbeat.

VMware License Checkup

Today I had to check up on some license keys for a customer. There was not a complete state of keys, enterprise accounts and support contracts so I started looking into how to collect data.

This may be common knowledge to many but VMware has this nifty tool on my.vmware.com:

https://www.vmware.com/support/serialNumberTrack.portal

Here you plop in your key and it returns support contract, type and which EA number it is connected to.

To get all the keys of a vCenter the following piece of PowerCLI can be used:

$licman = Get-View (Get-View ServiceInstance).Content.LicenseManager
$licman.Licenses | Select LicenseKey, EditionKey

This returns the list of keys on the vCenter easy to copy paste into the tracking tool. Nifty!

IO Issues on Cisco C220 M4

Hi All

As promised more content is finally available! Unfortunately I cannot share any screenshots from this actual issue I worked on – you will have to take my word for it.

At my new job I was tasked with solving some issues that had been observed on a remote site on two Cisco C220 M4’s with local disks. The hosts run a set of redundant services but nothing is shared between them except network equipment. The issues were 1) some times around 04.00 AM the software running in the virtual machines would throw an alarm briefly and 2) powering on/off or snapshotting a VM would cause the software to throw alarms as well but on other VMs than the one being powered on/off or snapshotted. The event log on both hosts showed intermittent I/O latency warnings not connected to any of the above issues but nothing alarming.

Now one note though. The software running on these VMs is very latency sensitive so something like snapshots could potentially be a problem in any case but powering on VM A should not affect VM B unless the host is hurting for resources which is not the case.

Before diving in I asked out in the vExpert Slack if anyone had seen issues like this before or had any ideas of what to look for. James Kilby and Bilal Ahmed were quick to throw some ideas on the table. James suggested that vswp file being created on power on might cause the problem and Bilal suggested looking at the network. With those things in mind I started debugging.

First off I had already decided to update the ESXi version to latest 5.5 U3 + patches – it was a little out dated. Also decided to firmware upgrade the servers with the latest 3.0.3c release for Standalone C220 M4s. I had also found the latest supported drivers for the Cisco RAID controller (lsi-mr3) and the Intel NIC (igb) to rule out any compatibility issues. Also it was my hope that an update would remove some of the I/O latency warnings.

Now before upgrading anything I tested to see if the problem was still there. It was. Powering on a test VM with no OS and just 1 core and 4 GB RAM – instant alarms kicking off in the application. However powering off did not cause any noticeable problems. I proceeded to firmware update as my hope was this would solve the issues. Firmware upgrading on a remote site through a small connection is painful! It took a while. But once the first host was updated I proceeded to test if the issues were still there. There were. Damnit. Time to dig deeper. I tried out James’ idea of vswp being the problem and setting a 100% reservation seemed to solve the problem. However this was not a viable solution as this would only solve the problem if the powering on VM has a reservation. If anyone powered on a VM without it, it would still affect all other VMs, regardless of reservation on those.

I booted up our favorite debugging tool ESXTOP and vent into HBA mode and set delay down to 2 seconds. I then observed the Cisco RAID controllers behavior during power on operations and that freaked me out. It would happily do anything between 100 and 1000 IOPS at 5 to 150 ms while not powering on. The latency would spike high but nothing I was that scared of on a small set of local 10k disks. However when powering on a VM without reservation the HBA would stop doing any operations for upwards of 4 refreshes in ESXTOP (at 2 second intervals!). All indicators showing 0. No IO was passed so no latency was observed. This scared me a bit. Latest firmware and supported drivers. Damn. We weren’t seeing the same issue on another site with Dell servers but they also had SSDs instead of 10k disks. Was this the 10k disks not performing enough?

We had a short talk internally about what to do. My boss suggested that we looked at this Cisco bug:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCut37134/?referring_site=bugquickviewredir

This bug references an issue for some C220 M4’s that were installed with a specific version of the 5.5 ESXi Cisco Custom ISO. It was not the ISO these hosts were installed with but the solution was to use a different driver than the one ESXi default selects for the Cisco RAID controller HBA – the lsi-mr3 driver. Instead it instructs to make sure that the megaraid-sas driver is installed and to remove the lsi-mr3 and lsi-msgpt3 drivers and reboot which will make the megaraid-sas the active driver. We decided to try this. Downloaded the latest support megaraid-sas driver for the server and remove the lsi-mr3 and lsimsgpt3 drivers. Reboot and wait.

After getting online again with one host we tested. Powered on the test VM – and nothing. No alarms. What a difference. Tried it again looking at ESXTOP. No drops in IO. It was now doing 8.000 IOPS @ 15ms no problem. Major difference. Mass powering on VMs 15 at a time had taken minutes before the actual power on task was done and the machines would start booting. It took seconds now.

So what is the moral here? Apparently it could benefit you as a Cisco UCS server user to use the megaraid-sas driver instead of the lsi-mr3 driver. Both are included on the Cisco Custom ISO but it defaults to the lsi-mr3 so you actively have to do something to change that.

2017 Update Post

Wauw. Almost a year has past since my last update – lots have happened. Not much time for write content.

So why the content drought? Well the primary reason is that on April 21st of this year, my wife gave birth to our two twin boys 5 weeks early and 3 weeks ahead of expected due date. With twins the birth is usually started 2 weeks (week 38) before normal because of slight increases in risk. However 3 weeeks ahead of that time the water broke and then you sorta have to go with the flow from there 🙂

That meant that I went on paternity leave for a looong time. The first 5 weeks we were on a neonatal intensive care unit at two different hospitals as the boys were not large enough to go home. During that time I was allowed to stay with my family and not have to work.

After the 5 weeks we came home and my real paternity leave started. I was luck enough to be able to take almost 5 months leave with pay by combining it with some holidays.

So from end April until October 1st I was off from work.. And had my hands full with the kids 🙂 Twins is a lot of work if anyone ever asks.

Just before returning to my job at the university October 1st I was offered a new job which i ended up accepting. I had one months notice at the university so went to work there handing over my assignments to other people and then on November 1st I started my new job as an IT System Consultant @ Lytzen IT.

Now getting a new job and having to get grips on everything also puts a dampener on things, content wise. I have been busy doing lots of exciting things and hopefully I will get some more time to do some updates. I have one thing lined up already for one of the next days hopefully.

Stay Tuned!

vExpert 2017!

Again this year I have been granted the title of vExpert. This is to me a great honor and I have been looking forward to the announcements and hoping to be amongst the many talented people on the list.

Robert Jensen has again this year made a complete list of all Danish vExperts. The list is available here: http://www.robert-jensen.dk/danish-vexperts-2017/

And if you are reading this Robert, I work for Aalborg University and this is the blog url! 🙂

 

Using tags for something interesting

Hello all

Thought I would try and share a bit of what I have been working on internally for a while now. As we are an organization that is still does not have all processes and data management in place I have been working on rewriting an Python based page I hacked together with a colleague to list information about initially virtual machines but it grew to include DNS, physical machines, NAT relations as well as relying on ARP data to resolve MAC to IP where needed. I finished version 1.0 of this new system written in Django 1.10 a few weeks back and am already working on a 1.1 release with a few feature requests.

Now the system grew out of a need to identify people and relations around virtual machines in our infrastructure and today heavily relies on Tags in the web client. We were already using tags to include VM’s in backup jobs in Veeam (something I can highly recommend doing!).

So what and how do we do it? We use 5 tag categories with the names Owner, Department, SysAdm, AppAdm and Service. These 5 tag types give a lot of information directly on the VM on who to contact and what service the VM belongs to. This information is then on a daily basis exported to the new system in the form of a CSV file at the moment. This CSV file along with a file from our NAT device, on from the datacenter routers with ARP, a list of DNS records, a CSV file from our Racktables installation for physical machines where the 5 tag categories are also defined and a list of systems with the SCOM agent installed.

The data is then imported into a data model defined in Django’s Object-relational mapping that tries to correlate some of the information from the different files. The end result is a web page where all systems and DNS records in theory are listed and can be searched, filtered, sorted etc. Where one can find a system(physical or virtual) based on the IP, DNS record (A, CNAME or MX), name, type, OS, you name it. This has some of the characteristics of a CMDB or CMS but instead of showing what it should be it is showing what it actually is at the time of the latest export. We have used this to help ourselves in the Infrastructure department as well as allowing some supports to find information to help route incidents and service requests to the correct groups in our service management system.

Below I have included a blurred screenshot from the system to show one of the views defined. A cut out from a simple system list. Note that the last 5 columns are the tag categories from vSphere/Racktables with the danish names we chose for them.

So that is an example of how we use tags in our day to day operations. We still in some cases miss the old Custom attributes (I know they are still in the API but not exposed in the web client) with the option of inputting variable information like expiry dates etc. Having to do something like that with tags would in my opinion be a mess (imagine a tag for every single date of expiry).

A bonus of doing this you can actually correlate this information with information from vRops via e.g PowerCLI. As an example one could send an email based on an alert in vRops to the addresses set via the SysAdm tags.

VMWorld Europe 2016 – Day 2

The general session on day 2 started with the story I’d how everything today is becoming digital in the digital transformation. Education,  x-ray and even flamingos at a zoo are digital. 

Users want simple consumption and IT wants enterprise security.  Users want any app on any cloud avaliable on any device. This is were Workspace One comes in delivering access to all apps from anywhere on any device. We saw a short demo of Skype for Business running inside a Horizon virtual desktop. 

Workspace One even has several apps to increase productivity from Boxer email client to a expense report assistance app. You can even show 3D renderings on a Samsung Android tablet powered by Horizon and Nvidia Grid. 

The SDDC

More info on vSphere 6.5 was shown like the ability to HA vCenter at the application level with a 5 min RTO. 6x the speed of operations compared to 5.5 yielding faster power ons. Max of 20.000 VMs per vCenter. And again the new HYML5 client which will have updates outside of the normal vCenter patch cycle for faster updates and new features. 

Encryption of VMs without guest agents and based on storage policies allows for more security

And the monster VM can now go to 6TB RAM to support SAP Hana and other in memory databases. 

vSphere Integrated Containers 

Allows for running containers on your existing vSphere infrastructure with a Docker compatible interface. A registry of containers as well as a new management portal out in beta. VRA7.2 will even allow for deploying containers from the service catalog as you would any service. 

VSAN 6.5

A new release tightly integrated in the vSphere stack. New management options and a new option to directly connect two nodes with a witness off-site for ROBO and SOHO deployment. iSCSI to physical or virtual machines is now also possible allowing for making those old MCSC clusters with shared disks as well as running physical workloads of of VSAN. 

5.000 users are running it now and 60% have business critical apps like SQL servers running of this. 

Danish supermarket chain Coop are using VSAN to run 1300 VMs. Everything that can run of VSAN does. 

You can even use VRA7 and policy based storage to allow users to request a change of storage and let the policy engine do the migrations necessary. 

Vendors 

I got around to a few vendors as well yesterday to talk about products. 

Mellanox

Showed me a few of their new features such as adapters running 10/25/40/50 and 100 g networks. Supporting all sorts of protocols from RoCe and NVMeoF which allows for RDMA like access to remote NVMe based storage. 

Mangstor

This lead me to Mangstor who along with Mellanox provide a solution that allows you to actually use the NVMeoF protocol against their box and get insane performance as either stand alone or as a caching layer between existing storage clusters like lustre for example. 

Intel 

Had a chat with Intel about their whitebox servers supporting VSAN which contains hot pluggable PCIe NVMe storage in both standard and hyper converged solutions. 

Nexenta 

Gave me a good demo and talk about the product and what it does for file services. With support for mixed access NFS and CIFS which I’m not quite sure works as smoothly as presented as well as replication and snapshot based data protection. Overall an interesting product with a lot of potential. 

The Party 

What every might be waiting for now is the party Wednesday night. Overall a bit lackluster with not much going on except drinks and food. However the band this year was a surprise for me. I was happy to see Empire of the sun had been hired to give the night it’s musical touch. Very nice! 

After the party I went straight to bed and slept like a rock. 

And now to last day! 

VMworld Europe 2016 – Day 1

Early morning day 2 of my VMworld 2016 trip seems like the time to do a short recap of yesterday.

Yesterday started with the General Session keynote where Pat Gelsinger and several others presented the view from VMware. Amongst his points I found the following things most interesting:

  • THE buzzword is Digital Transformation
  • Everyone is looking at Traditional vs Digital business
  • However only about 20% of companies are actively looking at doing this. 80% are stuck behind in traditional IT and spend time optimizing predictable processes.
  • Digital Business is the new Industrial Revolution

In 2016 – 10 years ago AWS was launched. Back there were about 29 million workloads running in IT. 2% of that was in the cloud mostly due to Salesforce. 98% was in traditional IT. Skip 5 years ahead now we have 80 million workloads and 7% in public cloud and 6% in private. Remaining 87% still in traditional perhaps virtualized IT. This year we are talking 15% public and 12% private cloud and 73% traditional of 160 million workloads. Pat’s research time have set a specific time and date for when cloud will be 50% (both public and private). That date is June 29th 2021 at 15:57 CEST. We will have about 255 million workloads by then. In 2030 50% of all workloads will be in public clouds. The hosting market is going to keep growing.

Also the devices we are connecting will keep growing. By 2021 we will have 8.7 billion laptops, phones, tablets etc connected. But looking at IoT by Q1 2019 there will be more IoT devices connected than laptops and phones etc and by 2021 18 billion IoT devices will be online.

In 2011 at VMworld in Copenhagen (please come back soon 🙂 ) the SDDC was introduced by Raghu Raghuram. Today we have it and keep expanding on it. So with today vSphere 6.5 and Virtual San 6.5 were announced for release as well as VMware Cloud Foundation as a single SDDC package and VMware Cross Cloud Services for managing your mutliple clouds.

vSphere 6.5 brings a lot of interesting new additions and updates – look here at the announcement. Some of the most interesting features from my view:

  • Native VC HA features with and Active, Passive, witness setup
  • HTML 5 web client for most deployments.
  • Better Appliance management
  • Encryption of VM data
  • And the VCSA is moving from SLES to Photon.

Updates on vCenter and hosts can be found here and here.

I got to stop by a few vendors at the Solutions exchange aswell and talk about new products:

Cohesity:

I talk to Frank Brix at the Cohesity booth who gave me a quick demo and look at their backup product. Very interesting hyper converged backup system that includes backup software for almost all need use cases and it scales linearly. Built-in deduplication and the possibility of presenting NFS/CIFS out of the deduped storage. Definitely worth a look if your are reviewing your backup infrastructure.

HDS:

Got a quick demo on Vvols and how to use it on our VSP G200 including how to move from the old VMFS to Vvols instead. Very easy and smooth process. I also got an update on the UCP platform that now allows for integration with an existing vCenter infrastructure. Very nice feature guys!

Cisco:

I went by the Cisco booth and got a great talk with Darren Williams about the Hyperflex platform and how it can be used in practice. Again a very interesting hyper-converged product with great potential.

Open Nebula:

I stopped by at OpenNebula to look at their vOneCloud product as an alternative to vRealize Automation now that VMware removed it from vCloud Suite Standard. It looks like a nice product – saw OpenNebula during my education back in 2011 I think while it was still version 1 or 2. They have a lot of great features but not totally on par with vRealize Automation – at least yet.

Veeam:

Got a quick walkthrough of the Veeam 9.5 features as well as some talk about Veeam Agent for Windows and Linux. Very nice to see them move to physical servers but there is still some ways to go before the can talk over all backup jobs.

 

Now for Day 2’s General Session!

Cisco UCS Manager AD authentication

Hello all

I have had this on my to-do for a while and finally got around to finishing it – using AD authentication on Cisco UCS Manager (UCSM). Now this is not something necessarily complicated but the official guides expect you to use a single AD domain and use sAMAccountName as the userid attribute. We have a large forest with a single root domain a lot of child domains all with parent-child trust to the root domain. We do not have sAMAccountName uniqueness across domains so instead we use userPrincipalName as the unique identifier for users. Users can also come from any of the child domains so to avoid having to add every domain we usually add a connection to Global Catalog instead. A note – the images below are from the UCSM 3.1 HTML5 interface but it is the same in the older 2.2 Java interface.

Now lets get into it. First things first we need to add domain controllers. I suggest you add two for redundancy purposes. Go to the Admin pane and down to User Management and unfold LDAP. Right click LDAP Providers and click Add.

In the image below I input some mock info but the important parts are to set the full DN of the user UCS should use to bind to AD. If you are using a multi domain forest set Base DN to the root domains Base DN and set the port to 3268 for GC LDAP and 3269 for GC LDAPS (remember to check the SSL box). Set the Filter to userPrincipalName=$userid. Input the password for the Bind account and select MS AD.

LDAP providerClick Next and you will be taken to the LDAP Group Rule and set Group Authorization Enabled and Group Recursion to Recursive. The rest should be default and now look like this:

LDAP group ruleWe now need to make a LDAP Provider Group. Right click the LDAP Provider Group and click Add. Give the group a name and add your domain controllers:

LDAP Provider GroupClick OK to finish creating the provider group. Now we need to add some group mappings for use with Group Authorization. Right click LDAP Group Maps and click Add. In the GUI below input the group DN and select which roles they should have. You can use the built in roles or create your own. Click OK to save.

LDAP Group MapsFinally, and this is where the magic happens, add a new Authentication Domain. Unfold Authentication under User Management and right click Authentication Domains and click Add. Give your Domain a name (you will see this when you login) and Select LDAP. Once LDAP is selected a drop-down will be show where you can select the LDAP Provider Group you created earlier. Once done click OK to save.

Authentication Domain

From now on when you access the login page you will see a drop down in which you can select either Native or the name of your Authentication domain. Select your Authentication domain and input your userPrincipalName and password in the fields and enjoy using AD login!