Category Archives: VMware

vSphere HA virtual machine monitoring error on VM

Posted on by — No Comments ↓

Hi all

Today I found a couple of VMs in a cluster, that had HA with VM monitoring enabled, that were showing a “vSphere HA virtual machine monitoring error” with a couple of different dates.

Looking into the event log of the VM via vCenter I could see the following events about every 20 seconds:

This indicated that HA VM monitoring wanted to reset the VM but failed. I tried searching for answers on first Google but with no luck. I remembered that the setup had vRealize Log Insight installed and collecting data so perhaps Log Insight had more logs to look at.

Made a simple filter on the VM name and found repeating logs starting with this error from “fdm” which is the HA component on the ESXi host.

This error looks bad to me. Not being able to find the MoRef of a running VM? Hmm. I asked out on Twitter if anyone had seen this before with out much luck. Think about it over lunch I figured that maybe it was the host that was running the VM that had gotten into some silly state about the VM and not knowing it’s MoRef. So the quick fix I tried was to simply do a compute and storage migration of the VM to a different host and a different datastore to clean up any stale references to files or the VM world running. The event log of the VM immediately stopped spamming the HA message and returned to normal after migration.

I cannot say for sure what caused the issue and what the root cause is but that at least solved it.

Simple network debugging – ESXi and VCSA

Posted on by — 1 Comment ↓

I ran into an old issue yesterday with ESXi 5.5 and vCenter disconnecting after 60 seconds. The issues is described in KB1029919 and is due to heartbeat UDP packets from ESXi not reaching vCenter on port 902 within 60 seconds or at all.

Now how to test this! Luckily the KB has a simple Python script that allows you to send UDP packets. Just edit the script and insert the IP of your vCenter server and run it. It will default send 10 packets of 100 bytes.

Now the KB mentions using Wireshark as an option on the vCenter side but when using VCSA that is not really an option. Luckily VCSA 6.5 comes with tcpdump installed! 5.5 and 6.0 don’t but worry not, VMware to the rescue. You can install tcpdump on 5.5 and 6.0 by following KB2084896. This gives you access to a client on ESXi to send UDP packets and a monitoring option on VCSA to see if the packets arrive.

All that needs to be done then is run the following:

#on the VCSA
tcpdump -i eth0 udp port 902 -vv | grep <srcIP>

#on ESXi
python udp_client.py

And then look at the results. You can lose the “| grep <srcIP>” if you want to see all packets but depending on your setup there may be many ESXi hosts sending heartbeat.

IO Issues on Cisco C220 M4

Posted on by — No Comments ↓

Hi All

As promised more content is finally available! Unfortunately I cannot share any screenshots from this actual issue I worked on – you will have to take my word for it.

At my new job I was tasked with solving some issues that had been observed on a remote site on two Cisco C220 M4’s with local disks. The hosts run a set of redundant services but nothing is shared between them except network equipment. The issues were 1) some times around 04.00 AM the software running in the virtual machines would throw an alarm briefly and 2) powering on/off or snapshotting a VM would cause the software to throw alarms as well but on other VMs than the one being powered on/off or snapshotted. The event log on both hosts showed intermittent I/O latency warnings not connected to any of the above issues but nothing alarming.

Now one note though. The software running on these VMs is very latency sensitive so something like snapshots could potentially be a problem in any case but powering on VM A should not affect VM B unless the host is hurting for resources which is not the case.

Before diving in I asked out in the vExpert Slack if anyone had seen issues like this before or had any ideas of what to look for. James Kilby and Bilal Ahmed were quick to throw some ideas on the table. James suggested that vswp file being created on power on might cause the problem and Bilal suggested looking at the network. With those things in mind I started debugging.

First off I had already decided to update the ESXi version to latest 5.5 U3 + patches – it was a little out dated. Also decided to firmware upgrade the servers with the latest 3.0.3c release for Standalone C220 M4s. I had also found the latest supported drivers for the Cisco RAID controller (lsi-mr3) and the Intel NIC (igb) to rule out any compatibility issues. Also it was my hope that an update would remove some of the I/O latency warnings.

Now before upgrading anything I tested to see if the problem was still there. It was. Powering on a test VM with no OS and just 1 core and 4 GB RAM – instant alarms kicking off in the application. However powering off did not cause any noticeable problems. I proceeded to firmware update as my hope was this would solve the issues. Firmware upgrading on a remote site through a small connection is painful! It took a while. But once the first host was updated I proceeded to test if the issues were still there. There were. Damnit. Time to dig deeper. I tried out James’ idea of vswp being the problem and setting a 100% reservation seemed to solve the problem. However this was not a viable solution as this would only solve the problem if the powering on VM has a reservation. If anyone powered on a VM without it, it would still affect all other VMs, regardless of reservation on those.

I booted up our favorite debugging tool ESXTOP and vent into HBA mode and set delay down to 2 seconds. I then observed the Cisco RAID controllers behavior during power on operations and that freaked me out. It would happily do anything between 100 and 1000 IOPS at 5 to 150 ms while not powering on. The latency would spike high but nothing I was that scared of on a small set of local 10k disks. However when powering on a VM without reservation the HBA would stop doing any operations for upwards of 4 refreshes in ESXTOP (at 2 second intervals!). All indicators showing 0. No IO was passed so no latency was observed. This scared me a bit. Latest firmware and supported drivers. Damn. We weren’t seeing the same issue on another site with Dell servers but they also had SSDs instead of 10k disks. Was this the 10k disks not performing enough?

We had a short talk internally about what to do. My boss suggested that we looked at this Cisco bug:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCut37134/?referring_site=bugquickviewredir

This bug references an issue for some C220 M4’s that were installed with a specific version of the 5.5 ESXi Cisco Custom ISO. It was not the ISO these hosts were installed with but the solution was to use a different driver than the one ESXi default selects for the Cisco RAID controller HBA – the lsi-mr3 driver. Instead it instructs to make sure that the megaraid-sas driver is installed and to remove the lsi-mr3 and lsi-msgpt3 drivers and reboot which will make the megaraid-sas the active driver. We decided to try this. Downloaded the latest support megaraid-sas driver for the server and remove the lsi-mr3 and lsimsgpt3 drivers. Reboot and wait.

After getting online again with one host we tested. Powered on the test VM – and nothing. No alarms. What a difference. Tried it again looking at ESXTOP. No drops in IO. It was now doing 8.000 IOPS @ 15ms no problem. Major difference. Mass powering on VMs 15 at a time had taken minutes before the actual power on task was done and the machines would start booting. It took seconds now.

So what is the moral here? Apparently it could benefit you as a Cisco UCS server user to use the megaraid-sas driver instead of the lsi-mr3 driver. Both are included on the Cisco Custom ISO but it defaults to the lsi-mr3 so you actively have to do something to change that.

vExpert 2017!

Posted on by — No Comments ↓

Again this year I have been granted the title of vExpert. This is to me a great honor and I have been looking forward to the announcements and hoping to be amongst the many talented people on the list.

Robert Jensen has again this year made a complete list of all Danish vExperts. The list is available here: http://www.robert-jensen.dk/danish-vexperts-2017/

And if you are reading this Robert, I work for Aalborg University and this is the blog url! 🙂

 

Using tags for something interesting

Posted on by — No Comments ↓

Hello all

Thought I would try and share a bit of what I have been working on internally for a while now. As we are an organization that is still does not have all processes and data management in place I have been working on rewriting an Python based page I hacked together with a colleague to list information about initially virtual machines but it grew to include DNS, physical machines, NAT relations as well as relying on ARP data to resolve MAC to IP where needed. I finished version 1.0 of this new system written in Django 1.10 a few weeks back and am already working on a 1.1 release with a few feature requests.

Now the system grew out of a need to identify people and relations around virtual machines in our infrastructure and today heavily relies on Tags in the web client. We were already using tags to include VM’s in backup jobs in Veeam (something I can highly recommend doing!).

So what and how do we do it? We use 5 tag categories with the names Owner, Department, SysAdm, AppAdm and Service. These 5 tag types give a lot of information directly on the VM on who to contact and what service the VM belongs to. This information is then on a daily basis exported to the new system in the form of a CSV file at the moment. This CSV file along with a file from our NAT device, on from the datacenter routers with ARP, a list of DNS records, a CSV file from our Racktables installation for physical machines where the 5 tag categories are also defined and a list of systems with the SCOM agent installed.

The data is then imported into a data model defined in Django’s Object-relational mapping that tries to correlate some of the information from the different files. The end result is a web page where all systems and DNS records in theory are listed and can be searched, filtered, sorted etc. Where one can find a system(physical or virtual) based on the IP, DNS record (A, CNAME or MX), name, type, OS, you name it. This has some of the characteristics of a CMDB or CMS but instead of showing what it should be it is showing what it actually is at the time of the latest export. We have used this to help ourselves in the Infrastructure department as well as allowing some supports to find information to help route incidents and service requests to the correct groups in our service management system.

Below I have included a blurred screenshot from the system to show one of the views defined. A cut out from a simple system list. Note that the last 5 columns are the tag categories from vSphere/Racktables with the danish names we chose for them.

So that is an example of how we use tags in our day to day operations. We still in some cases miss the old Custom attributes (I know they are still in the API but not exposed in the web client) with the option of inputting variable information like expiry dates etc. Having to do something like that with tags would in my opinion be a mess (imagine a tag for every single date of expiry).

A bonus of doing this you can actually correlate this information with information from vRops via e.g PowerCLI. As an example one could send an email based on an alert in vRops to the addresses set via the SysAdm tags.

VMworld Europe 2016 – Day 1

Posted on by — No Comments ↓

Early morning day 2 of my VMworld 2016 trip seems like the time to do a short recap of yesterday.

Yesterday started with the General Session keynote where Pat Gelsinger and several others presented the view from VMware. Amongst his points I found the following things most interesting:

  • THE buzzword is Digital Transformation
  • Everyone is looking at Traditional vs Digital business
  • However only about 20% of companies are actively looking at doing this. 80% are stuck behind in traditional IT and spend time optimizing predictable processes.
  • Digital Business is the new Industrial Revolution

In 2016 – 10 years ago AWS was launched. Back there were about 29 million workloads running in IT. 2% of that was in the cloud mostly due to Salesforce. 98% was in traditional IT. Skip 5 years ahead now we have 80 million workloads and 7% in public cloud and 6% in private. Remaining 87% still in traditional perhaps virtualized IT. This year we are talking 15% public and 12% private cloud and 73% traditional of 160 million workloads. Pat’s research time have set a specific time and date for when cloud will be 50% (both public and private). That date is June 29th 2021 at 15:57 CEST. We will have about 255 million workloads by then. In 2030 50% of all workloads will be in public clouds. The hosting market is going to keep growing.

Also the devices we are connecting will keep growing. By 2021 we will have 8.7 billion laptops, phones, tablets etc connected. But looking at IoT by Q1 2019 there will be more IoT devices connected than laptops and phones etc and by 2021 18 billion IoT devices will be online.

In 2011 at VMworld in Copenhagen (please come back soon 🙂 ) the SDDC was introduced by Raghu Raghuram. Today we have it and keep expanding on it. So with today vSphere 6.5 and Virtual San 6.5 were announced for release as well as VMware Cloud Foundation as a single SDDC package and VMware Cross Cloud Services for managing your mutliple clouds.

vSphere 6.5 brings a lot of interesting new additions and updates – look here at the announcement. Some of the most interesting features from my view:

  • Native VC HA features with and Active, Passive, witness setup
  • HTML 5 web client for most deployments.
  • Better Appliance management
  • Encryption of VM data
  • And the VCSA is moving from SLES to Photon.

Updates on vCenter and hosts can be found here and here.

I got to stop by a few vendors at the Solutions exchange aswell and talk about new products:

Cohesity:

I talk to Frank Brix at the Cohesity booth who gave me a quick demo and look at their backup product. Very interesting hyper converged backup system that includes backup software for almost all need use cases and it scales linearly. Built-in deduplication and the possibility of presenting NFS/CIFS out of the deduped storage. Definitely worth a look if your are reviewing your backup infrastructure.

HDS:

Got a quick demo on Vvols and how to use it on our VSP G200 including how to move from the old VMFS to Vvols instead. Very easy and smooth process. I also got an update on the UCP platform that now allows for integration with an existing vCenter infrastructure. Very nice feature guys!

Cisco:

I went by the Cisco booth and got a great talk with Darren Williams about the Hyperflex platform and how it can be used in practice. Again a very interesting hyper-converged product with great potential.

Open Nebula:

I stopped by at OpenNebula to look at their vOneCloud product as an alternative to vRealize Automation now that VMware removed it from vCloud Suite Standard. It looks like a nice product – saw OpenNebula during my education back in 2011 I think while it was still version 1 or 2. They have a lot of great features but not totally on par with vRealize Automation – at least yet.

Veeam:

Got a quick walkthrough of the Veeam 9.5 features as well as some talk about Veeam Agent for Windows and Linux. Very nice to see them move to physical servers but there is still some ways to go before the can talk over all backup jobs.

 

Now for Day 2’s General Session!

vExpert 2016!

Posted on by — No Comments ↓

Yay – again this year I was awarded vExpert and I am proud to be able to keep the title for another year.

Robert Jensen has made a nice list of all the Danes that were awarded this year – you can check it out here: http://www.robert-jensen.dk/2016/02/06/danish-vexperts-2016/

It feels great to be awarded again – really makes one want to work to continue to contribute as best as possible to the community.

The complete list of vExperts for 2016 is available here: http://blogs.vmware.com/vmtn/2016/02/vexpert-2016-award-announcement.html

 

Failed to get size of IP buffer error

Posted on by — No Comments ↓

Hello everyone

Just a brief post today. Back in the start of January we saw and older Server 2008 32-bit showing this error in the title. It would spam the alert in the event log of the server until it became inaccessible. Not much was to be found about the error but I did find this post from Alex575 who also saw the error in January.

As no answers had been made on the post I decided to follow it and try and work out a solution. We haven’t updated ESXi and Tools above 9359 since ESXi 5.5 U3 so I started thinking that maybe the new VMware Tools 10 package could solve the issues as the event log entries came from the Tools service (vmsvc).

We upgraded the servers tools version to 10245 (Version 10.0.5) and from crashing every 10 days it has as of yet not crashed (14 days and counting).

VMware Tools from Version 10 will ship outside of vSphere releases as blogged by Brian Graf here: https://blogs.vmware.com/vsphere/2015/09/vmware-tools-10-0-0-released.html

The 10.0.5 release can be downloaded here: https://my.vmware.com/group/vmware/details?downloadGroup=VMTOOLS1005&productId=491

PowerCLI: Datastore Cluster and Tags

Posted on by — No Comments ↓

I was trying to help out a colleague yesterday when I realized that a quick fix to the problem would be to tag the datastore clusters in our environment and get them based on these tags instead of trying to determine which datastore cluster to choose when deploying a VM from PowerCLI.

So I decided to do this quickly and will show what I did (code snippets are from my vSphere 6.0 lab but the it is the same on our 5.5 production).

New-TagCategory -Name "CDC" -Cardinality Single -EntityType DatastoreCluster
New-Tag -Name "DC2" -Category CDC
Get-DatastoreCluster DatastoreCluster | New-TagAssignment -Tag "DC2"

Now I hope we can agree that I have created a new TagCategory that applies to Datastore Clusters and allows for one tag per object. We have also created a tag in this category called “DC2”. Lastly we have added the tag to the datastore cluster “DatastoreCluster”. Now if I run the following I get what I would expect:

C:\> Get-DatastoreCluster DatastoreCluster | Get-TagAssignment

Tag                                      Entity
---                                      ------
CDC/DC2                                  DatastoreCluster
C:\>

But if I run this I get something that I did not expect

C:\> Get-DatastoreCluster -Tag "DC2"
C:\>

This means that it is not working the same as for Virtual Machines with the “get-vm” cmdlet:

C:\> New-TagCategory -Name "VMTest" -Cardinality Single -EntityType VirtualMachine
Name                                     Cardinality Description
----                                     ----------- -----------
VMTest                                   Single
C:\> New-Tag -Name "Test" -Category "VMTest"
Name                           Category                       Description
----                           --------                       -----------
Test                           VMTest
C:\> Get-VM testvm01 | New-TagAssignment Test
Tag                                      Entity
---                                      ------
VMTest/Test                              testvm01
C:\> get-vm | Get-TagAssignment
Tag                                      Entity
---                                      ------
VMTest/Test                              testvm01
C:\> get-vm -Tag "Test"
Name                 PowerState Num CPUs MemoryGB
----                 ---------- -------- --------
testvm01             PoweredOff 1        4,000

So I do not know if this is the way it was meant to work but I is definitely not what I expected!

vRealize Operations 6.1 is out!

Posted on by — 1 Comment ↓

As of midnight danish local time vRealize Operations 6.1 is out! This is great as we have been waiting for this release to fix some issues we have been having with our environment running on 6.0.2. Last communication from VMware Technical Support a month ago was that our two remaining problems would be fixed in this release.

I’ve look through the list of fixes but did not see it directly so hoping they still made it 🙂

Release notes can be found here.

UPDATE: Upgrading the VA-OS pak file worked but applying the VA pak file failed to complete. The logs showed that it was the conversion from xDB to cassandra that failed. VMware tech support were fast today and recommended rollback and applying 6.0.3 instead until further diagnostics could be made on 6.1 -> apparently we were the first to submit a case on 6.1 install 🙂