2017 Update Post

Wauw. Almost a year has past since my last update – lots have happened. Not much time for write content.

So why the content drought? Well the primary reason is that on April 21st of this year, my wife gave birth to our two twin boys 5 weeks early and 3 weeks ahead of expected due date. With twins the birth is usually started 2 weeks (week 38) before normal because of slight increases in risk. However 3 weeeks ahead of that time the water broke and then you sorta have to go with the flow from there 🙂

That meant that I went on paternity leave for a looong time. The first 5 weeks we were on a neonatal intensive care unit at two different hospitals as the boys were not large enough to go home. During that time I was allowed to stay with my family and not have to work.

After the 5 weeks we came home and my real paternity leave started. I was luck enough to be able to take almost 5 months leave with pay by combining it with some holidays.

So from end April until October 1st I was off from work.. And had my hands full with the kids 🙂 Twins is a lot of work if anyone ever asks.

Just before returning to my job at the university October 1st I was offered a new job which i ended up accepting. I had one months notice at the university so went to work there handing over my assignments to other people and then on November 1st I started my new job as an IT System Consultant @ Lytzen IT.

Now getting a new job and having to get grips on everything also puts a dampener on things, content wise. I have been busy doing lots of exciting things and hopefully I will get some more time to do some updates. I have one thing lined up already for one of the next days hopefully.

Stay Tuned!

vExpert 2015!

YAY!! Can’t really get my arms down yet. I was not sure I would make the cut this year but I did! So happy.

I was on vacation last week when the announcement of vExpert 2015 second half went out. A bit scared when I open the page and started scrolling only to realize that searching would probably be easier 🙂 So I did expecting not to find myself. But I did! So proud and humble that it happened and to me.

Now this announcement motivated to me to try and take my contributions a bit further. I will attempt to put out more content via this blog as often as possible and attending VMUGDK and trying to come up with more sessions to present. This is not my strongest side but it is a side that I believe I need to improve.

Thank you VMware for granting me this title! and thank you VMUGDK for the great danish VMware community!

Sysblog.dk is now a thing

Finally! I have had this domain on hand for almost a year with the intention on moving my virtualization blog to this domain. The name came to me one day and to my luck the domain was available!

I plan on continuing writing about virtualization and the things I work on at Aalborg University but also all the other interesting stuff I stumble upon in my daily life.

A few topics are already being lined up for the press regarding a few Windows issues I encountered when I got my new laptop at work. So stay tuned!

Since the last time

So it’s been a while since my last post, a lot has been going on. I have been through my first “employee performance interview” (Medarbejderudviklingssamtale or MUS in Danish). It was good and a lot of things were discussed in regard to this new organization. Some steps to increase my skills were also planned and I will get back to that later.

Since last time I attended VMWorld Europe 2013 in Barcelona! It was an awesome conference as always and I got a lot of new things with me home. One of the things I did different this year as compared to the previous two years was spend a lot more time on the Solutions Exchange. I focused primarily on storage vendors as I have taken fancy in new flash accelerated or all-flash storage systems. So I think I visited every booth with just the slightest connection to storage.

I also had the chance to discuss some of the new technologies coming out of VMware and also discuss the upgrade procedure for vSphere 5.5 when running with an SSO behind a load balancer. That was really useful and insightful and provided me with most of the information I need to perform an upgrade of the SSO and Web Client in our environment to vSphere 5.5 to relieve all the AD problems we have had. I will post a blog article on this later as there are still some hick-ups in the documentation and procedure that I need to test out and receive confirmation on from VMware support.

Our consolidation process has not been moving that much. Shortly after returning from Barcelona I took part in a live migration of VMs between our data center and a remote server room across a distance of about a kilometer. Without going into details about how everything was connected suffice to say that we had a single 10Gbit Ethernet connection between our one of our data center routers and one of the server room’s routers. We also had a single FC connection between a storage array in the data center and the blade chassis in the server room. This allowed us to evacuate a single blade in the server room and move it to an identical blade chassis after this we used another blade in the server room as the “transport host”. We vMotion VMs onto it as it could see both data center and server room storage arrays. The Storage vMotioned the VMs to the data center array and finally vMotion onto the host in the data center. Then one by one we evacuated all blades in the server room and moved them to the data center. The process took about 2 days including the move of a few physical hosts as well and was all in all very successful.

We had a single error during the move which caused an unexplained HA restart. The largest of the VMs (1TB storage spread on 4 different VMDKs) was set to change format to thin provisioned during the Storage vMotion. But at some point during migration we got an unexpected error (This was the actual message from the vSphere client). 30 seconds later HA spontaneously rebooted the VM even though Virtual Machine monitoring was disabled and the host didn’t crash. Luckily the VM handled the reboot well and it occurred close to midnight with no users online.

Right now my colleagues are planning the consolidation of two other VMware installation which will most likely be done with cold migrations. The amount of VMs is small and the fiber connections and licenses of these installations will not allow us to do a live migration. They are also planning a move similar to the one I worked on which we hope to complete some time in December. I am working on a cold migration of VMware installation as well where most of the VMs will be reinstalled on a new cluster rather than migrating them.

That was a status on what we are working on. Also back to the “I will get back to this later”. During the next month I will be working on a test installation of vCloud Automation Center to experiment with it and research if this is something we can use in our organization. The initial tests will be confined to the infrastructure department but if it works out it might be scaled up.

The process of consolidation

Hello readers.

This will be the first real post on this blog. Today I’m going to describe the process of consolidation of IT units with a specific focus on the virtualization part of this.

I’ll start up with some basic info. I work at Aalborg University (AAU) and have done so for almost 5 years, 4 of which as a part-time student employee. In September 2012, shortly after being hired full-time, the process of consolidating all IT units on Aalborg University startet with the hiring of a CIO to manage the new organization. Her work the following months resulted in the hiring of 4 new IT chiefs with responsibility of 4 different departments in the new organization; Process and Strategy, Applied IT and Development, Infrastructure Services and Support Services.

All employees of AAU’s different IT departments were then moved to this new organization this March. I was moved to the Infrastructure Services department and in this department, the Datacenter and Networking group. My and two colleagues will in the future be maintaining and developing AAU’s virtualization efforts. Primarily this is, as mentioned earlier, VMware but in the future we might look at other solutions as well.

Now to the more technical part. As I mentioned earlier AAU has 7 different vCenter servers  and 16 distinct clusters of 1 or more hosts (I know a single host cannot really be called a cluster 🙂 ) consisting of a total of a little over 50 hosts. On these 50 hosts somewhere around 1000 VM’s are running in some form. My old IT department had 2 of these clusters, each with their own vCenter server. We had a total of 13 hosts (8 and 5 respectively) running just under 300 of the virtual machines. We by far had the largest, newest and most developed environment.

So what is the golden solution for us then? We are working all over the Infrastructure department on consolidating all servers, storage, services and some network in two data centers and a backup location. We were lucky to last year get a completely new data center with 160kWh cooling and power capacity. This will be our primary data center and one of the older server rooms will be adapted to work as the secondary. We are required to have a backup location that is separate from the two data centers, this is the location where all backup from the two production locations will be stored.

In the new vSphere setup that we recently started using and migrating to we have maintained a design where each data center has a vCenter and when we are done 3 clusters; one cluster for infrastructure machines (AD, DNS, Exchange etc), one for customer machines (people external to the IT department needing virtualization) and one for testing purposes. I would draw you a graph showing the setup but currently I’m not using a machine with anything proper to draw it with:)

We settled (despite some warnings) on a vCenter setup consisting of 6 machines; 2 SSO nodes in HA (one in each data center with for now an Apache load balancer) with a database on a MSSQL 2012 cluster, 2 vCenter servers with their respective Inventory service as well and 2 Web Client servers that in the future will be placed behind a physical load balancer to provide some load distribution and availability of access. We are aware that database clusters are not supported on neither the SSO nor vCenter itself but having been running vCenter on an MSSQL 2008 cluster for a little over 3 years very little problems have been seen, most fixed by restarting the vCenter service.

The setup was tested in a small setup and shown to work but of course when deploying to production things change. We deployed the servers to a new data center networking setup meaning we needed to modify filters on a lot of networks to establish access from the new vCenter servers to the ESXi hosts on old networks. We needed to deploy on the new SQL cluster which was having networking problems. We then ran into some collation problems (we came to the conclusion that they were caused by failing over the empty databases in our SQL cluster) but with databases initialized the problem disappeared.

So after spending roughly 1½ weeks on installing and prepping the new environment with my colleagues we were ready to deploy. We then proceeded to test with one of the clusters by first moving one host from the old vCenter to the new and back. Success. A week later a maintenance window was planned and the entire cluster of 5 hosts including VMs was moved live from the old vCenter to the new with only a single IIS service having a hick-up in the process.

The details of the moving process will be described in a later post as there were some more or less complex steps needed to be taken for the process to run smoothly including moving VMs and VMkernels from distributed switches to virtual switches, copying resource pools and folders etc. Thank god for PowerCLI 🙂

That was it for the first post, hope you enjoyed reading it:)

And we have a go for launch!

So this blog is live! Very nice. I have been wanting to get into writing about the things we are doing at my work place with respect to virtualization. Some things just as a note to myself and other things to give others a tip on how we decided to solve some problems/assignments.

Not much to say yet but over the next couple of days I will try and write some things about the current process of consolidating 7 distinct vCenter installations and 16 separate clusters consisting of a total of about 56 hosts into one joined setup with 2 vCenters and and, the end of the tunnel, 6 clusters. It will contain some tips on the process of preparing for this and some of the steps that we have taken and also some designs on what we chose to do and at least some degree of argumentation of why we chose this 🙂

Until the have fun and virtualize!