The process of consolidation

Hello readers.

This will be the first real post on this blog. Today I’m going to describe the process of consolidation of IT units with a specific focus on the virtualization part of this.

I’ll start up with some basic info. I work at Aalborg University (AAU) and have done so for almost 5 years, 4 of which as a part-time student employee. In September 2012, shortly after being hired full-time, the process of consolidating all IT units on Aalborg University startet with the hiring of a CIO to manage the new organization. Her work the following months resulted in the hiring of 4 new IT chiefs with responsibility of 4 different departments in the new organization; Process and Strategy, Applied IT and Development, Infrastructure Services and Support Services.

All employees of AAU’s different IT departments were then moved to this new organization this March. I was moved to the Infrastructure Services department and in this department, the Datacenter and Networking group. My and two colleagues will in the future be maintaining and developing AAU’s virtualization efforts. Primarily this is, as mentioned earlier, VMware but in the future we might look at other solutions as well.

Now to the more technical part. As I mentioned earlier AAU has 7 different vCenter servers  and 16 distinct clusters of 1 or more hosts (I know a single host cannot really be called a cluster 🙂 ) consisting of a total of a little over 50 hosts. On these 50 hosts somewhere around 1000 VM’s are running in some form. My old IT department had 2 of these clusters, each with their own vCenter server. We had a total of 13 hosts (8 and 5 respectively) running just under 300 of the virtual machines. We by far had the largest, newest and most developed environment.

So what is the golden solution for us then? We are working all over the Infrastructure department on consolidating all servers, storage, services and some network in two data centers and a backup location. We were lucky to last year get a completely new data center with 160kWh cooling and power capacity. This will be our primary data center and one of the older server rooms will be adapted to work as the secondary. We are required to have a backup location that is separate from the two data centers, this is the location where all backup from the two production locations will be stored.

In the new vSphere setup that we recently started using and migrating to we have maintained a design where each data center has a vCenter and when we are done 3 clusters; one cluster for infrastructure machines (AD, DNS, Exchange etc), one for customer machines (people external to the IT department needing virtualization) and one for testing purposes. I would draw you a graph showing the setup but currently I’m not using a machine with anything proper to draw it with:)

We settled (despite some warnings) on a vCenter setup consisting of 6 machines; 2 SSO nodes in HA (one in each data center with for now an Apache load balancer) with a database on a MSSQL 2012 cluster, 2 vCenter servers with their respective Inventory service as well and 2 Web Client servers that in the future will be placed behind a physical load balancer to provide some load distribution and availability of access. We are aware that database clusters are not supported on neither the SSO nor vCenter itself but having been running vCenter on an MSSQL 2008 cluster for a little over 3 years very little problems have been seen, most fixed by restarting the vCenter service.

The setup was tested in a small setup and shown to work but of course when deploying to production things change. We deployed the servers to a new data center networking setup meaning we needed to modify filters on a lot of networks to establish access from the new vCenter servers to the ESXi hosts on old networks. We needed to deploy on the new SQL cluster which was having networking problems. We then ran into some collation problems (we came to the conclusion that they were caused by failing over the empty databases in our SQL cluster) but with databases initialized the problem disappeared.

So after spending roughly 1½ weeks on installing and prepping the new environment with my colleagues we were ready to deploy. We then proceeded to test with one of the clusters by first moving one host from the old vCenter to the new and back. Success. A week later a maintenance window was planned and the entire cluster of 5 hosts including VMs was moved live from the old vCenter to the new with only a single IIS service having a hick-up in the process.

The details of the moving process will be described in a later post as there were some more or less complex steps needed to be taken for the process to run smoothly including moving VMs and VMkernels from distributed switches to virtual switches, copying resource pools and folders etc. Thank god for PowerCLI 🙂

That was it for the first post, hope you enjoyed reading it:)