Additional NIC’s installed

During the SAN migration we placed an extra dual port NIC in the servers to make a total of eight Gigabit ports per server so we can build a virtual DMZ on two separate ports from the six ports that we use for the Service Console, VMotion and data center traffic.

It’s not essential to bring a DMZ into your VI this way (I guess you could put it on a VLAN and just make a vSwitch that only does that specific VLAN traffic) but it’s just something where we feel more comfortable having this traffic physically separated. A side effect is that it’s easier to explain to someone with less knowledge about virtualization which helps in certain situations (audits for example).

New host delivery confirmed

I received a short update from our supplier to let us know that the hosts that we ordered in November will be delivered this week. Kudos go to them for securing a replacement DL585 G2 chassis within a matter of days when one of the servers was diagnosed as being DOA (Dead On Arrival). The replacement unit was flown in today from the States so they could still hold their prognosed delivery date.

Because of the holiday season we’ll probably not be able to work on them next week so we’ll kick off the New Year with the completion of our planned cluster. Because the experiences are so good we have also decided to include an expansion of the cluster with another two hosts in next year’s budget as we are expecting our business growth of this year to continue and accelerate next year as well.

Power savings through virtualization

As we finally shut down our old SAN we were able to see the effect of virtualization on our power consumption today and we were quite amazed by the results. So far, 33% into our project, we have been able to reduce our power requirement in datacenter 1 by 3.7 KiloWatts (18.7 down to 15.0) and in datacenter 2 by approximately 1.5 KiloWatts. Unfortunately I am not able to calculate a true cost savings from this as I don’t have the pricing for our energy but when we finish it will be significant.

Because we are virtualizing everything to one production datacenter we will eventually be able to shut down datacenter 2 for production (which will mean less stringent requirements on cooling, UPS etc.). It will be interesting to see what the final outcome is for the project.

All is well and we still love ESX

Well, in the end it didn’t need an official response from VMware to solve the problem:

It seems that our troubleshooting efforts (rescanning, resignaturing and rebooting the servers somehow fixed the problem). We checked the /var/log/vmkernel and found that after setting LVM.EnableResignature to on and LVM.DisallowSnapshotLUN to off the host already saw the LUN’s as regular LUN’s in stead of snapshots but we weren’t able to rename them.

At first we thought the two problems to be related but this morning (after sleeping over it) we started to look in another direction and found an orphaned Shared Storage resource that probably wasn’t cleaned up correctly by ESX with the rescan of the storage.

After deleting this orphan we were able to rename all the Shared Storage resources. After a final reboot of the host all Shared Storage resources were detected as normal LUN’s so everything is fixed now. I have made a post in the VMWare forums with more details.

The fun thing this morning was that we were running full production while fixing one host. We just VMotioned everything over to the other host ! And then when we were done we just VMotioned everything back. And we had no complaints from users while doing that !

I mean, taking 40 servers down for maintenance in the middle of the morning would normally be impossible. After hating it yesterday when it was giving us problems, I am loving it today.

SAN migration final

Well, as expected Murphy had to show up after everything had been going so well in the first part.

After connecting the ESX hosts to the new SAN we ran into what seems to be a documented problem: a LUN that is incorrectly detected as being a snapshot and therefore acces is restricted to that LUN.

This gives the problem that in the VI client you cannot see that Shared Storage resources. The solution is to flag LVM.DisallowSnapshotLUN to zero and rescan the drives. The problem we had is that ESX still thinks these are snapshot LUN’s. After some discussion we decided to remove all VM’s and re-add them to the inventory. This seems to work.

We will have to wait on final say from VMWare support what we do about this.

SAN migration status

For those who are interested: we are now in our second day of the SAN migration from MSA1000’s to an EMC CX3-40.

So far everything is going OK, with a few hiccups yesterday evening. The speed of the migration (we use EMC’s SANCopy for the data move) has surprised us with seven out nine LUN’s ready by yesterday six o’clock in the evening. Two LUN’s from one MSA, with primarily fileserver data, didn’t copy over normally. We suspect because of the controller having difficulties with the I/O load. After the EMC consultant rescheduled the copy jobs and moved them to another storage processor on the CX they competed fine at nine o’clock yesterday.

All that is left now (…) is to patch the fibers to the new Cisco MDS9020 SAN switches, configure the zones for the servers / LUN’s and bring the servers online with their new storage.

SAN migration this weekend

If you were wondering why there seemed to be a lack of activity: we are migrating to our new SAN this weekend.

To have a relatively stable environment during our migration planning we only converted two fileservers (600GB and 500GB respectively). Although they are the largest fileservers we have and with this conversion all fileservers have now been virtualized the impact on the hosts has been minimal.

Memory usage is now 54% and 50% and CPU usage is 16% and 11% per host. Delivery of our new hosts has been confirmed for next week and since the licenses have all been bought and activated already we can quickly add these servers and carry on with the migrations.