Review Your Infrastructure Needs Regularly - or ....

Posted on 4 mins read

Let me give you a real case scenario. (I am not saying that any of the implementations in this example are best practice)

Default rack that was installed was a 42U rack.

Each rack was populated with no more than 16 Servers. (This was the most that they were willing to lose in the case of losing a full rack and also due to cooling issues per rack they were leaving 1U empty between each server).

Each rack has a 16 port LCD KVM keyboard tray (1 for each server).

Each rack has 24 1Gb Ports and also a 48port 10/100mb for backup network and management ports.

Up until now each Server by default had 1 NIC connected to a 1Gb port and the backup to the 100mb port - which left them with 16/24 1Gb ports populated.

Enter ESX.

Each ESX server has by default 6 NICS

  • vmnic0 + vmnic5 - SC + Vmotion
  • vmnic1 + vmnic3 - Iscsi/NFS storage
  • vmnc2 + vmnic4 - VM network traffic

Out of the 16 servers in each rack 2 ESX servers in a rack (the most they would risk is two hosts per rack)

Quick math again.

2 ESX hosts x 6 ports = 12 (All port have to be 1Gb)

OK 14 more servers in the rack - but one sec, there are only 12 ports free - which means that you can only put in 12 more servers.

The options were:

  • Run more 1GB cables to the Network Infrastructure - more switch ports which meant additional cost
  • Reduce the number of servers per rack to 14 servers per rack (not 16) - which meant 2 KVM ports were wasted and more empty U’s in each rack.
  • Run some production servers not on 1Gb infrastructure
  • Deploy 10Gbe - cost for 10Gbe NICS and modules for the Backbone infrastructure

What happened here? The racks were cabled several years ago - before Virtualization was something that they had in their datacenter - and they had what they thought more than enough place to grow in their network ports, and they did until they started to put in ESX servers with 6 NIC’s each.

Of course all of the options above have a cost with them - be it a cost that you have to lay out immediately or cost of not utilizing the resources they had to the full extent (rack space + KVM).

What was decided in the end, you may ask - well I do not have a definitive answer to that yet - because it is still under deliberation - as to which option they will choose.

What can you learn from this experience?

  1. You should review you datacenter infrastructure at regular intervals. What was suitable for you 2 years ago is not necessarily the same today.
  2. Bringing in a new technology will definitely have some benefits - but it will also have its caveats as well.
  3. Virtualization is not only something that touches one aspect of your infrastructure (be it your initial savings on costs of Hardware / Operating System Licensing) it affects everything:
    • Network - what ports and how many
    • Storage - is your storage fast / big enough?
    • Hardware - do you scale up / scale out
    • Operating Systems - Licensing models will change
    • Backup / Restore - will your system suffice?
    • BCP / DR - should you use new technologies (SRM)?
    • IT Procedures - Server provisioning - and lifecycle
    • Manpower - Maybe your headcount will shift according to your changing needs
    • Cooling - depending on your hardware your cooling needs will change
    • Electricity - ditto
    • Maintenance fees - for hardware will change - and software as well
    • OPEX / CAPEX - VMware Licenses - MS licenses - Hardware contracts
    • Rack Space - how much will you need in a rack?
    • User Education - they will get exactly what they pay for and do not need a gazillion cores :)

As you can see the list is long - and I am actually not sure that I have all the elements listed here.

I would be interested in hearing from you all if you have any additional elements that this will affect and your thoughts in this article.