Server Consolidation Plan: Risk of Biting Off Too Much!

In 2008 I took a virtualization engineer job with a large online company and immediately was tasked with helping the server consolidation project.

Like most companies that had been deploying VMware at this time (2008), there were multiple one-off environments, mix-matched server and storage configurations, and a slew of performance problems. For the record, server consolidation was just becoming a buzzword, so this wasn’t a surprise!

They also didn’t have any company-wide standards and every business unit was building their own virtual environments. Hence the root of the problem…

Under this project, the virtualization team was given the “green light” to standardize, consolidate and automate the virtual infrastructure. This project also included P2V and decommissioning as many physical servers as possible. At the time the Dell 1950 was the best low cost server to use for consolidation.

Lessons Learned

What I learned from this server consolidation project, which took nearly 18 months to complete, was – most managers don’t realize all the work it will take, and most companies will give up at the half-way mark because of the complexity.

Why do they quit half way?

Here are a couple of worst cases why server consolidation projects don’t finish. Both cases relate to the complexities that are not planned for that will cause extensive delays and scope-creep of the project:

1. Business critical systems that nobody wants to touch because the people who built them are no longer around to fix them if anything goes wrong – and there is no documentation. These are the same systems that never get security patches or reboots…
2. Applications that run on “spaghetti code” where IP addresses and user accounts are coded directly into the code. These systems generally are at the core of major services and will crash the service if they are messed with. They also do not get regular security patches or reboots…

Did I get your attention?

These are only a couple of examples that will cause problems during a  server consolidation project and there are more listed below for your consideration…

What it Takes to Finish!

Here’s a list of server consolidation project tips and unplanned problems to be prepared for:

• Make sure your server consolidation project team has the correct staff assigned (admins, engineers, developers, analysts, etc).
• Expect to take 3 times as long as you are planning (1. auditing and mapping servers and services 2. P2V migrations and rebuilding servers 3. Re-coding and fixing broken applications).
• Expect to run into complex problems with important applications (Example: database, application and web servers that are not centralized in one location).
• Expect to rebuild some of the servers that will not P2V (maybe 1 in 100 times there is a server that just won’t P2V).
• Expect there to be operating systems that are outdated (old Linux, Windows NT and 2000 systems should be upgraded).
• Expect applying updates will break some systems (you will most definitely run into systems; Windows and Linux that are never patched and when you patch them, they will break something).
• Expect that you will find spaghetti code (already explained).
• Expect to leave some physical servers as-is (already explained).
• Expect there will be some systems that have a physical dependency that will cause problems ( security, license, or attached FOB).
• Expect that you will have to lead through adversity (people are people and cause conflict with each other).
• Finally, also expect to run into problems with people that don’t want to virtualize (even though the project is approved, you will have people that drag their feet because they are against virtualization. They’ll want to test, test, test before they allow their servers to be virtualized, and don’t be surprised when they want VMs with the same VM resource requirements as big physical servers).

Summary:

Server consolidation is important to the bottom line so eventually you will do it…

Yes, you will be biting off more than most companies or IT departments want to chew! But remember you can finish what you start if you’re prepared for gotchas.

This post gives valuable insight and will help anyone that has never been involved in a server consolidation project.

Good luck!

Your turn:

Do you have tips or comments about server consolidation plans?