It is funny how things that happen in your personal life can teach a practical lesson that applies to your professional life. As a husband I have learned that it is not important to be right all of the time. This is an important skill to have in customer service where the customer is always right. As a father I have learned the value of patience, an important skill in management whether you have a team of one or one hundred. Many homeowners have learned that there are some jobs that call for a specialist when you are in over your head, a lesson learned the hard way by many IT professionals at 2 in the morning during an upgrade that went south. I learned that it is best to get a specialist involved early after my recent move to Cincinnati.
Our house has everything that we could ask for: four bedrooms, a three-car garage, a finished basement, and a huge yard with mature trees and plenty of wildlife. Springtime came, and the wildlife began to multiply. When wildlife starts to multiply, they bring their calculators and slide rules to warm, safe places to allow their babies to fine-tune their mathematical skills. A family of raccoons decided that our attic would be a great place to conduct these types of activities. The first sign that the new tenants were moving in was nothing more than a little scratching around in the attic. It was mostly at night, but not too big of a deal.
A few weeks later, we heard more scratching and “chirping” (that sound that raccoons make that sounds like a purr and meow) at 3:00 in the morning. I figured I could take care of the issue, so I got out the ladder and went up in the attic in the middle of the night to scare them off. Of course, they went right back to their antics as soon as I made it back to bed. After two weeks of this nonsense, my wife finally called out the exterminator. It took two more weeks to round all of the critters up, during which time it sounded like they were having a party up there complete with a disco ball and a DJ. Finally we had peace and quiet, but not before spending more than a thousand dollars rounding up the little monsters, cleaning up the mess they left in my attic, and patching the holes through which they entered to violate my humble abode. I now hate raccoons. I speed up when I see them on my street.
Most importantly, I could have saved money if I had called the specialist in earlier to fix the problem. I have seen this same story play out dozens of times in my professional life as a consultant. A customer puts in a new backup infrastructure that has everything that they need; plenty of horsepower on the media servers, plenty of capacity on the storage, plenty of bandwidth to back everything up, and de-duplication. Fast-forward a year later, the volume of data that they are backing up has grown 100%. The customer only planned on 30% growth year over year and had sized it for three years of growth. A lawsuit materializes and legal has told them that they have to keep daily full backups of email forever, or at least until the lawsuit is over. Email backups that used to run 1 hour, now run for 16. This overlaps with the production workday, causing email performance issues. Before they figure out what is going wrong, the “critters” in your data center have made your life miserable. It doesn’t happen all at once, it builds up slowly over time until it overwhelms you.
Unfortunately, the way this normally plays out is that the customer winds up hating their once beloved infrastructure. They want nothing more than to get it out of their data center and replace it with the next great thing. If they see the vendor that sold it to them in the parking lot, they speed up. The reality is that with regular monitoring and assessment by a trusted, outside consultant, the issues would have been identified much sooner, and solutions could have been implemented to prevent the problems before they got out of control.
The outside perspective of a specialist is important. The problem with managing the same infrastructure day in and day out is that you get used to the problems and restrictions imposed on the environment. As change happens, nobody has time to take a step back and consider the impact. Daily operational procedures stay the same, even as changes are imposed on the environment that can negatively impact those operations. I have seen customers that still have 9-track tapes in a vault somewhere because nobody wants to ask legal if it’s ok to destroy them.
An independent third party, that isn’t burdened with the knowledge of “the way it is” can help you prevent getting into trouble. Don’t let your data center get infested by critters. As Benjamin Franklin famously stated, “An ounce of prevention is worth a pound of cure”. Spending a little money on periodic visits by a trusted consultant can help you prevent small changes from becoming costly problems.