Check, check and triple check!

Well, a happy new year to everyone!

I remember when I first started doing storage that unallocating storage from a host was one of the most nerve racking things Id done professionally.  I used to check, double check and then check again before clicking that Apply button and would then sit at my desk for 10 minutes in case my phone rang.  If it did ring it would send my pulse racing in case it was one of the DBA’s telling me that their production database had gone down complaining of disk errors – considering that, it’s a wonder Im still in the game!

Granted, these days Im a little older and a lot less nervous.  Ive also unallocated a lot more storage than back in those first days.  However, I’ve never totally lost that rush, if you can call it that, and to be honest I think its healthy to be cautious when playing with storage.  

Anyway………. Im currently working on a project decommissioning two HDS USP 1100’s, and as part of the project all hosts that are currently accessing disk on the USP’s must be migrated to other storage arrays.  And as part of that there is a lot of unallocating being done.

The way we are approaching the migration is as follows –
The hosts are currently accessing storage on one of two USP 1100’s which are to be decommissioned.  We are adding in the new storage arrays to the correct zones so that the hosts can see and access both the storage on the USP and on the new storage arrays.  We are presenting identical LUNs from the new storage arrays and using Veritas Volume Manager on the hosts to create mirrors.  Once the mirrors are created we are splitting off the plexes on the USP, performing some quorum swings and then removing visibility of the USP’s.

The funny thing is, Im finding myself checking, double checking and triple checking everything that Im doing.  After all, some of the servers are live servers and the last thing I want to do is remove the wrong half of the mirror and then see everything fall over when I remove the USP’s form the zones.

Here is a quick list of all of the double checks Im having to do –
When creating the mirrors making sure that one half is on the USP and the other half not
Making sure I remove the correct half of the mirror
Making sure I unmap the correct storage ports from the Emulex cards
Making sure I remove the correct zone members when removing visibility of the USP’s
Making sure I delete the correct Host Groups on the USP’s

Im sure the list could go on, and to be honest it wouldn’t bother me if it was just a handful of servers, but we are migrating everything off of two pretty huge USP 1100’s so are frantically performing multiple migrations every day.  And at the moment and its taking an age because of how careful I’m having to be.  Still better safe than sorry!

Then of course there’s the eventual spinning down of the USP’s!  Thankfully it wont be my responsibility to flick the “Off” switch – that’s a job guaranteed to get any responsible persons heart rate going.  However, I imagine I will still feel some responsibility in declaring the USP’s clean and making sure we haven’t left any hosts accessing storage on them.  Still even with all that said, Im sure I’ll sleep well the night before they flip that switch

Nigel