When it gets hot, you need to cool it....but not to much...

In case someone missed my recent post on datacenter disasters in my neighbourhood, take a look at this.

Just about a month or two ago, I presented our management with a new proposal for our TSM environment. The primairy server grew beyond acceptable limits, and AIX4 clients kept failing backups because the TSM server's software level was too advanced. So my proposal was accepted, and very soon we would start to implement the new TSM layout. We made a separate TSM server using an older version, so the remaining AIX4 servers could still be backed up. We temporarily used the spare TSM server to act as the AIX4-backup TSM server, because the new design was to be implemented at a short notice.
So we wouldn't be exposed to possible TSM disasters for very a long time. Two weeks ago, an adjecent building had a little fire (go figure), which was under controle by the fire department in just a few minutes. It didn't affect our datacenter, because it was on a seperate floor in the adjecent building. The firefighters had to cut out some metal frames from the walls  and threw them onto the adjecent roof. Underneath that roof is our DC. Last week, some rain storms blazed over the country, leaving enormous puddles of water. Guess what. The stuff that was thrown onto our roof punctured the roofing, and made all the water enter our DC today. Some AIX server were standing right underneath the hole, including our primairy TSM server. They were flooded, and water was all over the circuitry.

Although my regular work day was over for about 4 hours, I was still waiting for our AIX admins to turn over the recovered AIX server, so I could start to rebuild the TSM server. Because the TSM backup/spare server was used for another purpose, we had to arrange some new hardware. This new hardware seems to only run LPAR's, so we had to recover the AIX server into a LPAR. This bedazzled our IBM CE too.

The DS4300 we are hosting our VTL data on, also got some splashes on it, but it still runs fine. We have about 55 TB of backup data on the box. Last weekend, we migrated all data from the wet VTL into a newly build VTL server and storage. I have to say that the Diligent Company was very helpfull and made the license delivery overnight, no questions asked. All the paperwork was done afterwards. Kudos to Diligent....

Although we have everything under control, or least we think we do, it was quite a disturbing week.
Events like this always proof that you never have everything under control.