So yesterday, we finished up our Spring 2016 disaster recovery exercise. The absolute best words that would be used to describe the exercise is utter failure.
I have been involved with many of these exercises over the years and there have been some doozies. Several have some close to failure, but none have actually succeeded… until this one.
Strangely, there really was very little that went right. There are, basically, two main pieces to our exercises. 1) the piece where the servers are restored and 2) the piece where the applications are “exercised” to make sure they work. I am in charge of the #2 piece. We never even got there. Very few servers were even restored. I hope they figured something out in their troubleshooting.
I guess the thing that bothers me the most is the prep-work that goes into these things. The logistics planning that ended up being for naught. Well, it wasn’t for naught. For that piece, we are, in fact, well prepared for an actual disaster. However, we didn’t even get that far for the exercise. Oddly, that was one of the most stressful and boring weeks of my DR career. With every passing day of nothing to exercise, the stress increased because that meant when things finally did “let loose”, #2 was going to be chaos… and exponentially so with every passing day of inactivity. It never let loose. The test ended in a mode of “okay, lets at least try to figure out what went wrong.”
I miss that part. I miss troubleshooting and figuring out what went wrong, but my services were not needed yesterday. I believe that people at work don’t see me as that guy anymore and I miss it.