A friend was recently sharing his experience in helping his company fix a botched RAID recovery. He was exhausted by the work and questioning the value of what he was learning from his employer.
But in our discussion we agreed that bad experience might be as valuable as good experience in some ways. It might not be as fun, and you can certainly hit diminishing returns on it. But the intense memories of those bad experiences are the basis for many of our most valuable instincts and judgements.
Well, at the time, it felt kinda nice to be able to help him put a happy face on his obnoxious job.
And then just this week I had the tables turned on me: A server I depended on failed because a battery swelled, and then warped and took out the motherboard. Even better, the backups only appeared to be working, but really weren't. And that's not all: there was some code on that server that wasn't yet in version control because of chaos on the team at the time - after existing in limbo for two years it was slated to be added this week. We were down for days replacing that server. And I suppose we were lucky - it could have been worse.
All this got me thinking about my glib advice to my friend about the value of crappy experience. Was there a silver lining for me in all this? Did this week add to my mental catalogue of situations to avoid? Well, just as an exercise I decided to write down what the catalogue might look like. At least just the part that deals with backups & recoveries. Here it is:
