Sooner later, something horrible and unexpected happens.
You can’t prepare for everything.
Oh, don’t get me wrong. You should certainly try. Examine every component of the system and imagine what would happen if it fails. It’s not enough to simply run the database in a cluster. Sometimes a process will fail part way through, leaving a system in an uncertain state. This can be a deplorable and embarrassing situation that should be avoided as much as possible.
Disasters will occur sooner or later. You will be remembered on how well you were prepared.
Assume every component in the system will have a failure at some point. Disk drives will fail. Network connections will drop. There will be a flood. A fire. A backhoe will cut through the fiber line. A hurricane rips the roof off the datacenter. Criminals steal the servers. Hackers break into the network. Terrorists destroy a building. An open window in winter causes a water pipe to freeze and break, flooding the floor and the eight floors below it.
All of these things have happened and can happen again. Or it could be something completely new. It pays to be prepared.
Today's Card 