This article was originally published on our old blog in 2012, but it is still true and worth repeating.
We normally operate two complete systems in different datacentres. The first one serves our customers and the second replicates all of the data, in real time, and is standing by to take over. If a disaster strikes the first location, the service is quickly switched to the second system.
This works for all kinds of failure, from hardware faults, through to network outages, power failures, ISP bankruptcies and natural disasters. If a Tyrannosaurus Rex ever ravages our main datacentre, rest assured that we have another one standing by, far from the trail of destruction.
So what has that got to do with server upgrades? Well – we use the same trick. We set up a new server to replicate the old one, and then at the moment of our choosing (in the middle of the night, when our customers are asleep), we switch the service across. Easy.
It actually took careful planning and a lot of hard work, but we like to make it look easy 🙂