Preparation pays

Disaster recovery only counts if you’ve tested it.

We keep a second, entirely independent set of infrastructure at a geographically separate disaster recovery site, ready to take over block production if the main one is lost. Setting it up and maintaining it is the easier part (although that requires care to keep it consistent and secure — more on that in a later post). Knowing it works when you need it is the harder part, so we rehearse failing over quarterly — not a paper exercise, but moving every pool across, forging a block on mainnet, then moving back and checking the records reconcile exactly.

The last rehearsal found two bugs in our automated failover processes. Neither would have shown up in normal running, and they would only surface during an actual outage — which, when you are under pressure, is the wrong moment to learn about them. Both are now fixed, and the checks that would have caught them earlier are in place.

There was no impact to block production, which is exactly the point. It’s why preparation pays, and it’s one of many reasons why delegating with ADAvault is a safe bet.