Context and Architecture
The platform contained several web applications with different release cadences, regional App Service deployments, Azure Front Door, SQL data, storage dependencies and external integrations. A single traffic switch could not make every dependency ready in the secondary region.
Business Risk
A regional loss could produce a misleading partial recovery: Front Door might route successfully while an application used the wrong database role, lacked a storage object, or called a regional integration that had not failed over. The key risk was declaring recovery from edge health alone.
Observable Evidence
- Application-to-database and application-to-storage dependency maps.
- Front Door origin health and routing configuration.
- SQL failover-group listener and replication state.
- Storage redundancy mode and secondary-read constraints.
- Configuration and secret differences between regions.
Hypotheses Considered
The design compared active-passive and active-active application tiers, operator-initiated versus automatic database failover, RA-GRS storage reads, and whether some applications should remain unavailable rather than return inconsistent results.
Root Cause of the Readiness Gap
The original plan treated regional routing as the recovery mechanism. Dependency review showed that traffic, data role, storage access, secrets and application validation were separate state transitions with different rollback consequences.
Permanent Design
The runbook established ordered gates: confirm incident scope, freeze risky deployments, verify secondary capacity and configuration, change the SQL role where approved, validate storage behavior, warm application instances, run synthetic transactions, then adjust Front Door routing.
Validation and Failback
Exercises used application-specific checks rather than a shared 200 response. Operators verified authentication, representative reads and writes, background jobs and integrations. Failback required replication health, a new change window and the same gates in reverse; it was not automatic after the primary region returned.
Operational Lessons
Regional disaster recovery is a dependency choreography, not a routing feature. Recovery objectives must be assigned per application and backed by rehearsed data and configuration transitions.