A client with strict uptime requirements wanted protection against an entire cloud provider going down, not just a single region. That meant genuine multi-cloud failover for their Postgres database, a meaningfully harder problem than multi-region redundancy within one provider, and one that most teams never actually need to solve. This client did: their contractual uptime commitments to their own customers were strict enough that a provider-wide outage, rare as those are, represented an unacceptable risk concentration.
Why multi-region within one provider wasn't enough
Multi-region redundancy inside a single cloud provider protects against the most common category of outage, a data center or availability zone going down, but it does nothing for the rarer, more consequential case of the provider's control plane, networking backbone, or global services having a bad day. Those events are infrequent, but when they happen they tend to take out every region at once, since the failure is architectural rather than physical. For most clients, that residual risk is acceptable given how rarely it materializes. For this client, given what was contractually at stake, it wasn't.
Logical replication across cloud providers worked, but required careful attention to network latency between providers, which shaped nearly every architectural decision that followed. Postgres's logical replication doesn't care where the subscriber lives, but the network path between two different clouds' infrastructure is a different animal from the low-latency backbone connecting regions within a single provider. We had to account for replication lag that could spike unpredictably during periods of cross-provider network congestion, and design our failover logic to tolerate that lag rather than assume it away.
Building the failover mechanism
Automating failover detection and DNS cutover reliably took more engineering effort than the replication setup itself, and we deliberately kept a human in the loop for the final failover decision rather than fully automating it. Detecting a genuine provider-wide outage, as opposed to a transient network blip or a problem in our own monitoring, is a harder classification problem than it sounds. A monitoring system that fails over too aggressively on false positives is arguably worse than one that's slow to react to a real outage, since spurious failovers introduce their own risk of data inconsistency and downtime.
Our approach settled on a tiered alerting system: automated monitoring detects and flags a likely outage within minutes, but the actual DNS cutover and primary promotion require a human to confirm the diagnosis before executing. That confirmation step adds a few minutes of latency to the worst-case recovery time, and we were explicit with the client that this was a deliberate tradeoff, not a limitation we'd fix given more time. A wrongly-triggered automated failover, promoting a replica that's several seconds behind the true primary during what turns out to be a transient blip, can cause more damage than the outage it was meant to protect against.
- Genuine multi-cloud failover protects against provider-wide outages that multi-region redundancy within one provider cannot.
- Logical replication across cloud providers works but requires designing around cross-provider network latency and lag variability.
- Automated outage detection paired with a human-confirmed cutover balanced speed against the risk of false-positive failovers.
- The engineering and operational cost of true multi-cloud failover is substantial and only justified by genuinely strict uptime commitments.
The operational cost that doesn't show up in the architecture diagram
What the architecture diagram doesn't capture is the ongoing operational burden of keeping a multi-cloud failover setup actually ready to fail over. Replication topology drifts if it isn't actively monitored, credentials and network configuration on the secondary provider need to stay current even though nothing is actively using them day to day, and the team needs to actually run failover drills periodically or risk discovering during a real incident that the runbook is stale. We scheduled quarterly failover drills with this client specifically because a failover mechanism nobody has exercised in six months is not meaningfully more reliable than no failover mechanism at all.
Cost was the other factor that's easy to underestimate going in. Running a warm standby on a second cloud provider means paying for meaningful compute and storage capacity that, in the best case, never gets used for anything. For most clients that ongoing cost is hard to justify against the actual probability of needing it. This client's business model made the calculation work, but it's a genuinely expensive insurance policy, and we say that directly to every client who asks about it rather than upselling architecture they don't need.
Who this is actually right for
True multi-cloud failover is expensive in engineering time and ongoing operational complexity. It's the right call for a narrow set of clients with genuinely strict uptime requirements, contractual penalties for extended downtime, or a business model where even a rare provider-wide outage would be catastrophic rather than merely inconvenient, and overkill for most everyone else. We've turned down requests to build this for clients where multi-region redundancy within a single well-chosen provider would deliver 95% of the protection at a fraction of the cost, because the honest answer to "can you build us multi-cloud failover" is sometimes "you could ask for this, but you probably shouldn't." That's not the answer that maximizes billable engineering hours, but it's the one that actually serves the client, and it's the one we give every time the math points that direction.
Because this question comes up more often than the number of clients who actually need the full solution, we've developed a more structured way of walking a client through the decision rather than relying on an ad hoc conversation each time. We ask about the actual cost of downtime in concrete terms, not a vague sense that "downtime is bad" but a real number: what does an hour of unavailability cost in lost revenue, contractual penalty, or reputational damage significant enough to affect the business's future. We ask about the client's tolerance for a failover event itself introducing a small amount of risk, since no failover mechanism is entirely free of the chance that the mechanism itself causes a problem the outage wouldn't have. And we ask, honestly, whether the team requesting the architecture has the operational maturity to run failover drills consistently, since a failover mechanism nobody drills is a false sense of security wearing an architecture diagram's clothing.
Most clients, walked through that framework honestly, arrive at multi-region-within-one-provider as the right answer for their actual situation, not because it's what we'd prefer to sell but because it's genuinely the more defensible choice for a business without truly catastrophic exposure to a rare, provider-wide event. The clients for whom true multi-cloud makes sense tend to self-identify clearly once the framework surfaces the real cost of downtime in specific numbers, since businesses with genuinely severe downtime costs usually already have a rough sense of that number before we ask.
If we were starting this project over, we'd invest earlier in the failover drill tooling itself rather than treating it as a follow-on task after the core replication architecture was working. We built the actual failover mechanism first and the tooling to safely rehearse it second, which meant the first few months of the system being technically complete were also months where nobody had actually exercised the full failover path under realistic conditions. A near-miss during an unrelated maintenance window, where a false alarm almost triggered a real failover before a human caught the misclassification, happened during exactly that gap, and it would have been caught earlier by a drill process if we'd built the rehearsal tooling in parallel with the mechanism itself rather than afterward.
We've since made building the drill and rehearsal tooling a required, parallel workstream on any future project involving automated or semi-automated failover, not an optional follow-up, specifically because of how close that near-miss came to being a real incident rather than a lesson learned safely in advance.
Part of this project that took more time than we initially scoped was making sure the client's own in-house engineering team, who would ultimately be responsible for maintaining this system day to day, understood not just how to operate it but why it was built the way it was, including the deliberate choice to keep a human in the loop for the final failover decision rather than fully automating it. That choice can look, from the outside, like an incomplete implementation rather than a deliberate one, and we wanted the client's team to be able to defend that design decision confidently to their own stakeholders rather than treating it as a gap we hadn't gotten around to closing.
We ended up writing a detailed design rationale document alongside the technical runbook, specifically covering the tradeoffs we weighed and why we landed where we did, since a system this operationally unusual needed more than an operations manual, it needed the client's team to genuinely understand the reasoning well enough to make good judgment calls of their own the first time a real incident didn't match the runbook's assumptions exactly. That extra documentation effort paid off a few months after handoff, when the client's on-call engineer faced a genuinely ambiguous partial-outage scenario the runbook hadn't explicitly anticipated, and made a sound judgment call quickly specifically because they understood the underlying design philosophy well enough to reason from first principles rather than searching for a matching runbook entry that didn't exist.