HA & Failover Key Points

High availability for PostgreSQL is not a single switch you flip, it is a system assembled from replication, an orchestrator that detects failure and elects a new leader, and routing that redirects clients to whichever node is currently primary.

Failover is what happens when that system decides the primary is gone and promotes a standby in its place, and the speed and safety of that decision is what separates a brief blip from a real outage.

This page builds the mental model behind that system before the more procedural pages in this section walk through Patroni configuration, proxy setup, and fencing in detail.

Summary

• Core Idea: HA combines a replicated standby, a consensus-based orchestrator, and client-facing routing so a primary failure results in automatic, safe promotion rather than manual intervention.
• Why It Matters: Without all three layers working together, a healthy standby still means a manual, error-prone, and slow recovery.
• Key Concepts: DCS (distributed configuration store), quorum, leader election, fencing, STONITH, RTO/RPO.
• When to Use: Any production database where a primary outage of more than a few minutes has real business cost.
• Limitations / Trade-offs: Automated failover trades some risk of a wrong or premature promotion for a large reduction in mean time to recovery.
• Related Topics: streaming replication, Patroni and etcd/Consul, split-brain prevention, connection proxies.

Foundations

High availability in this context means the system continues serving reads and writes within an agreed recovery window even after the primary PostgreSQL node fails.

That window has two names worth knowing precisely: RTO, the recovery time objective or how long an outage is allowed to last, and RPO, the recovery point objective or how much recent data loss is acceptable.

A standby alone does not deliver HA, since a standby that requires a human to notice the outage, verify the standby is current, and manually promote it can easily take longer to recover than the RTO a business actually needs.

Failover automates that judgment call: an orchestrator continuously watches the primary's health, and when it decides the primary is truly gone, it promotes the most current, healthy standby without waiting for a person.

A switchover is the planned cousin of failover, a deliberate, orchestrated leader change used for maintenance, where the outgoing primary cooperates in the handoff instead of simply disappearing.

The orchestrator's decisions have to be trustworthy under network partitions, which is why production PostgreSQL HA relies on a distributed configuration store (DCS) such as etcd or Consul rather than the orchestrator deciding alone.

A useful analogy is a group of firefighters agreeing by radio vote on who leads an operation, rather than each one independently deciding they are in charge, since a single node guessing "the primary must be down" from its own limited view is exactly how two primaries end up active at once.

Mechanics & Interactions

The DCS provides quorum, a voting mechanism where a majority of DCS nodes must agree before a leadership decision, such as promoting a new primary, is considered valid.

An orchestrator like Patroni 3.x holds a time-limited lease in the DCS representing "I am the current leader," and it must continuously renew that lease or lose leadership automatically.

If the primary's Patroni instance cannot reach the DCS, its lease expires, and any replica reachable by the surviving DCS quorum can be elected as the new leader, all without a human in the loop.

This is where split brain, two nodes each believing they are the primary, becomes a real risk rather than a theoretical one, since a primary that is merely network-isolated from the DCS may still be running and accepting writes even as a new leader is elected elsewhere.

Fencing is the set of techniques that guarantee the old primary genuinely stops accepting writes before or immediately after a new one takes over, ranging from a software watchdog that kills the local PostgreSQL process to hardware-level STONITH that powers off or network-isolates the node entirely.

-- Any promoted node should answer false; a lingering old primary answering
-- false too is the split-brain signal operators watch for during drills
SELECT pg_is_in_recovery();

Once a new leader is chosen, routing has to catch up: applications must stop sending writes to the old primary and start sending them to the new one, which is the job of a proxy layer like HAProxy or a Patroni-aware health check rather than the database itself.

Connection pools add their own wrinkle here, since a pooler like PgBouncer holding open connections to a now-demoted node will keep routing traffic there until its own health checks or a forced reconnect catch up with the new topology.

Clients sit at the end of this chain, and even a perfectly executed failover produces a brief window of connection resets, which is why application-level retry with backoff is treated as part of the HA design rather than an afterthought.

Advanced Considerations & Applications

The deepest operational lesson in PostgreSQL HA is that a system with excellent components can still fail as a whole if those components were never tested together under realistic failure conditions.

DCS sizing is the first thing to get right, since quorum math means an N-node DCS tolerates the loss of floor(N/2) nodes, which is why a 2-node etcd cluster provides no real protection and 3 or 5 nodes are the practical minimum for production.

Colocating the DCS on the same hosts as PostgreSQL is a subtle trap, since a single host or rack failure can then take out both the database node and enough DCS votes to lose quorum at the same time.

maximum_lag_on_failover-style guards exist because an automated system will otherwise happily promote a standby that is technically healthy but far enough behind to violate RPO, so orchestrators expose a byte-lag ceiling below which promotion is refused.

Game-day drills, deliberately triggering failover in a controlled way, are the only reliable method for discovering whether RTO assumptions hold once proxy reconnect times, pooler behavior, and application retry logic are all included in the measurement, not just the database promotion itself.

Cloud-managed offerings such as Multi-AZ deployments implement the same conceptual layers, DCS-equivalent consensus, fencing, and routing, but hide them behind a managed control plane, which trades operational control for a smaller surface area to configure incorrectly.

The pattern that separates HA on paper from HA in practice is treating the DCS, fencing, and routing as first-class parts of the design, then proving RTO and RPO with drills instead of trusting the theoretical uptime math alone.

Common Misconceptions

• "A standby means I already have HA." A standby without an orchestrator and routing layer still requires manual detection and promotion, which is exactly the slow, error-prone process automated failover exists to remove.
• "Failover and switchover are the same operation." Failover reacts to an unplanned primary loss, while switchover is a cooperative, planned handoff for maintenance, and the two carry very different risk profiles.
• "Two nodes are enough for automated failover." A 2-node DCS cannot form a majority after losing one node, which makes 3 or more nodes the practical floor for safe quorum-based decisions.
• "Fencing is optional if the network is usually reliable." Split brain only needs to happen once to cause real data divergence, and "usually reliable" networks still partition often enough that production clusters treat fencing as mandatory.
• "Higher uptime percentages are always achievable by adding more standbys." More standbys improve read capacity and failover candidates, but RTO is bounded by detection time, promotion time, and routing propagation, not by standby count alone.

FAQs

Related

• HA Basics - RTO/RPO targets and the HA stack in practice
• Patroni & etcd/Consul - configuring quorum-based leader election
• Split-Brain Prevention - fencing, quorum math, and STONITH
• Connection Proxies - routing clients to the current primary
• HA Testing - running failover game days
• Streaming Replication Explained - the replication layer HA builds on

Stack versions: This page was written for PostgreSQL 18.4 (stable major 18, maintenance line 17 also supported) and Patroni 3.x.