HA Best Practices

Practice failover quarterly; apps must retry transient errors. These rules keep Patroni clusters actually available during real failures.

How to Use This List

• Apply at cluster design and after every Patroni/etcd upgrade.
• Use as a game-day prep checklist.
• Share routing and retry sections with application teams.
• Store measured RTO from drills next to each item where relevant.

A - Architecture

• Three PostgreSQL nodes minimum for production HA. One leader plus two replicas.
• Odd-count etcd or Consul quorum (3 or 5). Never two-node DCS.
• Separate etcd from PostgreSQL hosts when possible. Reduces correlated failure.
• Enable Patroni Linux watchdog (mode: automatic). Reduces split-brain window.
• Set maximum_lag_on_failover from measured byte lag SLA. Block stale promotion.

B - Routing and Pooling

• Apps connect via stable write hostname through HAProxy + PgBouncer. No primary IP in config.
• HAProxy health checks Patroni REST /primary, not TCP 5432 alone. Prevents replica misroute.
• Run RECONNECT or reload PgBouncer in Patroni failover callback. Clear stale backends.
• Document read vs write endpoints separately. Read-your-writes needs primary route.
• Keep server_lifetime low enough for failover (300s or less). Stale connection cap.

C - Applications

• Retry transient connection errors with exponential backoff. Through failover window.
• Set connect_timeout in connection strings. Fail fast, retry sooner.
• Use idempotency keys for writes during retry windows. Prevent duplicate side effects.
• Alert on application DB error rate during leader change. DB may be up while apps are not.
• Load test with proxy path, not direct node connections. Matches production behavior.

D - Operations

• Quarterly unplanned failover drill in staging; annual in production with approval. Record RTO.
• Run patronictl switchover before OS patches on primary. Planned leader migration.
• Never pg_ctl promote outside Patroni. Avoid DCS state desync.
• Time-bound patronictl pause; document owner. Resume automatic failover promptly.
• Page on Patroni leader role change and etcd quorum loss. Automated failover still needs human awareness.

E - Replication Alignment

• Physical slot per replica; monitor inactive slots. WAL disk safety.
• At most one sync partner unless latency modeled. See streaming replication best practices.
• Replication and backups are complementary. HA does not replace pg_dump or pgBackRest.
• After major upgrade, rebuild Patroni cluster on single major version. No mixed 17/18 members.

FAQs

Related

• HA Basics - RTO/RPO primer
• Patroni & etcd/Consul - configuration
• Connection Proxies - HAProxy and PgBouncer
• HA Testing - game-day runbook

Stack versions: This page was written for PostgreSQL 18.4 (stable 18, maintenance 17), pgvector 0.8+, PgBouncer 1.x, Patroni 3.x, and PostGIS 3.5+.