Incident Response Best Practices

Runbooks in-repo; post-mortem every sev-1. Use this list during incident reviews and when onboarding DBAs to on-call.

How to Use This List

• Keep runbooks next to application code in git; version them like production config.
• Tick items after each Sev-1/2 incident to grow organizational memory.
• Convert repeat failures into alerts, timeouts, or CI guards within two sprints.
• Rehearse top-five scenarios quarterly on a staging cluster.

A - Preparation

• Runbooks live in-repo with tested commands. Copy-paste SQL and bash that worked in staging.
• On-call has read-only triage role. No shared superuser password in Slack.
• Severity matrix agreed with product and SRE. Everyone uses the same Sev definitions.
• Escalation tree documented. Named backups when primary DBA is unavailable.
• Pager routes to humans who can run psql. Not only generic infra on-call without DB access.

B - Detection and Triage

• Alerts tie to user-visible SLOs. Disk bytes alone is insufficient without WAL rate context.
• First query is pg_stat_activity. Before tuning queries, know who is connected and waiting.
• Capture lag in bytes and time. pg_wal_lsn_diff catches commit replay delays.
• Check replication slots on every disk alert. Inactive slots are a top primary killer.
• Log every pg_terminate_backend with ticket ID. Auditable termination policy.

C - Communication

• Status updates every 15 minutes during Sev-1/2. Even if status is "still investigating."
• Single incident commander. Prevents conflicting production changes.
• Customer comms separate from technical thread. No PII or query text in public posts.
• Declare all-clear only after error rate normalizes. Not when first fix is applied.

D - Recovery and Learning

• Post-mortem every Sev-1 within five business days. Blameless, action items with owners.
• Update runbook in same PR as post-mortem. Knowledge does not live in slides.
• PITR restore drill at least annually. Untested backups are assumptions.
• Review lock_timeout and idle_in_transaction policy after lock incidents. Defaults are not policy.

FAQs

How many runbooks do we need?

Start with five: disk full, lock storm, replication lag, connection exhaustion, and failed migration.

Should developers be on-call for Postgres?

Pair app on-call with DBA for Sev-1; app owners own connection pool and transaction boundaries.

What tools belong in every runbook?

psql connection strings (redacted pattern), Patroni/patronictl, disk and WAL queries, escalation contacts.

Do we need a war room for every incident?

Sev-1 yes; Sev-3 can use async ticket unless impact spreads.

How do we practice without production risk?

Game days on staging with injected lock holders and filled temp disks.

What metrics should never be missing?

Connection count, replication byte lag, slot retention bytes, disk free %, transactions per second, p95 query latency.

When is failover the right runbook?

Primary unrecoverable or confirmed corruption on primary, not mere slow queries.

How long to keep incident artifacts?

At least one year in git for compliance-heavy industries; redact PII.

Should we auto-terminate idle transactions?

Yes with idle_in_transaction_session_timeout after app review; document in runbook.

What should I read next?

Follow the Related links for scenario-specific depth.

Related

• Incident Basics - severity and first-15-minutes checklist
• Lock Storm Triage - blocker identification
• Replication Lag Emergency - slot and rebuild decisions
• Disk Full on PGDATA - WAL explosion runbook
• PITR Drill Runbook - restore practice

Stack versions: This page was written for PostgreSQL 18.4 (stable 18, maintenance 17), pgvector 0.8+, PgBouncer 1.x, Patroni 3.x, and PostGIS 3.5+.