Incident Basics

8 examples to get you started with Postgres incident response - 5 basic and 3 intermediate.

Prerequisites

# Confirm you can reach the primary and have a read-only role for triage
psql "$DATABASE_URL" -c "SELECT version();"
psql "$DATABASE_URL" -c "SELECT pg_is_in_recovery();"

• Access to pg_stat_activity, pg_locks, and replication views.
• An on-call rotation with a documented escalation path.
• Runbooks stored in git, not in a wiki nobody reads during outages.

Basic Examples

1. Severity Classification

Map user impact to a severity tier before you change production.

• Sev-1 includes replication slot WAL retention filling disk on the primary.
• Sev-2 includes connection pool exhaustion causing 503s.
• Reclassify when facts change; do not lock severity at the first pager message.

Related: Incident Response Best Practices - runbook hygiene

2. First-15-Minutes Checklist

Run this sequence before deep diagnosis.

-- 1. Is the instance accepting connections?
SELECT now() AS ts, pg_is_in_recovery() AS is_replica;
 
-- 2. Who is connected and what are they waiting on?
SELECT pid, usename, application_name, state, wait_event_type, wait_event,
       now() - query_start AS query_age, left(query, 120) AS query_snip
FROM pg_stat_activity
WHERE pid <> pg_backend_pid()
ORDER BY query_start NULLS LAST
LIMIT 30;
 
-- 3. Blocked sessions?
SELECT blocked.pid AS blocked_pid, blocking.pid AS blocking_pid,
       left(blocked.query, 80) AS blocked_q
FROM pg_stat_activity blocked
JOIN pg_stat_activity blocking ON blocking.pid = ANY(pg_blocking_pids(blocked.pid))
WHERE cardinality(pg_blocking_pids(blocked.pid)) > 0;

• Post status in the incident channel every 15 minutes during Sev-1/2.
• Assign roles: Incident Commander, Comms, Technical Lead, Scribe.
• Capture timestamps for every action; you need them for the post-mortem.

3. Communication Template

Keep stakeholders informed without leaking sensitive query text.

[INC-1234] Sev-2 - Elevated API latency on checkout service
Status: Investigating
Impact: ~15% of checkout requests timing out after 30s
Start: 2026-07-09 14:22 UTC
Actions: Checking connection pool saturation and lock waits on primary
Next update: 14:37 UTC

• Link to the status page if you have one.
• Do not paste production passwords, PII, or full EXPLAIN output in public channels.
• Tell product when you expect the next update, even if the answer is "still investigating."

4. Safe Read-Only Triage Role

Create a role that can diagnose without mutating data.

CREATE ROLE incident_reader LOGIN PASSWORD 'rotate-after-incident';
GRANT pg_read_all_stats TO incident_reader;
GRANT CONNECT ON DATABASE app TO incident_reader;
GRANT USAGE ON SCHEMA public TO incident_reader;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO incident_reader;
ALTER DEFAULT PRIVILEGES IN SCHEMA public
  GRANT SELECT ON TABLES TO incident_reader;

• Rotate credentials after the incident closes.
• Prefer read replicas for heavy diagnostic queries during load spikes.
• Never grant SUPERUSER to on-call accounts "just for the incident."

5. When to Fail Over

Failover is a business decision, not only a technical one.

# Patroni example: check cluster state before manual failover
patronictl -c /etc/patroni/patroni.yml list

• Fail over when the primary is unrecoverable or corruption is confirmed on the primary.
• Do not fail over for slow queries alone; fix the query or kill blockers first.
• Expect brief connection errors during DNS or proxy cutover; warn app teams.

Related: Replication Lag Emergency - lag triage before rebuild

Intermediate Examples

6. Incident Timeline Capture

Structured notes make post-mortems fast.

mkdir -p incidents/INC-1234
cat >> incidents/INC-1234/timeline.md <<'EOF'
| Time (UTC) | Actor | Action |
|------------|-------|--------|
| 14:22 | pager | Sev-2 opened, checkout latency |
| 14:25 | dba | pg_stat_activity shows 200 idle in transaction |
EOF

• Store artifacts in git: queries run, EXPLAIN output, config diffs.
• Redact customer data before committing.
• Link the timeline to the Jira/Linear ticket.

7. Coordinated Backend Termination

Terminate only with policy and logging.

-- Find long idle-in-transaction sessions (common incident root cause)
SELECT pid, usename, application_name, state,
       now() - xact_start AS xact_age, query
FROM pg_stat_activity
WHERE state = 'idle in transaction'
  AND now() - xact_start > interval '5 minutes';
 
-- Terminate one backend (requires appropriate role)
SELECT pg_terminate_backend(12345);

• Require two-person approval for pg_terminate_backend on Sev-1 unless automated runbook.
• Log pid, application_name, and ticket ID in the incident channel.
• After mass termination, watch for connection storms through PgBouncer.

8. Handoff to Post-Mortem

Close the incident loop with actionable follow-ups.

## Post-Mortem Required (Sev-1/2)
- Root cause (5 Whys)
- Time to detect / time to mitigate
- What went well
- Action items with owners and due dates
- Runbook updates merged to main

• Schedule the post-mortem within five business days.
• Blameless culture: focus on systems and process, not individuals.
• Every action item needs an owner; "team" is not an owner.

Related: Lock Storm Triage - when termination is the mitigation

Stack versions: This page was written for PostgreSQL 18.4 (stable 18, maintenance 17), pgvector 0.8+, PgBouncer 1.x, Patroni 3.x, and PostGIS 3.5+.