The Linux CLI
A PostgreSQL server does not run in isolation, it runs as a process on a Linux host that has its own disk, memory, and network limits, so reading the command line is really reading a second layer underneath the database itself.
This page builds the mental model for that reading: the three layers a triage session correlates, the doctrine that keeps an operator from changing things blindly, and where the CLI's evidence stops and a database-level decision begins.
Summary
- Core Idea: The Linux CLI gives an operator a second, independent view of a Postgres cluster - the operating system's view - that must be correlated with, not substituted for, the database's own view.
- Why It Matters: Symptoms that look database-specific, like a query storm, are often actually host-level exhaustion of disk, memory, or file descriptors wearing a Postgres costume.
- Key Concepts: three-layer correlation, triage doctrine, incident bundle, safe-stop versus kill, wait events.
- When to Use: Reach for this model the moment a system feels slow or unresponsive and you don't yet know whether the cause lives in the OS, the Postgres process, or the workload itself.
- Limitations / Trade-offs: CLI evidence tells you where to look, it rarely tells you the fix by itself, and acting on a single command's output without corroboration is how bad incidents get worse.
- Related Topics: IO and memory tooling, incident triage, disk-full recovery.
Foundations
Every triage session on a Postgres host is really working across three layers at once, even when it doesn't feel that way.
The first layer is the operating system: disk space, memory pressure, and network connections, all visible through tools like df, free, and ss that know nothing about SQL.
The second layer is the process: the actual postgres workers running on that host, visible through ps, pgrep, and systemctl, which know a process exists but not what query it's running.
The third layer is database state: pg_stat_activity, pg_locks, and the server logs, which know exactly what SQL is running but nothing about the host's free memory.
A useful analogy is a doctor reading vital signs, an X-ray, and a patient interview separately - each layer is accurate on its own terms, and a diagnosis only holds once all three agree.
The core discipline this page teaches is correlation: a disk-full symptom at the OS layer and a PANIC: could not write to file in the Postgres log are the same event seen from two layers, not two separate problems.
Learning to move fluently between these three layers, rather than living in just one of them, is what separates fast triage from guesswork.
Mechanics & Interactions
The reason CLI triage has a strict order - OS first, then process, then database - is that each layer's tools answer a different time window and a different kind of question.
OS-level commands like vmstat and iostat answer "what is happening right now on this host," independent of which database or query caused it.
Process-level commands like ps and systemctl status answer "is the Postgres process itself healthy," which matters because a restart loop can masquerade as a query performance problem.
Database-level views like pg_stat_activity answer "what is Postgres doing right now," including wait events that explain why a backend is slow rather than just that it's slow.
The triage doctrine that ties these together is simple: measure at all three layers before changing any configuration, restarting any service, or killing any process, because acting on one layer's evidence alone routinely misdiagnoses the other two.
This is also why an incident bundle script exists - capturing df, free, vmstat, and the relevant pg_stat_activity/pg_locks snapshots together, in one pass, before anything about the system changes.
# Minimal three-layer snapshot, taken together, not sequentially over minutes
df -h "$PGDATA"; free -h
ps -eo pid,rss,cmd --sort=-rss | grep postgres | head -5
psql "$DATABASE_URL" -c "SELECT wait_event_type, count(*) FROM pg_stat_activity GROUP BY 1;"Captured together, these three commands tell a coherent story; captured five minutes apart during a live incident, they can each describe a different moment.
Advanced Considerations & Applications
The doctrine of "measure before you touch" becomes harder to hold to under real incident pressure, which is exactly when it matters most.
A common failure mode is an operator seeing high CPU or IO and reaching straight for a GUC change or a failover, when the three-layer read would have shown a transient checkpoint spike that was already resolving on its own.
Escalation to failover is a database-level decision that CLI evidence can support but should rarely trigger alone - hardware failure, corruption, or an unrecoverable disk are the kinds of OS-layer findings that actually justify it.
The distinction between a graceful stop and a kill is where CLI mechanics meet real risk: systemctl stop lets Postgres complete its checkpoint, while kill -9 on the postmaster risks an incomplete checkpoint and a longer, riskier recovery on next start.
Access control follows the same layering logic - read-only OS and SQL commands are reasonable for any engineer to run, while anything that stops a process or changes a GUC belongs behind an on-call or buddy-system gate.
As infrastructure shifts toward containers, the same three-layer model still applies inside a kubectl exec session, it just replaces systemctl with the container runtime's own process visibility.
| Approach | Strength | Weakness | Best Fit |
|---|---|---|---|
Single-layer read (only pg_stat_activity, say) | Fast, requires no host access | Misses host-level root causes entirely, like disk or memory exhaustion | Quick sanity check when you already trust host health |
| Full three-layer correlation | Catches root causes a single layer would miss | Takes longer and requires broader access | Any real incident, or any symptom you can't explain from one layer alone |
| Act first, measure after | Feels fast under pressure | Frequently misdiagnoses transient spikes as real problems and causes unnecessary failovers | Never - included here only as the failure mode this page argues against |
Common Misconceptions
- "A query storm always means a database problem" - it often means the host ran out of memory or disk first, and Postgres is just the process most visibly affected.
- "
kill -9is a fast, safe way to restart a stuck Postgres" - it risks an incomplete checkpoint, turning a slow restart into a longer, riskier recovery. - "
pg_stat_activityalone tells you if the host is healthy" - it only knows about SQL-level state, and says nothing about disk space, memory pressure, or the process's own restart history. - "High CPU always means a query needs tuning" - checkpoint activity, autovacuum, and even host-level noisy neighbors can produce the same CPU signature as a bad query plan.
- "You need exotic tools to triage a Postgres host" -
df,free,vmstat, andpsare present on nearly every Linux image and cover most first-pass triage without any extra install.
FAQs
What are the "three layers" this page keeps referring to?
The operating system (disk, memory, network), the Postgres process itself (running, restarting, or crashed), and the database's own internal state (queries, locks, wait events).
Why does the order OS-then-process-then-database matter?
- Host exhaustion can produce symptoms that look database-specific.
- Checking the OS layer first prevents misdiagnosing a host problem as a query problem.
- Each layer answers a different time window, so checking them in isolation gives an incomplete picture.
Why capture an incident bundle instead of running commands one at a time?
Because system state changes minute to minute during an incident, and commands run five minutes apart can each be describing a different moment rather than one coherent story.
What's the real difference between a graceful stop and `kill -9`?
A graceful stop lets Postgres complete its current checkpoint before shutting down, while kill -9 interrupts that process and risks a longer, riskier recovery on the next start.
When does a CLI finding actually justify a failover?
When it points to something a restart can't fix, like hardware failure, filesystem corruption, or an unrecoverable disk - not for an ordinary lock storm or transient IO spike.
Should every engineer have write access to run these commands on production?
Read-only OS and SQL commands are reasonable for most engineers, while anything that stops a process or changes configuration should require an on-call role or a buddy.
Does this model still apply inside a Kubernetes pod?
Yes - kubectl exec gets you into the same three layers, it just substitutes the container runtime's process visibility for systemctl.
Why does high CPU not automatically mean a bad query?
Checkpoint writes, autovacuum, and host-level contention can all produce a similar CPU signature, which is exactly why single-layer conclusions are unreliable.
What tools does a minimal Postgres host actually need for triage?
df, free, vmstat, and ps cover most first-pass triage and are present on nearly every Linux image without extra installation.
Is it ever acceptable to skip the "measure first" step during a live incident?
The pressure to skip it is exactly when the risk of misdiagnosis is highest, which is why the doctrine exists as a default rather than a suggestion.
How does a disk-full symptom show up differently at each layer?
At the OS layer it's df reporting 100% usage, at the process layer it may show as a restart loop, and at the database layer it surfaces as a PANIC in the server log.
What's the biggest single mistake this page argues against?
Acting - restarting, killing a process, or failing over - on evidence from only one layer, before correlating it against the other two.
Related
- Linux CLI for DBAs Basics - the ten concrete commands this page's model organizes.
- IO & Memory Tools - the OS-layer toolset in depth.
- Incident Triage Skill - an agent skill built on this same triage doctrine.
- Disk Full on PGDATA - the disk-full scenario this page's three-layer example draws from.
- Lock Storm Triage - the database-layer half of a lock-storm investigation.
Stack versions: This page was written for PostgreSQL 18.4 running on Linux (systemd, Debian and RHEL layouts); it is otherwise conceptual and not tied to a specific tool version.