close

DEV Community

Cover image for What happens when a PostgreSQL backend crashes?
Franck Pachot
Franck Pachot

Posted on

What happens when a PostgreSQL backend crashes?

Your connection to PostgreSQL is handled by a dedicated backend process. If that process crashes, you might think only your session is affected. However, because backend processes share memory, PostgreSQL assumes the shared state may have been corrupted and immediately terminates all other connections. Recovery is then performed before new connections are accepted.

Here is a short test to demonstrate it and to see what is visible in the PostgreSQL server log and what is received in the application. I run an ephemeral Docker container, show the log file, start ten pgbench clients, and kill my own session:


docker exec -it $(
docker run -d --rm -e POSTGRES_PASSWORD=xxx postgres -c logging_collector=on
sleep 3
) psql -U postgres <<'SQL'

\! sleep 1 ; echo '\n 🐘 Start pgbench in the background...\n' ; sleep 1

\! pgbench -i -U postgres postgres 
\! pgbench -c 5 -T 60 -P 1 -U postgres postgres & sleep 10

\! sleep 1 ; echo '\n 🐘 Tail the logfile...\n' ; sleep 1

select current_setting('data_directory')||'/'||pg_current_logfile() log
\gset
\setenv log :log
\! tail -f "$log" | sed -e "s/^/πŸ“œ /" &

\! sleep 1 ; echo '\n 🐘 Crash the current process...\n' ; sleep 1

select pg_backend_pid() as pid
\gset 
\setenv pid :pid
\! kill -9 $pid

\! sleep 30

\q

SQL

Enter fullscreen mode Exit fullscreen mode

Here is the output. I've deliberately set it to run in an ephemeral Docker container because I don't want you to do the same on an existing database.

The container started, psql connected, and pgbench started:


psql (18.4 (Debian 18.4-1.pgdg13+1))
Type "help" for help.

postgres=#
postgres=# \! sleep 1 ; echo '\n 🐘 Start pgbench in the background...\n' ; sleep 1

 🐘 Start pgbench in the background...

postgres=#
postgres=# \! pgbench -i -U postgres postgres
dropping old tables...
NOTICE:  table "pgbench_accounts" does not exist, skipping
NOTICE:  table "pgbench_branches" does not exist, skipping
NOTICE:  table "pgbench_history" does not exist, skipping
NOTICE:  table "pgbench_tellers" does not exist, skipping
creating tables...
generating data (client-side)...
vacuuming...
creating primary keys...
done in 0.21 s (drop tables 0.00 s, create tables 0.01 s, client-side generate 0.14 s, vacuum 0.03 s, primary keys 0.03 s).
postgres=# \! pgbench -c 5 -T 60 -P 1 -U postgres postgres & sleep 10
pgbench (18.4 (Debian 18.4-1.pgdg13+1))
starting vacuum...end.
progress: 1.0 s, 504.0 tps, lat 9.719 ms stddev 7.949, 0 failed
progress: 2.0 s, 338.0 tps, lat 14.826 ms stddev 9.621, 0 failed
progress: 3.0 s, 348.0 tps, lat 14.335 ms stddev 9.640, 0 failed
progress: 4.0 s, 346.0 tps, lat 14.456 ms stddev 9.288, 0 failed
progress: 5.0 s, 357.0 tps, lat 13.976 ms stddev 9.178, 0 failed
progress: 6.0 s, 349.0 tps, lat 14.371 ms stddev 10.674, 0 failed
progress: 7.0 s, 357.0 tps, lat 14.010 ms stddev 9.035, 0 failed
progress: 8.0 s, 350.0 tps, lat 14.235 ms stddev 9.087, 0 failed
progress: 9.0 s, 341.0 tps, lat 14.788 ms stddev 9.204, 0 failed
progress: 10.0 s, 277.0 tps, lat 18.002 ms stddev 9.095, 0 failed

Enter fullscreen mode Exit fullscreen mode

Displaying the PostgreSQL log file while pgbench is still running:


postgres=# \! sleep 1 ; echo '\n 🐘 Tail the logfile...\n' ; sleep 1

 🐘 Tail the logfile...

progress: 11.0 s, 353.0 tps, lat 14.131 ms stddev 10.032, 0 failed
postgres=#
postgres=# select current_setting('data_directory')||'/'||pg_current_logfile() log
postgres-# \gset
postgres=# \setenv log :log
postgres=# \! tail -f "$log" | sed -e "s/^/πŸ“œ /" &
postgres=#
πŸ“œ 2026-07-06 17:06:18.098 UTC [1] LOG:  starting PostgreSQL 18.4 (Debian 18.4-1.pgdg13+1) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 14.2.0-19) 14.2.0, 64-bit
πŸ“œ 2026-07-06 17:06:18.099 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
πŸ“œ 2026-07-06 17:06:18.099 UTC [1] LOG:  listening on IPv6 address "::", port 5432
πŸ“œ 2026-07-06 17:06:18.101 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
πŸ“œ 2026-07-06 17:06:18.106 UTC [75] LOG:  database system was shut down at 2026-07-06 17:06:17 UTC
πŸ“œ 2026-07-06 17:06:18.109 UTC [1] LOG:  database system is ready to accept connections
progress: 12.0 s, 332.0 tps, lat 15.064 ms stddev 10.867, 0 failed

Enter fullscreen mode Exit fullscreen mode

Crashing the current backend with kill -9 while pgbench is still running and the log file is being tailed:


postgres=# \! sleep 1 ; echo '\n 🐘 Crash the current process...\n' ; sleep 1


 🐘 Crash the current process...

progress: 13.0 s, 357.0 tps, lat 13.976 ms stddev 9.172, 0 failed
postgres=#
postgres=# select pg_backend_pid() as pid
postgres-# \gset
postgres=# \setenv pid :pid
postgres=# \! kill -9 $pid
postgres=#
postgres=# \! sleep 30
πŸ“œ 2026-07-06 17:06:36.241 UTC [1] LOG:  client backend (PID 85) was terminated by signal 9: Killed
πŸ“œ 2026-07-06 17:06:36.241 UTC [1] DETAIL:  Failed process was running: select pg_backend_pid() as pid
πŸ“œ 2026-07-06 17:06:36.241 UTC [1] LOG:  terminating any other active server processes
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
pgbench: error: client 0 aborted in command 10 (SQL) of script 0; perhaps the backend died while processing
WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
pgbench: error: client 2 aborted in command 8 (SQL) of script 0; perhaps the backend died while processing
pgbench: error: client 3 aborted in command 8 (SQL) of script 0; perhaps the backend died while processing
πŸ“œ 2026-07-06 17:06:36.243 UTC [1] LOG:  all server processes terminated; reinitializing
pgbench: error: client 4 aborted in command 8 (SQL) of script 0; perhaps the backend died while processing
pgbench: error: client 1 aborted in command 7 (SQL) of script 0; perhaps the backend died while processing
transaction type: <builtin: TPC-B (sort of)>
scaling factor: 1
query mode: simple
number of clients: 5
number of threads: 1
maximum number of tries: 1
duration: 60 s
number of transactions actually processed: 4948
number of failed transactions: 0 (0.000%)
latency average = 14.106 ms
latency stddev = 9.783 ms
initial connection time = 6.526 ms
tps = 353.948496 (without initial connection time)
pgbench: error: Run was aborted; the above results are incomplete.
πŸ“œ 2026-07-06 17:06:36.250 UTC [117] LOG:  database system was interrupted; last known up at 2026-07-06 17:06:18 UTC
πŸ“œ 2026-07-06 17:06:36.390 UTC [117] LOG:  database system was not properly shut down; automatic recovery in progress
πŸ“œ 2026-07-06 17:06:36.392 UTC [117] LOG:  redo starts at 0/175F960
πŸ“œ 2026-07-06 17:06:36.433 UTC [117] LOG:  invalid record length at 0/2687F00: expected at least 24, got 0
πŸ“œ 2026-07-06 17:06:36.433 UTC [117] LOG:  redo done at 0/2687ED8 system usage: CPU: user: 0.03 s, system: 0.00 s, elapsed: 0.04 s
πŸ“œ 2026-07-06 17:06:36.435 UTC [118] LOG:  checkpoint starting: end-of-recovery immediate wait
πŸ“œ 2026-07-06 17:06:37.205 UTC [118] LOG:  checkpoint complete: wrote 2054 buffers (12.5%), wrote 3 SLRU buffers; 0 WAL file(s) added, 0 removed, 1 recycled; write=0.625 s, sync=0.138 s, total=0.771 s; sync files=51, longest=0.108 s, average=0.003 s; distance=15521 kB, estimate=15521 kB; lsn=0/2687F00, redo lsn=0/2687F00
πŸ“œ 2026-07-06 17:06:37.207 UTC [1] LOG:  database system is ready to accept connections


postgres=#
postgres=# \q

Enter fullscreen mode Exit fullscreen mode

This indicates that PostgreSQL intentionally terminated all other connections (as seen in the error messages from the five pgbench connections) when it detected a crash signal, and that it completed recovery before allowing new connections:

The logfile indicates the cause:

LOG:  client backend (PID 85) was terminated by signal 9: Killed
LOG:  terminating any other active server processes
Enter fullscreen mode Exit fullscreen mode

The application encountered the error:

WARNING:  terminating connection because of crash of another server process
DETAIL:  The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.
HINT:  In a moment you should be able to reconnect to the database and repeat your command.
Enter fullscreen mode Exit fullscreen mode

Unlike database engines that keep sessions separate, PostgreSQL backends share key memory areas like shared buffers, lock tables, and transaction status data. If a backend crashes unexpectedly, PostgreSQL cannot ensure these structures stay consistent. As a result, it adopts a cautious strategy: shutting down all backends and restarting from a reliable, known state.

This behavior is deliberate and is a fundamental safety feature of PostgreSQL. When a backend crashes, it is considered a possible sign of shared-memory corruption. To prevent serving inconsistent data, PostgreSQL shuts down all sessions, replays WAL during recovery, and only resumes accepting connections afterward.

Top comments (9)

Collapse
 
mads_hansen_27b33ebfee4c9 profile image
Mads Hansen

This is a useful demo because it shows a failure mode many app teams only learn during an incident.

The application-side lesson is worth spelling out: a backend crash should be treated differently from a normal connection error. If every session is terminated while Postgres restarts recovery, retry storms can make the incident noisier than the original crash.

I like pairing this kind of test with client behavior checks: exponential backoff, pool reset, transaction replay rules, and a clear distinction between idempotent reads and unsafe writes. The database recovery path is only half the story; the app recovery path decides how painful users feel it.

Collapse
 
franckpachot profile image
Franck Pachot

Thanks, yes, very good point. Connection pools should be configured to handle that.

Collapse
 
vinimabreu profile image
Vinicius Pereira

Worth adding where that SIGKILL usually comes from in production: almost nobody runs kill -9 by hand, the Linux OOM killer does it for them. A dedicated Postgres box with default memory overcommit and a few sessions spiking work_mem is enough, and the OOM killer likes to pick the fattest backend, which triggers exactly the full restart you demonstrated. The mitigations from the Postgres docs are vm.overcommit_memory=2 plus protecting the postmaster with oom_score_adj, so memory pressure surfaces as a local out-of-memory error (one query fails, transaction aborts) instead of a SIGKILL (every session dies). And on the application side, treat SQLSTATE 57P02 as retryable with backoff: the naive reconnect loop turns your own pool into a thundering herd that hits the database right in the middle of WAL replay.

Collapse
 
valentynkit profile image
Valentyn Kit

This is the shared-buffers process model showing its bill: one backend can scribble the shared segment, so Postgres can't trust any of it and takes the whole instance down. Curious how you'd contrast it with Oracle's PMON path, where a single dedicated-server kill gets cleaned up without dropping every other session.

Collapse
 
franckpachot profile image
Franck Pachot

I've never looked at the detail but Oracle has probably lots of safeguards when modifying shared structures. The multitenant architecture that powers Autonomous Database would not be possible without good isolation

Collapse
 
fabrice_chapuis profile image
Fabrice Chapuis

Fortunately, this mechanism does not come into play when using the function pg_terminate_backend()

Collapse
 
franckpachot profile image
Franck Pachot

Yes, great point. The cascading crash scenario only occurs with crash, like SIGKILL (this uncatchable kill -9 for example) and SIGQUIT (emergency termination sent by the postmaster to all backends). But, SIGTERM, as sent by pg_terminate_backend() is a graceful normal exit, and SIGINT (fast shutdown) is a controlled shutdown.

Collapse
 
_hm profile image
Hussein Mahdi

In a moment you should be able to reconnect β€” the most polite way a database has ever said I just nuked every session to protect your data. πŸ˜„ Great demo."

Some comments may only be visible to logged-in visitors. Sign in to view all comments.