pg-pool: enable TCP keepalive and retry once on connection-drop errors - #532
Open
16francej wants to merge 1 commit into
Open
pg-pool: enable TCP keepalive and retry once on connection-drop errors#53216francej wants to merge 1 commit into
16francej wants to merge 1 commit into
Conversation
Managed Postgres poolers/proxies can drop idle or in-flight connections (rebalancing, failover, idle-connection reaping). When that happens mid-query, pg surfaces 'Connection terminated unexpectedly' / 'Connection terminated', ECONNRESET/EPIPE socket errors, or 57P01 (terminating connection due to administrator command), and the request fails with a 500 even though the statement never ran anywhere. Repro: kill the Postgres backend mid-query; the caller gets a 500 with 'Connection terminated unexpectedly'. Fix: - set keepAlive: true on the pg.Pool so dead peers are detected and idle connections are less likely to be reaped by middleboxes - in the pool's simple-query path, retry once on a fresh client when the failure is a connection-drop error (new isConnectionDropError helper); anything else still throws immediately The retry is deliberately single-shot and limited to the pool-level query() path, where a checkout from the pool yields a fresh client.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Managed Postgres poolers/proxies can drop idle or in-flight connections (rebalancing, failover, idle-connection reaping); flaky networks do the same. When a connection dies mid-query,
pgsurfaces:Connection terminated unexpectedly/Connection terminatedECONNRESET/EPIPEsocket errors57P01(terminating connection due to administrator command)and the request fails with a 500 even though the statement never ran anywhere.
Repro: kill the Postgres backend mid-query → the caller gets a 500 with
Connection terminated unexpectedly.Fix
keepAlive: trueon thepg.Poolso dead peers are detected promptly and idle connections are less likely to be reaped by middleboxes.query()), retry once on a fresh client when the failure is a connection-drop error, via a new exportedisConnectionDropError(err)helper. Any other error still throws immediately.The retry is deliberately single-shot and limited to the pool-level
query()path, where each attempt checks out a fresh client from the pool (the dropped client is discarded bypg), so the retry never reuses a dead connection. Transactions (withPgTransaction) are untouched — replaying those is not reuse-safe.Tests
isConnectionDropError(positive shapes + non-drop errors + non-object inputs).pg-pool,pg-ca-options, andpersistence-init-retrysuites pass;tsc --noEmitclean.Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.