-
Notifications
You must be signed in to change notification settings - Fork 8
Description
Describe the bug
When using the wrapper with node-postgres (pg) driver and Aurora PostgreSQL, triggering a failover causes the Node.js process to crash due to two separate issues:
-
Unhandled
errorevent on internalpg.Pool:AwsPgInternalPoolClientcreates apg.Poolwithout attaching anerrorevent handler. During Aurora failover, idle connections in the pool are terminated by the server, which emits
errorevents on the pool. In Node.js, anEventEmitterthat emitserrorwithout a listener throws an uncaught exception and crashes the process. -
TypeError: this.targetClient.client.release is not a function: After a successful failover,AwsPGPooledConnection.release()callsthis.targetClient.client.release(), but the new client provided after failover may not have a
release()method (e.g., it's a direct connection rather than a pooled connection). This causes aTypeErrorthat crashes the process.
Expected Behavior
After Aurora failover, the wrapper should gracefully handle the connection switch without crashing the process. Specifically:
- Idle connection errors on the internal
pg.Poolshould be handled (swallowed or logged), allowing the failover plugin to handle reconnection. AwsPGPooledConnection.release()should check ifrelease()exists on the underlying client before calling it.
What plugins are used? What other connection properties were set?
auroraConnectionTracker,failover2,efm2
Current Behavior
Crash 1 - Unhandled pool error (process exits immediately):
node:events:497
throw er; // Unhandled 'error' event
^
Error: Connection terminated unexpectedly
at BoundPool.<anonymous> (node:events:497:17)
...
Emitted 'error' event on BoundPool instance at:
...
Crash 2 - release() TypeError (occurs after failover succeeds):
TypeError: this.targetClient.client.release is not a function
at AwsPGPooledConnection.release (aws-advanced-nodejs-wrapper/dist/pg/lib/client.js:...)
...
Both crashes cause the Node.js process to terminate, preventing the application from recovering after failover.
Reproduction Steps
- Set up an Aurora PostgreSQL cluster with at least 2 instances (writer + reader)
- Connect using
AwsPgPoolClientwithnode-postgresdriver:
import { AwsPoolConfig } from 'aws-advanced-nodejs-wrapper';
import { AwsPgPoolClient } from 'aws-advanced-nodejs-wrapper/pg';
const poolConfig = new AwsPoolConfig({ maxConnections: 10 });
const pool = new AwsPgPoolClient(
{
host: '<aurora-cluster-endpoint>',
user: '<user>',
password: '<password>',
database: '<database>',
plugins: 'auroraConnectionTracker,failover2,efm2',
},
poolConfig,
);
// Use with any ORM that calls release() in finally block (e.g., drizzle-orm)
// Keep the application running with active connections
// Trigger Aurora failover via AWS Console or CLI:
// aws rds failover-db-cluster --db-cluster-identifier <cluster-id>- Trigger a failover using
aws rds failover-db-cluster - The process crashes with one of the two errors above
Possible Solution
Fix 1 - Add error handler to internal pg.Pool in AwsPgInternalPoolClient (dist/pg/lib/icp/pg_internal_pool_client.js):
constructor(props) {
this.targetPool = new pkgPg.Pool(props);
// Handle idle client errors to prevent process crash during Aurora failover.
this.targetPool.on('error', (err) => {
// Intentionally swallowed - idle connection errors are expected during failover.
});
}Fix 2 - Add safety check in AwsPGPooledConnection.release() (dist/pg/lib/client.js):
async release() {
return this.pluginManager.execute(
this.pluginService.getCurrentHostInfo(),
this.properties,
'release',
async () => {
if (!this.targetClient) {
throw new UndefinedClientError();
}
this.pluginService.removeErrorListener(this.targetClient);
if (typeof this.targetClient.client?.release === 'function') {
return await this.targetClient.client.release();
}
},
null,
);
}Additional Information/Context
We discovered these issues while integrating the wrapper with drizzle-orm, which calls release() in a finally block after each transaction. However, these bugs are not specific to drizzle-orm - any application or ORM that uses pooled
connections and calls release() after queries will be affected.
We are currently working around both issues using pnpm patch.
The AWS Advanced NodeJs Wrapper version used
2.1.0
Node version used
24.13.0
Operating System and version
Amazon Linux 2 (ECS on Fargate)