Skip to content

Process crashes after Aurora failover with pg (node-postgres) driver - two issues #620

@azihsoyn

Description

@azihsoyn

Describe the bug

When using the wrapper with node-postgres (pg) driver and Aurora PostgreSQL, triggering a failover causes the Node.js process to crash due to two separate issues:

  1. Unhandled error event on internal pg.Pool: AwsPgInternalPoolClient creates a pg.Pool without attaching an error event handler. During Aurora failover, idle connections in the pool are terminated by the server, which emits
    error events on the pool. In Node.js, an EventEmitter that emits error without a listener throws an uncaught exception and crashes the process.

  2. TypeError: this.targetClient.client.release is not a function: After a successful failover, AwsPGPooledConnection.release() calls this.targetClient.client.release(), but the new client provided after failover may not have a
    release() method (e.g., it's a direct connection rather than a pooled connection). This causes a TypeError that crashes the process.

Expected Behavior

After Aurora failover, the wrapper should gracefully handle the connection switch without crashing the process. Specifically:

  1. Idle connection errors on the internal pg.Pool should be handled (swallowed or logged), allowing the failover plugin to handle reconnection.
  2. AwsPGPooledConnection.release() should check if release() exists on the underlying client before calling it.

What plugins are used? What other connection properties were set?

auroraConnectionTracker,failover2,efm2

Current Behavior

Crash 1 - Unhandled pool error (process exits immediately):

node:events:497
      throw er; // Unhandled 'error' event
      ^

Error: Connection terminated unexpectedly
    at BoundPool.<anonymous> (node:events:497:17)
    ...
Emitted 'error' event on BoundPool instance at:
    ...

Crash 2 - release() TypeError (occurs after failover succeeds):

TypeError: this.targetClient.client.release is not a function
    at AwsPGPooledConnection.release (aws-advanced-nodejs-wrapper/dist/pg/lib/client.js:...)
    ...

Both crashes cause the Node.js process to terminate, preventing the application from recovering after failover.

Reproduction Steps

  1. Set up an Aurora PostgreSQL cluster with at least 2 instances (writer + reader)
  2. Connect using AwsPgPoolClient with node-postgres driver:
import { AwsPoolConfig } from 'aws-advanced-nodejs-wrapper';
import { AwsPgPoolClient } from 'aws-advanced-nodejs-wrapper/pg';

const poolConfig = new AwsPoolConfig({ maxConnections: 10 });
const pool = new AwsPgPoolClient(
  {
    host: '<aurora-cluster-endpoint>',
    user: '<user>',
    password: '<password>',
    database: '<database>',
    plugins: 'auroraConnectionTracker,failover2,efm2',
  },
  poolConfig,
);

// Use with any ORM that calls release() in finally block (e.g., drizzle-orm)
// Keep the application running with active connections
// Trigger Aurora failover via AWS Console or CLI:
// aws rds failover-db-cluster --db-cluster-identifier <cluster-id>
  1. Trigger a failover using aws rds failover-db-cluster
  2. The process crashes with one of the two errors above

Possible Solution

Fix 1 - Add error handler to internal pg.Pool in AwsPgInternalPoolClient (dist/pg/lib/icp/pg_internal_pool_client.js):

constructor(props) {
  this.targetPool = new pkgPg.Pool(props);
  // Handle idle client errors to prevent process crash during Aurora failover.
  this.targetPool.on('error', (err) => {
    // Intentionally swallowed - idle connection errors are expected during failover.
  });
}

Fix 2 - Add safety check in AwsPGPooledConnection.release() (dist/pg/lib/client.js):

async release() {
  return this.pluginManager.execute(
    this.pluginService.getCurrentHostInfo(),
    this.properties,
    'release',
    async () => {
      if (!this.targetClient) {
        throw new UndefinedClientError();
      }
      this.pluginService.removeErrorListener(this.targetClient);
      if (typeof this.targetClient.client?.release === 'function') {
        return await this.targetClient.client.release();
      }
    },
    null,
  );
}

Additional Information/Context

We discovered these issues while integrating the wrapper with drizzle-orm, which calls release() in a finally block after each transaction. However, these bugs are not specific to drizzle-orm - any application or ORM that uses pooled
connections and calls release() after queries will be affected.

We are currently working around both issues using pnpm patch.

The AWS Advanced NodeJs Wrapper version used

2.1.0

Node version used

24.13.0

Operating System and version

Amazon Linux 2 (ECS on Fargate)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions