Migrate Postgres RDS from AWS to Philly Cluster

[TineoC]

Description

The goal of this task is to migrate the final AWS-hosted component—Postgres—to the Philly Cluster. This involves deploying a new pgcluster with pgvector features enabled and performing a data migration from the current AWS RDS instance.

Scope of Work

• Infrastructure Setup: Deploy pgcluster in both cfp-sandbox and cfp-live environments.
• Feature Enablement: Enable pgvector features on the new clusters.
• Data Migration: Migrate existing data from AWS RDS to Philly Cluster.
• Traffic Shifting: Implement a Blue-Green deployment to route a percentage of requests to the new pgcluster to avoid disruption.
• Validation: Monitor performance and parity before 100% cutover.

Strategy: Blue-Green Deployment

To avoid disruption in the dev and live environments, we will shift a percentage of requests from the current app to RDS and another to the new pgcluster.

Related Pull Requests

• PR # (To be linked): Infrastructure deployment for pgcluster
• PR # (To be linked): Configuration changes for pgvector and traffic splitting
• PR # (To be linked): Final cutover and RDS cleanup

Staheholders

• @taichan03
• @sahilds1
• @themightychris

[themightychris]

we've been wanting to deploy CNPG to the clusters and start time that to host databases for projects, instead of the usual creating a statefulset within each project. Anyone game to try helping stand that up as part of this?

[TineoC]

Yea I would first learn how to deploy PG vector on k8s first and then would see how this could be solved for the sandbox environment.

[themightychris]

The process will be very different between using CNPG and not using CNPG