Skip to content

Select fileds rathert * to read snapshot data in case schema changes.#4273

Open
loserwang1024 wants to merge 1 commit intoapache:masterfrom
loserwang1024:fix-postgres
Open

Select fileds rathert * to read snapshot data in case schema changes.#4273
loserwang1024 wants to merge 1 commit intoapache:masterfrom
loserwang1024:fix-postgres

Conversation

@loserwang1024
Copy link
Contributor

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates Postgres snapshot split scanning to select an explicit column list (instead of *) so snapshot reads are more robust when the underlying table schema changes (FLINK-39071).

Changes:

  • Extend PostgresQueryUtils.buildSplitScanQuery to optionally accept a column projection list.
  • Update PostgresScanFetchTask to build and pass a quoted column list from the Debezium Table schema for snapshot queries.
  • Simplify the internal query builder path by removing unused split-query branching/limit handling.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
.../utils/PostgresQueryUtils.java Adds a column-projection overload for split scan queries and simplifies select query construction.
.../fetch/PostgresScanFetchTask.java Builds an explicit quoted column list from the schema and uses it for snapshot split SELECT statements.
Comments suppressed due to low confidence (1)

flink-cdc-connect/flink-cdc-source-connectors/flink-connector-postgres-cdc/src/main/java/org/apache/flink/cdc/connectors/postgres/source/utils/PostgresQueryUtils.java:321

  • buildSelectWithRowLimits no longer applies any LIMIT clause (the limit parameter and logic were removed), so the method name is now misleading and the remaining buildSelectWithBoundaryRowLimits/get*Projection helpers appear to be unused. Consider either renaming this method to reflect its current behavior and removing the now-dead boundary/PK projection helpers, or reintroducing a limit parameter if limiting is still required elsewhere.
    private static String buildSelectWithRowLimits(
            TableId tableId,
            String projection,
            Optional<String> condition,
            Optional<String> orderBy) {
        final StringBuilder sql = new StringBuilder("SELECT ");

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants