Add support for spanner table options in import/export by n-d-joshi · Pull Request #3552 · GoogleCloudPlatform/DataflowTemplates

n-d-joshi · 2026-03-25T05:24:57Z

This PR adds:

Support for Import/Export of table options.
Adds an allowlist for the table options which currently only has fulltext_dictionary_table.

Syncing branch

gemini-code-assist · 2026-03-25T05:25:23Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the Spanner import/export functionality by enabling the handling of table options. This allows for a more complete and accurate representation of Spanner table definitions during data migration and schema synchronization processes, ensuring that critical table-level configurations are preserved.

Highlights

Spanner Table Options Support: Implemented comprehensive support for importing and exporting Spanner table options, ensuring that table-level configurations are preserved during data migration and schema synchronization.
Table Option Allowlist: Introduced a new TableOptionAllowlist to explicitly define and manage supported table options, with fulltext_dictionary_table as the initial entry.
DDL Conversion Enhancements: Updated AvroSchemaToDdlConverter and DdlToAvroSchemaConverter to correctly parse and generate DDL with table options from and to Avro schemas.
Information Schema Integration: Enhanced the InformationSchemaScanner to query and extract table options directly from Spanner's information_schema for both Google Standard SQL and PostgreSQL dialects.
DDL Representation Update: Modified the internal Table DDL representation to include tableOptions and updated the DDL pretty-printing logic to correctly incorporate these options.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

n-d-joshi · 2026-03-25T06:03:32Z

R: @darshan-sj

darshan-sj

The changes look good overall, I have left a few comments.

Please address these as well:

Please build the Import and export template with your code changes and run an export from Spanner database with your table option. Import the exported data into a new and empty Spanner database and make sure that the table option is created properly in the new database. Please document the results of the test here.
Please add integration test cases in ExportPipelineIT and ImportPipelineIT.

darshan-sj · 2026-03-26T05:17:36Z

+        String quoteChar =
+            dialect == Dialect.POSTGRESQL ? POSTGRESQL_LITERAL_QUOTE : GSQL_LITERAL_QUOTE;
+        options.add(
+            optionName + "=" + quoteChar + OPTION_STRING_ESCAPER.escape(optionValue) + quoteChar);


Why is NameUtils' quoteIdentifier or quoteLiteral not used here? - https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v1/src/main/java/com/google/cloud/teleport/spanner/common/NameUtils.java

Done. Changed it to NameUtils.

darshan-sj · 2026-03-26T05:23:09Z

+  @VisibleForTesting
+  Statement listTableOptionsSQL() {
+    switch (dialect) {
+      case GOOGLE_STANDARD_SQL:


Are all these columns and tables available in every production databases? Please confirm.

Please get these 2 queries reviewed and LGTMed by one of your team members.

Yes this should be available in all database as long as we expose the table_options table.

darshan-sj · 2026-03-26T05:32:05Z

                + " ) PRIMARY KEY (`id` ASC)"));
  }

+  @Test


I see some escaping is being done for the Table options in InformationSchemaScanner class. Please add test cases where escaping is exercised and escaping is not exercised.

I am following the same convention as the other tests in creating the table:

"CREATE TABLE `CustomDictionary` (" + " `Key` STRING(MAX) NOT NULL," + " `Value` ARRAY<STRING(MAX)> NOT NULL," + " ) PRIMARY KEY (`Key` ASC),\nOPTIONS (fulltext_dictionary_table=true)"));

This is not escaped?
Did I misunderstand the comment?

codecov · 2026-03-26T05:55:32Z

Codecov Report

❌ Patch coverage is 81.81818% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 52.18%. Comparing base (b8c4776) to head (a8a9479).
⚠️ Report is 92 commits behind head on main.

Files with missing lines	Patch %	Lines
...oud/teleport/spanner/ddl/TableOptionAllowlist.java	0.00%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3552      +/-   ##
============================================
- Coverage     52.19%   52.18%   -0.02%     
+ Complexity     6048     5649     -399     
============================================
  Files          1040     1041       +1     
  Lines         62985    63077      +92     
  Branches       6901     6917      +16     
============================================
+ Hits          32877    32917      +40     
- Misses        27882    27927      +45     
- Partials       2226     2233       +7

Components	Coverage Δ
spanner-templates	`72.23% <81.81%> (+0.04%)`	⬆️
spanner-import-export	`69.05% <81.81%> (+0.14%)`	⬆️
spanner-live-forward-migration	`80.47% <ø> (-0.02%)`	⬇️
spanner-live-reverse-replication	`77.87% <ø> (-0.02%)`	⬇️
spanner-bulk-migration	`89.18% <ø> (-0.01%)`	⬇️
gcs-spanner-dv	`85.32% <ø> (-0.03%)`	⬇️

Files with missing lines	Coverage Δ
...oud/teleport/spanner/AvroSchemaToDdlConverter.java	`86.80% <100.00%> (+0.02%)`	⬆️
...va/com/google/cloud/teleport/spanner/AvroUtil.java	`93.75% <ø> (ø)`
...oud/teleport/spanner/DdlToAvroSchemaConverter.java	`97.53% <100.00%> (+0.01%)`	⬆️
...a/com/google/cloud/teleport/spanner/ddl/Table.java	`89.76% <100.00%> (+0.41%)`	⬆️
...oud/teleport/spanner/ddl/TableOptionAllowlist.java	`0.00% <0.00%> (ø)`

... and 13 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

darshan-sj · 2026-04-10T06:01:06Z

+        options.add(
+            optionName + "=" + quoteChar + OPTION_STRING_ESCAPER.escape(optionValue) + quoteChar);
+      } else if (optionType.equalsIgnoreCase("character varying")) {
+        options.add(optionName + "='" + OPTION_STRING_ESCAPER.escape(optionValue) + "'");


I'm referring to this option_string_escaper string in the comment in AvroSchemaToDdlConverterTest.java‎ . Does the test case added there test different edge cases around this escaping?

n-d-joshi added 4 commits January 26, 2025 22:40

Merge pull request #1 from GoogleCloudPlatform/main

f3298e6

Syncing branch

Merge branch 'GoogleCloudPlatform:main' into main

1fbdbb8

Add support for spanner table options in import/export.

5f67b5e

Merge branch 'GoogleCloudPlatform:main' into main

fb34a87

n-d-joshi requested a review from a team as a code owner March 25, 2026 05:24

n-d-joshi requested review from manitgupta and rohitwali March 25, 2026 05:24

pull-request-size bot added the size/L label Mar 25, 2026

darshan-sj reviewed Mar 26, 2026

View reviewed changes

darshan-sj added the improvement Making existing code better label Mar 26, 2026

Addressing review comments and adding tests

a8a9479

darshan-sj reviewed Apr 10, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for spanner table options in import/export#3552

Add support for spanner table options in import/export#3552
n-d-joshi wants to merge 5 commits intoGoogleCloudPlatform:mainfrom
n-d-joshi:table-options

n-d-joshi commented Mar 25, 2026

Uh oh!

gemini-code-assist bot commented Mar 25, 2026

Uh oh!

n-d-joshi commented Mar 25, 2026

Uh oh!

darshan-sj left a comment

Uh oh!

darshan-sj Mar 26, 2026

Uh oh!

n-d-joshi Mar 26, 2026

Uh oh!

darshan-sj Mar 26, 2026

Uh oh!

xinbei-liu Apr 10, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

darshan-sj Mar 26, 2026

Uh oh!

n-d-joshi Mar 26, 2026

Uh oh!

codecov bot commented Mar 26, 2026 •

edited

Loading

Uh oh!

darshan-sj Apr 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

n-d-joshi commented Mar 25, 2026

Uh oh!

gemini-code-assist bot commented Mar 25, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

n-d-joshi commented Mar 25, 2026

Uh oh!

darshan-sj left a comment

Choose a reason for hiding this comment

Uh oh!

darshan-sj Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

n-d-joshi Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

darshan-sj Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

xinbei-liu Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

darshan-sj Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

n-d-joshi Mar 26, 2026

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

darshan-sj Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov bot commented Mar 26, 2026 •

edited

Loading