[Feature Request]: Cache the BigQuery table definition instead of calling tables.get() from every worker

[PraveenAkunuru]

What would you like to happen?

When calling BigQuery's tables.get method, today, we are calling it from every Dataflow worker. This is resulting in customers hitting Quota issues. Please can we cache the Table definition and re-use for all workers instead of calling the method from every worker.

https://cloud.google.com/bigquery/docs/reference/rest/v2/tables/get

beam/sdks/python/apache_beam/io/gcp/bigquery.py

Lines 902 to 903 in 9a1e7bf

	table = bq.get_table(
	table_ref.projectId, table_ref.datasetId, table_ref.tableId)

beam/sdks/python/apache_beam/io/gcp/bigquery_tools.py

Line 565 in 9a1e7bf

response = self.client.tables.Get(request)

Issue Priority

Priority: 2 (default / most feature requests should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[tamannakakkar93]

.take-issue

[Strikerrx01]

.take-issue