[Failing Test]: Spanner change stream tests failing on latest GCP BOM

[Abacn]

What happened?

see https://github.com/apache/beam/runs/48025477476 https://github.com/apache/beam/runs/48411385440

error message:

no result returned

java.lang.AssertionError: ParDo(ModsToString)/ParMultiDo(ModsToString).output: 
Expected: iterable with items [...] in any order
but: no item matches: "..." in []
	at org.apache.beam.sdk.testing.PAssert$PAssertionSite.capture(PAssert.java:176)
	at org.apache.beam.sdk.testing.PAssert.that(PAssert.java:461)
	at org.apache.beam.sdk.testing.PAssert.that(PAssert.java:453)
	at org.apache.beam.sdk.io.gcp.spanner.changestreams.it.SpannerChangeStreamIT.testReadSpannerChangeStreamImpl(SpannerChangeStreamIT.java:162)

java.lang.AssertionError: 
Expected: (stack trace contains string 'SpannerException' and stack trace contains string 'NOT_FOUND: Session not found')
     but: stack trace contains string 'NOT_FOUND: Session not found' was <org.apache.beam.sdk.Pipeline$PipelineExecutionException: com.google.cloud.spanner.SpannerException: INVALID_ARGUMENT: com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Multiplexed sessions may not be deleted>
Stacktrace was: org.apache.beam.sdk.Pipeline$PipelineExecutionException: com.google.cloud.spanner.SpannerException: INVALID_ARGUMENT: com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Multiplexed sessions may not be deleted

and other errors.

this is blocking GCP BOM upgrade #35854.

Issue Failure

Failure: Test is flaky

Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[changliiu]

According to the error message, the issue is around MultiplexedSessionDatabaseClient in the cloud spanner java client lib.

[changliiu]

The multiplex session issue is not relevant to the recent change stream connector v2 implementation.

[changliiu]

Example error message:

Expected: (stack trace contains string 'SpannerException' and stack trace contains string 'NOT_FOUND: Session not found')
     but: stack trace contains string 'NOT_FOUND: Session not found' was <org.apache.beam.sdk.Pipeline$PipelineExecutionException: com.google.cloud.spanner.SpannerException: INVALID_ARGUMENT: com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Multiplexed sessions may not be deleted>
Stacktrace was: org.apache.beam.sdk.Pipeline$PipelineExecutionException: com.google.cloud.spanner.SpannerException: INVALID_ARGUMENT: com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Multiplexed sessions may not be deleted

[Abacn]

To reproduce locally

• checkout Update BOM to 26.65.0 #35854 (git fetch upstream pull/35854/head:pr-35854 && git switch pr-35854)

• Run

./gradlew :sdks:java:io:google-cloud-platform:integrationTest \
--tests org.apache.beam.sdk.io.gcp.spanner.changestreams.it.SpannerChangeStreamIT \
-DintegrationTestPipelineOptions='["--project=<your project>","--instanceId=<spanner instance>"]'

It seems not fully deterministic. Assertion error happens if I run SpannerChangeStreamIT suite, but sometimes goes away when run single test under SpannerChangeStreamIT

[rahul2393]

@Abacn @changliiu Multiplex session can't be deleted so this is something not intended in tests