[Bug]: grpc: error while marshaling: string field contains invalid UTF-8

[jeschkies]

What happened?

I have a Dataflow pipeline using Beam for go. It started to fail with the following error

control.Send: Failed to respond: rpc error: code = Internal desc = grpc: error while marshaling: string field contains invalid UTF-8

For github.com/apache/beam/sdks/v2@v2.69.0/go/pkg/beam/core/runtime/harness/harness.go:143.

The logs are a little hard to grok for me so I'm not sure if this is caused the the Beam SDK or some parameter I'm passing.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[jeschkies]

The error disappear when I've addressed the warning that the service account running the workflow had no access to retrieve the Pub/Sub configuration.

I feels the flow should error earlier.

[chamikaramj]

Thanks for the feedback. Sounds like we should improve the error message in this case or elevate the warning you saw to an error. Can you provide the exact warning that helped ?

[jeschkies]

I ran into the issue again. This time the warning was

Pub/Sub subscription projects/.../subscriptions/... has a retry policy configured, which will not work as expected when Dataflow pulls from the subscription. See https://cloud.google.com/dataflow/docs/concepts/streaming-with-cloud-pubsub for further details.

Once I changed the retry config the error disappeared. I think this might be a red herring.