Skip to content

feat: Restore nullability when consuming substrait fields#22105

Open
neilconway wants to merge 2 commits into
apache:mainfrom
neilconway:neilc/feat-subtrait-nullable
Open

feat: Restore nullability when consuming substrait fields#22105
neilconway wants to merge 2 commits into
apache:mainfrom
neilconway:neilc/feat-subtrait-nullable

Conversation

@neilconway
Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Rationale for this change

The Substrait logical-plan consumer was discarding field nullability when reconstructing DataFusion schemas from Substrait struct types. Nullability matters because a Substrait plan may have been produced or optimized using non-null guarantees.

This also improves DataFusion <-> Substrait round-trip fidelity: required fields encoded by the producer are preserved when the plan is consumed again, instead of being widened to nullable.

What changes are included in this PR?

  • Preserve per-field nullability when converting Substrait struct types / NamedStruct schemas into DataFusion schemas.
  • Treat Substrait Required as non-nullable, and Nullable, Unspecified, or unknown nullability values as nullable.
  • Keep deprecated UserDefinedTypeReference non-nullable because it does not carry nullability metadata.
  • Enforce named-table ReadRel schema compatibility when the Substrait schema requires a field to be non-null but the resolved DataFusion table schema marks it nullable.
  • Extend compatibility checking recursively through nested Struct fields.
  • Leave List and Map child nullability compatibility as future work, since their child nullability is not faithfully reconstructed today.

Are these changes tested?

Yes; new tests added.

Are there any user-facing changes?

We are a bit stricter when consuming Substrait plans now, but that could prevent problems: for example, if a Substrait plan was produced under the assumption that a field x is non-nullable but the local DataFusion schema allows null values in x, executing the plan might produce unexpected results.

@github-actions github-actions Bot added the substrait Changes to the substrait crate label May 11, 2026
@neilconway
Copy link
Copy Markdown
Contributor Author

Initially motivated by the test failure in #22089, but seems worth improving as a separate task rather than just weakening the test assertions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

substrait Changes to the substrait crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Restore nullability support for consuming Substrait fields.

1 participant