[SPARK-57175][SQL] Extend nested column pruning to exists and forall over arrays of structs

[sunchao]

Why are the changes needed?

SPARK-57175 follows SPARK-57022, which added nested column pruning for transform over array<struct> inputs. The same optimization does not currently apply to the exists and forall higher-order array functions.

For example:

SELECT exists(rule_results, rule -> rule.rule_version > 10)
FROM events

If rule_results contains additional fields, Spark currently retains the full element struct in the scan schema even though the predicate only reads rule_version. This causes unnecessary Parquet and ORC input reads for wide array element schemas.

What changes were proposed in this PR?

• Share the nested-field collection path introduced for ArrayTransform with ArrayExists and ArrayForAll.
• Rewrite the bound lambda variable type and GetStructField ordinals against the projected element schema after pruning.
• Keep the conservative fallback when a lambda consumes the whole element.
• Add Catalyst and datasource tests covering schema discovery, ordinal rewrites, predicate-path schema merging, and whole-element fallback.

ArrayFilter and ArraySort remain out of scope because they return original input elements and require a different downstream-schema design.

Does this PR introduce any user-facing change?

Yes. Eligible queries using exists or forall over arrays of structs can read a narrower input schema. Query results and SQL APIs are unchanged.

How was this patch tested?

• JAVA_HOME=/opt/homebrew/opt/openjdk@17/libexec/openjdk.jdk/Contents/Home PATH=/opt/homebrew/opt/openjdk@17/bin:$PATH build/sbt "catalyst/testOnly org.apache.spark.sql.catalyst.expressions.SchemaPruningSuite" "sql/testOnly org.apache.spark.sql.execution.datasources.parquet.ParquetV1SchemaPruningSuite org.apache.spark.sql.execution.datasources.parquet.ParquetV2SchemaPruningSuite org.apache.spark.sql.execution.datasources.orc.OrcV1SchemaPruningSuite org.apache.spark.sql.execution.datasources.orc.OrcV2SchemaPruningSuite -- -z Array"
• git diff --check

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Codex (GPT-5)

[sunchao]

cc @cloud-fan @viirya @peter-toth

[viirya]

Nicely scoped follow-up to SPARK-57022. I traced the shared helper extraction through both SchemaPruning.getRootFields and ProjectionOverSchema, and verified the test fixtures and expected values. The change is correct and safe.

A few things I checked that are worth noting:

• ArrayExists.copy(argument =, function =) leaves followThreeValuedLogic untouched, so three-valued semantics are preserved even though ProjectionOverSchema wildcards that field in the pattern. Good.
• The whole-element fallback is correct: when the lambda consumes the full element, collectLambdaVariableFields returns None, so the helper falls back to recursing children and the full schema is retained. The two "do not prune" tests pin this down.
• Excluding ArrayFilter/ArraySort is both correct and safe. They return the original elements (dataType = argument.dataType), so pruning the argument's element struct would corrupt the output type. And leaving them on the default getRootFields branch can't cause silent mis-pruning, because SelectedField terminates only at an Attribute -- a GetStructField chain rooted at a NamedLambdaVariable yields no narrow root field, so the argument's full schema is preserved.
• The forall over an empty array returning true (Contact 1 has no friends) is a nice vacuous-truth edge to have covered.

Two optional nits, neither blocking:

1. In the datasource tests, exists is called fully-qualified (org.apache.spark.sql.functions.exists) while forall is bare. Understandable (bare exists collides with Scala's collection method), but the asymmetry reads oddly -- a short alias import or a one-line comment would make the intent clearer.
2. The Catalyst-level SchemaPruningSuite only covers the pruning case; the whole-element fallback is only exercised at the datasource layer. A unit-level assertion for the x -> x (returns None) path would round out coverage, though the datasource tests already cover it functionally.

LGTM.