Expose ExecutionPlan statistics across the FFI boundary

[mailmindlin]

Which issue does this PR close?

• Closes Expose ExecutionPlan statistics across the FFI boundary #22152

Rationale for this change

ExecutionPlan::partition_statistics and TableProvider::statistics are not currently transported across the DataFusion FFI boundary, so foreign plans and providers always report Statistics::new_unknown / None. This blocks optimizer rules that depend on statistics (e.g. join reordering, partition pruning) from working with out-of-process plugins, which defeats the point of exposing those hooks to plugin authors.

Statistics contains Precision<ScalarValue> for column min/max/sum. ScalarValue is a large enum that's impractical to mirror in #[repr(C)], so I reuse the existing datafusion_proto_common::Statistics prost encoding — the same pattern this crate already uses for filter expressions.

What changes are included in this PR?

• New datafusion_ffi::statistics module with [de]serialize_statistics helpers wrapping thedatafusion_proto_common::Statistics round-trip.
• New partition_statistics field on FFI_ExecutionPlan and corresponding ExecutionPlan::partition_statistics impl on ForeignExecutionPlan
• New statistics field on FFI_TableProvider and corresponding TableProvider::statistics impl on ForeignTableProvider. Since the trait returns Option<Statistics>, the implementation cannot propagate decode errors, it logs a log::warn! and triggers a debug_assert!.

This PR is expected to be merged after #22136 so it includes those changes.

Are these changes tested?

Yes:

• Unit tests in statistics.rs cover three round-trip cases: Statistics::new_unknown, fully-exact statistics with ScalarValue::Int32/Int64/Utf8 min/max/sum, and mixed Precision::Exact/Inexact/Absent values.
• A new round-trip integration test in execution_plan.rs exercises ForeignExecutionPlan::partition_statistics with both None and Some(idx) partitions, against a plan with no statistics (returns Statistics::new_unknown) and a plan with concrete statistics.
• A new round-trip integration test in table_provider.rs uses a thin TableWithStats wrapper over MemTable to verify both the None path and the concrete Statistics path through ForeignTableProvider::statistics.

Are there any user-facing changes?

This is a breaking ABI change for the datafusion-ffi crate:

• FFI_ExecutionPlan gains a partition_statistics field.
• FFI_TableProvider gains a statistics field.

Plugins compiled against earlier versions of datafusion-ffi will need to be recompiled. There are no breaking changes to the Rust trait surface or to Statistics itself; downstream ExecutionPlan / TableProvider implementations require no changes.

[github-actions]

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion-ffi v53.1.0 (current)
       Built [  60.582s] (current)
     Parsing datafusion-ffi v53.1.0 (current)
      Parsed [   0.059s] (current)
    Building datafusion-ffi v53.1.0 (baseline)
       Built [  58.321s] (baseline)
     Parsing datafusion-ffi v53.1.0 (baseline)
      Parsed [   0.059s] (baseline)
    Checking datafusion-ffi v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.252s] 222 checks: 220 pass, 1 fail, 1 warn, 30 skip

--- failure constructible_struct_adds_field: externally-constructible struct adds field ---

Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field FFI_ExecutionPlan.partition_statistics in /home/runner/work/datafusion/datafusion/datafusion/ffi/src/execution_plan.rs:82

--- warning repr_c_plain_struct_fields_reordered: struct fields reordered in repr(C) struct ---

Description:
A public repr(C) struct had its fields reordered. This can change the struct's memory layout, possibly breaking FFI use cases that depend on field position and order.
        ref: https://doc.rust-lang.org/reference/type-layout.html#reprc-structs
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/repr_c_plain_struct_fields_reordered.ron

Failed in:
  FFI_TableProvider.logical_codec moved from position 6 to 7, in /home/runner/work/datafusion/datafusion/datafusion/ffi/src/table_provider.rs:143
  FFI_TableProvider.version moved from position 9 to 10, in /home/runner/work/datafusion/datafusion/datafusion/ffi/src/table_provider.rs:153
  FFI_TableProvider.library_marker_id moved from position 11 to 12, in /home/runner/work/datafusion/datafusion/datafusion/ffi/src/table_provider.rs:162
  FFI_ExecutionPlan.clone moved from position 8 to 9, in /home/runner/work/datafusion/datafusion/datafusion/ffi/src/execution_plan.rs:89
  FFI_ExecutionPlan.release moved from position 9 to 10, in /home/runner/work/datafusion/datafusion/datafusion/ffi/src/execution_plan.rs:92
  FFI_ExecutionPlan.private_data moved from position 10 to 11, in /home/runner/work/datafusion/datafusion/datafusion/ffi/src/execution_plan.rs:96
  FFI_ExecutionPlan.library_marker_id moved from position 11 to 12, in /home/runner/work/datafusion/datafusion/datafusion/ffi/src/execution_plan.rs:101

     Summary semver requires new major version: 1 major and 0 minor checks failed
     Warning produced 1 major and 0 minor level warnings
    Finished [ 121.374s] datafusion-ffi

[Copilot]

Pull request overview

This PR plumbs ExecutionPlan::partition_statistics and TableProvider::statistics across the datafusion-ffi boundary so out-of-process plugins can report real statistics to the planner instead of always returning Statistics::new_unknown/None. It reuses the existing datafusion_proto_common::Statistics prost encoding (the same approach already used for filter expressions) rather than mirroring ScalarValue in #[repr(C)].

Changes:

• New datafusion_ffi::statistics module with serialize_statistics/deserialize_statistics helpers backed by datafusion-proto-common.
• New partition_statistics function pointer on FFI_ExecutionPlan and corresponding ForeignExecutionPlan::partition_statistics implementation (Result-propagating).
• New statistics function pointer on FFI_TableProvider and corresponding ForeignTableProvider::statistics implementation (logs + debug_assert! on decode error since the trait can only return Option).

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
datafusion/ffi/src/lib.rs	Registers the new statistics module.
datafusion/ffi/src/statistics.rs	New helpers wrapping prost round-trip with three unit round-trip tests.
datafusion/ffi/src/execution_plan.rs	Adds partition_statistics FFI field, wrapper, and ForeignExecutionPlan impl; extends EmptyExec test helper with with_statistics; adds a round-trip integration test.
datafusion/ffi/src/table_provider.rs	Adds statistics FFI field, wrapper, and ForeignTableProvider::statistics impl with log::warn!/debug_assert! on decode failure; adds a TableWithStats-based round-trip test.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[timsaucer]

Looking good, but a few things worth addressing

[timsaucer]

why FFI_Option<u64> instead of FFI_Option<usize>?

[mailmindlin]

I'm not sure I understand what you mean

[timsaucer]

Fascinating. If you type in FFI_Option<u64> without the back ticks then it doesn't show the <u64>. added the ticks