perf: avoid FFI import/export when passing batches between native plans#3930

Draft

andygrove wants to merge 22 commits intoapache:mainfrom

andygrove:batch-stash-shuffle-optimization

Commits on Apr 12, 2026

docs: add design spec for batch stash shuffle optimization
andygrove
committed
docs: add implementation plan for batch stash shuffle optimization
andygrove
committed
feat: add BatchStash registry for native batch handle passing
andygrove
committed
feat: add CometHandleBatchIterator Java class and JNI bridge
andygrove
committed
feat: add executePlanBatchHandle JNI function for stash-mode output
andygrove
committed
feat: add stash mode to CometExecIterator for batch handle output
andygrove
committed
feat: add handle-mode input path to ScanExec for batch stash retrieval
andygrove
committed
feat: detect CometHandleBatchIterator in planner and enable handle mode
andygrove
committed
feat: preserve CometExecIterator reference through shuffle dependency RDD
andygrove
committed
feat: integrate batch stash mode in CometNativeShuffleWriter
andygrove
committed
style: apply formatting fixes
andygrove
committed
fix: use scan source name for handle mode detection instead of JNI is_instance_of
andygrove
committed
chore: remove JVM crash log files
andygrove
committed
refactor: deduplicate executePlan logic and address code review findings
andygrove
committed
refactor: use protobuf field for batch stash handle mode instead of scan source name
andygrove
committed
chore: remove design docs from PR
andygrove
committed
fix: add batch_stash_handle field to Scan struct literals in tests
andygrove
committed
fix: use InputBatch::Complete for stashed batches to bypass schema reconciliation
andygrove
committed
feat: add spark.comet.exec.shuffle.batchStash.enabled config (default true)
andygrove
committed
fix: apply schema reconciliation for stashed batches with type mismatches
andygrove
committed

Commits on Apr 13, 2026

fix: address PR review feedback for batch stash
andygrove
committed

Commits on Apr 16, 2026

refactor: move batch stash from global static into ExecutionContext
andygrove
committed