[SPARK-56469][SQL][TESTS] BufferedRowsReader lifecycle hardening

[peter-toth]

What changes were proposed in this pull request?

Extracted from #55116.

The test-framework reader in InMemoryBaseTable now tracks a closed flag and throws IllegalStateException for reads, double-closes, or metric fetches on a closed reader. This ensures future tests catch reader lifecycle bugs that were previously hidden by the noop close().

Why are the changes needed?

Improve test framework.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Existing UTs.

Was this patch authored or co-authored using generative AI tooling?

No.

[peter-toth]

cc @dongjoon-hyun

[dongjoon-hyun]

+1, LGTM. Thank you, @peter-toth .

[dongjoon-hyun]

Currently SparkConnectSessionHolderSuite seems to hang for some reasons. Normally it finished quickly. Since it seems to be an unknown flaky test case, could you kill and re-trigger this test pipeline, please?

• https://github.com/peter-toth/spark/actions/runs/24402868746/job/71278240499

[info] SparkConnectPlannerSuite:
[info] *** Test still running after 12 minutes, 43 seconds: suite name: SparkConnectSessionHolderSuite, test name: python foreachBatch process: process terminates after query is stopped. 
[info] *** Test still running after 17 minutes, 43 seconds: suite name: SparkConnectSessionHolderSuite, test name: python foreachBatch process: process terminates after query is stopped. 
[info] *** Test still running after 22 minutes, 43 seconds: suite name: SparkConnectSessionHolderSuite, test name: python foreachBatch process: process terminates after query is stopped. 
[info] *** Test still running after 27 minutes, 43 seconds: suite name: SparkConnectSessionHolderSuite, test name: python foreachBatch process: process terminates after query is stopped. 
[info] *** Test still running after 32 minutes, 43 seconds: suite name: SparkConnectSessionHolderSuite, test name: python foreachBatch process: process terminates after query is stopped. 
[info] *** Test still running after 37 minutes, 43 seconds: suite name: SparkConnectSessionHolderSuite, test name: python foreachBatch process: process terminates after query is stopped. 
[info] *** Test still running after 42 minutes, 43 seconds: suite name: SparkConnectSessionHolderSuite, test name: python foreachBatch process: process terminates after query is stopped. 
[info] *** Test still running after 47 minutes, 43 seconds: suite name: SparkConnectSessionHolderSuite, test name: python foreachBatch process: process terminates after query is stopped. 
[info] *** Test still running after 52 minutes, 43 seconds: suite name: SparkConnectSessionHolderSuite, test name: python foreachBatch process: process terminates after query is stopped. 

[peter-toth]

I've just retriggered it.

[dongjoon-hyun]

Merged to master for Apache Spark 4.2.0.

[peter-toth]

Thank you @dongjoon-hyun!