Spark 4.0.x hive-1 jdk17 tests failing consistently in multiple PRs

[andygrove]

Describe the bug

It is getting frustrating trying to get PRs to pass all tests in CI. I see the Spark 4 / hive-1 / jdk17 tests failing even after re-running multiple times (after ~1 hour 20 minutes).

There are no failure. It looks like a hang or timeout. The last entries in the log look like this:

[info] HivePartitionFilteringSuite(2.0):
20:48:31.545 WARN org.apache.hadoop.hive.metastore.ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.0.0
20:48:32.044 WARN org.apache.hadoop.hive.ql.session.SessionState: METASTORE_FILTER_HOOK will be ignored, since hive.security.authorization.manager is set to instance of HiveAuthorizerFactory.
04:51:45.212 WARN org.apache.spark.sql.hive.client.Shim_v2_0: Caught Hive MetaException attempting to get partition metadata by filter from Hive. Falling back to fetching all partition metadata, which will degrade performance. Modifying your Hive metastore configuration to set hive.metastore.try.direct.sql to true (if it is not true already) may resolve this problem. Or you can enable spark.sql.hive.metastorePartitionPruningFastFallback to alleviate performance downgrade. Otherwise, to avoid degraded performance you can set spark.sql.hive.metastorePartitionPruningFall...
MetaException(message:Filtering is supported only on partition keys of type string)
	at org.apache.hadoop.hive.metastore.parser.ExpressionTree$FilterBuilder.setError(ExpressionTree.java:184)
	at org.apache.hadoop.hive.metastore.parser.ExpressionTree$LeafNode.getJdoFilterPushdownParam(ExpressionTree.java:439)
	at org.apache.hadoop.hive.metastore.parser.ExpressionTree$LeafNode.generateJDOFilterOverPartitions(ExpressionTree.java:356)
	at org.apache.hadoop.hive.metastore.parser.ExpressionTree$LeafNode.generateJDOFilter(ExpressionTree.java:278)
	at org.apache.hadoop.hive.metastore.parser.ExpressionTree.generateJDOFilterFragment(ExpressionTree.java:583)
	at org.apache.hadoop.hive.metastore.ObjectStore.makeQueryFilterString(ObjectStore.java:2704)
	at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsViaOrmFilter(ObjectStore.java:2312)
[info] - 2.0: getPartitionsByFilter returns all partitions when spark.sql.hive.metastorePartitionPruningFallbackOnException=true (5 seconds, 426 milliseconds)
[info] - 2.0: getPartitionsByFilter should fail when spark.sql.hive.metastorePartitionPruningFallbackOnException=false (69 milliseconds)
[info] - 2.0: getPartitionsByFilter: ds<=>20170101 (1 second, 466 milliseconds)
...
 05:12:55.055 WARN org.apache.spark.sql.hive.client.Shim_v3_0: Caught Hive MetaException attempting to get partition metadata by filter from Hive. Falling back to fetching all partition metadata, which will degrade performance. Modifying your Hive metastore configuration to set hive.metastore.try.direct.sql to true (if it is not true already) may resolve this problem. Or you can enable spark.sql.hive.metastorePartitionPruningFastFallback to alleviate performance downgrade. Otherwise, to avoid degraded performance you can set spark.sql.hive.metastorePartitionPruningFall...
MetaException(message:Filtering is supported only on partition keys of type string)
	at org.apache.hadoop.hive.metastore.parser.ExpressionTree$FilterBuilder.setError(ExpressionTree.java:184)
	at org.apache.hadoop.hive.metastore.parser.ExpressionTree$LeafNode.getJdoFilterPushdownParam(ExpressionTree.java:437)
	at org.apache.hadoop.hive.metastore.parser.ExpressionTree$LeafNode.generateJDOFilterOverPartitions(ExpressionTree.java:355)
	at org.apache.hadoop.hive.metastore.parser.ExpressionTree$LeafNode.generateJDOFilter(ExpressionTree.java:277)
	at org.apache.hadoop.hive.metastore.parser.ExpressionTree.generateJDOFilterFragment(ExpressionTree.java:581)
	at org.apache.hadoop.hive.metastore.ObjectStore.makeQueryFilterString(ObjectStore.java:3841)
	at org.apache.hadoop.hive.metastore.ObjectStore.getPartitionsViaOrmFilter(ObjectStore.java:3287)
...


### Steps to reproduce

_No response_

### Expected behavior

_No response_

### Additional context

_No response_

[coderfender]

Tagging PRs which are failing to make easier to trace :

#2932
#2600