Support metadata push mode in BaseSingleSegmentConversionExecutor by shounakmk219 · Pull Request #17632 · apache/pinot

shounakmk219 · 2026-02-04T15:36:22Z

Summary

Adds METADATA (and URI) segment push mode to BaseSingleSegmentConversionExecutor so single-segment conversion tasks can push via metadata (segment on output PinotFS + metadata to controller) instead of only TAR (HTTP upload). Moves shared segment-push logic into BaseTaskExecutor so both single- and multiple-segment conversion executors reuse the same helpers.

Changes

BaseTaskExecutor

Added shared segment push constants: SEGMENT_PUSH_DEFAULT_ATTEMPTS, SEGMENT_PUSH_DEFAULT_PARALLELISM, SEGMENT_PUSH_DEFAULT_RETRY_INTERVAL_MILLIS.
Added protected helpers used by both conversion executors:
- getPushJobSpec(configs) – builds PushJobSpec from task config.
- generateSegmentGenerationJobSpec(tableName, configs, pushJobSpec) – builds SegmentGenerationJobSpec for controller push.
- moveSegmentToOutputPinotFS(configs, localSegmentTarFile) – copies local segment tar to output PinotFS; requires output.segment.dir.uri.
- getSegmentPushCommonParams(tableNameWithType) – common HTTP params (parallel push protection, table name, table type).
- getSegmentPushMetadataHeaders(pinotTaskConfig, authProvider, segmentConversionResult) – headers for metadata push (ZK metadata modifier + auth).
- getSegmentPushType(configs) – resolves push mode from config (push.mode, default TAR); subclasses can override.

BaseSingleSegmentConversionExecutor

Supports configurable push mode (TAR vs METADATA/URI) via getSegmentPushType(configs) (overridable).
After conversion, upload step branches on push type: TAR → existing HTTP upload; METADATA/URI → move segment to output PinotFS (when output.segment.dir.uri and push.controllerUri are set) and call SegmentPushUtils.sendSegmentUriAndMetadata.
Uses base getSegmentPushCommonParams for upload parameters and the new base helpers for the METADATA path (no duplicated push logic).

BaseMultipleSegmentsConversionExecutor

Removed duplicate push logic: dropped local constants and implementations of getPushJobSpec, generateSegmentGenerationJobSpec, moveSegmentToOutputPinotFS, and getSegmentPushCommonParams; uses base implementations.
getSegmentPushCommonHeaders now delegates to base getSegmentPushMetadataHeaders.
pushSegment uses base getSegmentPushType(taskConfigs) for resolving push mode.

SegmentGenerationAndPushTaskExecutor

moveSegmentToOutputPinotFS overrides to handle missing output.segment.dir.uri (return local file URI) and delegates to super.moveSegmentToOutputPinotFS when the config is present.

MinionTaskUtils

new flag allowMetadataPushWithLocalFs to skip the override to TAR push mode when output segment dir is on local FS. This flag is introduced to enable testing of METADATA push mode and is not intended for production use (unless the local disk is somehow shared and all components have access to it).

Usage (single-segment METADATA push)

Set task config push.mode = METADATA (or override getSegmentPushType in a subclass).
Provide output.segment.dir.uri and push.controllerUri in task config for the METADATA path.

codecov-commenter · 2026-02-04T16:26:10Z

❌ 3 Tests Failed:

Tests completed	Failed	Passed	Skipped
12196	3	12193	59

View the full list of 3 ❄️ flaky test(s)

org.apache.pinot.integration.tests.KafkaConfluentSchemaRegistryAvroMessageDecoderRealtimeClusterIntegrationTest::setUp
Flake rate in main: 100.00% (Passed 0 times, Failed 76 times)
Stack Traces | 12.1s run time
Could not find a valid Docker environment. Please see logs and check configuration

org.apache.pinot.plugin.inputformat.json.confluent.JsonConfluentSchemaTest::@BeforeClass setup
Flake rate in main: 100.00% (Passed 0 times, Failed 49 times)
Stack Traces | 0.44s run time
Could not find a valid Docker environment. Please see logs and check configuration

org.apache.pinot.plugin.inputformat.json.confluent.JsonConfluentSchemaTest::setup
Flake rate in main: 100.00% (Passed 0 times, Failed 98 times)
Stack Traces | 0.44s run time
Could not find a valid Docker environment. Please see logs and check configuration

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

krishan1390

Have we tested it with METADATA push mode ?

krishan1390 · 2026-02-05T04:35:12Z

+                uploadURL, convertedTarredSegmentFile);
+            break;
+          case METADATA:
+          case URI:


BaseMultipleSegmentsConversionExecutor doesn't support URI mode. Is it fine to add here ?

Yeah let me revert that, not much roi compared to the increased testing scope for this PR

noob-se7en

Lets add a test for new code (metadata push via BaseSingleSegmentConverter).

noob-se7en · 2026-02-05T05:29:58Z

      List<Header> httpHeaders = new ArrayList<>();
      httpHeaders.add(ifMatchHeader);
      httpHeaders.add(refreshOnlyHeader);
      httpHeaders.add(segmentZKMetadataCustomMapModifierHeader);
      httpHeaders.addAll(AuthProviderUtils.toRequestHeaders(authProvider));


These headers are not required for metadata push?

uploadSegmentWithMetadata internally calls getSegmentPushMetadataHeaders to populate the required headers

…port-for-single-sce

shounakmk219 · 2026-02-12T14:48:05Z

Updated the code a bit to allow METADATA push mode even with local fs just for testing. Added tests for the tasks extending the BaseSingleSegmentConversionExecutor

…port-for-single-sce

swaminathanmanish

LGTM !

swaminathanmanish · 2026-02-13T05:20:42Z

-        LOGGER.warn("Local output dir found, defaulting to TAR: {}.", outputSegmentDirURI);
-        singleFileGenerationTaskConfig.put(BatchConfigProperties.PUSH_MODE,
-            BatchConfigProperties.SegmentPushType.TAR.toString());
+        boolean allowMetadataPushWithLocalFs = Boolean.parseBoolean(


Why is the new param needed. Instead of this extra param - ALLOW_METADATA_PUSH_WITH_LOCAL_FS, can we not just take the metadata push path when configured regardless of LOCAL_FS or not ?

Its risky for multi node setup where segment download url will point to a local path (mostly on the minion) and server will look for it on their own local fs for downloading the segment.

…port-for-single-sce

swaminathanmanish · 2026-02-16T05:44:47Z

Please note that there unrelated test failures. We should take it separately to address flaky tests.

KafkaConfluentSchemaRegistryAvroMessageDecoderRealtimeClusterIntegrationTest.setUp:292->BaseRealtimeClusterIntegrationTest.setUp:67->startKafka:88->startSchemaRegistry:99 » IllegalState Could not find a valid Docker environment. Please see logs and check configuration

Support metadata push mode in BaseSingleSegmentConversionExecutor

d3248ea

krishan1390 reviewed Feb 5, 2026

View reviewed changes

krishan1390 approved these changes Feb 5, 2026

View reviewed changes

noob-se7en approved these changes Feb 5, 2026

View reviewed changes

shounakmk219 added 8 commits February 11, 2026 12:26

Merge remote-tracking branch 'upstream/master' into metadata-push-sup…

2fd74d0

…port-for-single-sce

remove URI support

6aaccf0

Merge remote-tracking branch 'upstream/master' into metadata-push-sup…

6acda3b

…port-for-single-sce

bypass local fs specific logic for metadata push mode testing

781437f

inject relevant push configs for tasks extending ssce

519a937

extend existing tests to validate metadata push mode

c4d8037

cleanup

83777e3

Merge remote-tracking branch 'upstream/master' into metadata-push-sup…

7fb22f5

…port-for-single-sce

Merge remote-tracking branch 'upstream/master' into metadata-push-sup…

6f5334b

…port-for-single-sce

shounakmk219 added enhancement Improvement to existing functionality minion Related to Pinot Minion task framework labels Feb 12, 2026

swaminathanmanish approved these changes Feb 13, 2026

View reviewed changes

swaminathanmanish reviewed Feb 13, 2026

View reviewed changes

Merge remote-tracking branch 'upstream/master' into metadata-push-sup…

e24093f

…port-for-single-sce

swaminathanmanish merged commit 657e7f0 into apache:master Feb 16, 2026
36 of 48 checks passed

xiangfu0 added the ingestion Related to data ingestion pipeline label Mar 20, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support metadata push mode in BaseSingleSegmentConversionExecutor#17632

Support metadata push mode in BaseSingleSegmentConversionExecutor#17632
swaminathanmanish merged 11 commits intoapache:masterfrom
shounakmk219:metadata-push-support-for-single-sce

shounakmk219 commented Feb 4, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Feb 4, 2026 •

edited

Loading

Uh oh!

krishan1390 left a comment

Uh oh!

krishan1390 Feb 5, 2026

Uh oh!

shounakmk219 Feb 11, 2026

Uh oh!

noob-se7en left a comment

Uh oh!

noob-se7en Feb 5, 2026

Uh oh!

shounakmk219 Feb 11, 2026

Uh oh!

shounakmk219 commented Feb 12, 2026

Uh oh!

swaminathanmanish left a comment

Uh oh!

swaminathanmanish Feb 13, 2026

Uh oh!

shounakmk219 Feb 13, 2026

Uh oh!

swaminathanmanish commented Feb 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

shounakmk219 commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

BaseTaskExecutor

BaseSingleSegmentConversionExecutor

BaseMultipleSegmentsConversionExecutor

SegmentGenerationAndPushTaskExecutor

MinionTaskUtils

Usage (single-segment METADATA push)

Uh oh!

codecov-commenter commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 3 Tests Failed:

Uh oh!

krishan1390 left a comment

Choose a reason for hiding this comment

Uh oh!

krishan1390 Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

shounakmk219 Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

noob-se7en left a comment

Choose a reason for hiding this comment

Uh oh!

noob-se7en Feb 5, 2026

Choose a reason for hiding this comment

Uh oh!

shounakmk219 Feb 11, 2026

Choose a reason for hiding this comment

Uh oh!

shounakmk219 commented Feb 12, 2026

Uh oh!

swaminathanmanish left a comment

Choose a reason for hiding this comment

Uh oh!

swaminathanmanish Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

shounakmk219 Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

swaminathanmanish commented Feb 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

shounakmk219 commented Feb 4, 2026 •

edited

Loading

codecov-commenter commented Feb 4, 2026 •

edited

Loading