add minor compaction for Overlord based compaction supervisors using the MSQ engine by cecemei · Pull Request #19059 · apache/druid

cecemei · 2026-02-26T03:06:50Z

Description

This PR introduces minor compaction support in msq engine to Apache Druid, enabling compaction operations on only uncompacted segments within an interval while upgrading already-compacted segments for consistency.

Key Changes

CompactionMode Enum Extension

Added ALL_SEGMENTS mode (full compaction of all segments)
Added UNCOMPACTED_SEGMENTS_ONLY mode (compacts only new segments, upgrades existing ones)

MinorCompactionInputSpec
A new CompactionInputSpec implementation specifically for minor compaction:

Specifies uncompacted segments to compact within an interval
Required (non-nullable) interval and uncompactedSegments fields
Type: "uncompacted"
Replaces nullable field handling in CompactionIntervalSpec, making the intent of minor compaction explicit

MarkSegmentToUpgradeAction
A new task action that updates segment metadata without rewriting data, modifying partition numbers and shard specifications of already-compacted segments.
ShardSpec Interface Enhancements
Extended with mutation methods including withPartitionNum(), withCorePartitions(), and isNumChunkSupported() checks.
Policy-Level Control
Introduced minUncompactedBytesPercentForFullCompaction threshold in MostFragmentedIntervalFirstPolicy determining when incremental versus full compaction applies based on uncompacted-to-total segment byte ratios.
Builder Pattern Implementation
Replaced verbose constructors in UserCompactionTaskQueryTuningConfig with a builder pattern for improved readability.

Release Notes

This feature enables compacting newly ingested segments while upgrading already compacted data, with configuration through percentage thresholds in compaction policies. Support is currently limited to Overlord based compaction supervisors using the MSQ engine.

This PR has:

server/src/test/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinatorTest.java

+      final DataSegment segment = new DataSegment(
+          "foo",
+          Intervals.of("2023-01-0" + i + "/2023-01-0" + (i + 1)),
+          "2023-01-0" + i,
+          ImmutableMap.of("path", "a-" + i),
+          ImmutableList.of("dim1"),
+          ImmutableList.of("m1"),
+          new DimensionRangeShardSpec(List.of("dim1"), null, null, i - 1, 8),
+          9,
+          100
+      );


capistrant

This is awesome. left some minor comments

server/src/main/java/org/apache/druid/server/compaction/CompactionMode.java

server/src/main/java/org/apache/druid/server/compaction/MostFragmentedIntervalFirstPolicy.java

server/src/main/java/org/apache/druid/server/compaction/CompactionCandidate.java

capistrant · 2026-02-26T17:58:54Z

indexing-service/src/main/java/org/apache/druid/indexing/compact/CompactionJobQueue.java

@@ -282,18 +283,20 @@ private boolean startJobIfPendingAndReady(
    }

    // Check if the job is already running, completed or skipped


nit: this comment might need a refresh to reflect how the status handling has changed versus the former code. I was confused about why PENDING and COMPLETE resulted in a throw until I reviewed deriveCompactionStatus

there's no logic change here, just made the code more defensive. updated the comment here. ihmo, CompactionStatus is not the best class to use here.

actually i guess there's some logic change, the policy check has been moved to CompactionConfigBasedJobTemplate since creating the job needs to know the compaction mode (which is decided by the policy), but still the computeCompactionStatus only returns running/skipped/pending state.

capistrant · 2026-02-26T18:15:59Z

indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java

should we update the javadoc to call out how this method now offers special handling for incremental compaction that includes upgrading already compacted segments?

...ing-service/src/main/java/org/apache/druid/indexing/common/actions/SegmentUpgradeAction.java

...e/src/test/java/org/apache/druid/indexing/common/actions/MarkSegmentToUpgradeActionTest.java

processing/src/main/java/org/apache/druid/timeline/partition/LinearShardSpec.java

server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java

kfaraz

Leaving a partial review for the supporting changes.
Will review the core changes in CompactionTask and IndexerSQLMetadataStorageCoordinator shortly.

server/src/main/java/org/apache/druid/server/compaction/CompactionMode.java

...-tests/src/test/java/org/apache/druid/testing/embedded/compact/CompactionSupervisorTest.java

server/src/main/java/org/apache/druid/server/compaction/MostFragmentedIntervalFirstPolicy.java

server/src/main/java/org/apache/druid/server/coordinator/duty/CompactSegments.java

...ervice/src/main/java/org/apache/druid/indexing/compact/CompactionConfigBasedJobTemplate.java

kfaraz

@cecemei , I have left some more comments. Please let me know if anything seems ambiguous.

...rvice/src/main/java/org/apache/druid/indexing/common/actions/MarkSegmentToUpgradeAction.java

...ng-service/src/main/java/org/apache/druid/indexing/common/task/MinorCompactionInputSpec.java

indexing-service/src/main/java/org/apache/druid/indexing/common/task/UncompactedInputSpec.java

server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java

kfaraz · 2026-03-03T12:08:17Z

server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java

+      final ShardSpec shardSpec = oldSegment.getShardSpec().withPartitionNum(partitionNum);
      // Create upgraded segment with the correct interval, version and shard spec
      String lockVersion = upgradeSegmentToLockVersion.get(oldSegment.getId().toString());
      DataSegment dataSegment = DataSegment.builder(oldSegment)


Use the final shardSpec to build this DataSegment.

not sure i understand, the builder is already using the final shardSpec.

Yes, but the number of core partitions is updated later in the code flow, right?
That would effectively change the shard spec.
Even though the number of core partitions is not used to determine segment ID, it would still be cleaner to update the inner DataSegment object (or maybe construct it lazily) so that it uses the correct shard spec.

...-tests/src/test/java/org/apache/druid/testing/embedded/compact/CompactionSupervisorTest.java

…n/task/MinorCompactionInputSpec.java Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

…n/actions/MarkSegmentToUpgradeAction.java Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

kfaraz

@cecemei , the changes look good. I have just a couple of final queries. In particular, the number of core partitions to be used when committing the replace segments.

indexing-service/src/main/java/org/apache/druid/indexing/common/task/CompactionTask.java

kfaraz · 2026-03-09T07:07:21Z

...-tests/src/test/java/org/apache/druid/testing/embedded/compact/CompactionSupervisorTest.java

+    kafkaServer.deleteTopic(topic1);
+    cluster.callApi().postSupervisor(supervisor1.createSuspendedSpec());
+
+    long totalUsed = overlord.latchableEmitter().getMetricValues(


This need not give the correct value of total used segments currently present in the cache.
This metric is emitted after every cache sync, and as such the total aggregate would be higher than the actual number of segments.

the last value (not aggregate) should be number of used segments right?

Yes, but there can be race conditions. There might be some delay before the cache is synced with the metadata store. It is better to wait for the actual event you want.

kfaraz · 2026-03-09T07:09:25Z

...-tests/src/test/java/org/apache/druid/testing/embedded/compact/CompactionSupervisorTest.java

+    waitForAllCompactionTasksToFinish();
+
+    // wait for new segments have been updated to the cache
+    overlord.latchableEmitter().waitForEvent(


If the intent is to wait for the next cache sync, do the following instead:

overlord.latchableEmitter().waitForNextEvent( event -> event.hasMetricName("segment/metadataCache/sync/time") )

wouldn't that be the same to wait for the published segments to show up in the sync?

Yes, this check itself is correct. The problem is the value of totalUsed that this check depends on.

...-tests/src/test/java/org/apache/druid/testing/embedded/compact/CompactionSupervisorTest.java

server/src/main/java/org/apache/druid/metadata/IndexerSQLMetadataStorageCoordinator.java

…ataStorageCoordinator.java Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

kfaraz

LGTM 🚀 🚀

incremental-compaction-mode

682dd8b

github-actions bot added Area - Batch Ingestion Area - Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 labels Feb 26, 2026

github-advanced-security bot found potential problems Feb 26, 2026

View reviewed changes

cecemei requested a review from kfaraz February 26, 2026 03:37

cecemei marked this pull request as ready for review February 26, 2026 03:37

cecemei mentioned this pull request Feb 26, 2026

add incremental compaction support #18996

Draft

10 tasks

capistrant reviewed Feb 26, 2026

View reviewed changes

style

86ee6b8

cecemei added the Release Notes label Feb 27, 2026

pending

bdc28ac

kfaraz reviewed Feb 27, 2026

View reviewed changes

cecemei added 5 commits February 27, 2026 17:08

format

9a09e43

Merge branch 'master' into compact2.2

14294a2

build

3a0bd79

UncompactedInputSpec

3eac326

test

ba090ba

cecemei changed the title ~~add incremental compaction mode 2.2~~ add minor compaction for Overlord-based compaction supervisors using the MSQ engine Feb 28, 2026

cecemei added 5 commits February 27, 2026 18:43

uncompacted

64473a7

format

407e328

format2

df25187

format

0134e15

fix

c57b7f6

kfaraz mentioned this pull request Mar 3, 2026

Minor compaction #19016

Draft

9 tasks

kfaraz reviewed Mar 3, 2026

View reviewed changes

cecemei and others added 4 commits March 3, 2026 13:34

review

1d19c85

Update indexing-service/src/main/java/org/apache/druid/indexing/commo…

a7951be

…n/task/MinorCompactionInputSpec.java Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

Update indexing-service/src/main/java/org/apache/druid/indexing/commo…

4e888b7

…n/actions/MarkSegmentToUpgradeAction.java Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

set

2ed63ae

test

ed32beb

cecemei changed the title ~~add minor compaction for Overlord-based compaction supervisors using the MSQ engine~~ add minor compaction for Overlord based compaction supervisors using the MSQ engine Mar 4, 2026

cecemei requested review from capistrant and kfaraz March 4, 2026 01:11

cecemei added 6 commits March 4, 2026 10:49

review

aaf5ad4

minor

cd49144

bug

f732719

checkstyle

e703f29

format

1a055ba

Merge remote-tracking branch 'origin/master' into compact2.2

2b0b871

kfaraz reviewed Mar 9, 2026

View reviewed changes

cecemei and others added 4 commits March 9, 2026 11:19

Update server/src/main/java/org/apache/druid/metadata/IndexerSQLMetad…

fb628ee

…ataStorageCoordinator.java Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

Merge remote-tracking branch 'origin/master' into compact2.2

3d091c9

review

0131c04

review

826110c

kfaraz approved these changes Mar 9, 2026

View reviewed changes

cecemei added 3 commits March 10, 2026 16:49

Merge branch 'master' into compact2.2

2852f17

Merge branch 'master' into compact2.2

45b16b6

build

4ceeb51

cecemei merged commit 6badd78 into apache:master Mar 11, 2026
62 of 63 checks passed

github-actions bot added this to the 37.0.0 milestone Mar 11, 2026

kfaraz mentioned this pull request Mar 13, 2026

Emit metric dimension for minor compaction and other refactors #19151

Open

10 tasks

		@@ -282,18 +283,20 @@ private boolean startJobIfPendingAndReady(
		}

		// Check if the job is already running, completed or skipped

Conversation

cecemei commented Feb 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Check notice

Uh oh!

capistrant left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kfaraz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kfaraz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kfaraz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kfaraz left a comment

Choose a reason for hiding this comment

Uh oh!

cecemei commented Feb 26, 2026 •

edited

Loading