perf(metadata table) Metadata table archival should proceed until earliest commit to retain on data table by kbuci · Pull Request #18215 · apache/hudi

kbuci · 2026-02-17T23:47:46Z

Summary and Changelog

Summary: Metadata table (MDT) archival should be able to archive instants that are before the earliest commit to retain (in latest clean) on the data table, provided that there are no savepoints. This is since

the metadata table anyway cannot be safely restored to a point before the earliest commit to retain (ECTR) on the data table
For datasets with infrequent ingestion writes and duration based compaction scheduling on MDT, there can be 1.5-2x the number of instant files in MDT active timeline compared to data table timeline. This is since each write to the data table will not only create a deltacommit on MDT, but most likely compaction commit as well.

Changelog:

TimelineUtils
- getEarliestInstantForMetadataArchival now takes a third parameter: Option<String> earliestUncleanedDataTableInstantTimeOption (ECTR from the latest clean).
- Logic uses “earliest possible restore commit” (ECTR if present and on timeline, else first commit), “smallest” of that and inflight, and when not archiving beyond savepoint, the minimum of first savepointed write, earliest inflight, and earliest possible restore commit.
- Added getEarliestRetainedCommitFromLastClean(HoodieTableMetaClient) to read ECTR from the latest completed clean; throws HoodieIOException on read failure.
- Added findSmallestInstant(Option<HoodieInstant>, Option<HoodieInstant>) and getEarliestPossibleRestoreCommit(HoodieActiveTimeline, Option<String>).
HoodieTimeline / BaseHoodieTimeline
- New method findFirstSavepointedWrite(): returns the first write commit that has been savepointed (used when blocking MDT archival on savepoints).
TimelineArchiverV1 and TimelineArchiverV2
- When computing the earliest instant to retain for the metadata table, both archivers now pass the data table ECTR: TimelineUtils.getEarliestRetainedCommitFromLastClean(dataMetaClient) is supplied as the third argument to getEarliestInstantForMetadataArchival.
Tests
- TestTimelineUtils: All getEarliestInstantForMetadataArchival call sites updated to the 3-arg signature; added cases for ECTR present, ECTR not on timeline, and savepoints + ECTR (earlier than savepoints and between savepoints).
- HoodieTestTable: Added addIncrementalClean(String instantTime, String earliestCommitToRetain) for tests that need a clean with a given ECTR.
- TestHoodieTimelineArchiver: Added testArchivalInMetadataTableCanProceedUntilECTR to assert MDT archival proceeds up to the data table ECTR.

Impact

User-facing / API: No new public APIs. MDT archival may retain fewer instants (archive more)
Performance: Slightly less MDT timeline to scan when ECTR is present, due to more aggressive archival.

Risk Level

Low. It should be safe if instants on the data table before the ECTR don't have a corresponding deltacommit on MDT. This is since those instants must already have been committed, and cannot anyway be rolled back.

Documentation Update

None.

Contributor's checklist

Read through contributor's guide
Enough context is provided in the sections above
Adequate tests were added if applicable

hudi-bot · 2026-02-18T06:17:07Z

CI report:

31d7847 Azure: SUCCESS

Bot commands

@hudi-bot supports the following commands:

@hudi-bot run azure re-run the last Azure build

nsivabalan · 2026-02-23T07:59:20Z

hey @kbuci : but what does this buy us though. Its a minor optimization right. don't you think we can live w/ it and keep the code simple.

danny0405 · 2026-02-24T04:19:02Z

          TimelineUtils.getEarliestInstantForMetadataArchival(
-              dataMetaClient.getActiveTimeline(), config.shouldArchiveBeyondSavepoint());
+              dataMetaClient.getActiveTimeline(), config.shouldArchiveBeyondSavepoint(),
+              TimelineUtils.getEarliestRetainedCommitFromLastClean(dataMetaClient));


The dataMetaClient.getActiveTimeline() is passed as a param here, we can inline the fetch of TimelineUtils.getEarliestRetainedCommitFromLastClean(dataMetaClient) inside the method.

danny0405 · 2026-02-24T04:51:33Z

-      if (earliestCommit.get().compareTo(earliestInflight.get()) < 0) {
-        return earliestCommit;
+    if (shouldArchiveBeyondSavepoint) {
+      return findSmallestInstant(earliestInflight, earliestPossibleRestoreCommit);


in the original logic, when shouldArchiveBeyondSavepoint is true, #getFirstNonSavepointCommit is invoked to filter out the savepointed instant(because for data table these instants are skipped during archiving, there is case that these savepoints are very old, the skip is intentional).

In the new logic, earliestPossibleRestoreCommit does not skip the savepointed instants when the last clean retained instant is not there, should we still skip by checking dataTableActiveTimeline.findFirstSavepointedWrite() in this case.

🤖 Good catch. If shouldArchiveBeyondSavepoint is enabled, we need to ensure that old savepoints don't inadvertently become a hard floor for the archival watermark, as that would defeat the purpose of the config and cause the timeline to grow.

danny0405 · 2026-02-24T08:23:17Z

+  /**
+   * Returns the earliest commit to retain from the latest completed clean on the given table, if present.
+   */
+  public static Option<String> getEarliestRetainedCommitFromLastClean(HoodieTableMetaClient metaClient) {


we can reuse the logic in SavepointActionExecutor#getLastCommitRetained.

danny0405 · 2026-02-24T08:26:11Z

+        return earliestUncleanedDataTableInstant;
+      }
    }
+    return dataTableActiveTimeline.getCommitsTimeline().firstInstant();


when shouldArchiveBeyondSavepoint is true, should we skip the very old savepoints just like before here?

danny0405 · 2026-02-24T08:29:04Z

+    // - First save-pointed write
+    // - Earliest inflight
+    // - Earliest commit that dataset can potentially be restored to
+    return findSmallestInstant(dataTableActiveTimeline.findFirstSavepointedWrite(),


if rolling back a savepoint is not a real case, maybe the comparison of dataTableActiveTimeline.findFirstSavepointedWrite() is not necessary.

yihua

🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Style & Readability Review — The code changes look good overall, with a minor suggestion for a test helper's object creation.

yihua · 2026-04-05T05:22:11Z

    return addClean(instantTime, Option.empty(), cleanerPlan, metadata, false, false);
  }

+  public HoodieTestTable addIncrementalClean(String instantTime, String earliestCommitToRetain) throws IOException {


🤖 nit: For these complex objects like HoodieCleanerPlan and HoodieCleanStat with many parameters, consider using a builder pattern if they are created frequently with varying attributes. It can improve readability by explicitly naming the parameters.

yihua

🤖 This review was generated by an AI agent and may contain mistakes. Please verify any suggestions before applying.

Thanks for contributing! The logic to utilize ECTR for MDT archival is a great optimization to keep the MDT timeline lean and performant. The implementation is clean, but I left one inline comment regarding a potential edge case with missing write commits for savepoints.

yihua · 2026-04-05T05:28:17Z

+    }
+    HoodieInstant firstSavePoint = firstSavePointInstantOption.get();
+    return getWriteTimeline()
+        .filter(instant -> instant.requestedTime().equals(firstSavePoint.requestedTime()))


🤖 If the write commit corresponding to the savepoint is missing (e.g., due to a corrupted timeline or manual deletion), this will return Option.empty(). Consequently, TimelineUtils.getEarliestInstantForMetadataArchival will silently ignore the savepoint and might incorrectly allow archiving past it. Consider returning the savepoint instant itself directly (or as a fallback).

yihua · 2026-04-05T05:28:18Z

-      if (earliestCommit.get().compareTo(earliestInflight.get()) < 0) {
-        return earliestCommit;
+    if (shouldArchiveBeyondSavepoint) {
+      return findSmallestInstant(earliestInflight, earliestPossibleRestoreCommit);


🤖 Good catch. If shouldArchiveBeyondSavepoint is enabled, we need to ensure that old savepoints don't inadvertently become a hard floor for the archival watermark, as that would defeat the purpose of the config and cause the timeline to grow.

Krishen Bhan added 2 commits February 17, 2026 15:36

mdt archive until ectr

21510e0

throw exception

fa65993

github-actions Bot added the size:M PR with lines of changes in (100, 300] label Feb 17, 2026

Update TestHoodieTimelineArchiver.java

31d7847

kbuci changed the title ~~fix(metadata table) Metadata table archival should proceed past earliest commit to retain.~~ perf(metadata table) Metadata table archival should proceed until earliest commit to retain on data table Feb 18, 2026

kbuci marked this pull request as ready for review February 18, 2026 20:34

danny0405 self-assigned this Feb 23, 2026

danny0405 reviewed Feb 24, 2026

View reviewed changes

yihua reviewed Apr 5, 2026

View reviewed changes

Conversation

kbuci commented Feb 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary and Changelog

Impact

Risk Level

Documentation Update

Contributor's checklist

Uh oh!

hudi-bot commented Feb 18, 2026

CI report:

Uh oh!

nsivabalan commented Feb 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danny0405 Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yihua left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yihua left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

kbuci commented Feb 17, 2026 •

edited

Loading

danny0405 Feb 24, 2026 •

edited

Loading