[fix](azure) Fix incorrect modification timestamps in AzureObjStorage#61790
Open
dataroaring wants to merge 1 commit intoapache:masterfrom
Open
[fix](azure) Fix incorrect modification timestamps in AzureObjStorage#61790dataroaring wants to merge 1 commit intoapache:masterfrom
dataroaring wants to merge 1 commit intoapache:masterfrom
Conversation
The modification time for Azure blobs was computed incorrectly in three places: - globList LIST path (line 428): used getLastModified().getSecond() which returns the second-of-minute (0-59), not an epoch timestamp. - globListByGetProperties HEAD path (line 503): used toEpochSecond() which returns epoch seconds, off by 1000x from S3's toEpochMilli(). - listFiles path (line 564): same getSecond() bug as the glob path. All three are now fixed to use .toInstant().toEpochMilli(), consistent with S3ObjStorage which uses Instant.toEpochMilli(). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
There was a problem hiding this comment.
Pull request overview
Fixes incorrect RemoteFile.modificationTime values produced by the Azure object storage backend by normalizing Azure “last modified” timestamps to epoch milliseconds, aligning behavior with S3 and other file systems.
Changes:
- Fix
globListLIST-path timestamp bug (previously usedgetSecond()which is second-of-minute). - Fix deterministic-path
globListByGetProperties(HEAD/getProperties) timestamp unit from epoch seconds to epoch milliseconds. - Fix
listFilestimestamp bug (samegetSecond()issue).
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+428
to
+429
| isPrefix ? 0 : blobItem.getProperties().getLastModified() | ||
| .toInstant().toEpochMilli()); |
There was a problem hiding this comment.
Add/extend unit tests (e.g., AzureObjStorageTest’s mocked paged listing) to assert RemoteFile.modificationTime is an epoch-millisecond value derived from BlobItemProperties.getLastModified().toInstant().toEpochMilli(), so regressions back to second-of-minute/epoch-seconds are caught.
Comment on lines
503
to
505
| props.getLastModified() != null | ||
| ? props.getLastModified().toEpochSecond() : 0 | ||
| ? props.getLastModified().toInstant().toEpochMilli() : 0 | ||
| ); |
There was a problem hiding this comment.
Add test coverage for the deterministic-path getProperties/HEAD flow to validate the modification time unit is epoch milliseconds (not epoch seconds). Using a fixed OffsetDateTime in mocks will make the assertion deterministic.
Comment on lines
563
to
566
| props.getContentLength(), | ||
| props.getContentLength(), | ||
| props.getLastModified().getSecond(), | ||
| props.getLastModified().toInstant().toEpochMilli(), | ||
| null); |
There was a problem hiding this comment.
Consider adding a unit test around listFiles() (or a small isolated helper) to assert BlobItemProperties.getLastModified is converted to epoch milliseconds, since this path previously returned an incorrect value and is easy to regress.
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 28215 ms |
TPC-DS: Total hot run time: 168707 ms |
Contributor
FE UT Coverage ReportIncrement line coverage |
CalvinKirs
approved these changes
Mar 27, 2026
github-actions
bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix incorrect modification timestamps in
AzureObjStorageacross three code paths:globListLIST path: UsedgetLastModified().getSecond()which returns the second-of-minute (0–59), not an epoch timestampglobListByGetPropertiesHEAD path: UsedtoEpochSecond()which returns epoch seconds — off by 1000× from S3'stoEpochMilli()listFilespath: SamegetSecond()bug as the glob pathAll three are fixed to use
.toInstant().toEpochMilli(), consistent withS3ObjStoragewhich usesInstant.toEpochMilli().Found via review of #61775 (cherry-pick of #60414).
Test plan
RemoteFile.modifiedTime🤖 Generated with Claude Code