Add drop reason dimension to ingest/events/thrownAway metric #18855
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jtuglu1
force-pushed
the
ingest-dropped-event-metric-reason-dimension
branch
from
0a7d1ad to
76bbd91
Compare
jtuglu1
force-pushed
the
ingest-dropped-event-metric-reason-dimension
branch
from
76bbd91 to
36e0c72
Compare
jtuglu1
changed the title
jtuglu1
force-pushed
the
ingest-dropped-event-metric-reason-dimension
branch
3 times, most recently
from
c9c7e0d to
0ed4046
Compare
kfaraz
reviewed
Dec 19, 2025
Contributor
kfaraz
left a comment
kfaraz
left a comment
There was a problem hiding this comment.
Looks good, left some minor suggestions.
...service/src/main/java/org/apache/druid/indexing/common/stats/TaskRealtimeMetricsMonitor.java
Outdated
Show resolved
Hide resolved
indexing-service/src/main/java/org/apache/druid/indexing/common/task/RowFilter.java
Outdated
Show resolved
Hide resolved
indexing-service/src/main/java/org/apache/druid/indexing/common/task/RowFilter.java
Outdated
Show resolved
Hide resolved
indexing-service/src/main/java/org/apache/druid/indexing/common/task/RowFilter.java
Outdated
Show resolved
Hide resolved
processing/src/main/java/org/apache/druid/segment/incremental/ThrownAwayReason.java
Outdated
Show resolved
Hide resolved
processing/src/main/java/org/apache/druid/segment/incremental/ThrownAwayReason.java
Outdated
Show resolved
Hide resolved
processing/src/main/java/org/apache/druid/segment/incremental/ThrownAwayReason.java
Outdated
Show resolved
Hide resolved
processing/src/main/java/org/apache/druid/segment/incremental/ThrownAwayReason.java
Outdated
Show resolved
Hide resolved
processing/src/main/java/org/apache/druid/segment/incremental/ThrownAwayReason.java
Outdated
Show resolved
Hide resolved
...ce/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java
Outdated
Show resolved
Hide resolved
...ce/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java
Outdated
Show resolved
Hide resolved
...ce/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java
Outdated
Show resolved
Hide resolved
...ce/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java
Outdated
Show resolved
Hide resolved
...g-service/src/test/java/org/apache/druid/indexing/common/TaskRealtimeMetricsMonitorTest.java
Outdated
Show resolved
Hide resolved
jtuglu1
force-pushed
the
ingest-dropped-event-metric-reason-dimension
branch
from
9013d02 to
5ca36a7
Compare
jtuglu1
force-pushed
the
ingest-dropped-event-metric-reason-dimension
branch
3 times, most recently
from
9d8b12a to
1704dda
Compare
jtuglu1
force-pushed
the
ingest-dropped-event-metric-reason-dimension
branch
5 times, most recently
from
3a435d8 to
c8f8981
Compare
kfaraz
reviewed
Dec 22, 2025
Contributor
kfaraz
left a comment
kfaraz
left a comment
There was a problem hiding this comment.
Left some final suggestions.
...rvice/src/main/java/org/apache/druid/indexing/common/stats/DropwizardRowIngestionMeters.java
Outdated
Show resolved
Hide resolved
...service/src/main/java/org/apache/druid/indexing/common/stats/TaskRealtimeMetricsMonitor.java
Outdated
Show resolved
Hide resolved
indexing-service/src/main/java/org/apache/druid/indexing/common/task/InputRowFilter.java
Outdated
Show resolved
Hide resolved
...ce/src/main/java/org/apache/druid/indexing/seekablestream/SeekableStreamIndexTaskRunner.java
Outdated
Show resolved
Hide resolved
processing/src/main/java/org/apache/druid/segment/incremental/InputRowFilterResult.java
Show resolved
Hide resolved
processing/src/main/java/org/apache/druid/segment/incremental/InputRowFilterResult.java
Outdated
Show resolved
Hide resolved
processing/src/main/java/org/apache/druid/segment/incremental/InputRowFilterResult.java
Outdated
Show resolved
Hide resolved
processing/src/main/java/org/apache/druid/segment/incremental/RowIngestionMetersTotals.java
Outdated
Show resolved
Hide resolved
processing/src/main/java/org/apache/druid/segment/incremental/RowIngestionMetersTotals.java
Show resolved
Hide resolved
processing/src/main/java/org/apache/druid/segment/incremental/RowIngestionMetersTotals.java
Outdated
Show resolved
Hide resolved
abhishekrb19
approved these changes
Dec 22, 2025
Contributor
abhishekrb19
left a comment
abhishekrb19
left a comment
There was a problem hiding this comment.
LGTM, left a few minor comments - thanks!
processing/src/main/java/org/apache/druid/segment/incremental/InputRowFilterResult.java
Outdated
Show resolved
Hide resolved
processing/src/main/java/org/apache/druid/segment/incremental/InputRowFilterResult.java
Outdated
Show resolved
Hide resolved
processing/src/main/java/org/apache/druid/segment/incremental/InputRowFilterResult.java
Outdated
Show resolved
Hide resolved
processing/src/main/java/org/apache/druid/segment/incremental/InputRowFilterResult.java
Outdated
Show resolved
Hide resolved
processing/src/main/java/org/apache/druid/segment/incremental/InputRowFilterResult.java
Outdated
Show resolved
Hide resolved
jtuglu1
force-pushed
the
ingest-dropped-event-metric-reason-dimension
branch
from
78aaeae to
9a2d432
Compare
jtuglu1
force-pushed
the
ingest-dropped-event-metric-reason-dimension
branch
4 times, most recently
from
e2a2eea to
e489214
Compare
jtuglu1
force-pushed
the
ingest-dropped-event-metric-reason-dimension
branch
from
e489214 to
62440f3
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
I've found it hard to build observability on logically-dropped events (either due to transform, null, or temporal conditions).
This adds:
reasondimension to theingest/events/thrownAwaymetric for aggregating on thrown away reason.thrownAwayByReasonmap to row statistics task API response payload, so future consumers can make use of it should they need to.Release note
Add reason dimension to ingest/events/thrownAway metric. This allows for increased observability on why certain events are being logically excluded from ingest.
This PR has: