Conversation
| type Stats struct { | ||
| Accepted uint64 // Total number of accepted reports | ||
| Deduplicated uint64 // Total number of deduplicated reports when in HA | ||
| OutOfOrder uint64 // Total number of out-of-order reports seen |
go/stream.go
Outdated
| if ts.Equal(wm) { | ||
| s.stats.skipped.Add(1) | ||
| s.waterMarkMu.Unlock() | ||
| return nil | ||
| } |
There was a problem hiding this comment.
I don't follow. If we allow out of order, why are we skipping this one? Because it's a HA duplicate?
But if that's the intention, this will not work for out-of-order HA duplicates, causing uncertainty downstream.
If we want to catch those, we'd need a set of timestamps for which we have already received reports. We can have a background goroutine which trims the set by removing all entries older than, say, 5 seconds.
My advice here would be to either go with the set and have reliable filtering or remove the skip altogether when out-of-order reports are allowed. I'd prefer the former.
There was a problem hiding this comment.
good catch, i ended up rethinking the approach to handle these cases - extracted the dedup + watermark logic for clarity, which now maintains a fixed size buffer of 'seen' reports for dedup purposes
2 participants
add new config WsAllowOutOfOrder to all SDKs, default false.
this allows the client exporter to filter out HF report gaps resulting from out-of-order reports