Fix "metric was collected before" error#433
Fix "metric was collected before" error#433colinleroy wants to merge 2 commits intoprometheus-community:masterfrom
Conversation
Signed-Off-By: Colin Leroy-Mira <colin@colino.net>
The networksecurity.googleapis.com/https/request_count sometimes returns duplicate series. Filter them out to avoid errors. Signed-Off-By: Colin Leroy-Mira <colin@colino.net>
|
@colinleroy Lost other valid metrics. |
|
Were they metrics with at least one unique key/value label ? |
kgeckhart
left a comment
There was a problem hiding this comment.
Thanks for the PR. I have some concerns regarding the performance of the implementation as reflect deep equals in a double nested loop like this is going to be very expensive.
We have had to do something similar to this in the AWS CloudWatch exporter (https://github.com/kgeckhart/yet-another-cloudwatch-exporter/blob/master/pkg/promutil/migrate.go#L292-L316) but that implementation is much easier because it can hook in to the intermediate type used which has all the labels in a form that can be directly hashed.
The stackdriver exporter doesn't have a clean injection point or a clean representation. The easiest place is the metrics channel but at that point the data is already packaged as a prometheus.Metric where the labels are a []*LabelPair and must be copied to a new map[string]string to fingerprint. I don't have a quick answer ATM just thinking out loud.
Github issues that's likely related, #166
Hello,
Since one or two weeks, our stackdriver_exporter errors out with logs similar to:
After investigation, it comes from the fact that some of these metrics are returned twice by Stackdriver.
This PR adds a filtering to reportTimeSeriesMetrics() in order to avoid sending duplicate time series to the Prometheus Collector.
I hope this helps and I hope it's good enough, I'm not very well-versed in Go.
Thanks!