feat(dev/logs/metrics): Add sentry.timestamp.sequence definition to the logs and metrics specs#16492
feat(dev/logs/metrics): Add sentry.timestamp.sequence definition to the logs and metrics specs#16492
sentry.timestamp.sequence definition to the logs and metrics specs#16492Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
1 Skipped Deployment
|
sentry.log.sequence definition to the logs' specsentry.log.sequence definition to the logs' spec
develop-docs/sdk/telemetry/logs.mdx
Outdated
| The sequence number **MUST NOT**: | ||
| - Be intentionally reset during the process lifetime. |
There was a problem hiding this comment.
You only need it as a discriminator per timestamp, so maybe that's enough as a constraint?
There was a problem hiding this comment.
This might be a rare scenario, but If it does get reset during the lifetime of the SDK process then on environments like Cloudflare it will produce the same timestamps again since the time is frozen throughout their lifetime, so it won't be unique anymore. So, I wanted to close that door.
What do you think?
develop-docs/sdk/telemetry/logs.mdx
Outdated
| "sentry.message.parameter.1": { "value": 85, "type": "integer" }, | ||
| "sentry.message.parameter.2": { "value": 60, "type": "integer" } | ||
| "sentry.message.parameter.2": { "value": 60, "type": "integer" }, | ||
| "sentry.log.sequence": { "value": 0, "type": "integer" } |
There was a problem hiding this comment.
I wonder if this should be top-level, it feels heavily tied to the timestamp and a fundamental attribute of a log.
@k-fish wdyt?
There was a problem hiding this comment.
I namespaced it initially to signal to the user that they should not mess with this (they shouldn't be able anyways).
But yea I agree, it is tied to the logs timestamps heavily and has little to no-use outside of that.
develop-docs/sdk/telemetry/logs.mdx
Outdated
|
|
||
| The sequence number **MUST NOT**: | ||
| - Be intentionally reset during the process lifetime. | ||
| - Be used to modify or synthesize sub-millisecond timestamp precision. |
There was a problem hiding this comment.
Why? Adding a separate attribute means to do log ordering naturally we'll now have to either:
A. Include yet another column on every request in order to have logs ordered properly when a users sorts by time (log.sequence), we already are using timestamp and precise timestamp when doing log ordering. Adding a column adds additional query bytes to every request.
B. NOT sort properly by this monotonic counter, and have it as a free floating attribute that a user will have to know about and then sort by independently, with a count from beginning of the lifetime being hard to intuit.
Is it possible we send it and have relay transform the precise timestamp into a synthesized version where we add an annotation (meta) to indicate the log sequence value it was modified by and why (because their sdk doesn't support precise timestamps). That way we can put a warning in the timestamp tooltip that explains it, and encode the sequence into the precise timestamp where we have an extra 32 bits being essentially unused at millisecond precision? Thoughts?
There was a problem hiding this comment.
Is it possible we send it and have relay transform the precise timestamp into a synthesized version where we add an annotation (meta) to indicate the log sequence value it was modified by...
Nar suggested something similar about doing the synthesizing on the SDK level. I included that part to avoid lying to the user about the actual time. I think your suggestion is more transparent so we could do that, but it feels like it involves more changes downstream to the UI.
I have a few thoughts on this:
- We can't dynamically tell if the clock is frozen or not so we can't synthesize conditionally unless we send in another flag to tell relay to do that, but then again users cross-deploy a lot and they can have a node SDK running on Cloudflare, so the synthesizing would need to happen all the time. I initially wanted us to use this for the off chance some SDK got its clock frozen for whatever reason.
- We could skew the ms value but with 32 bit of space, it may not grow big enough to overflow into seconds range.
I would've liked for this to be completely transparent to the user, How considerable is the cost of adding an secondary sort integer to our queries?
There was a problem hiding this comment.
Talked with Kevan offline and we figured a good compromise is to do a reset of the counter whenever we detect that the clock did advance, this would keep the sequence numbers small for most cases which means it won't grow larger enough to skew the numbers.
We also will monitor how large the sequence numbers grows to see if we should modify our approach.
k-fish
left a comment
There was a problem hiding this comment.
General +1 that we need a monotonic counter if we can't get a more precise timestamp
develop-docs/sdk/telemetry/logs.mdx
Outdated
|
|
||
| <SpecSection id="log-ordering" status="candidate" since="1.16.0"> | ||
|
|
||
| Some runtimes (notably Cloudflare Workers) freeze timer APIs (`Date.now()`, `performance.now()`) during synchronous execution, causing multiple logs to share identical timestamps. To guarantee deterministic ordering, SDKs **MUST** attach a `sentry.log.sequence` integer attribute to every log. |
There was a problem hiding this comment.
re: sentry.log.sequence it actually is probably necessary for our other telemetry too, at least for metrics I know it could be a problem.
imo something like sentry.timestamp.sequence makes more sense to me, my $0.02
There was a problem hiding this comment.
Actually the clockfreeze thing on cloudflare does affect all of our telemetry.
- Traces and spans always display 0ms since
startTimeandendTimeare always the same, the counter wouldn't help with that tho. - Metrics are probably affected, but I haven't been exposed to them yet. I can explore further there if we want to make this a general telemetry attribute.
There was a problem hiding this comment.
Talked with Kevan offline and we will do the same thing for metrics, so the attribute name will change to sentry.timestamp.sequence
7a3e04f to
fbfa09e
Compare
sentry.log.sequence definition to the logs' specsentry.timestamp.sequence definition to the logs and metrics specs
sentry.timestamp.sequence definition to the logs and metrics specssentry.timestamp.sequence definition to the logs and metrics specs
k-fish
left a comment
There was a problem hiding this comment.
Add one final piece about sequence not being sent if not required (the sdk has better precision than 1ms or non frozen time) then 👍 from me
60b43dc to
578eb4d
Compare
There was a problem hiding this comment.
Pull request overview
This PR updates the SDK telemetry logs and metrics specifications to add a new attribute, sentry.timestamp.sequence, intended to enable deterministic ordering when multiple items share identical timestamps (e.g., in runtimes with frozen timers).
Changes:
- Bumped spec versions and added changelog entries for introducing
sentry.timestamp.sequence. - Added
sentry.timestamp.sequenceto the “default attributes” requirements and to envelope examples. - Introduced new “Log Ordering” / “Metric Ordering” sections describing how the sequence counter behaves.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 6 comments.
| File | Description |
|---|---|
| develop-docs/sdk/telemetry/metrics.mdx | Adds sentry.timestamp.sequence to default metric attributes, documents metric ordering rules, updates examples, and bumps spec version/changelog. |
| develop-docs/sdk/telemetry/logs.mdx | Adds sentry.timestamp.sequence to default log attributes, documents log ordering rules, updates examples, and bumps spec version/changelog. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| SDKs **MUST** attach the following attributes to every log: | ||
|
|
||
| | Attribute | Since | Description | | ||
| |-----------|-------|-------------| | ||
| | `sentry.environment` | 1.0.0 | The environment set in the SDK, if defined. | | ||
| | `sentry.release` | 1.0.0 | The release set in the SDK, if defined. | | ||
| | `sentry.sdk.name` | 1.0.0 | The name of the SDK that sent the log. | | ||
| | `sentry.sdk.version` | 1.0.0 | The version of the SDK that sent the log. | | ||
| | `sentry.replay_id` | 1.8.0 | The replay ID of the active replay when the log was collected. **MUST NOT** be set if no replay is active. | | ||
| | `sentry.timestamp.sequence` | 1.16.0 | A monotonically incrementing integer counter used to determine correct ordering of logs when timestamps are identical. See [Log Ordering](#log-ordering). | | ||
|
|
There was a problem hiding this comment.
The default-attributes table makes sentry.timestamp.sequence a required attribute for every log, but the “Log Ordering” section later says SDKs MAY omit it when timestamps are already deterministic. Please align these statements so SDK implementers don’t get conflicting guidance (always-required vs conditional).
| Some runtimes (notably Cloudflare Workers) freeze timer APIs (`Date.now()`, `performance.now()`) during synchronous execution, causing multiple logs to share identical timestamps. In those cases, to guarantee deterministic ordering, SDKs **MUST** attach a `sentry.timestamp.sequence` integer attribute to every log. | ||
|
|
||
| The sequence number **MUST**: | ||
| - Start at `0` when the SDK initializes. | ||
| - Increment by `1` for each log that is captured. | ||
| - Reset to `0` when: | ||
| - The SDK is re-initialized. | ||
| - The current log's integer millisecond differs from the previous log's integer millisecond (i.e., `floor(timestamp_seconds * 1000)` changes). | ||
|
|
||
| SDKs **MAY** omit `sentry.timestamp.sequence` if the runtime provides sub-millisecond timestamp precision and does not freeze timer APIs, since ordering is already deterministic from timestamps alone. | ||
|
|
||
| The sequence provides deterministic ordering within a single SDK instance. It does not guarantee ordering across independent processes or workers, which have separate counters. The reset behavior ensures the sequence only increments for consecutive logs that share the same timestamp. |
There was a problem hiding this comment.
This section requires sentry.timestamp.sequence only for frozen-timer runtimes, and also states SDKs MAY omit it in deterministic-timestamp environments. That contradicts the earlier “SDKs MUST attach … to every log” default-attributes requirement. Please clarify which rule is intended and adjust one of the sections accordingly.
| version: ">=1.0.0" | ||
| spec_changelog: | ||
| - version: 2.6.0 | ||
| date: 2026-02-06 |
There was a problem hiding this comment.
In the changelog, version 2.6.0 is dated 2026-02-06 but the previous 2.5.0 entry is dated 2026-02-12. This makes the timeline inconsistent with the version ordering; update the 2.6.0 date (or the 2.5.0 date) so newer versions don’t appear to precede older ones.
| date: 2026-02-06 | |
| date: 2026-02-19 |
| 5. `sentry.timestamp.sequence` — a monotonically incrementing integer counter used to determine correct ordering of metrics when timestamps are identical. See [Metric Ordering](#metric-ordering). | ||
|
|
There was a problem hiding this comment.
This section states SDKs MUST attach sentry.timestamp.sequence to every metric by default, but the later “Metric Ordering” section says SDKs MAY omit it when timestamps are already deterministic. Please reconcile these requirements (either make it always-required, or remove it from the unconditional default-attributes list and describe it as conditional).
| 5. `sentry.timestamp.sequence` — a monotonically incrementing integer counter used to determine correct ordering of metrics when timestamps are identical. See [Metric Ordering](#metric-ordering). | |
| SDKs that cannot otherwise guarantee deterministic ordering for metrics with identical timestamps **MUST** also attach `sentry.timestamp.sequence` — a monotonically incrementing integer counter used to determine correct ordering of metrics when timestamps are identical. See [Metric Ordering](#metric-ordering). |
| Some runtimes (notably Cloudflare Workers) freeze timer APIs (`Date.now()`, `performance.now()`) during synchronous execution, causing multiple metrics to share identical timestamps. In those cases, to guarantee deterministic ordering, SDKs **MUST** attach a `sentry.timestamp.sequence` integer attribute to every metric. | ||
|
|
||
| The sequence number **MUST**: | ||
| - Start at `0` when the SDK initializes. | ||
| - Increment by `1` for each metric that is captured. | ||
| - Reset to `0` when: | ||
| - The SDK is re-initialized. | ||
| - The current metric's integer millisecond differs from the previous metric's integer millisecond (i.e., `floor(timestamp_seconds * 1000)` changes). | ||
|
|
||
| SDKs **MAY** omit `sentry.timestamp.sequence` if the runtime provides sub-millisecond timestamp precision and does not freeze timer APIs, since ordering is already deterministic from timestamps alone. | ||
|
|
||
| The sequence provides deterministic ordering within a single SDK instance. It does not guarantee ordering across independent processes or workers, which have separate counters. The reset behavior ensures the sequence only increments for consecutive metrics that share the same timestamp. |
There was a problem hiding this comment.
This section says SDKs MUST attach sentry.timestamp.sequence “in those cases” (frozen timers), but elsewhere the spec now lists it as a default attribute for every metric. Also, the “SDKs MAY omit…” paragraph contradicts the unconditional requirement. Please clarify whether the attribute is always required or only required for runtimes without reliable sub-ms timestamps.
| "endpoint": { "value": "/api/users", "type": "string" }, | ||
| "method": { "value": "POST", "type": "string" }, | ||
| "sentry.sdk.name": { "value": "sentry.javascript.browser", "type": "string" }, | ||
| "sentry.sdk.version": { "value": "10.17.0", "type": "string" } | ||
| "sentry.sdk.version": { "value": "10.17.0", "type": "string" }, | ||
| "sentry.timestamp.sequence": { "value": 1, "type": "integer" } |
There was a problem hiding this comment.
In this full envelope example, the first metric item includes sentry.environment and sentry.release, but this item omits them even though they’re listed as default attributes (when defined) and would normally be consistent across metrics from the same SDK instance. Consider adding them here too, or removing them from the earlier item, so the example matches the spec rules.
This pull request updates the telemetry logs specification to introduce deterministic log ordering via a new attribute,
sentry.timestamp.sequence. This change addresses issues in environments where multiple logs can share identical timestamps, ensuring logs can always be ordered correctly.The attribute addresses an issue that came up in Cloudflare’s runtime which freezes all timers and clock functions, causing all timestamp calls to return the same value. This can lead to logs being displayed out of order if they don’t get sent in one request. Instead, they’ll appear in the order of arrival in ingest, which is subject to request latencies and out-of-order request completion.
This is a proposal as we have yet to see what product thinks about this.