Skip to content

feat(dev/logs/metrics): Add sentry.timestamp.sequence definition to the logs and metrics specs#16492

Open
logaretm wants to merge 6 commits intomasterfrom
awad/add-log-sequence-attr
Open

feat(dev/logs/metrics): Add sentry.timestamp.sequence definition to the logs and metrics specs#16492
logaretm wants to merge 6 commits intomasterfrom
awad/add-log-sequence-attr

Conversation

@logaretm
Copy link
Member

@logaretm logaretm commented Feb 20, 2026

This pull request updates the telemetry logs specification to introduce deterministic log ordering via a new attribute, sentry.timestamp.sequence. This change addresses issues in environments where multiple logs can share identical timestamps, ensuring logs can always be ordered correctly.

The attribute addresses an issue that came up in Cloudflare’s runtime which freezes all timers and clock functions, causing all timestamp calls to return the same value. This can lead to logs being displayed out of order if they don’t get sent in one request. Instead, they’ll appear in the order of arrival in ingest, which is subject to request latencies and out-of-order request completion.

This is a proposal as we have yet to see what product thinks about this.

@vercel
Copy link

vercel bot commented Feb 20, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
develop-docs Ready Ready Preview, Comment Feb 25, 2026 5:36pm
1 Skipped Deployment
Project Deployment Actions Updated (UTC)
sentry-docs Ignored Ignored Preview Feb 25, 2026 5:36pm

Request Review

@logaretm logaretm changed the title feat: Add sentry.log.sequence definition to the logs' spec feat(dev/logs): Add sentry.log.sequence definition to the logs' spec Feb 20, 2026
Comment on lines 309 to 310
The sequence number **MUST NOT**:
- Be intentionally reset during the process lifetime.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You only need it as a discriminator per timestamp, so maybe that's enough as a constraint?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be a rare scenario, but If it does get reset during the lifetime of the SDK process then on environments like Cloudflare it will produce the same timestamps again since the time is frozen throughout their lifetime, so it won't be unique anymore. So, I wanted to close that door.

What do you think?

"sentry.message.parameter.1": { "value": 85, "type": "integer" },
"sentry.message.parameter.2": { "value": 60, "type": "integer" }
"sentry.message.parameter.2": { "value": 60, "type": "integer" },
"sentry.log.sequence": { "value": 0, "type": "integer" }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this should be top-level, it feels heavily tied to the timestamp and a fundamental attribute of a log.

@k-fish wdyt?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I namespaced it initially to signal to the user that they should not mess with this (they shouldn't be able anyways).

But yea I agree, it is tied to the logs timestamps heavily and has little to no-use outside of that.


The sequence number **MUST NOT**:
- Be intentionally reset during the process lifetime.
- Be used to modify or synthesize sub-millisecond timestamp precision.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why? Adding a separate attribute means to do log ordering naturally we'll now have to either:

A. Include yet another column on every request in order to have logs ordered properly when a users sorts by time (log.sequence), we already are using timestamp and precise timestamp when doing log ordering. Adding a column adds additional query bytes to every request.
B. NOT sort properly by this monotonic counter, and have it as a free floating attribute that a user will have to know about and then sort by independently, with a count from beginning of the lifetime being hard to intuit.

Is it possible we send it and have relay transform the precise timestamp into a synthesized version where we add an annotation (meta) to indicate the log sequence value it was modified by and why (because their sdk doesn't support precise timestamps). That way we can put a warning in the timestamp tooltip that explains it, and encode the sequence into the precise timestamp where we have an extra 32 bits being essentially unused at millisecond precision? Thoughts?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible we send it and have relay transform the precise timestamp into a synthesized version where we add an annotation (meta) to indicate the log sequence value it was modified by...

Nar suggested something similar about doing the synthesizing on the SDK level. I included that part to avoid lying to the user about the actual time. I think your suggestion is more transparent so we could do that, but it feels like it involves more changes downstream to the UI.

I have a few thoughts on this:

  1. We can't dynamically tell if the clock is frozen or not so we can't synthesize conditionally unless we send in another flag to tell relay to do that, but then again users cross-deploy a lot and they can have a node SDK running on Cloudflare, so the synthesizing would need to happen all the time. I initially wanted us to use this for the off chance some SDK got its clock frozen for whatever reason.
  2. We could skew the ms value but with 32 bit of space, it may not grow big enough to overflow into seconds range.

I would've liked for this to be completely transparent to the user, How considerable is the cost of adding an secondary sort integer to our queries?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Talked with Kevan offline and we figured a good compromise is to do a reset of the counter whenever we detect that the clock did advance, this would keep the sequence numbers small for most cases which means it won't grow larger enough to skew the numbers.

We also will monitor how large the sequence numbers grows to see if we should modify our approach.

Copy link
Member

@k-fish k-fish left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

General +1 that we need a monotonic counter if we can't get a more precise timestamp


<SpecSection id="log-ordering" status="candidate" since="1.16.0">

Some runtimes (notably Cloudflare Workers) freeze timer APIs (`Date.now()`, `performance.now()`) during synchronous execution, causing multiple logs to share identical timestamps. To guarantee deterministic ordering, SDKs **MUST** attach a `sentry.log.sequence` integer attribute to every log.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

re: sentry.log.sequence it actually is probably necessary for our other telemetry too, at least for metrics I know it could be a problem.

imo something like sentry.timestamp.sequence makes more sense to me, my $0.02

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually the clockfreeze thing on cloudflare does affect all of our telemetry.

  • Traces and spans always display 0ms since startTime and endTime are always the same, the counter wouldn't help with that tho.
  • Metrics are probably affected, but I haven't been exposed to them yet. I can explore further there if we want to make this a general telemetry attribute.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Talked with Kevan offline and we will do the same thing for metrics, so the attribute name will change to sentry.timestamp.sequence

@logaretm logaretm force-pushed the awad/add-log-sequence-attr branch from 7a3e04f to fbfa09e Compare February 24, 2026 17:25
@logaretm logaretm changed the title feat(dev/logs): Add sentry.log.sequence definition to the logs' spec feat(dev/logs): Add sentry.timestamp.sequence definition to the logs and metrics specs Feb 24, 2026
@logaretm logaretm changed the title feat(dev/logs): Add sentry.timestamp.sequence definition to the logs and metrics specs feat(dev/logs/metrics): Add sentry.timestamp.sequence definition to the logs and metrics specs Feb 24, 2026
@logaretm logaretm requested review from Dav1dde and k-fish February 24, 2026 20:13
Copy link
Member

@k-fish k-fish left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add one final piece about sequence not being sent if not required (the sdk has better precision than 1ms or non frozen time) then 👍 from me

@logaretm logaretm force-pushed the awad/add-log-sequence-attr branch from 60b43dc to 578eb4d Compare February 25, 2026 17:29
@logaretm logaretm marked this pull request as ready for review February 25, 2026 18:57
Copilot AI review requested due to automatic review settings February 25, 2026 18:57
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the SDK telemetry logs and metrics specifications to add a new attribute, sentry.timestamp.sequence, intended to enable deterministic ordering when multiple items share identical timestamps (e.g., in runtimes with frozen timers).

Changes:

  • Bumped spec versions and added changelog entries for introducing sentry.timestamp.sequence.
  • Added sentry.timestamp.sequence to the “default attributes” requirements and to envelope examples.
  • Introduced new “Log Ordering” / “Metric Ordering” sections describing how the sequence counter behaves.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 6 comments.

File Description
develop-docs/sdk/telemetry/metrics.mdx Adds sentry.timestamp.sequence to default metric attributes, documents metric ordering rules, updates examples, and bumps spec version/changelog.
develop-docs/sdk/telemetry/logs.mdx Adds sentry.timestamp.sequence to default log attributes, documents log ordering rules, updates examples, and bumps spec version/changelog.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines 150 to 160
SDKs **MUST** attach the following attributes to every log:

| Attribute | Since | Description |
|-----------|-------|-------------|
| `sentry.environment` | 1.0.0 | The environment set in the SDK, if defined. |
| `sentry.release` | 1.0.0 | The release set in the SDK, if defined. |
| `sentry.sdk.name` | 1.0.0 | The name of the SDK that sent the log. |
| `sentry.sdk.version` | 1.0.0 | The version of the SDK that sent the log. |
| `sentry.replay_id` | 1.8.0 | The replay ID of the active replay when the log was collected. **MUST NOT** be set if no replay is active. |
| `sentry.timestamp.sequence` | 1.16.0 | A monotonically incrementing integer counter used to determine correct ordering of logs when timestamps are identical. See [Log Ordering](#log-ordering). |

Copy link

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The default-attributes table makes sentry.timestamp.sequence a required attribute for every log, but the “Log Ordering” section later says SDKs MAY omit it when timestamps are already deterministic. Please align these statements so SDK implementers don’t get conflicting guidance (always-required vs conditional).

Copilot uses AI. Check for mistakes.
Comment on lines +301 to +312
Some runtimes (notably Cloudflare Workers) freeze timer APIs (`Date.now()`, `performance.now()`) during synchronous execution, causing multiple logs to share identical timestamps. In those cases, to guarantee deterministic ordering, SDKs **MUST** attach a `sentry.timestamp.sequence` integer attribute to every log.

The sequence number **MUST**:
- Start at `0` when the SDK initializes.
- Increment by `1` for each log that is captured.
- Reset to `0` when:
- The SDK is re-initialized.
- The current log's integer millisecond differs from the previous log's integer millisecond (i.e., `floor(timestamp_seconds * 1000)` changes).

SDKs **MAY** omit `sentry.timestamp.sequence` if the runtime provides sub-millisecond timestamp precision and does not freeze timer APIs, since ordering is already deterministic from timestamps alone.

The sequence provides deterministic ordering within a single SDK instance. It does not guarantee ordering across independent processes or workers, which have separate counters. The reset behavior ensures the sequence only increments for consecutive logs that share the same timestamp.
Copy link

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section requires sentry.timestamp.sequence only for frozen-timer runtimes, and also states SDKs MAY omit it in deterministic-timestamp environments. That contradicts the earlier “SDKs MUST attach … to every log” default-attributes requirement. Please clarify which rule is intended and adjust one of the sections accordingly.

Copilot uses AI. Check for mistakes.
version: ">=1.0.0"
spec_changelog:
- version: 2.6.0
date: 2026-02-06
Copy link

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the changelog, version 2.6.0 is dated 2026-02-06 but the previous 2.5.0 entry is dated 2026-02-12. This makes the timeline inconsistent with the version ordering; update the 2.6.0 date (or the 2.5.0 date) so newer versions don’t appear to precede older ones.

Suggested change
date: 2026-02-06
date: 2026-02-19

Copilot uses AI. Check for mistakes.
Comment on lines +120 to 121
5. `sentry.timestamp.sequence` — a monotonically incrementing integer counter used to determine correct ordering of metrics when timestamps are identical. See [Metric Ordering](#metric-ordering).

Copy link

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section states SDKs MUST attach sentry.timestamp.sequence to every metric by default, but the later “Metric Ordering” section says SDKs MAY omit it when timestamps are already deterministic. Please reconcile these requirements (either make it always-required, or remove it from the unconditional default-attributes list and describe it as conditional).

Suggested change
5. `sentry.timestamp.sequence` — a monotonically incrementing integer counter used to determine correct ordering of metrics when timestamps are identical. See [Metric Ordering](#metric-ordering).
SDKs that cannot otherwise guarantee deterministic ordering for metrics with identical timestamps **MUST** also attach `sentry.timestamp.sequence` — a monotonically incrementing integer counter used to determine correct ordering of metrics when timestamps are identical. See [Metric Ordering](#metric-ordering).

Copilot uses AI. Check for mistakes.
Comment on lines +198 to +209
Some runtimes (notably Cloudflare Workers) freeze timer APIs (`Date.now()`, `performance.now()`) during synchronous execution, causing multiple metrics to share identical timestamps. In those cases, to guarantee deterministic ordering, SDKs **MUST** attach a `sentry.timestamp.sequence` integer attribute to every metric.

The sequence number **MUST**:
- Start at `0` when the SDK initializes.
- Increment by `1` for each metric that is captured.
- Reset to `0` when:
- The SDK is re-initialized.
- The current metric's integer millisecond differs from the previous metric's integer millisecond (i.e., `floor(timestamp_seconds * 1000)` changes).

SDKs **MAY** omit `sentry.timestamp.sequence` if the runtime provides sub-millisecond timestamp precision and does not freeze timer APIs, since ordering is already deterministic from timestamps alone.

The sequence provides deterministic ordering within a single SDK instance. It does not guarantee ordering across independent processes or workers, which have separate counters. The reset behavior ensures the sequence only increments for consecutive metrics that share the same timestamp.
Copy link

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section says SDKs MUST attach sentry.timestamp.sequence “in those cases” (frozen timers), but elsewhere the spec now lists it as a default attribute for every metric. Also, the “SDKs MAY omit…” paragraph contradicts the unconditional requirement. Please clarify whether the attribute is always required or only required for runtimes without reliable sub-ms timestamps.

Copilot uses AI. Check for mistakes.
Comment on lines 588 to +592
"endpoint": { "value": "/api/users", "type": "string" },
"method": { "value": "POST", "type": "string" },
"sentry.sdk.name": { "value": "sentry.javascript.browser", "type": "string" },
"sentry.sdk.version": { "value": "10.17.0", "type": "string" }
"sentry.sdk.version": { "value": "10.17.0", "type": "string" },
"sentry.timestamp.sequence": { "value": 1, "type": "integer" }
Copy link

Copilot AI Feb 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In this full envelope example, the first metric item includes sentry.environment and sentry.release, but this item omits them even though they’re listed as default attributes (when defined) and would normally be consistent across metrics from the same SDK instance. Consider adding them here too, or removing them from the earlier item, so the example matches the spec rules.

Copilot uses AI. Check for mistakes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants