Revamp Observability section #640
Open
abhijaisrivastava15 wants to merge 18 commits into
Open
Conversation
Pages rewritten with text matched to actual platform UI (Error Feed Demo project, Past 30D filter): - llm-tracing.mdx: 10 fresh screenshots; removed phantom Spans tab, fixed column list, restored Cost in span detail header. - session.mdx: 6 screenshots; corrected sessions grid columns (was 9 fictional, real is 6), fixed bulk-action list, rewrote replay flow to match the 3-step Create Scenarios dialog. - users.mdx: 5 screenshots; reduced 14-column claim to real 6, removed phantom bulk actions, documented per-user detail page. - evals.mdx: 2 screenshots; replaced multi-step wizard claim with the actual Create Task page (Spans/Traces/Sessions, Live Preview). - charts.mdx: 2 screenshots; fixed chart count (4 system + per-eval), switched description from dropdown to inline-button date range. - dashboard.mdx: 3 screenshots; documented dashboards list -> click to open -> +Add Widget flow. - alerts.mdx: 2 screenshots; renamed 'Create Alert' to '+ New Alert', fixed list columns. All screenshots are 3024x1614 retina captures via Puppeteer over CDP. Text written for engineer / PM / non-tech reader. Also adds the new 'Observability (Updated)' nav group in navigation.ts pointing at observe-two routes.
Voice Observability page rewritten against the live platform using a real voice project (Fast Food Order Agent_replay_agent_09_04_26 — Vapi inbound calls). 5 fresh Puppeteer screenshots captured: - voice-agent-definitions.png — Agent Definitions list with Voice Inbound / Voice Outbound / Chat agents and Vapi provider details. - voice-create-form.png — Create flow first step (3-step wizard: Basic Info -> Configuration -> Behaviour). - voice-projects-list.png — Tracing project list showing voice projects alongside text projects. - voice-tracing-overview.png — Voice project's call log table with voice-specific columns: Call Details, Status, Duration, Avg Latency, Turn Count, Tokens. - voice-call-detail.png — Call detail drawer with transcript, recording waveform, speaker filters, and the Call Analytics tab showing Talk Ratio, Latency Pipeline, and Cost Breakdown. Text fixes vs prior version: - Replaced single-form claim with the actual 3-step wizard. - Documented the full call-detail drawer (header chips, 4 tabs, analytics breakdown) — previous text only mentioned "transcript and recording URLs". - Updated provider list note (Vapi confirmed; Retell + others noted). - Added voice-specific column descriptions for the Tracing table.
LiveKit sets trace attributes that show up in the Tracing view but are not yet selectable as Filter or Breakdown options in Dashboard widgets. Verified the platform fix (PR #124, branch fix/th-4660-livekit-dashboard-attrs, 4 commits incl. af6b170 by Azain on 2026-04-28) is not merged into main. Adding the limitation note back to dashboard.mdx so users know to fall back to Tracing for LiveKit-specific filtering until the fix lands. Will be removed once PR #124 merges.
The new observe-two docs (rewrites + verified screenshots from earlier
commits on this branch) become the canonical /docs/observe/* docs.
The old hand-written /docs/observe/* pages are deleted.
Changes:
- Delete src/pages/docs/observe/ (the old pages and old manual-tracing).
- Rename src/pages/docs/observe-two/ -> src/pages/docs/observe/.
- Delete public/images/docs/observe/{1..5}.png, .webp (old assets).
- Rename public/images/docs/observe-two/ -> public/images/docs/observe/
(preserves all 36 verified screenshots taken on this branch).
- Update every internal reference inside the moved pages from
/docs/observe-two/ -> /docs/observe/ and /images/docs/observe-two/
-> /images/docs/observe/ (151 occurrences).
- src/lib/navigation.ts: remove the old "Observability" group; rename
the "Observability (Updated)" group to "Observability" and swap
its hrefs from /docs/observe-two/ to /docs/observe/.
Also fixes 3 external broken links that were already pointing at
nonexistent /docs/observe/ routes:
- src/pages/index.astro: /docs/observe/quickstart ->
/docs/observe/features/quickstart.
- src/pages/docs/simulation/features/voice-replay.mdx (x2):
/docs/observe/voice/set-up -> /docs/observe/features/voice.
- src/pages/docs/sdk/annotation-queues.mdx:
/docs/observe/features/annotation-queue-using-sdk ->
/docs/annotations/sdk/annotation-queue-using-sdk (the actual
location per navigation.ts).
Concept pages under /docs/tracing/concepts/ are referenced from the
new observe docs and are intentionally left untouched.
Verified: zero remaining "observe-two" references in src/ or public/.
…660) Incorporates the platform fix for TH-4660 / PR #124 into the docs. By the time these docs ship, the fix should be merged and live. Removes the "Known limitation (LiveKit users)" note added in 852fa89 and replaces it with positive guidance: - Filter section now explicitly says every span attribute visible in Tracing is filterable in Dashboard widgets, including LiveKit specific fields (room name, participant identity, custom). - Breakdown section reads similarly — group by any span attribute. Net effect: docs read as if LiveKit-set attributes Just Work as Filter and Breakdown options in dashboards.
The Overview page (/docs/observe) was still pointing at the old /images/observe_dashboard.png (a light-mode screenshot from the previous docs era). Swapped to /images/docs/observe/llm-tracing-overview.png — the dark-mode Tracing screenshot captured for the new Tracing page. Matches the rest of the rewritten observe docs visually.
…lign headings - Zoom-cropped 10 popup/dropdown screenshots so the modal is the subject rather than a tiny corner of a full dashboard view (date-range, filter, display, bulk-actions across llm-tracing/sessions/users) - Renamed top-level headings on all 8 observe feature pages to match the rest of the docs: 'What is this page?' -> 'About', and 'When would you use this?' -> 'When to use' Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
JayaSurya-27
requested changes
May 11, 2026
- Deleted charts.mdx and its two screenshots - Removed Charts entry from sidebar navigation - Removed Charts cards from Observe Overview, Dashboards, Alerts, Evals - Updated inline copy on evals.mdx and dashboard.mdx to point readers to Dashboards instead of Charts Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Contributor
|
HI @abhijaisrivastava15 can you make sure it follows the playbook i shared on the slack group |
Author
|
Yes, working on this |
Align the Observe/traceAI section with the Product Docs Playbook: - Rewrite in-scope pages to page-type skeletons + snake_case frontmatter; nav restructure (traceAI / Framework integrations / Reference / Troubleshooting). - Add Mermaid component (is:inline CDN import) so concept diagrams render. - Fix CodeGroup FOUC: SSR-hide non-active panels before JS initializes. - Reshoot screenshots from the Error Feed Demo project (synthetic data) as WebP. Accuracy pass against SDK source (future-agi/traceAI) + the live platform: - users: Users list has 6 real columns (User ID, First/Last Active, No. of Traces, No. of Sessions, Actions); move cost/evals/guardrails to the detail view. - semantic-conventions: FiSpanKindValues lists all 14 SDK values. - evals: 'Row limit'/matching rows (drop unverified spans_limit/default-1000); observation_type + filter fields confirmed against live eval tasks. Verified: astro build clean, 0 broken links, no banned words. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The mental-model diagram nested the retriever span under the tool span, which is an unrealistic trace shape. Rebalance: the agent runs a chain and a tool; the chain contains the retriever and LLM spans. Same teaching point (a child span can have children), accurate structure, symmetric layout. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Mermaid outputs a fixed-width block <svg> that sat flush-left. Add margin-inline:auto so diagrams center within their box site-wide. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add loading="lazy" to every <img> except the first (hero) on each page (alerts, users, evals, dashboard, voice, llm-tracing) — 13 images. - Remove duplicated sentences in trace-filter-syntax and export-formats. - Fix stale Users hero alt text to match the real column set. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The example trace nested a retriever span under a tool span (tool.check_order_status -> retriever.order_db) — the same unrealistic shape removed from the Spans diagram, and it misused the retriever span kind for a DB lookup. Route the retriever under a generate_reply chain doing genuine knowledge-base retrieval; keeps the nesting the prose describes, accurate span kinds, consistent with the Spans diagram. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
…source Verified against the platform backend (tracer/models), not just the SDK: - evals: restore `spans_limit` (default 1000) — the field is real (EvalTask.spans_limit = IntegerField(default=1000)); the UI labels it 'Row limit'. Reverts an over-correction that dropped the field name. - voice: add Eleven Labs to the supported providers. ProviderChoices and a dedicated tracer/utils/eleven_labs.py handler confirm it alongside Vapi and Retell; 'only Vapi and Retell' was incomplete. Also confirmed correct (no change): user.id.type = email/phone/uuid/custom (UserIdType), eval statuses = pending/running/completed/paused/failed/deleted (EvalTaskStatus). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Verified the voice/call columns against the frontend source (agents/helper.jsx baseColumns): Call Details, Status, Duration, Avg Latency, Turn Count, Tokens, Cost are the visible defaults (Talk Ratio is hidden). The doc listed the first six correctly but omitted Cost; added it. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
JayaSurya-27
requested changes
May 27, 2026
- Overview (index): lead with what Observe does / how it helps; drop the 'monitoring surface' jargon and the 'groups them' wording. - Concepts intro: the traceAI SDK is built on OpenTelemetry (not an either/or); spans form a trace via a shared trace ID (FutureAGI does not 'group' them); remove the 'nothing to observe' filler line. - Traces: drop 'in FutureAGI' (traces/OpenTelemetry are not ours); frame the support-agent example with 'for example'; clarify the tree, that the trace is the whole request and the root span is what started it; name 'eval scores' + link. - Spans: clarify spans are not optional or an alternative to traces. - Quickstart: rename 'Not seeing data?' to Troubleshooting. - Evals: retitle to 'Running Evals' (generic). Still open: trace/session-level eval content (needs @atharva-bhange).
Add a 'What you can evaluate' section to Running Evals covering the three eval levels — span, trace (with spans.0.* attribute referencing), and session — plus the context feature for handing an evaluator extra trace/span data. Content reviewed and approved by Atharva. Closes the last open review comment.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Revamp the Observe / tracing docs to conform to the new Product Docs Playbook (
future-agi/internal-docs→product-docs-playbook/). Every in-scope page is rewritten to its playbook page-type skeleton with snake_case frontmatter, the sidebar is restructured under traceAI, Mermaid concept diagrams are added, and screenshots are reshot from the Error Feed Demo project as WebP.What changed
Overview & quickstart
observe/index.mdx→ product-overview shapeobserve/features/quickstart.mdx→ quickstart shape (expected output, "Confirm the trace", "Not seeing data?")Feature deep-dives (
observe/features/) — rewritten to the feature-deep-dive skeletonllm-tracing), Sessions (session), Users (users), Dashboards (dashboard), Alerts (alerts), Production eval overlays (evals), Voice (voice)Concepts (
tracing/concepts/) — rewritten to the concept skeleton + Mermaid diagramsindex,traces,spans,otel,traceai, and a newsessions-and-userspageManual instrumentation how-tos (
observe/features/manual-tracing/, 14 pages) — reshaped to how-to skeletons (Prerequisites / Steps / Verify), CodeGroup tabs, verified API surface.New reference pages (
observe/reference/)trace-filter-syntax,dashboard-metric-definitions,export-formatsNew troubleshooting pages (
observe/troubleshooting/)no-traces-appearing,missing-attributes,dashboard-numbers-look-wrong,alerts-did-not-fireComponents / infra
Mermaid.astro(renders diagrams from CDN viais:inline), registered invite-docs-transform.mjsCodeGroup.astro— SSR-hide non-active panels to remove the tab flash (FOUC)navigation.ts— sidebar group renamed Observability → traceAI; added the Concepts (Sessions and Users), Reference, and Troubleshooting groupsloading="lazy"Accuracy pass
Every data claim was checked against the SDK source (
future-agi/traceAI) and the live platform:FiSpanKindValueslists all 14 SDK valuesspans_limit/ "default 1000");observation_typeand the other filter fields confirmed against live eval tasksTest plan
astro buildclean (935 pages)audit-links— 0 broken nav / content linksPending (out of scope for this PR to resolve)
user.id.typevalues (email/phone/uuid/custom); voice providers ("Vapi/Retell only") + voice columns; the full eval status enum