docs(tracing): add Long-running workers section#2839
docs(tracing): add Long-running workers section#2839nanookclaw wants to merge 1 commit intoopenai:mainfrom
Conversation
Document force_flush() pattern for Celery workers, FastAPI background tasks, RQ, Dramatiq, and other long-lived processes where traces are never exported because process shutdown (which normally triggers the BatchTraceProcessor flush) never occurs between tasks. Closes openai#2135
github-actions
bot
added
the
documentation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: e0b9672564
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| are buffered by the default `BatchTraceProcessor` and only flushed automatically on process | ||
| shutdown. Because these processes never exit between tasks, buffered traces may never be exported | ||
| to the Traces dashboard. |
There was a problem hiding this comment.
This section states that BatchTraceProcessor flushes automatically only on process shutdown and that traces in long-lived workers may never export, but the runtime currently does periodic background exports (_run flushes on schedule delay and queue threshold) even without shutdown. As written, this can mislead users into adding per-task force_flush() calls that block each job and unnecessarily reduce throughput. Please update the wording to reflect that force_flush() is for immediate delivery guarantees, not because automatic export never happens.
Useful? React with 👍 / 👎.
2 participants
Summary
In long-lived processes (Celery workers, FastAPI background tasks, RQ, Dramatiq), traces created with
agents.trace()are buffered byBatchTraceProcessorbut never exported — the flush that normally happens on process shutdown never fires between tasks.This is a well-known pain point (#2135, 5 comments, reported Nov 2025), but
docs/tracing.mdcurrently has no mention of it. The section under "Additional notes" is just "View free traces at OpenAI Traces dashboard."This PR adds a Long-running workers section that:
get_trace_provider().force_flush()pattern using the existing public APIforce_flush()is blocking (call it after the trace context exits)Relationship to PR #2735
PR #2735 proposes adding a dedicated
flush_traces()top-level convenience function. This docs PR is complementary — it documents the underlying mechanism thatflush_traces()would wrap, using the already-publicget_trace_provider().force_flush()API. If #2735 ships, this section can be updated to use the simpler import.Code references
get_trace_provider():src/agents/tracing/__init__.pyTraceProvider.force_flush():src/agents/tracing/provider.py:119BatchTraceProcessor.force_flush():src/agents/tracing/processors.py:547Closes #2135