Add model_name, agent_name, and response_id to RequestUsage for better tracking#2114
Add model_name, agent_name, and response_id to RequestUsage for better tracking#2114ihower wants to merge 1 commit intoopenai:mainfrom
Conversation
| """The base interface for calling an LLM.""" | ||
|
|
||
| # The model name. Subclasses can set this in __init__. | ||
| model: str = "" |
There was a problem hiding this comment.
in fact, it seems available models have self.model but this could be potentially a breaking change for some custom models. it will require a minor version upgrade and release note annoucements
There was a problem hiding this comment.
Can we get rid of this property this time? When a developer uses a ModelProvider, this property could be absent, and it sounds surprisingly inconsistent.
src/agents/usage.py
Outdated
| def add( | ||
| self, | ||
| other: Usage, | ||
| model_name: str, |
There was a problem hiding this comment.
these properties are in the request usage data, so i don't think these data should be passed this way. at least, we should avoid passing these in order to keep flexibility of future enhancement (meaning having additional-properties argument, which can accept more properties in the future)
|
@seratch Thanks for the code review. I’ve updated the PR so that all of these fields go into a single Regarding Are you concerned about custom models that don’t inherit from If we want to avoid touching the Model class to fully eliminate this risk, I can switch to a runtime check when calling add(), something like: if hasattr(model, "model") and model.model:
metadata["model_name"] = model.modelI can update the PR if you prefer this approach. Just let me know. Thanks! |
|
This PR is stale because it has been open for 10 days with no activity. |
|
There is no rush but once this project starts working on 0.7 series changes, resolving the conflicts would be appreciated |
8ecd03b to
8ec30bd
Compare
|
@seratch Thanks! I’ve resolved the conflicts and rebased onto the latest main. |
8ec30bd to
4bd5a49
Compare
|
Rebased again on latest main to resolve new conflicts. |
| """The base interface for calling an LLM.""" | ||
|
|
||
| # The model name. Subclasses can set this in __init__. | ||
| model: str = "" |
There was a problem hiding this comment.
Can we get rid of this property this time? When a developer uses a ModelProvider, this property could be absent, and it sounds surprisingly inconsistent.
src/agents/run.py
Outdated
| context_wrapper.usage.add( | ||
| new_response.usage, | ||
| metadata={ | ||
| "model_name": model.model, |
There was a problem hiding this comment.
As mentioned above, I'd like to avoid having model_name this time. Also, I've added endpoint: str | None = None (responses.create, responses.compact, or None) to TS SDK for supporting responses.compact API: openai/openai-agents-js#785 We'll add the same to the Usage data class in Python SDK. So, for these SDKs' consistency plus better developer experience, I'd like to avoid dict[str, Any] here and simply having agent_name: str | None and response_id: str: None to request usage data.
There was a problem hiding this comment.
I’ve removed the metadata dict and now put the fields directly on Usage.
I think model_name is the most important part here. The original issue was about tracking model cost.
Since the base Model provides a default value of "", this should not be a breaking change.
If a developer uses a custom provider and doesn’t set a model name, it should be easy to add once when they notice. This is also consistent with the other built-in Model implementations in the SDK, which all provide a model name.
Thank you for reconsidering.
4bd5a49 to
1439c06
Compare
Resolved: #2100
This PR adds three useful fields to
RequestUsage:model_name,agent_name, andresponse_id.These fields make it easier to:
Note
model_nameis taken from theModel#modelattribute. All real subclasses ofModel(such asOpenAIResponsesModel,OpenAIChatCompletionsModel, andLitellmModel) do initialize amodelattribute.However, the parent
Modelclass does not define this field, which leads to type-checking errors. So this PR adds the attribute to the base class. To avoid breaking customModelimplementations (for example,FakeModelin tests that do not set a model name), the default value is an blank string.