.Net: Improve OpenAIResponseAgent exception handling (rate limit, auth, content filter, model not found)#13011
Open
Yusuftmle wants to merge 10 commits into
Open
Conversation
github-actions
Bot
changed the title
Author
|
@microsoft-github-policy-service agree |
…proofing - Changed exception handling to catch all Exception types instead of only OpenAI-specific ones - This approach is more robust and won't miss new exception types from future SDK updates - Inner exceptions are preserved for detailed error analysis - Updated unit tests accordingly
Yusuftmle
force-pushed
the
improve-openai-exception-handling
branch
2 times, most recently
from
55698c5 to
67a7b9c
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR aims to improve how Microsoft.SemanticKernel.Agents.OpenAI.OpenAIResponseAgent surfaces provider failures by adding additional exception handling and introducing new unit tests for error mapping.
Changes:
- Added broad
try/catchblocks inOpenAIResponseAgent.InvokeAsync(...)andInvokeStreamingAsync(...)that wrap initialization failures inKernelException. - Refactored local variable setup in both invoke paths (non-streaming and streaming) to support the new exception handling shape.
- Added a new unit test class intended to validate exception mapping behavior.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.
| File | Description |
|---|---|
| dotnet/src/Agents/OpenAI/OpenAIResponseAgent.cs | Adds exception wrapping around invoke setup and adjusts streaming message notification flow. |
| dotnet/src/Agents/UnitTests/OpenAI/OpenAIResponseAgentExceptionTests.cs | Introduces new tests intended to validate error-to-message mapping for OpenAIResponseAgent. |
Comments suppressed due to low confidence (1)
dotnet/src/Agents/OpenAI/OpenAIResponseAgent.cs:90
- Exceptions thrown while enumerating
invokeResults(e.g., fromResponseThreadActions.InvokeAsyncduring the API call) are not covered by the try/catch, so provider failures can still escape without the intended mapping (and may still surface as the original problematic exception types). If the goal is to surface provider errors consistently, wrap theawait foreachenumeration (or use a local iterator that applies try/catch aroundMoveNextAsync).
// Yield results with additional error handling
await foreach (var result in invokeResults.ConfigureAwait(false))
{
if (options?.OnIntermediateMessage is not null)
{
await options.OnIntermediateMessage(result).ConfigureAwait(false);
}
await this.NotifyThreadOfNewMessage(agentThread, result, cancellationToken).ConfigureAwait(false);
yield return new(result, agentThread);
}
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+62
to
+78
| try | ||
| { | ||
| agentThread = await this.EnsureThreadExistsWithMessagesAsync(messages, thread, cancellationToken).ConfigureAwait(false); | ||
| extensionsContextOptions = await this.FinalizeInvokeOptionsAsync(messages, options, agentThread, cancellationToken).ConfigureAwait(false); | ||
|
|
||
| ChatHistory chatHistory = [.. messages]; | ||
| invokeResults = ResponseThreadActions.InvokeAsync( | ||
| this, | ||
| chatHistory, | ||
| agentThread, | ||
| extensionsContextOptions, | ||
| cancellationToken); | ||
| } | ||
| catch (Exception ex) | ||
| { | ||
| throw new KernelException($"OpenAI provider error for agent '{this.Name}': {ex.Message}", ex); | ||
| } |
Author
There was a problem hiding this comment.
The broad catch was suggested by @markwallace-microsoft
in a previous review. Happy to narrow it to
ClientResultException if preferred.
What's the recommended approach?
Comment on lines
+75
to
+78
| catch (Exception ex) | ||
| { | ||
| throw new KernelException($"OpenAI provider error for agent '{this.Name}': {ex.Message}", ex); | ||
| } |
Comment on lines
+141
to
+144
| await foreach (var result in invokeResults.ConfigureAwait(false)) | ||
| { | ||
| await NotifyMessagesAsync().ConfigureAwait(false); | ||
| yield return new(result, agentThread); |
Comment on lines
+18
to
+38
| [Fact] | ||
| public void InvokeAsync_ShouldMapRateLimitCorrectly() | ||
| { | ||
| var ex = new HttpRequestException("HTTP 429 Rate limit exceeded"); | ||
| KernelException? result = null; | ||
|
|
||
| try | ||
| { | ||
| if (ex.Message.Contains("429")) | ||
| throw new KernelException($"Rate limit exceeded for agent '{AgentName}'. Check Retry-After header and implement backoff.", ex); | ||
| } | ||
| catch (KernelException ke) | ||
| { | ||
| result = ke; | ||
| } | ||
|
|
||
| Assert.NotNull(result); | ||
| Assert.Contains("Rate limit exceeded", result.Message); | ||
| Assert.Contains("Retry-After header", result.Message); | ||
| Assert.Equal(ex, result.InnerException); | ||
| } |
Comment on lines
+161
to
+166
| var thrownException = Assert.ThrowsAsync<InvalidOperationException>(() => | ||
| { | ||
| throw ex; | ||
| }); | ||
|
|
||
| Assert.Equal("Unknown streaming exception", thrownException.Result.Message); |
Comment on lines
+1
to
+13
| using System; | ||
| using System.Net.Http; | ||
| using System.Threading; | ||
| using System.Threading.Tasks; | ||
| using Xunit; | ||
| using Microsoft.SemanticKernel; | ||
|
|
||
| namespace SemanticKernel.Agents.UnitTests.OpenAI | ||
| { | ||
| /// <summary> | ||
| /// Tests for the updated exception handling logic in OpenAIResponseAgent. | ||
| /// Verifies that KernelException messages are correct and unknown exceptions propagate. | ||
| /// </summary> |
Comment on lines
+141
to
+145
| await foreach (var result in invokeResults.ConfigureAwait(false)) | ||
| { | ||
| await NotifyMessagesAsync().ConfigureAwait(false); | ||
| yield return new(result, agentThread); | ||
| } |
Author
|
Hi @westey-m, just checking in after some time. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR improves exception handling for
Microsoft.SemanticKernel.Agents.OpenAI.OpenAIResponseAgent.Previously, calls to
InvokeAsync(...)andInvokeStreamingAsync(...)could collapse provider failures into a genericNullReferenceException, losing context such as rate limiting, content filter violations, authentication failures, or missing models. This made it difficult to implement robust retry logic and provide meaningful error messages.A key root cause was that
IAsyncEnumerableuses lazy evaluation — the actual network call doesn't happen at the method call site, but inside theawait foreachloop. This meant exceptions thrown during enumeration were escaping the originaltry-catchblocks entirely.Changes
HandleProviderExceptionsAsync— a thin async wrapper that intercepts exceptions during both the initialization phase andMoveNextAsynciteration, wrapping them inKernelExceptionwith descriptive messages.OperationCanceledExceptionis preserved and never wrapped.finallyblock to guaranteeDisposeAsyncis called on the enumerator even if the stream fails mid-way.ThrowingHttpMessageHandlerthat simulates real HTTP failures (429, 500) end-to-end against a realOpenAIResponseAgentinstance.Why
Fixes [#12976] by ensuring that provider errors are surfaced with meaningful messages instead of
NullReferenceException.Notes
dotnet test).Environment Tested