feat: Implement Arrow Flight Middleware support

[CurtHagenlocher]

Enhancement: Apache Arrow Flight Middleware in C#

Co-authored-by: @HackPoint
Original change in apache/arrow#46316.

Overview

This Pull Request enhances middleware support for Apache Arrow Flight using C#, focusing on improved metadata header management and propagation for better observability and extensibility. It also provides handling for HTTP/HTTPS communication.

Rationale for this Change

Effective middleware is critical for managing metadata headers, ensuring accurate request/response handling, and simplifying debugging in distributed systems. By improving middleware capabilities, we enhance reliability and observability, significantly benefiting developers and operational teams managing complex Flight-based applications.

What's Included in this PR?

• Middleware enhancements supporting complete metadata header propagation.
• Middleware lifecycle hooks for better request/response management.
• Comprehensive integration tests validating middleware functionality.
• Documentation updates reflecting middleware improvements.

Key Features

• Complete Header Propagation: Ensures accurate propagation of gRPC metadata headers throughout middleware lifecycle events.
• HTTP/HTTPS Handling: Supports middleware integration and metadata propagation for HTTP and HTTPS communications.
• Middleware Lifecycle Management: Supports reliable middleware hooks (OnBeforeSendingHeaders, OnHeadersReceived, OnCallCompleted).
• Enhanced Testing: Adds comprehensive integration tests to verify correct middleware behavior.

Impact

• Improves middleware reliability and simplifies debugging.
• Enhances transparency in gRPC and HTTP/S communication within Flight-based applications.

Are These Changes Tested?

Testing Overview

Unit Tests:

• Added tests for middleware lifecycle event execution (e.g., OnBeforeSendingHeaders, OnHeadersReceived, OnCallCompleted).
• Verified internal logic for capturing and storing gRPC metadata headers.

Integration Tests:

• Tested end-to-end with a real Flight client and in-memory server setup.
• Validated propagation of custom headers (e.g., x-server-header, Set-Cookie) between client and server.

End-to-End Tests:

• Simulated real-world Flight requests to ensure headers are processed consistently across middleware layers.
• Confirmed correct invocation order and middleware behavior under different server responses.

Example Test Cases:

• Verify that OnHeadersReceived correctly captures server-sent headers.
• Ensure custom client middleware modifies request headers as expected.
• Validate that OnCallCompleted is triggered on both success and error cases.

Checklist

• Implementation completed
• Tests added and passing

Closes GitHub Issue: #138

[HackPoint]

@CurtHagenlocher
Hi Curt — what’s your preferred approach for cleaning up the lint issues? Should I fork the repo and open a PR, or clone and work on a branch if I have permissions?

[CurtHagenlocher]

@CurtHagenlocher Hi Curt — what’s your preferred approach for cleaning up the lint issues? Should I fork the repo and open a PR, or clone and work on a branch if I have permissions?

I'm not great at Git/Github strategies. If I were you, I'd probably get a copy of my branch by cloning it and then just overwriting the files in a branch of your own.

[HackPoint]

@CurtHagenlocher I ran dotnet build -p:EnforceCodeStyleInBuild=true and didn’t see any issues. Which tool/command should I run to surface the linting errors? If there aren’t any, could you please merge it?

[CurtHagenlocher]

@CurtHagenlocher I ran dotnet build -p:EnforceCodeStyleInBuild=true and didn’t see any issues. Which tool/command should I run to surface the linting errors? If there aren’t any, could you please merge it?

Oh, I'm sorry if that wasn't clear but I'd fixed the linter errors on my branch in order to get the PR to green.

[HackPoint]

@CurtHagenlocher Hi Curt, can you please merge the PR then? I'm waiting for a year for this to be merged.