Add headerForward to MCPRemoteProxy CRD

[jhrozek]

Summary

Add support for forwarding custom headers to remote MCP servers via MCPRemoteProxy. Headers can be specified as plaintext values or resolved from Kubernetes Secrets at runtime.

New CRD Fields

spec:
  headerForward:
    addPlaintextHeaders:     # Static headers (non-sensitive)
      X-Tenant-ID: "tenant-123"
    addHeadersFromSecret:    # Headers from K8s Secrets
      - headerName: "X-API-Key"
        valueSecretRef:
          name: api-key-secret
          key: api-key

Commits

1. Allow TOOLHIVE_SECRETS_PROVIDER env var to bypass SetupCompleted check

Previously, GetProviderType() checked SetupCompleted before checking the environment variable, causing Kubernetes deployments to fail when using environment-injected secrets.

Reorders the checks so the environment variable is checked first, allowing Kubernetes deployments to specify TOOLHIVE_SECRETS_PROVIDER=environment without requiring local secrets setup.

2. Add headerForward to MCPRemoteProxy CRD

Implementation:

• Operator creates env vars with SecretKeyRef for secret-backed headers
• Operator sets TOOLHIVE_SECRETS_PROVIDER=environment to enable the EnvironmentProvider in the runner
• RunConfig stores secret identifiers (not values)
• Runner resolves secrets via EnvironmentProvider at startup
• Header forward middleware adds headers to outgoing requests

Security:

• Secret values never stored in ConfigMaps (only identifiers)
• kubectl describe shows <set to key 'x' in secret 'y'>
• resolvedHeaders field is not serialized to disk

Example Manifests

Secret-backed headers only

apiVersion: toolhive.stacklok.dev/v1alpha1
kind: MCPRemoteProxy
metadata:
  name: secret-headers-proxy
spec:
  remoteURL: http://mcp-backend.default.svc.cluster.local:8080/mcp
  transport: streamable-http
  oidcConfig:
    type: kubernetes
    kubernetes:
      issuer: https://kubernetes.default.svc.cluster.local
      audience: https://kubernetes.default.svc.cluster.local
  headerForward:
    addHeadersFromSecret:
      - headerName: "X-API-Key"
        valueSecretRef:
          name: api-key-secret
          key: api-key
      - headerName: "Authorization"
        valueSecretRef:
          name: auth-token-secret
          key: token

Mixed plaintext and secret headers

apiVersion: toolhive.stacklok.dev/v1alpha1
kind: MCPRemoteProxy
metadata:
  name: mixed-headers-proxy
spec:
  remoteURL: http://mcp-backend.default.svc.cluster.local:8080/mcp
  transport: streamable-http
  oidcConfig:
    type: kubernetes
    kubernetes:
      issuer: https://kubernetes.default.svc.cluster.local
      audience: https://kubernetes.default.svc.cluster.local
  headerForward:
    addPlaintextHeaders:
      X-Tenant-ID: "tenant-456"
      X-Request-Source: "toolhive-proxy"
    addHeadersFromSecret:
      - headerName: "X-API-Key"
        valueSecretRef:
          name: api-key-secret
          key: api-key

Test plan

Tested in kind cluster using yardstick MCP server (patched to log incoming headers):

• Plaintext headers only
• Secret-backed headers only
• Mixed plaintext and secret headers
• Multiple secrets, multiple keys from same secret
• Missing secret (graceful failure with clear error)

Verified headers appear in yardstick logs:

Header: X-Api-Key: sk-test-api-key-12345
Header: Authorization: Bearer super-secret-token-67890
Header: X-Tenant-Id: tenant-456

[codecov]

Codecov Report

❌ Patch coverage is 95.91837% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.88%. Comparing base (e6d3bde) to head (c8699d0).

Files with missing lines	Patch %	Lines
...v-operator/controllers/mcpremoteproxy_runconfig.go	85.71%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3458      +/-   ##
==========================================
- Coverage   64.90%   64.88%   -0.03%     
==========================================
  Files         394      394              
  Lines       38402    38449      +47     
==========================================
+ Hits        24926    24947      +21     
- Misses      11530    11556      +26     
  Partials     1946     1946              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.