RFC-0055: MCPServerEntry CRD for Direct Remote MCP Server Backends

[JAORMX]

Summary

Introduces MCPServerEntry, a new lightweight CRD (short name: mcpentry) that allows VirtualMCPServer to connect directly to remote MCP servers without deploying MCPRemoteProxy infrastructure. MCPServerEntry is a pod-less configuration resource that declares a remote MCP endpoint and belongs to an MCPGroup.

Motivation

vMCP currently requires MCPRemoteProxy (which spawns thv-proxyrunner pods) to reach remote MCP servers. This creates three problems:

1. Forced auth on public remotes (#3104): MCPRemoteProxy requires OIDC even when vMCP already handles client auth
2. Dual auth boundary confusion (#4109): externalAuthConfigRef serves two conflicting purposes (vMCP-to-proxy AND proxy-to-remote)
3. Resource waste: Every remote backend needs a Deployment + Service + Pod just to make an HTTP call

Design Highlights

• Zero infrastructure: MCPServerEntry deploys no pods, services, or deployments — pure configuration
• Single auth boundary: externalAuthConfigRef has one unambiguous purpose (auth to the remote)
• Validation-only controller: Validates references (MCPGroup, MCPExternalAuthConfig) but never probes remote URLs
• Both discovery modes: Works with static (operator-generated ConfigMap) and dynamic (vMCP runtime discovery) from THV-0014
• Header forwarding: Reuses existing headerForward pattern from THV-0026
• Custom CA support: caBundleRef for private remote servers with internal CAs
• Planned supersession: Will be superseded by MCPRemoteEndpoint (THV-0067), which unifies direct and proxy modes into a single CRD

Related RFCs

• THV-0008 (Virtual MCP Server)
• THV-0009 (Remote MCP Proxy)
• THV-0010 (MCPGroup CRD)
• THV-0014 (K8s-Aware vMCP)
• THV-0026 (Header Passthrough)
• THV-0067 (MCPRemoteEndpoint — future unified CRD)

Test plan

• Review RFC structure and completeness
• Validate CRD design against existing patterns
• Verify security considerations (auth boundaries, SSRF mitigations)
• Confirm MCPRemoteEndpoint supersession path is documented
• Review comparison guide (MCPRemoteProxy vs MCPServerEntry)

[yrobla]

This is a well-structured RFC with clear motivation, thorough security considerations, and good alternatives analysis. The naming convention (following Istio ServiceEntry) is a nice touch. A few issues worth addressing before implementation:

• groupRef type inconsistency (string vs object pattern) needs resolution
• caBundleRef resource type (Secret vs ConfigMap) is ambiguous across sections
• SSRF mitigation has a gap: HTTPS-only validation doesn't block private IP ranges
• Auth resolution timing in ListWorkloadsInGroup() description is unclear
• Static mode CA bundle propagation is unspecified
• toolConfigRef deferral creates an unacknowledged migration regression for MCPRemoteProxy users

[yrobla]

looks as a very good improvement to me

[amirejaz]

This looks good to me. Added one comment

[amirejaz]

Nit: could we use a real public MCP server here, such as mcp-spec? context7 is currently being used as the example for both private and public MCP servers, which feels a bit confusing.

[amirejaz]

Question: Would it make sense to rename MCPServerEntry to MCPRemoteServerEntry to improve clarity?

[jerm-dro]

Looks great! I appreciate the alternatives section.

[lorr1]

We need this!! I love it.

[reyortiz3]

(naming): if it's for remote MCPs only, why not "MCPRemoteServer"? just by the "MCPServerEntry" it wasn't clear to me it was for remote only

[ChrisJBurns]

Thanks for the detailed RFC — the three problems are real and worth solving, and the security thinking (no operator-side probing, HTTPS enforcement, credential separation) is solid throughout.

The core question this review raises is: does solving these three problems require a new CRD, or can they be addressed by extending what already exists?

After reading the RFC against the actual codebase, the case for a new CRD is weaker than it appears:

• Problem #1 (forced OIDC on public remotes) can be fixed by making oidcConfig optional on MCPRemoteProxy — a one-field change (see comment 2).
• Problem #2 (dual auth boundary confusion) turns out not to exist in the code — externalAuthConfigRef and oidcConfig already serve separate purposes in pkg/vmcp/workloads/k8s.go (see comment 1).
• Problem #3 (pod waste) can be solved by adding a direct: true flag to MCPRemoteProxy that skips ensureDeployment() and ensureService() and uses remoteURL as the backend URL directly — touching ~4 files vs. the RFC's ~20 (see comments 3, 4).

The RFC does consider alternatives but doesn't seriously evaluate the most natural one: extending MCPRemoteProxy itself. Its alternatives section argues against extending MCPServer (fair) and against config-only approaches (fair), but never evaluates a lightweight mode on the resource that already exists for this exact purpose (comment 3).

Beyond the alternatives analysis, there are several concrete concerns: the operational cost of a new CRD is significantly underestimated (comment 4); the new CRD creates naming confusion alongside MCPServer and MCPRemoteProxy (comment 9); the MCPGroup status model would need restructuring for a third backend type (comment 5); the CA bundle volume-mounting introduces new operator complexity (comment 8); and the SSRF trade-off of moving outbound calls into the vMCP pod deserves explicit acknowledgement (comment 7).

Suggested path forward: Before investing in a new CRD, implement the simpler approach — make oidcConfig optional and add direct: true to MCPRemoteProxy. If real-world usage reveals that the RBAC separation story (platform teams vs. product teams managing backends independently) is a genuine need that can't be satisfied by MCPRemoteProxy's existing RBAC surface, revisit the CRD proposal with that as the primary motivation rather than the pod-waste argument.

If the team decides a new CRD is still the right call after considering the above, comments 5–12 cover the design-level issues that should be resolved before implementation begins.

🤖 Review assisted by Claude Code

[jhrozek]

+1 to what @ChrisJBurns said, I think we should consider reusing the existing CRDs and be more explicit in why it's not an option. There's also the cognitive load on the users, the documentation load etc..

[JAORMX]

+1 to what @ChrisJBurns said, I think we should consider reusing the existing CRDs and be more explicit in why it's not an option. There's also the cognitive load on the users, the documentation load etc..

I actually disagree with this. I think re-using existing CRDs would do us a diservice and further increase technical depth by using APIs that are not fit for purpose because they had very different intentions.

[ChrisJBurns]

Bug: Field name doesn't match existing type

addHeadersFromSecrets (plural) doesn't match the existing HeaderForwardConfig type which uses addHeadersFromSecret (singular). See cmd/thv-operator/api/v1alpha1/mcpremoteproxy_types.go line 20:

type HeaderForwardConfig struct {
    AddPlaintextHeaders  map[string]string    `json:"addPlaintextHeaders,omitempty"`
    AddHeadersFromSecret []HeaderFromSecret   `json:"addHeadersFromSecret,omitempty"`
}

This appears in multiple YAML examples throughout the RFC (lines 240, 310, 531). Should be addHeadersFromSecret everywhere to match the existing type.

[ChrisJBurns]

Bug: caBundleRef shape doesn't match existing CABundleSource type

The RFC shows a flat {name, key} struct:

caBundleRef:
  name: internal-ca-bundle
  key: ca.crt

But the existing CABundleSource type in mcpserver_types.go (line 659) wraps a corev1.ConfigMapKeySelector:

type CABundleSource struct {
    ConfigMapRef *corev1.ConfigMapKeySelector `json:"configMapRef,omitempty"`
}

So the actual YAML shape should be:

caBundleRef:
  configMapRef:
    name: internal-ca-bundle
    key: ca.crt

Recommend reusing the existing CABundleSource type for consistency across CRDs. This affects all YAML examples with caBundleRef (lines 249, 315).

[ChrisJBurns]

Naming: Condition types don't match codebase convention

The RFC uses -Valid suffix (GroupRefValid, AuthConfigValid, CABundleValid), but the codebase consistently uses -Validated:

• MCPServer: GroupRefValidated, ExternalAuthConfigValidated, CABundleRefValidated
• MCPRemoteProxy: GroupRefValidated, ExternalAuthConfigValidated, ToolConfigValidated

For consistency, these should be:

• GroupRefValidated (not GroupRefValid)
• ExternalAuthConfigValidated (not AuthConfigValid)
• CABundleRefValidated (not CABundleValid)

Also, the Go constant names should follow the existing prefix pattern: ConditionTypeMCPServerEntryGroupRefValidated, etc.

[ChrisJBurns]

Missing: .status.phase field

Every existing CRD in the codebase has a .status.phase field with a typed enum:

• MCPServer: Pending, Running, Failed, Terminating, Stopped
• MCPRemoteProxy: Pending, Ready, Failed, Terminating
• VirtualMCPServer: Pending, Ready, Degraded, Failed
• MCPGroup: Ready, Pending, Failed

The RFC status only lists conditions and observedGeneration but no phase. For a validation-only resource, sensible phases would be: Pending, Ready, Failed.

Also missing: kubebuilder print column markers. Every existing CRD specifies +kubebuilder:printcolumn for Phase/Ready, URL, Transport, Age, etc. The RFC should specify these for kubectl get mcpentry output.

[ChrisJBurns]

Design: Use a spec field instead of annotation for HTTPS override

The existing codebase uses insecureAllowHTTP: bool spec fields (e.g., InlineOIDCConfig.InsecureAllowHTTP in mcpserver_types.go line 719, and in oidc_validation.go line 59). The annotation-based approach (toolhive.stacklok.dev/allow-insecure) is a divergence with security implications:

1. RBAC escalation: Annotations can be set by anyone with patch rights on metadata, even without update on spec
2. No CEL validation: Annotations bypass kubebuilder validation markers — a typo like "tru" silently enforces HTTPS
3. Visibility: Annotations don't show up in kubectl get print columns

Recommend using a spec field (e.g., insecureAllowHTTP: bool with +kubebuilder:default=false) consistent with the existing pattern.

[ChrisJBurns]

Missing: Multi-upstream ExternalAuthConfig rejection + header forwarding interaction

Multi-upstream: Both MCPServer and MCPRemoteProxy explicitly reject multi-upstream ExternalAuthConfig (see ConditionReasonExternalAuthConfigMultiUpstream). The RFC should specify that MCPServerEntry does the same.

Header forwarding + token exchange interaction: What happens when both externalAuthConfigRef (token exchange → sets Authorization) and headerForward.addHeadersFromSecrets (also sets Authorization) are configured? The existing types have no CEL validation preventing this conflict.

Recommend either:

• Define clear precedence rules (e.g., externalAuthConfigRef wins for any header it sets)
• Or add CEL/webhook validation rejecting headerForward entries that set Authorization when externalAuthConfigRef is also present
• Document that headerForward is for non-auth headers (tenant IDs, correlation IDs) while externalAuthConfigRef handles all authentication

[ChrisJBurns]

Incomplete: MCPGroup controller changes are substantial

The MCPGroup controller update is described in one paragraph but requires significant work:

1. New status fields: ServerEntries []string + ServerEntryCount int (CRD schema change)
2. New findReferencingMCPServerEntries() method
3. New populateServerEntryStatus() method
4. Update handleDeletion() to iterate MCPServerEntry resources (without this, deleting an MCPGroup leaves MCPServerEntries with stale GroupRefValid=True conditions)
5. New watch in SetupWithManager() for MCPServerEntry
6. New field index for spec.groupRef on MCPServerEntry
7. Update ConditionTypeMCPServersChecked message (or rename to MembersChecked)

The RFC should enumerate these explicitly so implementation scope is clear.

[ChrisJBurns]

Implementation gap: No existing per-backend TLS configuration code path

The current newBackendTransport() in pkg/vmcp/client/client.go (line 90) clones http.DefaultTransport with no per-backend TLS customization. For caBundleRef to work, a new mechanism is needed to inject custom CA bundles per backend into the HTTP client factory.

This needs to flow through BackendTarget (adding a CABundle []byte field or similar) or be injected at the transport factory level. The RFC should call out this gap since it's a meaningful implementation change to the vMCP client infrastructure.

Also worth noting: for static mode, the CA bundle ConfigMap must be mounted as a volume into the VirtualMCPServer pod (not the nonexistent MCPServerEntry pod). The mount path convention (e.g., /etc/toolhive/ca-bundles/<entry-name>/ca.crt) and how it's referenced in the generated backend ConfigMap should be specified.

[ChrisJBurns]

Note: Session recovery is more important for entry backends

The existing mcpSession explicitly documents a "Phase 1 limitation — no reconnection: if the underlying transport drops... all subsequent operations on this backend will fail." For in-cluster backends this is rarely hit (pods are colocated), but MCPServerEntry backends over the internet will experience session drops much more frequently due to:

• Network interruptions
• Remote server restarts/redeployments
• Remote server session TTL expiration
• TLS renegotiation failures

The MCP spec states servers may terminate sessions at any time and clients should re-initialize when receiving HTTP 404 for a known session ID. Consider documenting this as a known limitation with a follow-up plan, or addressing session re-establishment as part of this RFC's implementation phases.

[ChrisJBurns]

Missing: WorkloadKindMCPServerEntry and WorkloadType constants

The existing WorkloadReference enum in mcpoidcconfig_types.go (line 148-158) defines:

WorkloadKindMCPServer        = "MCPServer"
WorkloadKindVirtualMCPServer = "VirtualMCPServer"
WorkloadKindMCPRemoteProxy   = "MCPRemoteProxy"

And pkg/vmcp/workloads/discoverer.go defines WorkloadTypeMCPServer and WorkloadTypeMCPRemoteProxy.

Both need new constants for MCPServerEntry, plus the +kubebuilder:validation:Enum on WorkloadReference.Kind must be updated. The MCPExternalAuthConfig controller's ReferencingWorkloads tracking also needs to recognize MCPServerEntry references.

Also: BackendReconciler.fetchBackendResource() currently tries MCPServer then MCPRemoteProxy sequentially. A third try-fetch is fine for three types but the RFC should acknowledge this pattern doesn't scale well beyond that.