Add E2E tests for authServerRef on MCPServer and MCPRemoteProxy

[tgrunnagle]

Description

Create a Ginkgo/Gomega E2E test suite in test/e2e/thv-operator/authserver/ that exercises the new authServerRef field on both MCPServer and MCPRemoteProxy resources against a real Kubernetes cluster. These tests validate that the operator correctly resolves authServerRef, generates the expected RunConfig ConfigMap entries, detects conflicts and type mismatches, and preserves backward compatibility with the existing externalAuthConfigRef path. This is Phase 2 of RFC-0050 and builds on the CRD and controller changes delivered in #4640.

Context

#4640 adds the authServerRef field, conflict validation, type validation, and watch handler updates. Without E2E tests, there is no verification that these changes work correctly in a real Kubernetes environment end-to-end -- from resource creation through reconciliation to ConfigMap generation. The E2E suite covers the full operator reconciliation path, including status phase transitions and error condition reporting, which cannot be verified by unit tests alone.

Dependencies: #4640 (CRD types, controller logic, and unit tests)
Blocks: None

Acceptance Criteria

• New test directory test/e2e/thv-operator/authserver/ exists with suite_test.go, helpers.go, and test files
• E2E: MCPServer with authServerRef pointing to type: embeddedAuthServer reaches Running phase and the <name>-runconfig ConfigMap contains embedded_auth_server_config
• E2E: MCPServer with both authServerRef and externalAuthConfigRef pointing to type: embeddedAuthServer configs enters Failed phase with an error condition indicating conflict
• E2E: MCPServer with authServerRef pointing to a non-embeddedAuthServer type (e.g., unauthenticated) enters Failed phase with an error condition indicating type mismatch
• E2E: MCPServer with legacy externalAuthConfigRef pointing to type: embeddedAuthServer (no authServerRef) still reaches Running phase (backward compatibility)
• E2E: MCPRemoteProxy with authServerRef pointing to type: embeddedAuthServer reaches Ready phase and the ConfigMap contains embedded_auth_server_config
• E2E: MCPRemoteProxy with authServerRef (embeddedAuthServer) + externalAuthConfigRef (awsSts) reaches Ready phase and the ConfigMap contains both embedded_auth_server_config and aws_sts_config
• E2E: MCPRemoteProxy with both authServerRef and externalAuthConfigRef pointing to type: embeddedAuthServer enters Failed phase with an error condition indicating conflict
• E2E: MCPRemoteProxy with authServerRef pointing to a non-embeddedAuthServer type enters Failed phase with an error condition indicating type mismatch
• E2E: MCPRemoteProxy with legacy externalAuthConfigRef pointing to type: embeddedAuthServer (no authServerRef) still reaches Ready phase (backward compatibility)
• All tests pass
• Code reviewed and approved

Technical Approach

Recommended Implementation

Create a new Ginkgo test suite under test/e2e/thv-operator/authserver/ following the established pattern from test/e2e/thv-operator/virtualmcp/. The suite connects to a real Kubernetes cluster via kubeconfig, creates test resources in the default namespace, and uses Eventually polling to wait for status transitions.

Each test case creates the necessary MCPExternalAuthConfig resources (embedded auth server, AWS STS, unauthenticated), then creates the MCPServer or MCPRemoteProxy referencing them via authServerRef and/or externalAuthConfigRef. Assertions verify:

1. Happy path tests: Poll for Running/Ready phase, then fetch the <name>-runconfig ConfigMap and unmarshal the runconfig.json data to verify the expected auth config fields are present.
2. Error tests (conflict, type mismatch): Poll for Failed phase and check that status conditions contain the expected error message substring.
3. Backward compatibility tests: Create resources using only externalAuthConfigRef with type: embeddedAuthServer and verify they still reach Running/Ready phase.

The MCPRemoteProxy combined test (authServerRef + externalAuthConfigRef with awsSts) is the primary use case motivating RFC-0050 and deserves particular attention -- verify both embedded_auth_server_config and aws_sts_config keys exist in the RunConfig JSON.

Use AfterAll cleanup blocks to delete all test resources. Use Ordered containers so resources are created once and assertions run in sequence.

Patterns & Frameworks

• Ginkgo/Gomega BDD-style: Follow the Describe/Context/It structure used in test/e2e/thv-operator/virtualmcp/
• Suite bootstrap: Mirror virtualmcp/suite_test.go for kubeconfig loading, scheme registration, and client creation
• Eventually polling: Use gomega.Eventually with 2-3 minute timeouts and 1-second polling for status checks
• JustAfterEach state dump: Include the same failure diagnostic dump pattern from virtualmcp/suite_test.go for debugging test failures
• Resource cleanup: Use AfterAll with client.Delete to clean up test resources; ignore NotFound errors

Code Pointers

• test/e2e/thv-operator/virtualmcp/suite_test.go - Suite bootstrap pattern to replicate: kubeconfig loading, scheme registration, k8s client setup, JustAfterEach state dump
• test/e2e/thv-operator/virtualmcp/helpers.go - Helper function patterns (wait for ready, check pods, etc.) to adapt for MCPServer/MCPRemoteProxy status checks
• test/e2e/thv-operator/virtualmcp/virtualmcp_external_auth_test.go - Pattern for creating MCPExternalAuthConfig resources and MCPServer resources that reference them
• test/e2e/thv-operator/virtualmcp/virtualmcp_authserver_config_test.go - Pattern for testing auth server configuration validation via status conditions
• cmd/thv-operator/api/v1alpha1/mcpserver_types.go - MCPServer phase constants (MCPServerPhaseRunning, MCPServerPhaseFailed) and condition types
• cmd/thv-operator/api/v1alpha1/mcpremoteproxy_types.go - MCPRemoteProxy phase constants (MCPRemoteProxyPhaseReady, MCPRemoteProxyPhaseFailed) and condition types
• cmd/thv-operator/controllers/mcpserver_runconfig.go (line 52) - ConfigMap naming convention: fmt.Sprintf("%s-runconfig", m.Name)
• pkg/runner/config.go - RunConfig struct with EmbeddedAuthServerConfig and AWSStsConfig fields for JSON unmarshaling assertions

Component Interfaces

Test helper signatures to implement in helpers.go:

// WaitForMCPServerPhase polls until an MCPServer reaches the expected phase
func WaitForMCPServerPhase(
    ctx context.Context,
    c client.Client,
    name, namespace string,
    expectedPhase mcpv1alpha1.MCPServerPhase,
    timeout, pollingInterval time.Duration,
)

// WaitForMCPRemoteProxyPhase polls until an MCPRemoteProxy reaches the expected phase
func WaitForMCPRemoteProxyPhase(
    ctx context.Context,
    c client.Client,
    name, namespace string,
    expectedPhase mcpv1alpha1.MCPRemoteProxyPhase,
    timeout, pollingInterval time.Duration,
)

// GetRunConfigFromConfigMap fetches the runconfig ConfigMap and returns the parsed JSON
func GetRunConfigFromConfigMap(
    ctx context.Context,
    c client.Client,
    resourceName, namespace string,
) (map[string]interface{}, error)

// ExpectMCPServerConditionMessage waits for a condition with a message containing the substring
func ExpectMCPServerConditionMessage(
    ctx context.Context,
    c client.Client,
    name, namespace string,
    conditionType string,
    messageSubstring string,
    timeout, pollingInterval time.Duration,
)

MCPExternalAuthConfig test fixtures:

// Embedded auth server config for authServerRef tests
embeddedAuthConfig := &mcpv1alpha1.MCPExternalAuthConfig{
    ObjectMeta: metav1.ObjectMeta{Name: "test-embedded-auth", Namespace: namespace},
    Spec: mcpv1alpha1.MCPExternalAuthConfigSpec{
        Type: mcpv1alpha1.ExternalAuthTypeEmbeddedAuthServer,
        EmbeddedAuthServer: &mcpv1alpha1.EmbeddedAuthServerConfig{
            Issuer: "http://localhost:9090",
            UpstreamProviders: []mcpv1alpha1.UpstreamProviderConfig{
                {Name: "test", Type: mcpv1alpha1.UpstreamProviderTypeOIDC,
                 OIDCConfig: &mcpv1alpha1.OIDCUpstreamConfig{
                     IssuerURL: "https://accounts.google.com",
                     ClientID:  "test-client-id",
                 }},
            },
        },
    },
}

// AWS STS config for combined auth test
awsStsConfig := &mcpv1alpha1.MCPExternalAuthConfig{
    ObjectMeta: metav1.ObjectMeta{Name: "test-aws-sts", Namespace: namespace},
    Spec: mcpv1alpha1.MCPExternalAuthConfigSpec{
        Type: mcpv1alpha1.ExternalAuthTypeAWSSts,
        AWSSts: &mcpv1alpha1.AWSStsConfig{
            Region:          "us-east-1",
            FallbackRoleArn: "arn:aws:iam::123456789012:role/test-role",
        },
    },
}

MCPServer with authServerRef:

server := &mcpv1alpha1.MCPServer{
    ObjectMeta: metav1.ObjectMeta{Name: "test-authserverref", Namespace: namespace},
    Spec: mcpv1alpha1.MCPServerSpec{
        Image:     images.GofetchServerImage,
        Transport: "streamable-http",
        AuthServerRef: &corev1.TypedLocalObjectReference{
            Kind: "MCPExternalAuthConfig",
            Name: "test-embedded-auth",
        },
    },
}

Testing Strategy

MCPServer Tests

• Happy path: MCPServer with authServerRef -> Running phase, ConfigMap runconfig.json contains embedded_auth_server_config key
• Conflict: MCPServer with authServerRef + externalAuthConfigRef both referencing type: embeddedAuthServer -> Failed phase, error message mentions "cannot both configure"
• Type mismatch: MCPServer with authServerRef pointing to type: unauthenticated -> Failed phase, error message mentions expected type
• Backward compatibility: MCPServer with externalAuthConfigRef pointing to type: embeddedAuthServer (no authServerRef) -> Running phase

MCPRemoteProxy Tests

• Happy path: MCPRemoteProxy with authServerRef -> Ready phase, ConfigMap contains embedded_auth_server_config
• Combined auth (primary use case): MCPRemoteProxy with authServerRef (embeddedAuthServer) + externalAuthConfigRef (awsSts) -> Ready phase, ConfigMap contains both embedded_auth_server_config and aws_sts_config
• Conflict: MCPRemoteProxy with both refs to type: embeddedAuthServer -> Failed phase
• Type mismatch: MCPRemoteProxy with authServerRef pointing to non-embeddedAuthServer type -> Failed phase
• Backward compatibility: MCPRemoteProxy with externalAuthConfigRef to type: embeddedAuthServer (no authServerRef) -> Ready phase

Edge Cases

• Verify that authServerRef with apiGroup omitted (nil) works correctly (the controller should accept this)
• Verify resource cleanup does not leave orphaned ConfigMaps

Out of Scope

• Unit tests for controller logic (covered by Add authServerRef CRD types, controller logic, and unit tests #4640)
• Architecture documentation updates (Update operator architecture documentation for authServerRef #4642)
• Chainsaw tests (this suite uses Ginkgo/Gomega directly, following the virtualmcp pattern)
• Changes to CLI-mode toolhive (thv run)
• Testing auth server runtime behavior (JWT issuance, token validation) -- only ConfigMap generation is verified
• Testing actual AWS STS calls -- only ConfigMap content is verified

References

• RFC-0050: Dedicated Auth Server Reference for MCP Resources
• Existing E2E test pattern: virtualmcp suite
• Related epic: stacklok/stacklok-epics#256