USHIFT-6801: C2CC: IPSec test

docs/user/howto_c2cc_ipsec.md

@@ -0,0 +1,154 @@
+# Encrypting C2CC Traffic with IPsec
+
+MicroShift Cluster-to-Cluster Connectivity (C2CC) routes cross-cluster pod and service traffic as raw IP between nodes.
+This traffic traverses the physical network unencrypted by default.
+You can use IPsec to protect it using standard Linux tools.
+
+MicroShift does not configure or manage IPsec.
+Setting up and maintaining the IPsec tunnels is the responsibility of the system administrator.
+This guide provides a minimal working example using Libreswan in tunnel mode to help you get started.
+
+For comprehensive IPsec/VPN documentation, see [Setting up an IPsec VPN](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_networking/setting-up-an-ipsec-vpn_configuring-and-managing-networking) in the RHEL documentation.
+
+## Prerequisites
+
+- Two or more RHEL hosts running MicroShift with C2CC configured (non-overlapping pod and service CIDRs, `clusterToCluster.remoteClusters` populated in each node's config).
+- IP connectivity between the hosts on the underlay network.
+- Libreswan installed on every host:
+
+```bash
+sudo dnf install -y libreswan
+```
+
+## Firewall
+
+Open the firewall for IKE negotiation and ESP:
+
+```bash
+sudo firewall-cmd --permanent --zone=public --add-service=ipsec
+sudo firewall-cmd --reload
+```
+
+This allows UDP ports 500 and 4500 (IKE/NAT-T) and IP protocol 50 (ESP).
+
+## Generate a Pre-Shared Key
+
+Generate a shared secret on one host and distribute it to all others through a secure channel:
+
+```bash
+openssl rand -hex 32
+```
+
+## Configure Libreswan
+
+The examples below assume a two-cluster setup:
+
+| Host    | Underlay IP   | Pod CIDR       | Service CIDR   |
+|---------|---------------|----------------|----------------|
+| Host A  | 192.168.1.10  | 10.42.0.0/16   | 10.43.0.0/16   |
+| Host B  | 192.168.1.20  | 10.45.0.0/16   | 10.46.0.0/16   |
+
+### Secrets
+
+On **each** host, create `/etc/ipsec.d/c2cc.secrets`:
+
+```conf
+192.168.1.10 192.168.1.20 : PSK "<your-hex-key>"
+```
+
+Set permissions:
+
+```bash
+sudo chmod 600 /etc/ipsec.d/c2cc.secrets
+sudo restorecon -v /etc/ipsec.d/c2cc.secrets
+```
+
+### Connection definition
+
+On **Host A**, create `/etc/ipsec.d/c2cc-tunnel.conf`:
+
+```conf
+conn c2cc-to-host-b
+    type=tunnel
+    authby=secret
+    left=192.168.1.10
+    right=192.168.1.20
+    leftsubnets={10.42.0.0/16 10.43.0.0/16}
+    rightsubnets={10.45.0.0/16 10.46.0.0/16}
+    auto=start
+    ike=aes256-sha2_256-modp2048
+    esp=aes256-sha2_256
+    failureshunt=drop
+    negotiationshunt=drop
+    ikev2=insist
+```
+
+On **Host B**, create the same file with `left`/`right` and subnet values swapped:
+
+```conf
+conn c2cc-to-host-a
+    type=tunnel
+    authby=secret
+    left=192.168.1.20
+    right=192.168.1.10
+    leftsubnets={10.45.0.0/16 10.46.0.0/16}
+    rightsubnets={10.42.0.0/16 10.43.0.0/16}
+    auto=start
+    ike=aes256-sha2_256-modp2048
+    esp=aes256-sha2_256
+    failureshunt=drop
+    negotiationshunt=drop
+    ikev2=insist
+```
+
+Key parameters:
+
+- **`type=tunnel`** -- Tunnel mode encrypts the original IP packet and wraps it in a new IP header. This is required because C2CC traffic uses pod/service CIDRs as source and destination, which are not routable on the underlay.
+- **`leftsubnets` / `rightsubnets`** -- Must match the pod and service CIDRs configured in MicroShift. Each `{cidr1 cidr2}` pair creates one child SA per local/remote CIDR combination.
+- **`auto=start`** -- Bring the tunnel up automatically when the IPsec service starts.
+- **`failureshunt=drop` / `negotiationshunt=drop`** -- Drop traffic that matches the tunnel selectors if the SA fails or is still negotiating, preventing fallback to plaintext.
+- **`ikev2=insist`** -- Require IKEv2. IKEv1 is not recommended.
+
+### Three or more clusters
+
+For a full mesh of N clusters, each host needs a connection definition and a secrets entry for every remote host.
+For example, with three hosts, Host A would have two `conn` blocks (one for Host B, one for Host C) and two secrets entries.
+
+## Start IPsec
+
+Initialize the NSS database (first time only) and start the service:
+
+```bash
+sudo ipsec checknss
+sudo systemctl enable --now ipsec
+```
+
+## Verify the Tunnels
+
+Check that tunnel SAs are established:
+
+```bash
+sudo ipsec trafficstatus
+```
+
+You should see output containing `type=ESP` entries for each subnet pair.
+For a two-cluster setup with 2 local CIDRs and 2 remote CIDRs, expect 4 child SAs.
+
+Verify XFRM state is populated:
+
+```bash
+ip xfrm state
+```
+
+You can also capture packets on the wire to confirm ESP encapsulation:
+
+```bash
+sudo tcpdump -i enp1s0 -c 10 esp
+```
+
+## Considerations
+
+- **IPsec adds overhead.** ESP tunnel mode adds approximately 36-52 bytes per packet. If you experience MTU issues, verify that path MTU discovery is working or adjust MTU settings accordingly.
+- **Tunnel recovery.** If IPsec is restarted on one host, tunnels renegotiate automatically when `auto=start` is set. No MicroShift restart is required.
+- **Certificates.** This guide uses pre-shared keys for simplicity. For production deployments, consider certificate-based authentication. See the [RHEL VPN documentation](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_networking/setting-up-an-ipsec-vpn_configuring-and-managing-networking) for details.
+- **Policy enforcement.** The example connection definitions include `failureshunt=drop` and `negotiationshunt=drop` to prevent traffic from falling back to plaintext when the tunnel is down or still negotiating. If you remove these options, traffic matching the tunnel selectors will be sent unencrypted whenever the SA is unavailable.

test/image-blueprints-bootc/el10/layer2-presubmit/group2/rhel102-bootc-source-ipsec.containerfile

@@ -0,0 +1,10 @@
+FROM localhost/rhel102-bootc-source:latest
+
+# Install Libreswan (IPsec) and tcpdump for C2CC IPsec E2E tests.
+# Libreswan provides the ipsec service and pluto daemon.
+# tcpdump is used to capture and verify ESP-encapsulated packets.
+RUN dnf install -y libreswan tcpdump && \
+    dnf clean all
+
+# Pre-configure firewall for IKE (UDP 500/4500) and ESP (protocol 50).
+RUN firewall-offline-cmd --zone=public --add-service=ipsec

test/image-blueprints-bootc/el9/layer2-presubmit/group2/rhel98-bootc-source-ipsec.containerfile

@@ -0,0 +1,10 @@
+FROM localhost/rhel98-bootc-source:latest
+
+# Install Libreswan (IPsec) and tcpdump for C2CC IPsec E2E tests.
+# Libreswan provides the ipsec service and pluto daemon.
+# tcpdump is used to capture and verify ESP-encapsulated packets.
+RUN dnf install -y libreswan tcpdump && \
+    dnf clean all
+
+# Pre-configure firewall for IKE (UDP 500/4500) and ESP (protocol 50).
+RUN firewall-offline-cmd --zone=public --add-service=ipsec

test/resources/c2cc.resource

@@ -32,6 +32,11 @@ ${KUBECONFIG_C}             ${EMPTY}
 &{C2CC_KUBECONFIGS}         &{EMPTY}
 &{C2CC_SSH_IDS}             &{EMPTY}
 @{C2CC_REMOTE_ALIASES}      @{EMPTY}
+&{NAMESPACES}               cluster-a=${EMPTY}    cluster-b=${EMPTY}    cluster-c=${EMPTY}
+&{DOMAIN_MAP}
+...                         cluster-a=${CLUSTER_A_DOMAIN}
+...                         cluster-b=${CLUSTER_B_DOMAIN}
+...                         cluster-c=${CLUSTER_C_DOMAIN}
 
 
 *** Keywords ***
@@ -295,3 +300,93 @@ Cleanup Test Workloads
     FOR    ${alias}    IN    cluster-a    cluster-b    cluster-c
         Oc On Cluster    ${alias}    oc delete namespace ${NAMESPACES}[${alias}] --timeout=60s
     END
+
+Get Hello Pod IP
+    [Documentation]    Get the pod IP of hello-microshift on the given cluster.
+    [Arguments]    ${alias}
+    ${ip}=    Oc On Cluster    ${alias}
+    ...    oc get pod hello-microshift -n ${NAMESPACES}[${alias}] -o jsonpath='{.status.podIP}'
+    RETURN    ${ip}
+
+Get Hello Service IP
+    [Documentation]    Get the ClusterIP of the hello-microshift service on the given cluster.
+    [Arguments]    ${alias}
+    ${ip}=    Oc On Cluster    ${alias}
+    ...    oc get svc hello-microshift -n ${NAMESPACES}[${alias}] -o jsonpath='{.spec.clusterIP}'
+    RETURN    ${ip}
+
+Get Curl Pod IP
+    [Documentation]    Get the pod IP of curl-pod on the given cluster.
+    [Arguments]    ${alias}
+    ${ip}=    Oc On Cluster    ${alias}
+    ...    oc get pod curl-pod -n ${NAMESPACES}[${alias}] -o jsonpath='{.status.podIP}'
+    RETURN    ${ip}
+
+Curl From Cluster
+    [Documentation]    Exec curl from curl-pod on the given cluster to the target IP and port.
+    [Arguments]    ${alias}    ${ip}    ${port}
+    ${stdout}=    Oc On Cluster    ${alias}
+    ...    oc exec curl-pod -n ${NAMESPACES}[${alias}] -- curl -sS --max-time 10 http://${ip}:${port}/cgi-bin/hello
+    RETURN    ${stdout}
+
+Test Connectivity Between Clusters
+    [Documentation]    Verify pod on ${source} can reach ${endpoint_type} IP on ${destination}.
+    [Arguments]    ${source}    ${destination}    ${endpoint_type}
+    IF    '${endpoint_type}' == 'pod'
+        ${ip_dest}=    Get Hello Pod IP    ${destination}
+    ELSE IF    '${endpoint_type}' == 'service'
+        ${ip_dest}=    Get Hello Service IP    ${destination}
+    ELSE
+        Fail    Invalid endpoint_type: ${endpoint_type}. Must be 'pod' or 'service'.
+    END
+    ${stdout}=    Curl From Cluster    ${source}    ${ip_dest}    8080
+    Should Contain    ${stdout}    Hello from
+
+Test Source IP Preserved Between Clusters
+    [Documentation]    Verify ${source} to ${destination} pod-to-${endpoint_type} traffic preserves the source pod IP (no SNAT).
+    [Arguments]    ${source}    ${destination}    ${endpoint_type}
+    ${curl_pod_ip}=    Get Curl Pod IP    ${source}
+    IF    '${endpoint_type}' == 'pod'
+        ${ip_dest}=    Get Hello Pod IP    ${destination}
+    ELSE IF    '${endpoint_type}' == 'service'
+        ${ip_dest}=    Get Hello Service IP    ${destination}
+    ELSE
+        Fail    Invalid endpoint_type: ${endpoint_type}. Must be 'pod' or 'service'.
+    END
+    ${stdout}=    Curl From Cluster    ${source}    ${ip_dest}    8080
+    Should Contain    ${stdout}    source: ${curl_pod_ip}
+
+DNS Resolve From Cluster
+    [Documentation]    Resolve a DNS name from curl-pod on the given cluster. Retries for up to 60s.
+    [Arguments]    ${alias}    ${fqdn}
+    Wait Until Keyword Succeeds    12x    5s
+    ...    DNS Lookup Should Succeed    ${alias}    ${fqdn}
+
+DNS Lookup Should Succeed
+    [Documentation]    Resolve a DNS name from curl-pod using getent hosts.
+    [Arguments]    ${alias}    ${fqdn}
+    ${stdout}=    Oc On Cluster    ${alias}
+    ...    oc exec curl-pod -n ${NAMESPACES}[${alias}] -- getent hosts ${fqdn}
+    Should Not Be Empty    ${stdout}
+
+Curl DNS From Cluster
+    [Documentation]    Curl a service by DNS name from curl-pod on the given cluster.
+    [Arguments]    ${alias}    ${fqdn}    ${port}
+    ${stdout}=    Wait Until Keyword Succeeds    12x    5s
+    ...    Curl DNS Should Succeed    ${alias}    ${fqdn}    ${port}
+    RETURN    ${stdout}
+
+Curl DNS Should Succeed
+    [Documentation]    Single attempt to curl a DNS name from curl-pod.
+    [Arguments]    ${alias}    ${fqdn}    ${port}
+    ${stdout}=    Oc On Cluster    ${alias}
+    ...    oc exec curl-pod -n ${NAMESPACES}[${alias}] -- curl -sS --max-time 10 http://${fqdn}:${port}/cgi-bin/hello
+    Should Contain    ${stdout}    Hello from
+    RETURN    ${stdout}
+
+Curl Remote Service Via DNS
+    [Documentation]    Verify pod on ${source} can reach a service on ${destination} using the remote DNS name.
+    [Arguments]    ${source}    ${destination}
+    ${stdout}=    Curl DNS From Cluster    ${source}
+    ...    hello-microshift.${NAMESPACES}[${destination}].svc.${DOMAIN_MAP}[${destination}]    8080
+    Should Contain    ${stdout}    Hello from