diff --git a/docs/user/howto_c2cc_ipsec.md b/docs/user/howto_c2cc_ipsec.md new file mode 100644 index 0000000000..cdb60accbd --- /dev/null +++ b/docs/user/howto_c2cc_ipsec.md @@ -0,0 +1,154 @@ +# Encrypting C2CC Traffic with IPsec + +MicroShift Cluster-to-Cluster Connectivity (C2CC) routes cross-cluster pod and service traffic as raw IP between nodes. +This traffic traverses the physical network unencrypted by default. +You can use IPsec to protect it using standard Linux tools. + +MicroShift does not configure or manage IPsec. +Setting up and maintaining the IPsec tunnels is the responsibility of the system administrator. +This guide provides a minimal working example using Libreswan in tunnel mode to help you get started. + +For comprehensive IPsec/VPN documentation, see [Setting up an IPsec VPN](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_networking/setting-up-an-ipsec-vpn_configuring-and-managing-networking) in the RHEL documentation. + +## Prerequisites + +- Two or more RHEL hosts running MicroShift with C2CC configured (non-overlapping pod and service CIDRs, `clusterToCluster.remoteClusters` populated in each node's config). +- IP connectivity between the hosts on the underlay network. +- Libreswan installed on every host: + +```bash +sudo dnf install -y libreswan +``` + +## Firewall + +Open the firewall for IKE negotiation and ESP: + +```bash +sudo firewall-cmd --permanent --zone=public --add-service=ipsec +sudo firewall-cmd --reload +``` + +This allows UDP ports 500 and 4500 (IKE/NAT-T) and IP protocol 50 (ESP). + +## Generate a Pre-Shared Key + +Generate a shared secret on one host and distribute it to all others through a secure channel: + +```bash +openssl rand -hex 32 +``` + +## Configure Libreswan + +The examples below assume a two-cluster setup: + +| Host | Underlay IP | Pod CIDR | Service CIDR | +|---------|---------------|----------------|----------------| +| Host A | 192.168.1.10 | 10.42.0.0/16 | 10.43.0.0/16 | +| Host B | 192.168.1.20 | 10.45.0.0/16 | 10.46.0.0/16 | + +### Secrets + +On **each** host, create `/etc/ipsec.d/c2cc.secrets`: + +```conf +192.168.1.10 192.168.1.20 : PSK "" +``` + +Set permissions: + +```bash +sudo chmod 600 /etc/ipsec.d/c2cc.secrets +sudo restorecon -v /etc/ipsec.d/c2cc.secrets +``` + +### Connection definition + +On **Host A**, create `/etc/ipsec.d/c2cc-tunnel.conf`: + +```conf +conn c2cc-to-host-b + type=tunnel + authby=secret + left=192.168.1.10 + right=192.168.1.20 + leftsubnets={10.42.0.0/16 10.43.0.0/16} + rightsubnets={10.45.0.0/16 10.46.0.0/16} + auto=start + ike=aes256-sha2_256-modp2048 + esp=aes256-sha2_256 + failureshunt=drop + negotiationshunt=drop + ikev2=insist +``` + +On **Host B**, create the same file with `left`/`right` and subnet values swapped: + +```conf +conn c2cc-to-host-a + type=tunnel + authby=secret + left=192.168.1.20 + right=192.168.1.10 + leftsubnets={10.45.0.0/16 10.46.0.0/16} + rightsubnets={10.42.0.0/16 10.43.0.0/16} + auto=start + ike=aes256-sha2_256-modp2048 + esp=aes256-sha2_256 + failureshunt=drop + negotiationshunt=drop + ikev2=insist +``` + +Key parameters: + +- **`type=tunnel`** -- Tunnel mode encrypts the original IP packet and wraps it in a new IP header. This is required because C2CC traffic uses pod/service CIDRs as source and destination, which are not routable on the underlay. +- **`leftsubnets` / `rightsubnets`** -- Must match the pod and service CIDRs configured in MicroShift. Each `{cidr1 cidr2}` pair creates one child SA per local/remote CIDR combination. +- **`auto=start`** -- Bring the tunnel up automatically when the IPsec service starts. +- **`failureshunt=drop` / `negotiationshunt=drop`** -- Drop traffic that matches the tunnel selectors if the SA fails or is still negotiating, preventing fallback to plaintext. +- **`ikev2=insist`** -- Require IKEv2. IKEv1 is not recommended. + +### Three or more clusters + +For a full mesh of N clusters, each host needs a connection definition and a secrets entry for every remote host. +For example, with three hosts, Host A would have two `conn` blocks (one for Host B, one for Host C) and two secrets entries. + +## Start IPsec + +Initialize the NSS database (first time only) and start the service: + +```bash +sudo ipsec checknss +sudo systemctl enable --now ipsec +``` + +## Verify the Tunnels + +Check that tunnel SAs are established: + +```bash +sudo ipsec trafficstatus +``` + +You should see output containing `type=ESP` entries for each subnet pair. +For a two-cluster setup with 2 local CIDRs and 2 remote CIDRs, expect 4 child SAs. + +Verify XFRM state is populated: + +```bash +ip xfrm state +``` + +You can also capture packets on the wire to confirm ESP encapsulation: + +```bash +sudo tcpdump -i enp1s0 -c 10 esp +``` + +## Considerations + +- **IPsec adds overhead.** ESP tunnel mode adds approximately 36-52 bytes per packet. If you experience MTU issues, verify that path MTU discovery is working or adjust MTU settings accordingly. +- **Tunnel recovery.** If IPsec is restarted on one host, tunnels renegotiate automatically when `auto=start` is set. No MicroShift restart is required. +- **Certificates.** This guide uses pre-shared keys for simplicity. For production deployments, consider certificate-based authentication. See the [RHEL VPN documentation](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_networking/setting-up-an-ipsec-vpn_configuring-and-managing-networking) for details. +- **Policy enforcement.** The example connection definitions include `failureshunt=drop` and `negotiationshunt=drop` to prevent traffic from falling back to plaintext when the tunnel is down or still negotiating. If you remove these options, traffic matching the tunnel selectors will be sent unencrypted whenever the SA is unavailable. diff --git a/test/image-blueprints-bootc/el10/layer2-presubmit/group2/rhel102-bootc-source-ipsec.containerfile b/test/image-blueprints-bootc/el10/layer2-presubmit/group2/rhel102-bootc-source-ipsec.containerfile new file mode 100644 index 0000000000..5040c3227d --- /dev/null +++ b/test/image-blueprints-bootc/el10/layer2-presubmit/group2/rhel102-bootc-source-ipsec.containerfile @@ -0,0 +1,10 @@ +FROM localhost/rhel102-bootc-source:latest + +# Install Libreswan (IPsec) and tcpdump for C2CC IPsec E2E tests. +# Libreswan provides the ipsec service and pluto daemon. +# tcpdump is used to capture and verify ESP-encapsulated packets. +RUN dnf install -y libreswan tcpdump && \ + dnf clean all + +# Pre-configure firewall for IKE (UDP 500/4500) and ESP (protocol 50). +RUN firewall-offline-cmd --zone=public --add-service=ipsec diff --git a/test/image-blueprints-bootc/el9/layer2-presubmit/group2/rhel98-bootc-source-ipsec.containerfile b/test/image-blueprints-bootc/el9/layer2-presubmit/group2/rhel98-bootc-source-ipsec.containerfile new file mode 100644 index 0000000000..739a0c5a44 --- /dev/null +++ b/test/image-blueprints-bootc/el9/layer2-presubmit/group2/rhel98-bootc-source-ipsec.containerfile @@ -0,0 +1,10 @@ +FROM localhost/rhel98-bootc-source:latest + +# Install Libreswan (IPsec) and tcpdump for C2CC IPsec E2E tests. +# Libreswan provides the ipsec service and pluto daemon. +# tcpdump is used to capture and verify ESP-encapsulated packets. +RUN dnf install -y libreswan tcpdump && \ + dnf clean all + +# Pre-configure firewall for IKE (UDP 500/4500) and ESP (protocol 50). +RUN firewall-offline-cmd --zone=public --add-service=ipsec diff --git a/test/resources/c2cc.resource b/test/resources/c2cc.resource index 0cc408bc8e..3a5c02e288 100644 --- a/test/resources/c2cc.resource +++ b/test/resources/c2cc.resource @@ -32,6 +32,11 @@ ${KUBECONFIG_C} ${EMPTY} &{C2CC_KUBECONFIGS} &{EMPTY} &{C2CC_SSH_IDS} &{EMPTY} @{C2CC_REMOTE_ALIASES} @{EMPTY} +&{NAMESPACES} cluster-a=${EMPTY} cluster-b=${EMPTY} cluster-c=${EMPTY} +&{DOMAIN_MAP} +... cluster-a=${CLUSTER_A_DOMAIN} +... cluster-b=${CLUSTER_B_DOMAIN} +... cluster-c=${CLUSTER_C_DOMAIN} *** Keywords *** @@ -295,3 +300,93 @@ Cleanup Test Workloads FOR ${alias} IN cluster-a cluster-b cluster-c Oc On Cluster ${alias} oc delete namespace ${NAMESPACES}[${alias}] --timeout=60s END + +Get Hello Pod IP + [Documentation] Get the pod IP of hello-microshift on the given cluster. + [Arguments] ${alias} + ${ip}= Oc On Cluster ${alias} + ... oc get pod hello-microshift -n ${NAMESPACES}[${alias}] -o jsonpath='{.status.podIP}' + RETURN ${ip} + +Get Hello Service IP + [Documentation] Get the ClusterIP of the hello-microshift service on the given cluster. + [Arguments] ${alias} + ${ip}= Oc On Cluster ${alias} + ... oc get svc hello-microshift -n ${NAMESPACES}[${alias}] -o jsonpath='{.spec.clusterIP}' + RETURN ${ip} + +Get Curl Pod IP + [Documentation] Get the pod IP of curl-pod on the given cluster. + [Arguments] ${alias} + ${ip}= Oc On Cluster ${alias} + ... oc get pod curl-pod -n ${NAMESPACES}[${alias}] -o jsonpath='{.status.podIP}' + RETURN ${ip} + +Curl From Cluster + [Documentation] Exec curl from curl-pod on the given cluster to the target IP and port. + [Arguments] ${alias} ${ip} ${port} + ${stdout}= Oc On Cluster ${alias} + ... oc exec curl-pod -n ${NAMESPACES}[${alias}] -- curl -sS --max-time 10 http://${ip}:${port}/cgi-bin/hello + RETURN ${stdout} + +Test Connectivity Between Clusters + [Documentation] Verify pod on ${source} can reach ${endpoint_type} IP on ${destination}. + [Arguments] ${source} ${destination} ${endpoint_type} + IF '${endpoint_type}' == 'pod' + ${ip_dest}= Get Hello Pod IP ${destination} + ELSE IF '${endpoint_type}' == 'service' + ${ip_dest}= Get Hello Service IP ${destination} + ELSE + Fail Invalid endpoint_type: ${endpoint_type}. Must be 'pod' or 'service'. + END + ${stdout}= Curl From Cluster ${source} ${ip_dest} 8080 + Should Contain ${stdout} Hello from + +Test Source IP Preserved Between Clusters + [Documentation] Verify ${source} to ${destination} pod-to-${endpoint_type} traffic preserves the source pod IP (no SNAT). + [Arguments] ${source} ${destination} ${endpoint_type} + ${curl_pod_ip}= Get Curl Pod IP ${source} + IF '${endpoint_type}' == 'pod' + ${ip_dest}= Get Hello Pod IP ${destination} + ELSE IF '${endpoint_type}' == 'service' + ${ip_dest}= Get Hello Service IP ${destination} + ELSE + Fail Invalid endpoint_type: ${endpoint_type}. Must be 'pod' or 'service'. + END + ${stdout}= Curl From Cluster ${source} ${ip_dest} 8080 + Should Contain ${stdout} source: ${curl_pod_ip} + +DNS Resolve From Cluster + [Documentation] Resolve a DNS name from curl-pod on the given cluster. Retries for up to 60s. + [Arguments] ${alias} ${fqdn} + Wait Until Keyword Succeeds 12x 5s + ... DNS Lookup Should Succeed ${alias} ${fqdn} + +DNS Lookup Should Succeed + [Documentation] Resolve a DNS name from curl-pod using getent hosts. + [Arguments] ${alias} ${fqdn} + ${stdout}= Oc On Cluster ${alias} + ... oc exec curl-pod -n ${NAMESPACES}[${alias}] -- getent hosts ${fqdn} + Should Not Be Empty ${stdout} + +Curl DNS From Cluster + [Documentation] Curl a service by DNS name from curl-pod on the given cluster. + [Arguments] ${alias} ${fqdn} ${port} + ${stdout}= Wait Until Keyword Succeeds 12x 5s + ... Curl DNS Should Succeed ${alias} ${fqdn} ${port} + RETURN ${stdout} + +Curl DNS Should Succeed + [Documentation] Single attempt to curl a DNS name from curl-pod. + [Arguments] ${alias} ${fqdn} ${port} + ${stdout}= Oc On Cluster ${alias} + ... oc exec curl-pod -n ${NAMESPACES}[${alias}] -- curl -sS --max-time 10 http://${fqdn}:${port}/cgi-bin/hello + Should Contain ${stdout} Hello from + RETURN ${stdout} + +Curl Remote Service Via DNS + [Documentation] Verify pod on ${source} can reach a service on ${destination} using the remote DNS name. + [Arguments] ${source} ${destination} + ${stdout}= Curl DNS From Cluster ${source} + ... hello-microshift.${NAMESPACES}[${destination}].svc.${DOMAIN_MAP}[${destination}] 8080 + Should Contain ${stdout} Hello from diff --git a/test/resources/ipsec.resource b/test/resources/ipsec.resource new file mode 100644 index 0000000000..b87f9edd22 --- /dev/null +++ b/test/resources/ipsec.resource @@ -0,0 +1,132 @@ +*** Settings *** +Documentation Keywords for IPsec (Libreswan) operations in C2CC test scenarios. +... Provides tunnel verification, XFRM counter checks, ESP packet capture, +... SA management, and nftables IPsec enforcement helpers. + +Library String +Resource c2cc.resource + + +*** Variables *** +${IPSEC_SA_TIMEOUT} 60s +${IPSEC_SA_RETRY} 5s + + +*** Keywords *** +Verify IPsec Tunnel Is Established + [Documentation] Verify that at least one IPsec tunnel is established on the given cluster. + [Arguments] ${alias} + ${stdout}= Command On Cluster ${alias} ipsec trafficstatus + Should Contain ${stdout} type=ESP + +Verify All IPsec Tunnels On Cluster + [Documentation] Verify that the expected number of IPsec tunnels are established. + [Arguments] ${alias} ${expected_count}=2 + ${stdout}= Command On Cluster ${alias} ipsec trafficstatus + ${count}= Get Count ${stdout} type=ESP + Should Be True ${count} >= ${expected_count} + ... Expected ${expected_count} IPsec tunnels but found ${count} + +Stop IPsec Service On Cluster + [Documentation] Stop the ipsec systemd service on the given cluster. + [Arguments] ${alias} + Disruptive Command On Cluster ${alias} systemctl stop ipsec + +Start IPsec Service On Cluster + [Documentation] Start the ipsec systemd service on the given cluster. + [Arguments] ${alias} + Command On Cluster ${alias} systemctl start ipsec + +Restart IPsec Service On Cluster + [Documentation] Restart the ipsec systemd service on the given cluster. + [Arguments] ${alias} + Command On Cluster ${alias} systemctl restart ipsec + +Wait For IPsec Tunnel Reestablishment + [Documentation] Wait for IPsec tunnels to be re-established after disruption. + [Arguments] ${alias} ${expected_count}=2 + Wait Until Keyword Succeeds ${IPSEC_SA_TIMEOUT} ${IPSEC_SA_RETRY} + ... Verify All IPsec Tunnels On Cluster ${alias} ${expected_count} + +Get XFRM Byte Counters + [Documentation] Return the total XFRM SA byte counter (sum across all ESP SAs). + [Arguments] ${alias} + ${stdout}= Command On Cluster ${alias} + ... ip -s xfrm state | awk '/bytes/{gsub(/[^0-9]/,"",$1); sum+=$1} END{print sum+0}' + RETURN ${stdout} + +Verify XFRM Counters Incremented + [Documentation] Verify XFRM byte counters have incremented from a baseline value. + [Arguments] ${alias} ${baseline_bytes} + ${current_bytes}= Get XFRM Byte Counters ${alias} + ${baseline_int}= Convert To Integer ${baseline_bytes} + ${current_int}= Convert To Integer ${current_bytes} + Should Be True ${current_int} > ${baseline_int} + ... XFRM byte counter did not increment: baseline=${baseline_int}, current=${current_int} + +Start Tcpdump For ESP On Cluster + [Documentation] Start tcpdump in the background capturing ESP packets. + ... Returns the pcap file path. Uses timeout to auto-exit. + [Arguments] ${alias} ${iface}=enp1s0 ${timeout}=15 + VAR ${pcap_file}= /tmp/esp-capture-${alias}.pcap + Disruptive Command On Cluster ${alias} rm -f ${pcap_file} + Command On Cluster ${alias} + ... nohup timeout ${timeout} tcpdump -i ${iface} -w ${pcap_file} esp &>/dev/null & + Sleep 2s reason=Wait for tcpdump to initialize packet capture + RETURN ${pcap_file} + +Wait For Tcpdump And Verify ESP + [Documentation] Wait briefly for traffic to be captured, then verify ESP packets in pcap. + [Arguments] ${alias} ${pcap_file} + Sleep 5s reason=Wait for ESP packets to be captured + ${stdout}= Command On Cluster ${alias} + ... tcpdump -r ${pcap_file} 2>/dev/null | head -5 + Should Contain ${stdout} ESP + Disruptive Command On Cluster ${alias} rm -f ${pcap_file} + +Flush IPsec SAs On Cluster + [Documentation] Flush all IPsec Security Associations on the given cluster. + [Arguments] ${alias} + Disruptive Command On Cluster ${alias} ip xfrm state flush + +Verify No XFRM SAs Exist + [Documentation] Verify no XFRM SAs are present on the given cluster. + [Arguments] ${alias} + ${stdout}= Command On Cluster ${alias} ip xfrm state list + Should Be Empty ${stdout} + +Add NFTables IPsec Enforcement Rules + [Documentation] Add nftables rules that drop non-IPsec traffic destined for local pod CIDRs. + ... Uses 'meta ipsec missing' match (kernel 5.10+) to detect unencrypted packets. + ... In tunnel mode, decrypted packets have pod/service CIDRs as dst — plaintext + ... packets arriving without ESP are caught by this rule. + [Arguments] ${alias} ${local_pod_cidr} + Command On Cluster ${alias} nft add table inet c2cc_ipsec_test + Command On Cluster ${alias} + ... nft 'add chain inet c2cc_ipsec_test enforce { type filter hook input priority -150; policy accept; }' + Command On Cluster ${alias} + ... nft add rule inet c2cc_ipsec_test enforce ip daddr ${local_pod_cidr} meta ipsec missing counter drop + +Remove NFTables IPsec Enforcement Rules + [Documentation] Remove C2CC IPsec enforcement nftables table and all its rules. + [Arguments] ${alias} + Disruptive Command On Cluster + ... ${alias} + ... nft delete table inet c2cc_ipsec_test 2>/dev/null || true + +Curl Should Fail From Cluster + [Documentation] Verify curl from curl-pod times out or fails (no "Hello from" in response). + [Arguments] ${alias} ${ip} ${port} ${ns} + ${stdout}= Oc On Cluster ${alias} + ... oc exec curl-pod -n ${ns} -- curl -sS --max-time 5 http://${ip}:${port}/cgi-bin/hello 2>&1 || true + ... allow_fail=${TRUE} + Should Not Contain ${stdout} Hello from + +Curl From Host Should Fail + [Documentation] Curl directly from the host (not from a pod) to a remote pod IP. + ... Expects failure — host-originated traffic is not matched by tunnel-mode + ... IPsec selectors scoped to pod/service CIDRs. + [Arguments] ${alias} ${pod_ip} ${port} + ${stdout}= Disruptive Command On Cluster ${alias} + ... curl -sS --max-time 5 http://${pod_ip}:${port}/cgi-bin/hello 2>&1 || true + Should Not Contain ${stdout} Hello from diff --git a/test/scenarios-bootc/el10/presubmits/el102-src@c2cc-ipsec.sh b/test/scenarios-bootc/el10/presubmits/el102-src@c2cc-ipsec.sh new file mode 100644 index 0000000000..c91fb271ec --- /dev/null +++ b/test/scenarios-bootc/el10/presubmits/el102-src@c2cc-ipsec.sh @@ -0,0 +1,295 @@ +#!/bin/bash + +# Sourced from scenario.sh and uses functions defined there. +# +# Sets up 3 MicroShift clusters with C2CC and Libreswan IPsec (tunnel mode +# protecting pod/service CIDRs). Each host maintains IPsec tunnels forming a +# full mesh. Tests validate ESP encapsulation, connectivity, policy +# enforcement, plaintext rejection, and MTU behaviour. + +# IPsec tests have ordering dependencies (setup verification must pass before +# enforcement tests), so disable randomization. +export TEST_RANDOMIZATION=none + +# Cluster A (host1): default MicroShift CIDRs +CLUSTER_A_POD_CIDR="10.42.0.0/16" +CLUSTER_A_SVC_CIDR="10.43.0.0/16" +CLUSTER_A_DOMAIN="cluster-a.remote" + +# Cluster B (host2): non-overlapping CIDRs +CLUSTER_B_POD_CIDR="10.45.0.0/16" +CLUSTER_B_SVC_CIDR="10.46.0.0/16" +CLUSTER_B_DOMAIN="cluster-b.remote" + +# Cluster C (host3): non-overlapping CIDRs +CLUSTER_C_POD_CIDR="10.48.0.0/16" +CLUSTER_C_SVC_CIDR="10.49.0.0/16" +CLUSTER_C_DOMAIN="cluster-c.remote" + +wait_for_greenboot_on_hosts() { + local junit_label=$1 + local host + for host in host1 host2 host3; do + local host_ip full_host + host_ip=$(get_vm_property "${host}" ip) + full_host=$(full_vm_name "${host}") + if ! wait_for_greenboot "${full_host}" "${host_ip}"; then + record_junit "${host}" "${junit_label}" "FAILED" + return 1 + fi + record_junit "${host}" "${junit_label}" "OK" + done +} + +configure_c2cc_host() { + local host=$1 + shift + # Remaining args are sets of 4: remote_ip remote_pod_cidr remote_svc_cidr remote_domain + + run_command_on_vm "${host}" "sudo mkdir -p /etc/microshift/config.d" + + local yaml_content + yaml_content="clusterToCluster:"$'\n'" remoteClusters:" + local firewall_cidrs=() + + while [ $# -gt 0 ]; do + local remote_ip=$1 + local remote_pod_cidr=$2 + local remote_svc_cidr=$3 + local remote_domain=$4 + shift 4 + + yaml_content+=$'\n'" - nextHop: ${remote_ip}" + yaml_content+=$'\n'" clusterNetwork:" + yaml_content+=$'\n'" - ${remote_pod_cidr}" + yaml_content+=$'\n'" serviceNetwork:" + yaml_content+=$'\n'" - ${remote_svc_cidr}" + yaml_content+=$'\n'" domain: ${remote_domain}" + + firewall_cidrs+=("${remote_pod_cidr}" "${remote_svc_cidr}") + done + + run_command_on_vm "${host}" "sudo tee /etc/microshift/config.d/50-c2cc.yaml > /dev/null <&2; return 1; } + host2_ip=$(get_vm_property host2 ip) || { echo "failed to get host2 ip" >&2; return 1; } + host3_ip=$(get_vm_property host3 ip) || { echo "failed to get host3 ip" >&2; return 1; } + readonly host1_ip host2_ip host3_ip + + wait_for_greenboot_on_hosts "c2cc_ipsec_pre_greenboot" + + configure_c2cc_host host1 \ + "${host2_ip}" "${CLUSTER_B_POD_CIDR}" "${CLUSTER_B_SVC_CIDR}" "${CLUSTER_B_DOMAIN}" \ + "${host3_ip}" "${CLUSTER_C_POD_CIDR}" "${CLUSTER_C_SVC_CIDR}" "${CLUSTER_C_DOMAIN}" + + configure_c2cc_host host2 \ + "${host1_ip}" "${CLUSTER_A_POD_CIDR}" "${CLUSTER_A_SVC_CIDR}" "${CLUSTER_A_DOMAIN}" \ + "${host3_ip}" "${CLUSTER_C_POD_CIDR}" "${CLUSTER_C_SVC_CIDR}" "${CLUSTER_C_DOMAIN}" + + configure_c2cc_host host3 \ + "${host1_ip}" "${CLUSTER_A_POD_CIDR}" "${CLUSTER_A_SVC_CIDR}" "${CLUSTER_A_DOMAIN}" \ + "${host2_ip}" "${CLUSTER_B_POD_CIDR}" "${CLUSTER_B_SVC_CIDR}" "${CLUSTER_B_DOMAIN}" + + wait_for_greenboot_on_hosts "c2cc_ipsec_greenboot" +} + +# configure_ipsec_host writes the PSK and connection configs, initializes the +# NSS database, and starts the ipsec service on a single host. +# Libreswan, tcpdump, and firewall rules are pre-installed in the +# rhel102-bootc-source-ipsec container image. +# +# Uses tunnel mode with subnet selectors to protect C2CC traffic (pod/service +# CIDRs). MicroShift C2CC routes cross-cluster traffic as raw IP via the +# host's physical interface — there is no Geneve tunnel between hosts. +# +# Arguments: +# $1 — VM name (host1, host2, host3) +# $2 — this host's IP +# $3 — local pod CIDR +# $4 — local service CIDR +# $5 — pre-shared key (hex string) +# $6..N — sets of 4: remote_ip remote_name remote_pod_cidr remote_svc_cidr +configure_ipsec_host() { + local -r host=$1 + local -r host_ip=$2 + local -r local_pod_cidr=$3 + local -r local_svc_cidr=$4 + local -r psk=$5 + shift 5 + + local secrets_content="" + local conn_content="" + while [ $# -gt 0 ]; do + local remote_ip=$1 + local remote_name=$2 + local remote_pod_cidr=$3 + local remote_svc_cidr=$4 + shift 4 + + secrets_content+="${host_ip} ${remote_ip} : PSK \"${psk}\""$'\n' + + conn_content+="conn c2cc-to-${remote_name}"$'\n' + conn_content+=" type=tunnel"$'\n' + conn_content+=" authby=secret"$'\n' + conn_content+=" left=${host_ip}"$'\n' + conn_content+=" right=${remote_ip}"$'\n' + conn_content+=" leftsubnets={${local_pod_cidr} ${local_svc_cidr}}"$'\n' + conn_content+=" rightsubnets={${remote_pod_cidr} ${remote_svc_cidr}}"$'\n' + conn_content+=" auto=start"$'\n' + conn_content+=" ike=aes256-sha2_256-modp2048"$'\n' + conn_content+=" esp=aes256-sha2_256"$'\n' + conn_content+=" failureshunt=drop"$'\n' + conn_content+=" negotiationshunt=drop"$'\n' + conn_content+=" ikev2=insist"$'\n' + conn_content+=$'\n' + done + + run_command_on_vm "${host}" "sudo tee /etc/ipsec.d/c2cc.secrets > /dev/null < /dev/null </dev/null | grep -c 'type=ESP' || true") + count=$(echo "${count}" | tail -1 | tr -d '\r') + if [ "${count}" -ge "${expected_count}" ]; then + record_junit "${host}" "ipsec_tunnels" "OK" + return 0 + fi + sleep 2 + attempts=$((attempts + 1)) + done + record_junit "${host}" "ipsec_tunnels" "FAILED" + return 1 +} + +configure_ipsec() { + local host1_ip host2_ip host3_ip + host1_ip=$(get_vm_property host1 ip) || { echo "failed to get host1 ip" >&2; return 1; } + host2_ip=$(get_vm_property host2 ip) || { echo "failed to get host2 ip" >&2; return 1; } + host3_ip=$(get_vm_property host3 ip) || { echo "failed to get host3 ip" >&2; return 1; } + readonly host1_ip host2_ip host3_ip + + local psk + psk=$(openssl rand -hex 32) || { echo "failed to generate PSK" >&2; return 1; } + readonly psk + + configure_ipsec_host host1 "${host1_ip}" "${CLUSTER_A_POD_CIDR}" "${CLUSTER_A_SVC_CIDR}" "${psk}" \ + "${host2_ip}" host2 "${CLUSTER_B_POD_CIDR}" "${CLUSTER_B_SVC_CIDR}" \ + "${host3_ip}" host3 "${CLUSTER_C_POD_CIDR}" "${CLUSTER_C_SVC_CIDR}" + + configure_ipsec_host host2 "${host2_ip}" "${CLUSTER_B_POD_CIDR}" "${CLUSTER_B_SVC_CIDR}" "${psk}" \ + "${host1_ip}" host1 "${CLUSTER_A_POD_CIDR}" "${CLUSTER_A_SVC_CIDR}" \ + "${host3_ip}" host3 "${CLUSTER_C_POD_CIDR}" "${CLUSTER_C_SVC_CIDR}" + + configure_ipsec_host host3 "${host3_ip}" "${CLUSTER_C_POD_CIDR}" "${CLUSTER_C_SVC_CIDR}" "${psk}" \ + "${host1_ip}" host1 "${CLUSTER_A_POD_CIDR}" "${CLUSTER_A_SVC_CIDR}" \ + "${host2_ip}" host2 "${CLUSTER_B_POD_CIDR}" "${CLUSTER_B_SVC_CIDR}" + + # Each host has 2 remote hosts × 4 subnet pairs (2 local × 2 remote CIDRs) = 8 child SAs. + for host in host1 host2 host3; do + if ! wait_for_ipsec_tunnels "${host}" 8; then + return 1 + fi + done +} + +scenario_create_vms() { + prepare_kickstart host1 kickstart-bootc.ks.template rhel102-bootc-source-ipsec + prepare_kickstart host2 kickstart-bootc.ks.template rhel102-bootc-source-ipsec + prepare_kickstart host3 kickstart-bootc.ks.template rhel102-bootc-source-ipsec + + local -r host2_ks_dir="${SCENARIO_INFO_DIR}/${SCENARIO}/vms/host2" + cat >> "${host2_ks_dir}/post-microshift.cfg" <>/etc/microshift/config.yaml <> "${host3_ks_dir}/post-microshift.cfg" <>/etc/microshift/config.yaml < /dev/null <&2; return 1; } + host2_ip=$(get_vm_property host2 ip) || { echo "failed to get host2 ip" >&2; return 1; } + host3_ip=$(get_vm_property host3 ip) || { echo "failed to get host3 ip" >&2; return 1; } + readonly host1_ip host2_ip host3_ip + + wait_for_greenboot_on_hosts "c2cc_ipsec_pre_greenboot" + + configure_c2cc_host host1 \ + "${host2_ip}" "${CLUSTER_B_POD_CIDR}" "${CLUSTER_B_SVC_CIDR}" "${CLUSTER_B_DOMAIN}" \ + "${host3_ip}" "${CLUSTER_C_POD_CIDR}" "${CLUSTER_C_SVC_CIDR}" "${CLUSTER_C_DOMAIN}" + + configure_c2cc_host host2 \ + "${host1_ip}" "${CLUSTER_A_POD_CIDR}" "${CLUSTER_A_SVC_CIDR}" "${CLUSTER_A_DOMAIN}" \ + "${host3_ip}" "${CLUSTER_C_POD_CIDR}" "${CLUSTER_C_SVC_CIDR}" "${CLUSTER_C_DOMAIN}" + + configure_c2cc_host host3 \ + "${host1_ip}" "${CLUSTER_A_POD_CIDR}" "${CLUSTER_A_SVC_CIDR}" "${CLUSTER_A_DOMAIN}" \ + "${host2_ip}" "${CLUSTER_B_POD_CIDR}" "${CLUSTER_B_SVC_CIDR}" "${CLUSTER_B_DOMAIN}" + + wait_for_greenboot_on_hosts "c2cc_ipsec_greenboot" +} + +# configure_ipsec_host writes the PSK and connection configs, initializes the +# NSS database, and starts the ipsec service on a single host. +# Libreswan, tcpdump, and firewall rules are pre-installed in the +# rhel98-bootc-source-ipsec container image. +# +# Uses tunnel mode with subnet selectors to protect C2CC traffic (pod/service +# CIDRs). MicroShift C2CC routes cross-cluster traffic as raw IP via the +# host's physical interface — there is no Geneve tunnel between hosts. +# +# Arguments: +# $1 — VM name (host1, host2, host3) +# $2 — this host's IP +# $3 — local pod CIDR +# $4 — local service CIDR +# $5 — pre-shared key (hex string) +# $6..N — sets of 4: remote_ip remote_name remote_pod_cidr remote_svc_cidr +configure_ipsec_host() { + local -r host=$1 + local -r host_ip=$2 + local -r local_pod_cidr=$3 + local -r local_svc_cidr=$4 + local -r psk=$5 + shift 5 + + local secrets_content="" + local conn_content="" + while [ $# -gt 0 ]; do + local remote_ip=$1 + local remote_name=$2 + local remote_pod_cidr=$3 + local remote_svc_cidr=$4 + shift 4 + + secrets_content+="${host_ip} ${remote_ip} : PSK \"${psk}\""$'\n' + + conn_content+="conn c2cc-to-${remote_name}"$'\n' + conn_content+=" type=tunnel"$'\n' + conn_content+=" authby=secret"$'\n' + conn_content+=" left=${host_ip}"$'\n' + conn_content+=" right=${remote_ip}"$'\n' + conn_content+=" leftsubnets={${local_pod_cidr} ${local_svc_cidr}}"$'\n' + conn_content+=" rightsubnets={${remote_pod_cidr} ${remote_svc_cidr}}"$'\n' + conn_content+=" auto=start"$'\n' + conn_content+=" ike=aes256-sha2_256-modp2048"$'\n' + conn_content+=" esp=aes256-sha2_256"$'\n' + conn_content+=" failureshunt=drop"$'\n' + conn_content+=" negotiationshunt=drop"$'\n' + conn_content+=" ikev2=insist"$'\n' + conn_content+=$'\n' + done + + run_command_on_vm "${host}" "sudo tee /etc/ipsec.d/c2cc.secrets > /dev/null < /dev/null </dev/null | grep -c 'type=ESP' || true") + count=$(echo "${count}" | tail -1 | tr -d '\r') + if [ "${count}" -ge "${expected_count}" ]; then + record_junit "${host}" "ipsec_tunnels" "OK" + return 0 + fi + sleep 2 + attempts=$((attempts + 1)) + done + record_junit "${host}" "ipsec_tunnels" "FAILED" + return 1 +} + +configure_ipsec() { + local -r host1_ip=$(get_vm_property host1 ip) + local -r host2_ip=$(get_vm_property host2 ip) + local -r host3_ip=$(get_vm_property host3 ip) + + local -r psk=$(openssl rand -hex 32) + + configure_ipsec_host host1 "${host1_ip}" "${CLUSTER_A_POD_CIDR}" "${CLUSTER_A_SVC_CIDR}" "${psk}" \ + "${host2_ip}" host2 "${CLUSTER_B_POD_CIDR}" "${CLUSTER_B_SVC_CIDR}" \ + "${host3_ip}" host3 "${CLUSTER_C_POD_CIDR}" "${CLUSTER_C_SVC_CIDR}" + + configure_ipsec_host host2 "${host2_ip}" "${CLUSTER_B_POD_CIDR}" "${CLUSTER_B_SVC_CIDR}" "${psk}" \ + "${host1_ip}" host1 "${CLUSTER_A_POD_CIDR}" "${CLUSTER_A_SVC_CIDR}" \ + "${host3_ip}" host3 "${CLUSTER_C_POD_CIDR}" "${CLUSTER_C_SVC_CIDR}" + + configure_ipsec_host host3 "${host3_ip}" "${CLUSTER_C_POD_CIDR}" "${CLUSTER_C_SVC_CIDR}" "${psk}" \ + "${host1_ip}" host1 "${CLUSTER_A_POD_CIDR}" "${CLUSTER_A_SVC_CIDR}" \ + "${host2_ip}" host2 "${CLUSTER_B_POD_CIDR}" "${CLUSTER_B_SVC_CIDR}" + + # Each host has 2 remote hosts × 4 subnet pairs (2 local × 2 remote CIDRs) = 8 child SAs. + for host in host1 host2 host3; do + if ! wait_for_ipsec_tunnels "${host}" 8; then + return 1 + fi + done +} + +scenario_create_vms() { + prepare_kickstart host1 kickstart-bootc.ks.template rhel98-bootc-source-ipsec + prepare_kickstart host2 kickstart-bootc.ks.template rhel98-bootc-source-ipsec + prepare_kickstart host3 kickstart-bootc.ks.template rhel98-bootc-source-ipsec + + local -r host2_ks_dir="${SCENARIO_INFO_DIR}/${SCENARIO}/vms/host2" + cat >> "${host2_ks_dir}/post-microshift.cfg" <>/etc/microshift/config.yaml <> "${host3_ks_dir}/post-microshift.cfg" <>/etc/microshift/config.yaml </dev/null | curl -sS --max-time 15 --data-binary @- http://${ip}:8080/cgi-bin/hello' + Should Contain ${stdout} Hello from diff --git a/test/suites/c2cc/connectivity.robot b/test/suites/c2cc/connectivity.robot index e645045243..ad37aa8cce 100644 --- a/test/suites/c2cc/connectivity.robot +++ b/test/suites/c2cc/connectivity.robot @@ -14,10 +14,6 @@ Suite Teardown Teardown Test Tags c2cc -*** Variables *** -&{NAMESPACES} cluster-a=${EMPTY} cluster-b=${EMPTY} cluster-c=${EMPTY} - - *** Test Cases *** Test Cross Cluster Connectivity [Documentation] Verify pods on all clusters can reach pods/services on all other clusters. @@ -69,60 +65,3 @@ Teardown Teardown All Remote Clusters Remove Kubeconfig Logout MicroShift Host - -Test Connectivity Between Clusters - [Documentation] Verify pod on ${source} can reach ${endpoint_type} IP on ${destination} - [Arguments] ${source} ${destination} ${endpoint_type} - IF '${endpoint_type}' == 'pod' - ${ip_dest}= Get Hello Pod IP ${destination} - ELSE IF '${endpoint_type}' == 'service' - ${ip_dest}= Get Hello Service IP ${destination} - ELSE - Fail Invalid endpoint_type: ${endpoint_type}. Must be 'pod' or 'service'. - END - - ${stdout}= Curl From Cluster ${source} ${ip_dest} 8080 - Should Contain ${stdout} Hello from - -Test Source IP Preserved Between Clusters - [Documentation] Verify ${source} to ${destination} pod-to-${endpoint_type} traffic preserves the source pod IP (no SNAT). - [Arguments] ${source} ${destination} ${endpoint_type} - ${curl_pod_ip}= Get Curl Pod IP ${source} - IF '${endpoint_type}' == 'pod' - ${ip_dest}= Get Hello Pod IP ${destination} - ELSE IF '${endpoint_type}' == 'service' - ${ip_dest}= Get Hello Service IP ${destination} - ELSE - Fail Invalid endpoint_type: ${endpoint_type}. Must be 'pod' or 'service'. - END - - ${stdout}= Curl From Cluster ${source} ${ip_dest} 8080 - Should Contain ${stdout} source: ${curl_pod_ip} - -Get Hello Pod IP - [Documentation] Get the pod IP of hello-microshift on the given cluster. - [Arguments] ${alias} - ${ip}= Oc On Cluster ${alias} - ... oc get pod hello-microshift -n ${NAMESPACES}[${alias}] -o jsonpath='{.status.podIP}' - RETURN ${ip} - -Get Hello Service IP - [Documentation] Get the ClusterIP of the hello-microshift service on the given cluster. - [Arguments] ${alias} - ${ip}= Oc On Cluster ${alias} - ... oc get svc hello-microshift -n ${NAMESPACES}[${alias}] -o jsonpath='{.spec.clusterIP}' - RETURN ${ip} - -Get Curl Pod IP - [Documentation] Get the pod IP of curl-pod on the given cluster. - [Arguments] ${alias} - ${ip}= Oc On Cluster ${alias} - ... oc get pod curl-pod -n ${NAMESPACES}[${alias}] -o jsonpath='{.status.podIP}' - RETURN ${ip} - -Curl From Cluster - [Documentation] Exec curl from curl-pod on the given cluster to the target IP and port. - [Arguments] ${alias} ${ip} ${port} - ${stdout}= Oc On Cluster ${alias} - ... oc exec curl-pod -n ${NAMESPACES}[${alias}] -- curl -sS --max-time 10 http://${ip}:${port}/cgi-bin/hello - RETURN ${stdout} diff --git a/test/suites/c2cc/dns.robot b/test/suites/c2cc/dns.robot index 21438b79a4..c33b57286e 100644 --- a/test/suites/c2cc/dns.robot +++ b/test/suites/c2cc/dns.robot @@ -15,11 +15,6 @@ Suite Teardown Teardown Test Tags c2cc -*** Variables *** -&{NAMESPACES} cluster-a=${EMPTY} cluster-b=${EMPTY} cluster-c=${EMPTY} -&{DOMAIN_MAP} cluster-a=${CLUSTER_A_DOMAIN} cluster-b=${CLUSTER_B_DOMAIN} cluster-c=${CLUSTER_C_DOMAIN} - - *** Test Cases *** Test Corefile Contains C2CC Server Block [Documentation] Verify every cluster's Corefile has a server block for every other cluster domain. @@ -69,38 +64,3 @@ Teardown Teardown All Remote Clusters Remove Kubeconfig Logout MicroShift Host - -Curl Remote Service Via DNS - [Documentation] Verify pod on ${source} can reach a service on ${destination} using the remote DNS name. - [Arguments] ${source} ${destination} - ${stdout}= Curl DNS From Cluster ${source} - ... hello-microshift.${NAMESPACES}[${destination}].svc.${DOMAIN_MAP}[${destination}] 8080 - Should Contain ${stdout} Hello from - -DNS Resolve From Cluster - [Documentation] Resolve a DNS name from curl-pod on the given cluster. Retries for up to 60s. - [Arguments] ${alias} ${fqdn} - Wait Until Keyword Succeeds 12x 5s - ... DNS Lookup Should Succeed ${alias} ${fqdn} - -DNS Lookup Should Succeed - [Documentation] Resolve a DNS name from curl-pod using getent hosts. - [Arguments] ${alias} ${fqdn} - ${stdout}= Oc On Cluster ${alias} - ... oc exec curl-pod -n ${NAMESPACES}[${alias}] -- getent hosts ${fqdn} - Should Not Be Empty ${stdout} - -Curl DNS From Cluster - [Documentation] Curl a service by DNS name from curl-pod on the given cluster. - [Arguments] ${alias} ${fqdn} ${port} - ${stdout}= Wait Until Keyword Succeeds 12x 5s - ... Curl DNS Should Succeed ${alias} ${fqdn} ${port} - RETURN ${stdout} - -Curl DNS Should Succeed - [Documentation] Single attempt to curl a DNS name from curl-pod. - [Arguments] ${alias} ${fqdn} ${port} - ${stdout}= Oc On Cluster ${alias} - ... oc exec curl-pod -n ${NAMESPACES}[${alias}] -- curl -sS --max-time 10 http://${fqdn}:${port}/cgi-bin/hello - Should Contain ${stdout} Hello from - RETURN ${stdout}