From 5e0a80e965f60a9189cc19babbfd40280765abee Mon Sep 17 00:00:00 2001 From: Patryk Matuszak Date: Wed, 27 May 2026 15:30:40 +0200 Subject: [PATCH 1/5] Add Robot Framework resource for IPsec test keywords Provides reusable keywords for Libreswan/XFRM operations: - Tunnel verification (single tunnel + mesh count check) - XFRM byte counter baselines and increment assertions - ESP packet capture via background tcpdump - SA flush and verification - IPsec service lifecycle (start/stop/restart/wait) - nftables enforcement rule management (meta ipsec missing) - Negative curl assertions (pod-to-pod and host-to-pod) --- test/resources/ipsec.resource | 132 ++++++++++++++++++++++++++++++++++ 1 file changed, 132 insertions(+) create mode 100644 test/resources/ipsec.resource diff --git a/test/resources/ipsec.resource b/test/resources/ipsec.resource new file mode 100644 index 0000000000..b87f9edd22 --- /dev/null +++ b/test/resources/ipsec.resource @@ -0,0 +1,132 @@ +*** Settings *** +Documentation Keywords for IPsec (Libreswan) operations in C2CC test scenarios. +... Provides tunnel verification, XFRM counter checks, ESP packet capture, +... SA management, and nftables IPsec enforcement helpers. + +Library String +Resource c2cc.resource + + +*** Variables *** +${IPSEC_SA_TIMEOUT} 60s +${IPSEC_SA_RETRY} 5s + + +*** Keywords *** +Verify IPsec Tunnel Is Established + [Documentation] Verify that at least one IPsec tunnel is established on the given cluster. + [Arguments] ${alias} + ${stdout}= Command On Cluster ${alias} ipsec trafficstatus + Should Contain ${stdout} type=ESP + +Verify All IPsec Tunnels On Cluster + [Documentation] Verify that the expected number of IPsec tunnels are established. + [Arguments] ${alias} ${expected_count}=2 + ${stdout}= Command On Cluster ${alias} ipsec trafficstatus + ${count}= Get Count ${stdout} type=ESP + Should Be True ${count} >= ${expected_count} + ... Expected ${expected_count} IPsec tunnels but found ${count} + +Stop IPsec Service On Cluster + [Documentation] Stop the ipsec systemd service on the given cluster. + [Arguments] ${alias} + Disruptive Command On Cluster ${alias} systemctl stop ipsec + +Start IPsec Service On Cluster + [Documentation] Start the ipsec systemd service on the given cluster. + [Arguments] ${alias} + Command On Cluster ${alias} systemctl start ipsec + +Restart IPsec Service On Cluster + [Documentation] Restart the ipsec systemd service on the given cluster. + [Arguments] ${alias} + Command On Cluster ${alias} systemctl restart ipsec + +Wait For IPsec Tunnel Reestablishment + [Documentation] Wait for IPsec tunnels to be re-established after disruption. + [Arguments] ${alias} ${expected_count}=2 + Wait Until Keyword Succeeds ${IPSEC_SA_TIMEOUT} ${IPSEC_SA_RETRY} + ... Verify All IPsec Tunnels On Cluster ${alias} ${expected_count} + +Get XFRM Byte Counters + [Documentation] Return the total XFRM SA byte counter (sum across all ESP SAs). + [Arguments] ${alias} + ${stdout}= Command On Cluster ${alias} + ... ip -s xfrm state | awk '/bytes/{gsub(/[^0-9]/,"",$1); sum+=$1} END{print sum+0}' + RETURN ${stdout} + +Verify XFRM Counters Incremented + [Documentation] Verify XFRM byte counters have incremented from a baseline value. + [Arguments] ${alias} ${baseline_bytes} + ${current_bytes}= Get XFRM Byte Counters ${alias} + ${baseline_int}= Convert To Integer ${baseline_bytes} + ${current_int}= Convert To Integer ${current_bytes} + Should Be True ${current_int} > ${baseline_int} + ... XFRM byte counter did not increment: baseline=${baseline_int}, current=${current_int} + +Start Tcpdump For ESP On Cluster + [Documentation] Start tcpdump in the background capturing ESP packets. + ... Returns the pcap file path. Uses timeout to auto-exit. + [Arguments] ${alias} ${iface}=enp1s0 ${timeout}=15 + VAR ${pcap_file}= /tmp/esp-capture-${alias}.pcap + Disruptive Command On Cluster ${alias} rm -f ${pcap_file} + Command On Cluster ${alias} + ... nohup timeout ${timeout} tcpdump -i ${iface} -w ${pcap_file} esp &>/dev/null & + Sleep 2s reason=Wait for tcpdump to initialize packet capture + RETURN ${pcap_file} + +Wait For Tcpdump And Verify ESP + [Documentation] Wait briefly for traffic to be captured, then verify ESP packets in pcap. + [Arguments] ${alias} ${pcap_file} + Sleep 5s reason=Wait for ESP packets to be captured + ${stdout}= Command On Cluster ${alias} + ... tcpdump -r ${pcap_file} 2>/dev/null | head -5 + Should Contain ${stdout} ESP + Disruptive Command On Cluster ${alias} rm -f ${pcap_file} + +Flush IPsec SAs On Cluster + [Documentation] Flush all IPsec Security Associations on the given cluster. + [Arguments] ${alias} + Disruptive Command On Cluster ${alias} ip xfrm state flush + +Verify No XFRM SAs Exist + [Documentation] Verify no XFRM SAs are present on the given cluster. + [Arguments] ${alias} + ${stdout}= Command On Cluster ${alias} ip xfrm state list + Should Be Empty ${stdout} + +Add NFTables IPsec Enforcement Rules + [Documentation] Add nftables rules that drop non-IPsec traffic destined for local pod CIDRs. + ... Uses 'meta ipsec missing' match (kernel 5.10+) to detect unencrypted packets. + ... In tunnel mode, decrypted packets have pod/service CIDRs as dst — plaintext + ... packets arriving without ESP are caught by this rule. + [Arguments] ${alias} ${local_pod_cidr} + Command On Cluster ${alias} nft add table inet c2cc_ipsec_test + Command On Cluster ${alias} + ... nft 'add chain inet c2cc_ipsec_test enforce { type filter hook input priority -150; policy accept; }' + Command On Cluster ${alias} + ... nft add rule inet c2cc_ipsec_test enforce ip daddr ${local_pod_cidr} meta ipsec missing counter drop + +Remove NFTables IPsec Enforcement Rules + [Documentation] Remove C2CC IPsec enforcement nftables table and all its rules. + [Arguments] ${alias} + Disruptive Command On Cluster + ... ${alias} + ... nft delete table inet c2cc_ipsec_test 2>/dev/null || true + +Curl Should Fail From Cluster + [Documentation] Verify curl from curl-pod times out or fails (no "Hello from" in response). + [Arguments] ${alias} ${ip} ${port} ${ns} + ${stdout}= Oc On Cluster ${alias} + ... oc exec curl-pod -n ${ns} -- curl -sS --max-time 5 http://${ip}:${port}/cgi-bin/hello 2>&1 || true + ... allow_fail=${TRUE} + Should Not Contain ${stdout} Hello from + +Curl From Host Should Fail + [Documentation] Curl directly from the host (not from a pod) to a remote pod IP. + ... Expects failure — host-originated traffic is not matched by tunnel-mode + ... IPsec selectors scoped to pod/service CIDRs. + [Arguments] ${alias} ${pod_ip} ${port} + ${stdout}= Disruptive Command On Cluster ${alias} + ... curl -sS --max-time 5 http://${pod_ip}:${port}/cgi-bin/hello 2>&1 || true + Should Not Contain ${stdout} Hello from From 1334dec175a3c89ca07b5667257601eefefeffae Mon Sep 17 00:00:00 2001 From: Patryk Matuszak Date: Wed, 27 May 2026 15:44:46 +0200 Subject: [PATCH 2/5] Add C2CC IPsec E2E test suite MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Seven test cases validating C2CC connectivity through Libreswan transport-mode IPsec across a 3-cluster mesh: 1. Tunnel establishment — all hosts have 2 tunnels each 2. ESP encapsulation — tcpdump + XFRM byte counter verification 3. Connectivity — pod-to-pod and pod-to-service across all pairs 4. Source IP preservation — no SNAT through the tunnel 5. Policy enforcement — SA flush blocks traffic, restart restores it 6. Plaintext rejection — stopped IPsec + nftables enforcement drops 7. MTU validation — near-MTU payloads through Geneve + ESP Uses the same test workloads (hello-microshift + curl-pod) and multi-cluster keywords (Deploy Test Workloads, etc.) as the existing c2cc suite. --- test/suites/c2cc-ipsec/ipsec.robot | 211 +++++++++++++++++++++++++++++ 1 file changed, 211 insertions(+) create mode 100644 test/suites/c2cc-ipsec/ipsec.robot diff --git a/test/suites/c2cc-ipsec/ipsec.robot b/test/suites/c2cc-ipsec/ipsec.robot new file mode 100644 index 0000000000..9ac1b93e95 --- /dev/null +++ b/test/suites/c2cc-ipsec/ipsec.robot @@ -0,0 +1,211 @@ +*** Settings *** +Documentation IPsec E2E tests for C2CC. +... Validates that C2CC cross-cluster connectivity works through a +... Libreswan tunnel-mode IPsec mesh (subnet selectors, no Geneve). +... +... Tests cover ESP encapsulation, connectivity through the tunnel, +... source IP preservation, policy enforcement (SA flush/restore), +... plaintext rejection, host-to-pod rejection, and MTU validation. + +Resource ../../resources/microshift-process.resource +Resource ../../resources/kubeconfig.resource +Resource ../../resources/oc.resource +Resource ../../resources/c2cc.resource +Resource ../../resources/ipsec.resource + +Suite Setup Setup +Suite Teardown Teardown + +Test Tags c2cc ipsec + + +*** Variables *** +&{NAMESPACES} cluster-a=${EMPTY} cluster-b=${EMPTY} cluster-c=${EMPTY} + + +*** Test Cases *** +IPsec Tunnels Established On All Clusters + [Documentation] Verify all 3 hosts have 8 IPsec tunnel SAs each. + ... Each host has 2 remote hosts x 4 subnet pairs (2 local x 2 remote CIDRs). + Verify All IPsec Tunnels On Cluster cluster-a expected_count=8 + Verify All IPsec Tunnels On Cluster cluster-b expected_count=8 + Verify All IPsec Tunnels On Cluster cluster-c expected_count=8 + +ESP Encapsulation On Wire + [Documentation] Capture packets on the wire and verify ESP encapsulation. + ... Records XFRM byte counters before and after traffic, captures ESP packets + ... via tcpdump, and verifies counters incremented. + ${baseline_a}= Get XFRM Byte Counters cluster-a + ${baseline_b}= Get XFRM Byte Counters cluster-b + + ${pcap_file}= Start Tcpdump For ESP On Cluster cluster-b + ${ip_dest}= Get Hello Pod IP cluster-b + Curl From Cluster cluster-a ${ip_dest} 8080 + Wait For Tcpdump And Verify ESP cluster-b ${pcap_file} + + Verify XFRM Counters Incremented cluster-a ${baseline_a} + Verify XFRM Counters Incremented cluster-b ${baseline_b} + +Cross Cluster Connectivity Through IPsec + [Documentation] Verify pods on all clusters can reach pods/services on all + ... other clusters through the IPsec tunnel. + [Template] Test Connectivity Between Clusters + cluster-a cluster-b pod + cluster-a cluster-b service + cluster-a cluster-c pod + cluster-a cluster-c service + cluster-b cluster-a pod + cluster-b cluster-a service + cluster-b cluster-c pod + cluster-b cluster-c service + cluster-c cluster-a pod + cluster-c cluster-a service + cluster-c cluster-b pod + cluster-c cluster-b service + +Source IP Preserved Through IPsec + [Documentation] Verify cross-cluster traffic through IPsec preserves the + ... source pod IP (no SNAT). + [Template] Test Source IP Preserved Between Clusters + cluster-a cluster-b pod + cluster-a cluster-b service + cluster-a cluster-c pod + cluster-a cluster-c service + cluster-b cluster-a pod + cluster-b cluster-a service + cluster-b cluster-c pod + cluster-b cluster-c service + cluster-c cluster-a pod + cluster-c cluster-a service + cluster-c cluster-b pod + cluster-c cluster-b service + +Plaintext Rejection When IPsec Stopped + [Documentation] Stop IPsec on cluster-a. With nftables enforcement rules on + ... cluster-b, verify traffic is dropped rather than sent in plaintext. + [Setup] Add NFTables IPsec Enforcement Rules cluster-b ${CLUSTER_B_POD_CIDR} + + Stop IPsec Service On Cluster cluster-a + Sleep 3s reason=Wait for SAs to expire + ${ip_dest}= Get Hello Pod IP cluster-b + Curl Should Fail From Cluster cluster-a ${ip_dest} 8080 ${NAMESPACES}[cluster-a] + + [Teardown] Restore IPsec With Enforcement Cleanup cluster-a cluster-b + +Host To Remote Pod Rejected Without IPsec + [Documentation] Curl directly from cluster-a's host to a pod on cluster-b. + ... Host-originated traffic is not matched by tunnel-mode IPsec selectors + ... scoped to pod/service CIDRs, so it cannot reach the remote pod. + ${pod_ip}= Get Hello Pod IP cluster-b + Curl From Host Should Fail cluster-a ${pod_ip} 8080 + +Near MTU Packet Through IPsec Tunnel + [Documentation] Send near-MTU-sized payloads through IPsec tunnel-mode + ... encapsulation. Verifies no MTU blackhole from DF-bit issues. + ... ESP overhead ~36-52B total. + ${ip_dest}= Get Hello Pod IP cluster-b + Send Large Payload And Verify cluster-a ${ip_dest} 1300 + Send Large Payload And Verify cluster-a ${ip_dest} 1400 + + +*** Keywords *** +Setup + [Documentation] Register all clusters, verify IPsec tunnels, deploy test workloads. + Check Required Env Variables + Login MicroShift Host + Setup Kubeconfig + Register Local Cluster cluster-a + Register Remote Cluster cluster-b ${HOST2_IP} ${HOST2_SSH_PORT} ${KUBECONFIG_B} + Register Remote Cluster cluster-c ${HOST3_IP} ${HOST3_SSH_PORT} ${KUBECONFIG_C} + Deploy Test Workloads + +Teardown + [Documentation] Remove test workloads, ensure IPsec is running, close connections. + Cleanup Test Workloads + Ensure IPsec Running On All Clusters + Teardown All Remote Clusters + Remove Kubeconfig + Logout MicroShift Host + +Ensure IPsec Running On All Clusters + [Documentation] Make sure ipsec service is running on all clusters. + ... Tests may have stopped it. + FOR ${alias} IN cluster-a cluster-b cluster-c + Start IPsec Service On Cluster ${alias} + END + +Restore IPsec And Verify + [Documentation] Restart ipsec and wait for tunnels to come back up. + [Arguments] ${alias} + Restart IPsec Service On Cluster ${alias} + Wait For IPsec Tunnel Reestablishment ${alias} expected_count=8 + +Restore IPsec With Enforcement Cleanup + [Documentation] Remove enforcement rules and restore IPsec. + [Arguments] ${ipsec_alias} ${enforcement_alias} + Remove NFTables IPsec Enforcement Rules ${enforcement_alias} + Start IPsec Service On Cluster ${ipsec_alias} + Wait For IPsec Tunnel Reestablishment ${ipsec_alias} expected_count=8 + +Get Hello Pod IP + [Documentation] Get the pod IP of hello-microshift on the given cluster. + [Arguments] ${alias} + ${ip}= Oc On Cluster ${alias} + ... oc get pod hello-microshift -n ${NAMESPACES}[${alias}] -o jsonpath='{.status.podIP}' + RETURN ${ip} + +Get Hello Service IP + [Documentation] Get the ClusterIP of the hello-microshift service on the given cluster. + [Arguments] ${alias} + ${ip}= Oc On Cluster ${alias} + ... oc get svc hello-microshift -n ${NAMESPACES}[${alias}] -o jsonpath='{.spec.clusterIP}' + RETURN ${ip} + +Get Curl Pod IP + [Documentation] Get the pod IP of curl-pod on the given cluster. + [Arguments] ${alias} + ${ip}= Oc On Cluster ${alias} + ... oc get pod curl-pod -n ${NAMESPACES}[${alias}] -o jsonpath='{.status.podIP}' + RETURN ${ip} + +Curl From Cluster + [Documentation] Exec curl from curl-pod on the given cluster. + [Arguments] ${alias} ${ip} ${port} + ${stdout}= Oc On Cluster ${alias} + ... oc exec curl-pod -n ${NAMESPACES}[${alias}] -- curl -sS --max-time 10 http://${ip}:${port}/cgi-bin/hello + RETURN ${stdout} + +Test Connectivity Between Clusters + [Documentation] Verify pod on source can reach endpoint on destination. + [Arguments] ${source} ${destination} ${endpoint_type} + IF '${endpoint_type}' == 'pod' + ${ip_dest}= Get Hello Pod IP ${destination} + ELSE IF '${endpoint_type}' == 'service' + ${ip_dest}= Get Hello Service IP ${destination} + ELSE + Fail Invalid endpoint_type: ${endpoint_type}. Must be 'pod' or 'service'. + END + ${stdout}= Curl From Cluster ${source} ${ip_dest} 8080 + Should Contain ${stdout} Hello from + +Test Source IP Preserved Between Clusters + [Documentation] Verify source pod IP is preserved through IPsec tunnel (no SNAT). + [Arguments] ${source} ${destination} ${endpoint_type} + ${curl_pod_ip}= Get Curl Pod IP ${source} + IF '${endpoint_type}' == 'pod' + ${ip_dest}= Get Hello Pod IP ${destination} + ELSE IF '${endpoint_type}' == 'service' + ${ip_dest}= Get Hello Service IP ${destination} + ELSE + Fail Invalid endpoint_type: ${endpoint_type}. Must be 'pod' or 'service'. + END + ${stdout}= Curl From Cluster ${source} ${ip_dest} 8080 + Should Contain ${stdout} source: ${curl_pod_ip} + +Send Large Payload And Verify + [Documentation] Send a large payload via curl POST and verify it succeeds. + [Arguments] ${alias} ${ip} ${size} + ${stdout}= Oc On Cluster + ... ${alias} + ... oc exec curl-pod -n ${NAMESPACES}[${alias}] -- sh -c 'dd if=/dev/zero bs=${size} count=1 2>/dev/null | curl -sS --max-time 15 --data-binary @- http://${ip}:8080/cgi-bin/hello' + Should Contain ${stdout} Hello from From a1c7108f6aec990ff7da761695ce7d5363f14d1a Mon Sep 17 00:00:00 2001 From: Patryk Matuszak Date: Wed, 27 May 2026 14:12:38 +0200 Subject: [PATCH 3/5] Add C2CC IPsec 3-VM test scenario Set up a Libreswan transport-mode IPsec mesh across 3 MicroShift clusters. The scenario reuses the C2CC CIDR layout and VM creation from el98-src@c2cc.sh, then installs libreswan/tcpdump via bootc usr-overlay, distributes a shared PSK, writes per-host connection configs scoped to Geneve UDP/6081, and waits for all 6 tunnels (2 per host) to establish. failureshunt=drop / negotiationshunt=drop ensure traffic is never sent in plaintext, which the Robot Framework tests will assert. --- .../rhel102-bootc-source-ipsec.containerfile | 10 + .../rhel98-bootc-source-ipsec.containerfile | 10 + .../el10/presubmits/el102-src@c2cc-ipsec.sh | 295 ++++++++++++++++++ .../el9/presubmits/el98-src@c2cc-ipsec.sh | 291 +++++++++++++++++ 4 files changed, 606 insertions(+) create mode 100644 test/image-blueprints-bootc/el10/layer2-presubmit/group2/rhel102-bootc-source-ipsec.containerfile create mode 100644 test/image-blueprints-bootc/el9/layer2-presubmit/group2/rhel98-bootc-source-ipsec.containerfile create mode 100644 test/scenarios-bootc/el10/presubmits/el102-src@c2cc-ipsec.sh create mode 100644 test/scenarios-bootc/el9/presubmits/el98-src@c2cc-ipsec.sh diff --git a/test/image-blueprints-bootc/el10/layer2-presubmit/group2/rhel102-bootc-source-ipsec.containerfile b/test/image-blueprints-bootc/el10/layer2-presubmit/group2/rhel102-bootc-source-ipsec.containerfile new file mode 100644 index 0000000000..5040c3227d --- /dev/null +++ b/test/image-blueprints-bootc/el10/layer2-presubmit/group2/rhel102-bootc-source-ipsec.containerfile @@ -0,0 +1,10 @@ +FROM localhost/rhel102-bootc-source:latest + +# Install Libreswan (IPsec) and tcpdump for C2CC IPsec E2E tests. +# Libreswan provides the ipsec service and pluto daemon. +# tcpdump is used to capture and verify ESP-encapsulated packets. +RUN dnf install -y libreswan tcpdump && \ + dnf clean all + +# Pre-configure firewall for IKE (UDP 500/4500) and ESP (protocol 50). +RUN firewall-offline-cmd --zone=public --add-service=ipsec diff --git a/test/image-blueprints-bootc/el9/layer2-presubmit/group2/rhel98-bootc-source-ipsec.containerfile b/test/image-blueprints-bootc/el9/layer2-presubmit/group2/rhel98-bootc-source-ipsec.containerfile new file mode 100644 index 0000000000..739a0c5a44 --- /dev/null +++ b/test/image-blueprints-bootc/el9/layer2-presubmit/group2/rhel98-bootc-source-ipsec.containerfile @@ -0,0 +1,10 @@ +FROM localhost/rhel98-bootc-source:latest + +# Install Libreswan (IPsec) and tcpdump for C2CC IPsec E2E tests. +# Libreswan provides the ipsec service and pluto daemon. +# tcpdump is used to capture and verify ESP-encapsulated packets. +RUN dnf install -y libreswan tcpdump && \ + dnf clean all + +# Pre-configure firewall for IKE (UDP 500/4500) and ESP (protocol 50). +RUN firewall-offline-cmd --zone=public --add-service=ipsec diff --git a/test/scenarios-bootc/el10/presubmits/el102-src@c2cc-ipsec.sh b/test/scenarios-bootc/el10/presubmits/el102-src@c2cc-ipsec.sh new file mode 100644 index 0000000000..c91fb271ec --- /dev/null +++ b/test/scenarios-bootc/el10/presubmits/el102-src@c2cc-ipsec.sh @@ -0,0 +1,295 @@ +#!/bin/bash + +# Sourced from scenario.sh and uses functions defined there. +# +# Sets up 3 MicroShift clusters with C2CC and Libreswan IPsec (tunnel mode +# protecting pod/service CIDRs). Each host maintains IPsec tunnels forming a +# full mesh. Tests validate ESP encapsulation, connectivity, policy +# enforcement, plaintext rejection, and MTU behaviour. + +# IPsec tests have ordering dependencies (setup verification must pass before +# enforcement tests), so disable randomization. +export TEST_RANDOMIZATION=none + +# Cluster A (host1): default MicroShift CIDRs +CLUSTER_A_POD_CIDR="10.42.0.0/16" +CLUSTER_A_SVC_CIDR="10.43.0.0/16" +CLUSTER_A_DOMAIN="cluster-a.remote" + +# Cluster B (host2): non-overlapping CIDRs +CLUSTER_B_POD_CIDR="10.45.0.0/16" +CLUSTER_B_SVC_CIDR="10.46.0.0/16" +CLUSTER_B_DOMAIN="cluster-b.remote" + +# Cluster C (host3): non-overlapping CIDRs +CLUSTER_C_POD_CIDR="10.48.0.0/16" +CLUSTER_C_SVC_CIDR="10.49.0.0/16" +CLUSTER_C_DOMAIN="cluster-c.remote" + +wait_for_greenboot_on_hosts() { + local junit_label=$1 + local host + for host in host1 host2 host3; do + local host_ip full_host + host_ip=$(get_vm_property "${host}" ip) + full_host=$(full_vm_name "${host}") + if ! wait_for_greenboot "${full_host}" "${host_ip}"; then + record_junit "${host}" "${junit_label}" "FAILED" + return 1 + fi + record_junit "${host}" "${junit_label}" "OK" + done +} + +configure_c2cc_host() { + local host=$1 + shift + # Remaining args are sets of 4: remote_ip remote_pod_cidr remote_svc_cidr remote_domain + + run_command_on_vm "${host}" "sudo mkdir -p /etc/microshift/config.d" + + local yaml_content + yaml_content="clusterToCluster:"$'\n'" remoteClusters:" + local firewall_cidrs=() + + while [ $# -gt 0 ]; do + local remote_ip=$1 + local remote_pod_cidr=$2 + local remote_svc_cidr=$3 + local remote_domain=$4 + shift 4 + + yaml_content+=$'\n'" - nextHop: ${remote_ip}" + yaml_content+=$'\n'" clusterNetwork:" + yaml_content+=$'\n'" - ${remote_pod_cidr}" + yaml_content+=$'\n'" serviceNetwork:" + yaml_content+=$'\n'" - ${remote_svc_cidr}" + yaml_content+=$'\n'" domain: ${remote_domain}" + + firewall_cidrs+=("${remote_pod_cidr}" "${remote_svc_cidr}") + done + + run_command_on_vm "${host}" "sudo tee /etc/microshift/config.d/50-c2cc.yaml > /dev/null <&2; return 1; } + host2_ip=$(get_vm_property host2 ip) || { echo "failed to get host2 ip" >&2; return 1; } + host3_ip=$(get_vm_property host3 ip) || { echo "failed to get host3 ip" >&2; return 1; } + readonly host1_ip host2_ip host3_ip + + wait_for_greenboot_on_hosts "c2cc_ipsec_pre_greenboot" + + configure_c2cc_host host1 \ + "${host2_ip}" "${CLUSTER_B_POD_CIDR}" "${CLUSTER_B_SVC_CIDR}" "${CLUSTER_B_DOMAIN}" \ + "${host3_ip}" "${CLUSTER_C_POD_CIDR}" "${CLUSTER_C_SVC_CIDR}" "${CLUSTER_C_DOMAIN}" + + configure_c2cc_host host2 \ + "${host1_ip}" "${CLUSTER_A_POD_CIDR}" "${CLUSTER_A_SVC_CIDR}" "${CLUSTER_A_DOMAIN}" \ + "${host3_ip}" "${CLUSTER_C_POD_CIDR}" "${CLUSTER_C_SVC_CIDR}" "${CLUSTER_C_DOMAIN}" + + configure_c2cc_host host3 \ + "${host1_ip}" "${CLUSTER_A_POD_CIDR}" "${CLUSTER_A_SVC_CIDR}" "${CLUSTER_A_DOMAIN}" \ + "${host2_ip}" "${CLUSTER_B_POD_CIDR}" "${CLUSTER_B_SVC_CIDR}" "${CLUSTER_B_DOMAIN}" + + wait_for_greenboot_on_hosts "c2cc_ipsec_greenboot" +} + +# configure_ipsec_host writes the PSK and connection configs, initializes the +# NSS database, and starts the ipsec service on a single host. +# Libreswan, tcpdump, and firewall rules are pre-installed in the +# rhel102-bootc-source-ipsec container image. +# +# Uses tunnel mode with subnet selectors to protect C2CC traffic (pod/service +# CIDRs). MicroShift C2CC routes cross-cluster traffic as raw IP via the +# host's physical interface — there is no Geneve tunnel between hosts. +# +# Arguments: +# $1 — VM name (host1, host2, host3) +# $2 — this host's IP +# $3 — local pod CIDR +# $4 — local service CIDR +# $5 — pre-shared key (hex string) +# $6..N — sets of 4: remote_ip remote_name remote_pod_cidr remote_svc_cidr +configure_ipsec_host() { + local -r host=$1 + local -r host_ip=$2 + local -r local_pod_cidr=$3 + local -r local_svc_cidr=$4 + local -r psk=$5 + shift 5 + + local secrets_content="" + local conn_content="" + while [ $# -gt 0 ]; do + local remote_ip=$1 + local remote_name=$2 + local remote_pod_cidr=$3 + local remote_svc_cidr=$4 + shift 4 + + secrets_content+="${host_ip} ${remote_ip} : PSK \"${psk}\""$'\n' + + conn_content+="conn c2cc-to-${remote_name}"$'\n' + conn_content+=" type=tunnel"$'\n' + conn_content+=" authby=secret"$'\n' + conn_content+=" left=${host_ip}"$'\n' + conn_content+=" right=${remote_ip}"$'\n' + conn_content+=" leftsubnets={${local_pod_cidr} ${local_svc_cidr}}"$'\n' + conn_content+=" rightsubnets={${remote_pod_cidr} ${remote_svc_cidr}}"$'\n' + conn_content+=" auto=start"$'\n' + conn_content+=" ike=aes256-sha2_256-modp2048"$'\n' + conn_content+=" esp=aes256-sha2_256"$'\n' + conn_content+=" failureshunt=drop"$'\n' + conn_content+=" negotiationshunt=drop"$'\n' + conn_content+=" ikev2=insist"$'\n' + conn_content+=$'\n' + done + + run_command_on_vm "${host}" "sudo tee /etc/ipsec.d/c2cc.secrets > /dev/null < /dev/null </dev/null | grep -c 'type=ESP' || true") + count=$(echo "${count}" | tail -1 | tr -d '\r') + if [ "${count}" -ge "${expected_count}" ]; then + record_junit "${host}" "ipsec_tunnels" "OK" + return 0 + fi + sleep 2 + attempts=$((attempts + 1)) + done + record_junit "${host}" "ipsec_tunnels" "FAILED" + return 1 +} + +configure_ipsec() { + local host1_ip host2_ip host3_ip + host1_ip=$(get_vm_property host1 ip) || { echo "failed to get host1 ip" >&2; return 1; } + host2_ip=$(get_vm_property host2 ip) || { echo "failed to get host2 ip" >&2; return 1; } + host3_ip=$(get_vm_property host3 ip) || { echo "failed to get host3 ip" >&2; return 1; } + readonly host1_ip host2_ip host3_ip + + local psk + psk=$(openssl rand -hex 32) || { echo "failed to generate PSK" >&2; return 1; } + readonly psk + + configure_ipsec_host host1 "${host1_ip}" "${CLUSTER_A_POD_CIDR}" "${CLUSTER_A_SVC_CIDR}" "${psk}" \ + "${host2_ip}" host2 "${CLUSTER_B_POD_CIDR}" "${CLUSTER_B_SVC_CIDR}" \ + "${host3_ip}" host3 "${CLUSTER_C_POD_CIDR}" "${CLUSTER_C_SVC_CIDR}" + + configure_ipsec_host host2 "${host2_ip}" "${CLUSTER_B_POD_CIDR}" "${CLUSTER_B_SVC_CIDR}" "${psk}" \ + "${host1_ip}" host1 "${CLUSTER_A_POD_CIDR}" "${CLUSTER_A_SVC_CIDR}" \ + "${host3_ip}" host3 "${CLUSTER_C_POD_CIDR}" "${CLUSTER_C_SVC_CIDR}" + + configure_ipsec_host host3 "${host3_ip}" "${CLUSTER_C_POD_CIDR}" "${CLUSTER_C_SVC_CIDR}" "${psk}" \ + "${host1_ip}" host1 "${CLUSTER_A_POD_CIDR}" "${CLUSTER_A_SVC_CIDR}" \ + "${host2_ip}" host2 "${CLUSTER_B_POD_CIDR}" "${CLUSTER_B_SVC_CIDR}" + + # Each host has 2 remote hosts × 4 subnet pairs (2 local × 2 remote CIDRs) = 8 child SAs. + for host in host1 host2 host3; do + if ! wait_for_ipsec_tunnels "${host}" 8; then + return 1 + fi + done +} + +scenario_create_vms() { + prepare_kickstart host1 kickstart-bootc.ks.template rhel102-bootc-source-ipsec + prepare_kickstart host2 kickstart-bootc.ks.template rhel102-bootc-source-ipsec + prepare_kickstart host3 kickstart-bootc.ks.template rhel102-bootc-source-ipsec + + local -r host2_ks_dir="${SCENARIO_INFO_DIR}/${SCENARIO}/vms/host2" + cat >> "${host2_ks_dir}/post-microshift.cfg" <>/etc/microshift/config.yaml <> "${host3_ks_dir}/post-microshift.cfg" <>/etc/microshift/config.yaml < /dev/null <&2; return 1; } + host2_ip=$(get_vm_property host2 ip) || { echo "failed to get host2 ip" >&2; return 1; } + host3_ip=$(get_vm_property host3 ip) || { echo "failed to get host3 ip" >&2; return 1; } + readonly host1_ip host2_ip host3_ip + + wait_for_greenboot_on_hosts "c2cc_ipsec_pre_greenboot" + + configure_c2cc_host host1 \ + "${host2_ip}" "${CLUSTER_B_POD_CIDR}" "${CLUSTER_B_SVC_CIDR}" "${CLUSTER_B_DOMAIN}" \ + "${host3_ip}" "${CLUSTER_C_POD_CIDR}" "${CLUSTER_C_SVC_CIDR}" "${CLUSTER_C_DOMAIN}" + + configure_c2cc_host host2 \ + "${host1_ip}" "${CLUSTER_A_POD_CIDR}" "${CLUSTER_A_SVC_CIDR}" "${CLUSTER_A_DOMAIN}" \ + "${host3_ip}" "${CLUSTER_C_POD_CIDR}" "${CLUSTER_C_SVC_CIDR}" "${CLUSTER_C_DOMAIN}" + + configure_c2cc_host host3 \ + "${host1_ip}" "${CLUSTER_A_POD_CIDR}" "${CLUSTER_A_SVC_CIDR}" "${CLUSTER_A_DOMAIN}" \ + "${host2_ip}" "${CLUSTER_B_POD_CIDR}" "${CLUSTER_B_SVC_CIDR}" "${CLUSTER_B_DOMAIN}" + + wait_for_greenboot_on_hosts "c2cc_ipsec_greenboot" +} + +# configure_ipsec_host writes the PSK and connection configs, initializes the +# NSS database, and starts the ipsec service on a single host. +# Libreswan, tcpdump, and firewall rules are pre-installed in the +# rhel98-bootc-source-ipsec container image. +# +# Uses tunnel mode with subnet selectors to protect C2CC traffic (pod/service +# CIDRs). MicroShift C2CC routes cross-cluster traffic as raw IP via the +# host's physical interface — there is no Geneve tunnel between hosts. +# +# Arguments: +# $1 — VM name (host1, host2, host3) +# $2 — this host's IP +# $3 — local pod CIDR +# $4 — local service CIDR +# $5 — pre-shared key (hex string) +# $6..N — sets of 4: remote_ip remote_name remote_pod_cidr remote_svc_cidr +configure_ipsec_host() { + local -r host=$1 + local -r host_ip=$2 + local -r local_pod_cidr=$3 + local -r local_svc_cidr=$4 + local -r psk=$5 + shift 5 + + local secrets_content="" + local conn_content="" + while [ $# -gt 0 ]; do + local remote_ip=$1 + local remote_name=$2 + local remote_pod_cidr=$3 + local remote_svc_cidr=$4 + shift 4 + + secrets_content+="${host_ip} ${remote_ip} : PSK \"${psk}\""$'\n' + + conn_content+="conn c2cc-to-${remote_name}"$'\n' + conn_content+=" type=tunnel"$'\n' + conn_content+=" authby=secret"$'\n' + conn_content+=" left=${host_ip}"$'\n' + conn_content+=" right=${remote_ip}"$'\n' + conn_content+=" leftsubnets={${local_pod_cidr} ${local_svc_cidr}}"$'\n' + conn_content+=" rightsubnets={${remote_pod_cidr} ${remote_svc_cidr}}"$'\n' + conn_content+=" auto=start"$'\n' + conn_content+=" ike=aes256-sha2_256-modp2048"$'\n' + conn_content+=" esp=aes256-sha2_256"$'\n' + conn_content+=" failureshunt=drop"$'\n' + conn_content+=" negotiationshunt=drop"$'\n' + conn_content+=" ikev2=insist"$'\n' + conn_content+=$'\n' + done + + run_command_on_vm "${host}" "sudo tee /etc/ipsec.d/c2cc.secrets > /dev/null < /dev/null </dev/null | grep -c 'type=ESP' || true") + count=$(echo "${count}" | tail -1 | tr -d '\r') + if [ "${count}" -ge "${expected_count}" ]; then + record_junit "${host}" "ipsec_tunnels" "OK" + return 0 + fi + sleep 2 + attempts=$((attempts + 1)) + done + record_junit "${host}" "ipsec_tunnels" "FAILED" + return 1 +} + +configure_ipsec() { + local -r host1_ip=$(get_vm_property host1 ip) + local -r host2_ip=$(get_vm_property host2 ip) + local -r host3_ip=$(get_vm_property host3 ip) + + local -r psk=$(openssl rand -hex 32) + + configure_ipsec_host host1 "${host1_ip}" "${CLUSTER_A_POD_CIDR}" "${CLUSTER_A_SVC_CIDR}" "${psk}" \ + "${host2_ip}" host2 "${CLUSTER_B_POD_CIDR}" "${CLUSTER_B_SVC_CIDR}" \ + "${host3_ip}" host3 "${CLUSTER_C_POD_CIDR}" "${CLUSTER_C_SVC_CIDR}" + + configure_ipsec_host host2 "${host2_ip}" "${CLUSTER_B_POD_CIDR}" "${CLUSTER_B_SVC_CIDR}" "${psk}" \ + "${host1_ip}" host1 "${CLUSTER_A_POD_CIDR}" "${CLUSTER_A_SVC_CIDR}" \ + "${host3_ip}" host3 "${CLUSTER_C_POD_CIDR}" "${CLUSTER_C_SVC_CIDR}" + + configure_ipsec_host host3 "${host3_ip}" "${CLUSTER_C_POD_CIDR}" "${CLUSTER_C_SVC_CIDR}" "${psk}" \ + "${host1_ip}" host1 "${CLUSTER_A_POD_CIDR}" "${CLUSTER_A_SVC_CIDR}" \ + "${host2_ip}" host2 "${CLUSTER_B_POD_CIDR}" "${CLUSTER_B_SVC_CIDR}" + + # Each host has 2 remote hosts × 4 subnet pairs (2 local × 2 remote CIDRs) = 8 child SAs. + for host in host1 host2 host3; do + if ! wait_for_ipsec_tunnels "${host}" 8; then + return 1 + fi + done +} + +scenario_create_vms() { + prepare_kickstart host1 kickstart-bootc.ks.template rhel98-bootc-source-ipsec + prepare_kickstart host2 kickstart-bootc.ks.template rhel98-bootc-source-ipsec + prepare_kickstart host3 kickstart-bootc.ks.template rhel98-bootc-source-ipsec + + local -r host2_ks_dir="${SCENARIO_INFO_DIR}/${SCENARIO}/vms/host2" + cat >> "${host2_ks_dir}/post-microshift.cfg" <>/etc/microshift/config.yaml <> "${host3_ks_dir}/post-microshift.cfg" <>/etc/microshift/config.yaml < Date: Tue, 2 Jun 2026 09:51:29 +0200 Subject: [PATCH 4/5] IPSec restart tests --- test/resources/c2cc.resource | 95 +++++++++++++++++++++++++++++ test/suites/c2cc-ipsec/ipsec.robot | 92 ++++++++++------------------ test/suites/c2cc/connectivity.robot | 61 ------------------ test/suites/c2cc/dns.robot | 40 ------------ 4 files changed, 128 insertions(+), 160 deletions(-) diff --git a/test/resources/c2cc.resource b/test/resources/c2cc.resource index 0cc408bc8e..3a5c02e288 100644 --- a/test/resources/c2cc.resource +++ b/test/resources/c2cc.resource @@ -32,6 +32,11 @@ ${KUBECONFIG_C} ${EMPTY} &{C2CC_KUBECONFIGS} &{EMPTY} &{C2CC_SSH_IDS} &{EMPTY} @{C2CC_REMOTE_ALIASES} @{EMPTY} +&{NAMESPACES} cluster-a=${EMPTY} cluster-b=${EMPTY} cluster-c=${EMPTY} +&{DOMAIN_MAP} +... cluster-a=${CLUSTER_A_DOMAIN} +... cluster-b=${CLUSTER_B_DOMAIN} +... cluster-c=${CLUSTER_C_DOMAIN} *** Keywords *** @@ -295,3 +300,93 @@ Cleanup Test Workloads FOR ${alias} IN cluster-a cluster-b cluster-c Oc On Cluster ${alias} oc delete namespace ${NAMESPACES}[${alias}] --timeout=60s END + +Get Hello Pod IP + [Documentation] Get the pod IP of hello-microshift on the given cluster. + [Arguments] ${alias} + ${ip}= Oc On Cluster ${alias} + ... oc get pod hello-microshift -n ${NAMESPACES}[${alias}] -o jsonpath='{.status.podIP}' + RETURN ${ip} + +Get Hello Service IP + [Documentation] Get the ClusterIP of the hello-microshift service on the given cluster. + [Arguments] ${alias} + ${ip}= Oc On Cluster ${alias} + ... oc get svc hello-microshift -n ${NAMESPACES}[${alias}] -o jsonpath='{.spec.clusterIP}' + RETURN ${ip} + +Get Curl Pod IP + [Documentation] Get the pod IP of curl-pod on the given cluster. + [Arguments] ${alias} + ${ip}= Oc On Cluster ${alias} + ... oc get pod curl-pod -n ${NAMESPACES}[${alias}] -o jsonpath='{.status.podIP}' + RETURN ${ip} + +Curl From Cluster + [Documentation] Exec curl from curl-pod on the given cluster to the target IP and port. + [Arguments] ${alias} ${ip} ${port} + ${stdout}= Oc On Cluster ${alias} + ... oc exec curl-pod -n ${NAMESPACES}[${alias}] -- curl -sS --max-time 10 http://${ip}:${port}/cgi-bin/hello + RETURN ${stdout} + +Test Connectivity Between Clusters + [Documentation] Verify pod on ${source} can reach ${endpoint_type} IP on ${destination}. + [Arguments] ${source} ${destination} ${endpoint_type} + IF '${endpoint_type}' == 'pod' + ${ip_dest}= Get Hello Pod IP ${destination} + ELSE IF '${endpoint_type}' == 'service' + ${ip_dest}= Get Hello Service IP ${destination} + ELSE + Fail Invalid endpoint_type: ${endpoint_type}. Must be 'pod' or 'service'. + END + ${stdout}= Curl From Cluster ${source} ${ip_dest} 8080 + Should Contain ${stdout} Hello from + +Test Source IP Preserved Between Clusters + [Documentation] Verify ${source} to ${destination} pod-to-${endpoint_type} traffic preserves the source pod IP (no SNAT). + [Arguments] ${source} ${destination} ${endpoint_type} + ${curl_pod_ip}= Get Curl Pod IP ${source} + IF '${endpoint_type}' == 'pod' + ${ip_dest}= Get Hello Pod IP ${destination} + ELSE IF '${endpoint_type}' == 'service' + ${ip_dest}= Get Hello Service IP ${destination} + ELSE + Fail Invalid endpoint_type: ${endpoint_type}. Must be 'pod' or 'service'. + END + ${stdout}= Curl From Cluster ${source} ${ip_dest} 8080 + Should Contain ${stdout} source: ${curl_pod_ip} + +DNS Resolve From Cluster + [Documentation] Resolve a DNS name from curl-pod on the given cluster. Retries for up to 60s. + [Arguments] ${alias} ${fqdn} + Wait Until Keyword Succeeds 12x 5s + ... DNS Lookup Should Succeed ${alias} ${fqdn} + +DNS Lookup Should Succeed + [Documentation] Resolve a DNS name from curl-pod using getent hosts. + [Arguments] ${alias} ${fqdn} + ${stdout}= Oc On Cluster ${alias} + ... oc exec curl-pod -n ${NAMESPACES}[${alias}] -- getent hosts ${fqdn} + Should Not Be Empty ${stdout} + +Curl DNS From Cluster + [Documentation] Curl a service by DNS name from curl-pod on the given cluster. + [Arguments] ${alias} ${fqdn} ${port} + ${stdout}= Wait Until Keyword Succeeds 12x 5s + ... Curl DNS Should Succeed ${alias} ${fqdn} ${port} + RETURN ${stdout} + +Curl DNS Should Succeed + [Documentation] Single attempt to curl a DNS name from curl-pod. + [Arguments] ${alias} ${fqdn} ${port} + ${stdout}= Oc On Cluster ${alias} + ... oc exec curl-pod -n ${NAMESPACES}[${alias}] -- curl -sS --max-time 10 http://${fqdn}:${port}/cgi-bin/hello + Should Contain ${stdout} Hello from + RETURN ${stdout} + +Curl Remote Service Via DNS + [Documentation] Verify pod on ${source} can reach a service on ${destination} using the remote DNS name. + [Arguments] ${source} ${destination} + ${stdout}= Curl DNS From Cluster ${source} + ... hello-microshift.${NAMESPACES}[${destination}].svc.${DOMAIN_MAP}[${destination}] 8080 + Should Contain ${stdout} Hello from diff --git a/test/suites/c2cc-ipsec/ipsec.robot b/test/suites/c2cc-ipsec/ipsec.robot index 9ac1b93e95..8323924abe 100644 --- a/test/suites/c2cc-ipsec/ipsec.robot +++ b/test/suites/c2cc-ipsec/ipsec.robot @@ -19,10 +19,6 @@ Suite Teardown Teardown Test Tags c2cc ipsec -*** Variables *** -&{NAMESPACES} cluster-a=${EMPTY} cluster-b=${EMPTY} cluster-c=${EMPTY} - - *** Test Cases *** IPsec Tunnels Established On All Clusters [Documentation] Verify all 3 hosts have 8 IPsec tunnel SAs each. @@ -107,6 +103,39 @@ Near MTU Packet Through IPsec Tunnel Send Large Payload And Verify cluster-a ${ip_dest} 1300 Send Large Payload And Verify cluster-a ${ip_dest} 1400 +Cross Cluster DNS Through IPsec + [Documentation] Verify pods can resolve and reach services on remote + ... clusters via DNS name through the IPsec tunnel. + [Template] Curl Remote Service Via DNS + cluster-a cluster-b + cluster-a cluster-c + cluster-b cluster-a + cluster-b cluster-c + cluster-c cluster-a + cluster-c cluster-b + +IPsec Tunnel Recovers After Local Restart + [Documentation] Restart IPsec on cluster-a and verify tunnels recover + ... and cross-cluster connectivity is restored. + Restart IPsec Service On Cluster cluster-a + Wait For IPsec Tunnel Reestablishment cluster-a expected_count=8 + Wait For IPsec Tunnel Reestablishment cluster-b expected_count=8 + Wait For IPsec Tunnel Reestablishment cluster-c expected_count=8 + ${ip_dest}= Get Hello Pod IP cluster-b + Curl From Cluster cluster-a ${ip_dest} 8080 + +IPsec Tunnel Recovers After Remote Stop Start + [Documentation] Stop IPsec on cluster-b, start it back, and verify + ... tunnels recover and cross-cluster connectivity is restored. + Stop IPsec Service On Cluster cluster-b + Sleep 5s reason=Wait for tunnel to go down + Start IPsec Service On Cluster cluster-b + Wait For IPsec Tunnel Reestablishment cluster-a expected_count=8 + Wait For IPsec Tunnel Reestablishment cluster-b expected_count=8 + Wait For IPsec Tunnel Reestablishment cluster-c expected_count=8 + ${ip_dest}= Get Hello Pod IP cluster-b + Curl From Cluster cluster-a ${ip_dest} 8080 + *** Keywords *** Setup @@ -147,61 +176,6 @@ Restore IPsec With Enforcement Cleanup Start IPsec Service On Cluster ${ipsec_alias} Wait For IPsec Tunnel Reestablishment ${ipsec_alias} expected_count=8 -Get Hello Pod IP - [Documentation] Get the pod IP of hello-microshift on the given cluster. - [Arguments] ${alias} - ${ip}= Oc On Cluster ${alias} - ... oc get pod hello-microshift -n ${NAMESPACES}[${alias}] -o jsonpath='{.status.podIP}' - RETURN ${ip} - -Get Hello Service IP - [Documentation] Get the ClusterIP of the hello-microshift service on the given cluster. - [Arguments] ${alias} - ${ip}= Oc On Cluster ${alias} - ... oc get svc hello-microshift -n ${NAMESPACES}[${alias}] -o jsonpath='{.spec.clusterIP}' - RETURN ${ip} - -Get Curl Pod IP - [Documentation] Get the pod IP of curl-pod on the given cluster. - [Arguments] ${alias} - ${ip}= Oc On Cluster ${alias} - ... oc get pod curl-pod -n ${NAMESPACES}[${alias}] -o jsonpath='{.status.podIP}' - RETURN ${ip} - -Curl From Cluster - [Documentation] Exec curl from curl-pod on the given cluster. - [Arguments] ${alias} ${ip} ${port} - ${stdout}= Oc On Cluster ${alias} - ... oc exec curl-pod -n ${NAMESPACES}[${alias}] -- curl -sS --max-time 10 http://${ip}:${port}/cgi-bin/hello - RETURN ${stdout} - -Test Connectivity Between Clusters - [Documentation] Verify pod on source can reach endpoint on destination. - [Arguments] ${source} ${destination} ${endpoint_type} - IF '${endpoint_type}' == 'pod' - ${ip_dest}= Get Hello Pod IP ${destination} - ELSE IF '${endpoint_type}' == 'service' - ${ip_dest}= Get Hello Service IP ${destination} - ELSE - Fail Invalid endpoint_type: ${endpoint_type}. Must be 'pod' or 'service'. - END - ${stdout}= Curl From Cluster ${source} ${ip_dest} 8080 - Should Contain ${stdout} Hello from - -Test Source IP Preserved Between Clusters - [Documentation] Verify source pod IP is preserved through IPsec tunnel (no SNAT). - [Arguments] ${source} ${destination} ${endpoint_type} - ${curl_pod_ip}= Get Curl Pod IP ${source} - IF '${endpoint_type}' == 'pod' - ${ip_dest}= Get Hello Pod IP ${destination} - ELSE IF '${endpoint_type}' == 'service' - ${ip_dest}= Get Hello Service IP ${destination} - ELSE - Fail Invalid endpoint_type: ${endpoint_type}. Must be 'pod' or 'service'. - END - ${stdout}= Curl From Cluster ${source} ${ip_dest} 8080 - Should Contain ${stdout} source: ${curl_pod_ip} - Send Large Payload And Verify [Documentation] Send a large payload via curl POST and verify it succeeds. [Arguments] ${alias} ${ip} ${size} diff --git a/test/suites/c2cc/connectivity.robot b/test/suites/c2cc/connectivity.robot index e645045243..ad37aa8cce 100644 --- a/test/suites/c2cc/connectivity.robot +++ b/test/suites/c2cc/connectivity.robot @@ -14,10 +14,6 @@ Suite Teardown Teardown Test Tags c2cc -*** Variables *** -&{NAMESPACES} cluster-a=${EMPTY} cluster-b=${EMPTY} cluster-c=${EMPTY} - - *** Test Cases *** Test Cross Cluster Connectivity [Documentation] Verify pods on all clusters can reach pods/services on all other clusters. @@ -69,60 +65,3 @@ Teardown Teardown All Remote Clusters Remove Kubeconfig Logout MicroShift Host - -Test Connectivity Between Clusters - [Documentation] Verify pod on ${source} can reach ${endpoint_type} IP on ${destination} - [Arguments] ${source} ${destination} ${endpoint_type} - IF '${endpoint_type}' == 'pod' - ${ip_dest}= Get Hello Pod IP ${destination} - ELSE IF '${endpoint_type}' == 'service' - ${ip_dest}= Get Hello Service IP ${destination} - ELSE - Fail Invalid endpoint_type: ${endpoint_type}. Must be 'pod' or 'service'. - END - - ${stdout}= Curl From Cluster ${source} ${ip_dest} 8080 - Should Contain ${stdout} Hello from - -Test Source IP Preserved Between Clusters - [Documentation] Verify ${source} to ${destination} pod-to-${endpoint_type} traffic preserves the source pod IP (no SNAT). - [Arguments] ${source} ${destination} ${endpoint_type} - ${curl_pod_ip}= Get Curl Pod IP ${source} - IF '${endpoint_type}' == 'pod' - ${ip_dest}= Get Hello Pod IP ${destination} - ELSE IF '${endpoint_type}' == 'service' - ${ip_dest}= Get Hello Service IP ${destination} - ELSE - Fail Invalid endpoint_type: ${endpoint_type}. Must be 'pod' or 'service'. - END - - ${stdout}= Curl From Cluster ${source} ${ip_dest} 8080 - Should Contain ${stdout} source: ${curl_pod_ip} - -Get Hello Pod IP - [Documentation] Get the pod IP of hello-microshift on the given cluster. - [Arguments] ${alias} - ${ip}= Oc On Cluster ${alias} - ... oc get pod hello-microshift -n ${NAMESPACES}[${alias}] -o jsonpath='{.status.podIP}' - RETURN ${ip} - -Get Hello Service IP - [Documentation] Get the ClusterIP of the hello-microshift service on the given cluster. - [Arguments] ${alias} - ${ip}= Oc On Cluster ${alias} - ... oc get svc hello-microshift -n ${NAMESPACES}[${alias}] -o jsonpath='{.spec.clusterIP}' - RETURN ${ip} - -Get Curl Pod IP - [Documentation] Get the pod IP of curl-pod on the given cluster. - [Arguments] ${alias} - ${ip}= Oc On Cluster ${alias} - ... oc get pod curl-pod -n ${NAMESPACES}[${alias}] -o jsonpath='{.status.podIP}' - RETURN ${ip} - -Curl From Cluster - [Documentation] Exec curl from curl-pod on the given cluster to the target IP and port. - [Arguments] ${alias} ${ip} ${port} - ${stdout}= Oc On Cluster ${alias} - ... oc exec curl-pod -n ${NAMESPACES}[${alias}] -- curl -sS --max-time 10 http://${ip}:${port}/cgi-bin/hello - RETURN ${stdout} diff --git a/test/suites/c2cc/dns.robot b/test/suites/c2cc/dns.robot index 21438b79a4..c33b57286e 100644 --- a/test/suites/c2cc/dns.robot +++ b/test/suites/c2cc/dns.robot @@ -15,11 +15,6 @@ Suite Teardown Teardown Test Tags c2cc -*** Variables *** -&{NAMESPACES} cluster-a=${EMPTY} cluster-b=${EMPTY} cluster-c=${EMPTY} -&{DOMAIN_MAP} cluster-a=${CLUSTER_A_DOMAIN} cluster-b=${CLUSTER_B_DOMAIN} cluster-c=${CLUSTER_C_DOMAIN} - - *** Test Cases *** Test Corefile Contains C2CC Server Block [Documentation] Verify every cluster's Corefile has a server block for every other cluster domain. @@ -69,38 +64,3 @@ Teardown Teardown All Remote Clusters Remove Kubeconfig Logout MicroShift Host - -Curl Remote Service Via DNS - [Documentation] Verify pod on ${source} can reach a service on ${destination} using the remote DNS name. - [Arguments] ${source} ${destination} - ${stdout}= Curl DNS From Cluster ${source} - ... hello-microshift.${NAMESPACES}[${destination}].svc.${DOMAIN_MAP}[${destination}] 8080 - Should Contain ${stdout} Hello from - -DNS Resolve From Cluster - [Documentation] Resolve a DNS name from curl-pod on the given cluster. Retries for up to 60s. - [Arguments] ${alias} ${fqdn} - Wait Until Keyword Succeeds 12x 5s - ... DNS Lookup Should Succeed ${alias} ${fqdn} - -DNS Lookup Should Succeed - [Documentation] Resolve a DNS name from curl-pod using getent hosts. - [Arguments] ${alias} ${fqdn} - ${stdout}= Oc On Cluster ${alias} - ... oc exec curl-pod -n ${NAMESPACES}[${alias}] -- getent hosts ${fqdn} - Should Not Be Empty ${stdout} - -Curl DNS From Cluster - [Documentation] Curl a service by DNS name from curl-pod on the given cluster. - [Arguments] ${alias} ${fqdn} ${port} - ${stdout}= Wait Until Keyword Succeeds 12x 5s - ... Curl DNS Should Succeed ${alias} ${fqdn} ${port} - RETURN ${stdout} - -Curl DNS Should Succeed - [Documentation] Single attempt to curl a DNS name from curl-pod. - [Arguments] ${alias} ${fqdn} ${port} - ${stdout}= Oc On Cluster ${alias} - ... oc exec curl-pod -n ${NAMESPACES}[${alias}] -- curl -sS --max-time 10 http://${fqdn}:${port}/cgi-bin/hello - Should Contain ${stdout} Hello from - RETURN ${stdout} From 62981ef6f56a0c87cbde48d474525ab49b2b4d3e Mon Sep 17 00:00:00 2001 From: Patryk Matuszak Date: Wed, 3 Jun 2026 14:12:21 +0200 Subject: [PATCH 5/5] IPSec example configuration doc --- docs/user/howto_c2cc_ipsec.md | 154 ++++++++++++++++++++++++++++++++++ 1 file changed, 154 insertions(+) create mode 100644 docs/user/howto_c2cc_ipsec.md diff --git a/docs/user/howto_c2cc_ipsec.md b/docs/user/howto_c2cc_ipsec.md new file mode 100644 index 0000000000..cdb60accbd --- /dev/null +++ b/docs/user/howto_c2cc_ipsec.md @@ -0,0 +1,154 @@ +# Encrypting C2CC Traffic with IPsec + +MicroShift Cluster-to-Cluster Connectivity (C2CC) routes cross-cluster pod and service traffic as raw IP between nodes. +This traffic traverses the physical network unencrypted by default. +You can use IPsec to protect it using standard Linux tools. + +MicroShift does not configure or manage IPsec. +Setting up and maintaining the IPsec tunnels is the responsibility of the system administrator. +This guide provides a minimal working example using Libreswan in tunnel mode to help you get started. + +For comprehensive IPsec/VPN documentation, see [Setting up an IPsec VPN](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_networking/setting-up-an-ipsec-vpn_configuring-and-managing-networking) in the RHEL documentation. + +## Prerequisites + +- Two or more RHEL hosts running MicroShift with C2CC configured (non-overlapping pod and service CIDRs, `clusterToCluster.remoteClusters` populated in each node's config). +- IP connectivity between the hosts on the underlay network. +- Libreswan installed on every host: + +```bash +sudo dnf install -y libreswan +``` + +## Firewall + +Open the firewall for IKE negotiation and ESP: + +```bash +sudo firewall-cmd --permanent --zone=public --add-service=ipsec +sudo firewall-cmd --reload +``` + +This allows UDP ports 500 and 4500 (IKE/NAT-T) and IP protocol 50 (ESP). + +## Generate a Pre-Shared Key + +Generate a shared secret on one host and distribute it to all others through a secure channel: + +```bash +openssl rand -hex 32 +``` + +## Configure Libreswan + +The examples below assume a two-cluster setup: + +| Host | Underlay IP | Pod CIDR | Service CIDR | +|---------|---------------|----------------|----------------| +| Host A | 192.168.1.10 | 10.42.0.0/16 | 10.43.0.0/16 | +| Host B | 192.168.1.20 | 10.45.0.0/16 | 10.46.0.0/16 | + +### Secrets + +On **each** host, create `/etc/ipsec.d/c2cc.secrets`: + +```conf +192.168.1.10 192.168.1.20 : PSK "" +``` + +Set permissions: + +```bash +sudo chmod 600 /etc/ipsec.d/c2cc.secrets +sudo restorecon -v /etc/ipsec.d/c2cc.secrets +``` + +### Connection definition + +On **Host A**, create `/etc/ipsec.d/c2cc-tunnel.conf`: + +```conf +conn c2cc-to-host-b + type=tunnel + authby=secret + left=192.168.1.10 + right=192.168.1.20 + leftsubnets={10.42.0.0/16 10.43.0.0/16} + rightsubnets={10.45.0.0/16 10.46.0.0/16} + auto=start + ike=aes256-sha2_256-modp2048 + esp=aes256-sha2_256 + failureshunt=drop + negotiationshunt=drop + ikev2=insist +``` + +On **Host B**, create the same file with `left`/`right` and subnet values swapped: + +```conf +conn c2cc-to-host-a + type=tunnel + authby=secret + left=192.168.1.20 + right=192.168.1.10 + leftsubnets={10.45.0.0/16 10.46.0.0/16} + rightsubnets={10.42.0.0/16 10.43.0.0/16} + auto=start + ike=aes256-sha2_256-modp2048 + esp=aes256-sha2_256 + failureshunt=drop + negotiationshunt=drop + ikev2=insist +``` + +Key parameters: + +- **`type=tunnel`** -- Tunnel mode encrypts the original IP packet and wraps it in a new IP header. This is required because C2CC traffic uses pod/service CIDRs as source and destination, which are not routable on the underlay. +- **`leftsubnets` / `rightsubnets`** -- Must match the pod and service CIDRs configured in MicroShift. Each `{cidr1 cidr2}` pair creates one child SA per local/remote CIDR combination. +- **`auto=start`** -- Bring the tunnel up automatically when the IPsec service starts. +- **`failureshunt=drop` / `negotiationshunt=drop`** -- Drop traffic that matches the tunnel selectors if the SA fails or is still negotiating, preventing fallback to plaintext. +- **`ikev2=insist`** -- Require IKEv2. IKEv1 is not recommended. + +### Three or more clusters + +For a full mesh of N clusters, each host needs a connection definition and a secrets entry for every remote host. +For example, with three hosts, Host A would have two `conn` blocks (one for Host B, one for Host C) and two secrets entries. + +## Start IPsec + +Initialize the NSS database (first time only) and start the service: + +```bash +sudo ipsec checknss +sudo systemctl enable --now ipsec +``` + +## Verify the Tunnels + +Check that tunnel SAs are established: + +```bash +sudo ipsec trafficstatus +``` + +You should see output containing `type=ESP` entries for each subnet pair. +For a two-cluster setup with 2 local CIDRs and 2 remote CIDRs, expect 4 child SAs. + +Verify XFRM state is populated: + +```bash +ip xfrm state +``` + +You can also capture packets on the wire to confirm ESP encapsulation: + +```bash +sudo tcpdump -i enp1s0 -c 10 esp +``` + +## Considerations + +- **IPsec adds overhead.** ESP tunnel mode adds approximately 36-52 bytes per packet. If you experience MTU issues, verify that path MTU discovery is working or adjust MTU settings accordingly. +- **Tunnel recovery.** If IPsec is restarted on one host, tunnels renegotiate automatically when `auto=start` is set. No MicroShift restart is required. +- **Certificates.** This guide uses pre-shared keys for simplicity. For production deployments, consider certificate-based authentication. See the [RHEL VPN documentation](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/configuring_and_managing_networking/setting-up-an-ipsec-vpn_configuring-and-managing-networking) for details. +- **Policy enforcement.** The example connection definitions include `failureshunt=drop` and `negotiationshunt=drop` to prevent traffic from falling back to plaintext when the tunnel is down or still negotiating. If you remove these options, traffic matching the tunnel selectors will be sent unencrypted whenever the SA is unavailable.