The new instance is not added to {@code supportedProviders}, so it + * will never be returned by {@link #getProvider(String)} and will not pollute the + * global provider registry. Use this for dynamic / extension-backed providers + * whose names are only known at runtime (e.g. NetworkOrchestrator extensions).
+ * + * @param name the provider name (typically the extension name) + * @return a transient {@link Provider} instance with the given name + */ + public static Provider createTransientProvider(String name) { + return new Provider(name); + } + @Override public String toString() { return new ToStringBuilder(this, ToStringStyle.SHORT_PREFIX_STYLE) .append("name", name) .toString(); } + + @Override + public boolean equals(Object obj) { + if (this == obj) return true; + if (!(obj instanceof Provider)) return false; + Provider provider = (Provider) obj; + return this.name.equals(provider.name); + } + + @Override + public int hashCode() { + return name.hashCode(); + } } public static class Capability { diff --git a/api/src/main/java/com/cloud/network/NetworkModel.java b/api/src/main/java/com/cloud/network/NetworkModel.java index c212e6319eb4..7e1a07ebeb69 100644 --- a/api/src/main/java/com/cloud/network/NetworkModel.java +++ b/api/src/main/java/com/cloud/network/NetworkModel.java @@ -187,6 +187,8 @@ public interface NetworkModel { boolean canElementEnableIndividualServices(Provider provider); + boolean canElementEnableIndividualServicesByName(String providerName); + boolean areServicesSupportedInNetwork(long networkId, Service... services); boolean isNetworkSystem(Network network); @@ -237,6 +239,18 @@ public interface NetworkModel { String getDefaultGuestTrafficLabel(long dcId, HypervisorType vmware); + /** + * Resolves a provider name to a {@link Provider} instance. + * For known static providers, delegates to {@link Provider#getProvider(String)}. + * For dynamically-registered NetworkOrchestrator extension providers whose names + * are not in the static registry, returns a transient {@link Provider} with the + * given name so callers can still dispatch correctly. + * + * @param providerName the provider name from {@code ntwk_service_map} or similar + * @return a {@link Provider} instance, or {@code null} if not resolvable + */ + Provider resolveProvider(String providerName); + /** * @param providerName * @return diff --git a/api/src/main/java/com/cloud/network/NetworkService.java b/api/src/main/java/com/cloud/network/NetworkService.java index 53692f932a4e..0c684046fd8f 100644 --- a/api/src/main/java/com/cloud/network/NetworkService.java +++ b/api/src/main/java/com/cloud/network/NetworkService.java @@ -155,7 +155,7 @@ PhysicalNetwork createPhysicalNetwork(Long zoneId, String vnetRange, String netw PairThis is the preferred lookup when multiple extensions are registered on + * the same physical network: the provider name stored in + * {@code ntwk_service_map} is used to pinpoint the exact extension that + * handles a given network.
+ * + * @param physicalNetworkId the physical network ID + * @param providerName the provider name (must equal the extension name) + * @return the matching {@link Extension}, or {@code null} + */ + Extension getExtensionForPhysicalNetworkAndProvider(long physicalNetworkId, String providerName); + + /** + * Returns ALL {@code extension_resource_map_details} (including hidden) for + * the specific extension registered on the given physical network. Used by + * {@code NetworkExtensionElement} to inject device credentials into the script + * environment for the correct extension when multiple different extensions are + * registered on the same physical network. + * + * @param physicalNetworkId the physical network ID + * @param extensionId the extension ID + * @return all key/value details including non-display ones, or an empty map + */ + MapIf {@code physicalNetworkId} is {@code null}, the method searches across all + * physical networks that have extensions registered and returns the capabilities for + * the first matching extension.
+ * + * @param physicalNetworkId physical network ID, or {@code null} for offering-level queries + * @param providerName provider / extension name + * @return capabilities map, or the default capabilities if no matching extension is found + */ + MapThis interface is looked up by {@code ExtensionsManagerImpl} to dispatch + * {@code runCustomAction} requests whose resource type is {@code Network}.
+ */ +public interface NetworkCustomActionProvider { + + /** + * Returns {@code true} if this provider handles networks whose physical + * network has an ExternalNetwork service provider registered. + * + * @param network the target network + * @return {@code true} if this provider can handle the network + */ + boolean canHandleCustomAction(Network network); + + /** + * Runs a named custom action against the external network device that + * manages the given network. + * + * @param network the CloudStack network on which to run the action + * @param actionName the action name (e.g. {@code "reboot-device"}, {@code "dump-config"}) + * @param parameters optional parameters supplied by the caller + * @return output from the action script, or {@code null} on failure + */ + String runCustomAction(Network network, String actionName, MapThe base list ({@link #networkElements}, wired by Spring) is extended + * at runtime with one transient {@link NetworkExtensionElement} per + * registered {@code NetworkOrchestrator} extension. This keeps the + * Spring bean list free from {@code NetworkExtensionElement} and allows + * dynamic discovery of extensions without a restart.
+ */ + private ListTwo well-known optional keys control which host network interfaces the + * wrapper script uses to create bridges and veth pairs:
+ *Example registration for a KVM-namespace backend:
+ *+ * cmk registerExtension id=<ext-uuid> resourcetype=PhysicalNetwork \ + * resourceid=<phys-uuid> \ + * details[0].key=hosts details[0].value=192.168.1.10,192.168.1.11 \ + * details[1].key=port details[1].value=22 \ + * details[2].key=username details[2].value=root \ + * details[3].key=sshkey details[3].value="$(cat ~/.ssh/id_rsa)" \ + * details[4].key=guest.network.device details[4].value=eth1 \ + * details[5].key=public.network.device details[5].value=eth1 + *+ * + *
Example per-network details (KVM-namespace backend):
+ *{"host":"192.168.1.10","namespace":"cs-net-42"}
+ *
+ *
+ * {
+ * "services": ["SourceNat", "StaticNat", "PortForwarding", "Firewall"],
+ * "capabilities": {
+ * "SourceNat": { "SupportedSourceNatTypes": "peraccount", "RedundantRouter": "false" }
+ * }
+ * }
+ *
+ */
+public class NetworkExtensionElement extends AdapterBase implements
+ NetworkElement, SourceNatServiceProvider, StaticNatServiceProvider,
+ PortForwardingServiceProvider, IpDeployer, NetworkCustomActionProvider,
+ DhcpServiceProvider, DnsServiceProvider, FirewallServiceProvider,
+ UserDataServiceProvider, LoadBalancingServiceProvider,
+ VpcProvider, NetworkACLServiceProvider, AggregatedCommandExecutor {
+
+ private static final MapFor VPC tier networks the extension details are inherited from the VPC-level + * details (stored in {@code vpc_details}) so all tiers in the same VPC share + * the same host/namespace binding. The script's {@code ensure-network-device} + * is only called at the VPC level (see {@link #ensureExtensionDetails(Vpc)}).
+ */ + protected void ensureExtensionDetails(Network network) { + if (network.getVpcId() != null) { + Vpc vpc = vpcDao.findById(network.getVpcId()); + ensureExtensionDetails(vpc); + return; + } + + // Isolated network: run ensure-network-device to select / validate the host. + MapTwo JSON blobs are always appended as named CLI arguments:
+ *--action-params '{"key1":"value1","key2":"value2"}'
+ * The wrapper script receives the `--action-params` JSON string and forwards
+ * it unchanged to hook scripts as the `--action-params` CLI argument; hook
+ * scripts should parse the JSON themselves (for example using `jq` or a
+ * small shell/awk parser).
+ */
+ public String runCustomAction(Network network, String actionName, MapLookup order (first match wins):
+ *":"", ...}
+ StringBuilder json = new StringBuilder("{");
+ boolean first = true;
+ for (Map.Entry e : dhcpOptions.entrySet()) {
+ if (!first) json.append(",");
+ json.append("\"").append(e.getKey()).append("\":\"")
+ .append(e.getValue() != null ? e.getValue().replace("\"", "\\\"") : "")
+ .append("\"");
+ first = false;
+ }
+ json.append("}");
+ String extensionIp = ensureExtensionIp(network);
+ List args = new ArrayList<>();
+ args.add("--network-id"); args.add(String.valueOf(network.getId()));
+ args.add("--nic-id"); args.add(String.valueOf(nicId));
+ args.add("--options"); args.add(json.toString());
+ args.add("--extension-ip"); args.add(safeStr(extensionIp));
+ args.addAll(getVpcIdArgs(network));
+ try {
+ return executeScript(network, "set-dhcp-options", args.toArray(new String[0]));
+ } catch (Exception e) {
+ logger.warn("setExtraDhcpOptions failed for network {}: {}", network.getId(), e.getMessage());
+ return false;
+ }
+ }
+
+ @Override
+ public boolean removeDhcpEntry(Network network, NicProfile nic, VirtualMachineProfile vmProfile)
+ throws ResourceUnavailableException {
+ if (!canHandle(network, Service.Dhcp)) {
+ return false;
+ }
+ logger.debug("removeDhcpEntry: network={} mac={} ip={}", network.getId(),
+ nic.getMacAddress(), nic.getIPv4Address());
+ String extensionIp = ensureExtensionIp(network);
+ List args = new ArrayList<>();
+ args.add("--network-id"); args.add(String.valueOf(network.getId()));
+ args.add("--mac"); args.add(safeStr(nic.getMacAddress()));
+ args.add("--ip"); args.add(safeStr(nic.getIPv4Address()));
+ args.add("--extension-ip"); args.add(safeStr(extensionIp));
+ args.addAll(getVpcIdArgs(network));
+ return executeScript(network, "remove-dhcp-entry", args.toArray(new String[0]));
+ }
+
+ // ---- DnsServiceProvider ----
+
+ @Override
+ public boolean addDnsEntry(Network network, NicProfile nic, VirtualMachineProfile vm,
+ DeployDestination dest, ReservationContext context)
+ throws ConcurrentOperationException, InsufficientCapacityException, ResourceUnavailableException {
+ if (!canHandle(network, Service.Dns)) {
+ return false;
+ }
+ String hostname = vm.getHostName();
+ logger.debug("addDnsEntry: network={} hostname={} ip={}", network.getId(),
+ hostname, nic.getIPv4Address());
+ String extensionIp = ensureExtensionIp(network);
+ List args = new ArrayList<>();
+ args.add("--network-id"); args.add(String.valueOf(network.getId()));
+ args.add("--ip"); args.add(safeStr(nic.getIPv4Address()));
+ args.add("--hostname"); args.add(safeStr(hostname));
+ args.add("--extension-ip"); args.add(safeStr(extensionIp));
+ args.addAll(getVpcIdArgs(network));
+ return executeScript(network, "add-dns-entry", args.toArray(new String[0]));
+ }
+
+ @Override
+ public boolean configDnsSupportForSubnet(Network network, NicProfile nic, VirtualMachineProfile vm,
+ DeployDestination dest, ReservationContext context)
+ throws ConcurrentOperationException, InsufficientCapacityException, ResourceUnavailableException {
+ if (!canHandle(network, Service.Dns)) {
+ return false;
+ }
+ logger.debug("configDnsSupportForSubnet: network={}", network.getId());
+ String extensionIp = ensureExtensionIp(network);
+ List args = new ArrayList<>();
+ args.add("--network-id"); args.add(String.valueOf(network.getId()));
+ args.add("--gateway"); args.add(safeStr(network.getGateway()));
+ args.add("--cidr"); args.add(safeStr(network.getCidr()));
+ args.add("--dns"); args.add(safeStr(getNetworkDns(network)));
+ args.add("--vlan"); args.add(safeStr(getVlanId(network)));
+ args.add("--domain"); args.add(safeStr(network.getNetworkDomain()));
+ args.add("--extension-ip"); args.add(safeStr(extensionIp));
+ args.addAll(getVpcIdArgs(network));
+ return executeScript(network, "config-dns-subnet", args.toArray(new String[0]));
+ }
+
+ @Override
+ public boolean removeDnsSupportForSubnet(Network network) throws ResourceUnavailableException {
+ if (!canHandle(network, Service.Dns)) {
+ return false;
+ }
+ logger.debug("removeDnsSupportForSubnet: network={}", network.getId());
+ String extensionIp = ensureExtensionIp(network);
+ List args = new ArrayList<>();
+ args.add("--network-id"); args.add(String.valueOf(network.getId()));
+ args.add("--extension-ip"); args.add(safeStr(extensionIp));
+ args.addAll(getVpcIdArgs(network));
+ return executeScript(network, "remove-dns-subnet", args.toArray(new String[0]));
+ }
+
+ // ---- UserDataServiceProvider ----
+
+ @Override
+ public boolean addPasswordAndUserdata(Network network, NicProfile nic, VirtualMachineProfile profile,
+ DeployDestination dest, ReservationContext context)
+ throws ConcurrentOperationException, InsufficientCapacityException, ResourceUnavailableException {
+ if (!canHandle(network, Service.UserData)) {
+ return false;
+ }
+
+ VirtualMachine vm = profile.getVirtualMachine();
+
+ // SSH public key from VM instance details
+ String sshPublicKey = null;
+ try {
+ VMInstanceDetailVO sshKeyDetail = vmInstanceDetailsDao.findDetail(profile.getId(), VmDetailConstants.SSH_PUBLIC_KEY);
+ if (sshKeyDetail != null) {
+ sshPublicKey = sshKeyDetail.getValue();
+ }
+ } catch (Exception e) {
+ logger.debug("Could not fetch SSH public key for VM {}: {}", profile.getId(), e.getMessage());
+ }
+
+ // Service offering display name
+ String serviceOfferingName = "";
+ try {
+ serviceOfferingName = profile.getServiceOffering().getDisplayText();
+ } catch (Exception e) {
+ logger.debug("Could not fetch service offering for VM {}: {}", profile.getId(), e.getMessage());
+ }
+
+ // Is Windows guest?
+ boolean isWindows = false;
+ try {
+ isWindows = guestOSCategoryDao
+ .findById(guestOSDao.findById(vm.getGuestOSId()).getCategoryId())
+ .getName().equalsIgnoreCase("Windows");
+ } catch (Exception e) {
+ logger.debug("Could not determine OS type for VM {}: {}", profile.getId(), e.getMessage());
+ }
+
+ // Hypervisor hostname – prefer dest host, fall back to current host
+ String destHostname = null;
+ try {
+ if (dest != null && dest.getHost() != null) {
+ destHostname = VirtualMachineManager.getHypervisorHostname(dest.getHost().getName());
+ } else if (vm.getHostId() != null) {
+ destHostname = VirtualMachineManager.getHypervisorHostname(
+ hostDao.findById(vm.getHostId()).getName());
+ }
+ } catch (Exception e) {
+ logger.debug("Could not resolve hypervisor hostname for VM {}: {}", profile.getId(), e.getMessage());
+ }
+
+ // Password from the VM profile parameter (set by UserVmManager before deployment)
+ String password = (String) profile.getParameter(VirtualMachineProfile.Param.VmPassword);
+
+ // Use this NIC's IP — the metadata server in each namespace identifies requesters
+ // by REMOTE_ADDR, which will be the VM's IP on THIS network (not necessarily the
+ // default NIC IP), so we always key metadata by the NIC's IP on this network.
+ String nicIpAddress = nic.getIPv4Address();
+
+ logger.debug("addPasswordAndUserdata: network={} ip={} hasPassword={} hasSshKey={}",
+ network.getId(), nicIpAddress,
+ password != null && !password.isEmpty(),
+ sshPublicKey != null && !sshPublicKey.isEmpty());
+
+ final UserVmVO userVm = userVmDao.findById(vm.getId());
+ if (userVm == null) {
+ throw new CloudRuntimeException("Could not find UserVmVO for VM " + vm.getId());
+ }
+
+ // Generate the full metadata set (userdata, meta-data/*, password) in one go
+ List vmData = networkModel.generateVmData(
+ userVm.getUserData(),
+ userVm.getUserDataDetails(),
+ serviceOfferingName,
+ vm.getDataCenterId(),
+ profile.getInstanceName(),
+ profile.getHostName(),
+ profile.getId(),
+ profile.getUuid(),
+ nicIpAddress,
+ sshPublicKey,
+ password,
+ isWindows,
+ destHostname);
+
+ if (vmData == null || vmData.isEmpty()) {
+ logger.debug("addPasswordAndUserdata: no VM data generated for network={} ip={}", network.getId(), nicIpAddress);
+ return true;
+ }
+
+ // Serialise vmData as JSON array.
+ // For the userdata entry CloudStack stores user-data base64-encoded; decode it so the
+ // wrapper writes the actual bytes. All other fields are plain strings. In both cases we
+ // then re-encode with Base64 so the single --vm-data argument is shell-safe.
+ StringBuilder json = new StringBuilder("[");
+ boolean first = true;
+ for (String[] entry : vmData) {
+ String dir = entry[NetworkModel.CONFIGDATA_DIR];
+ String file = entry[NetworkModel.CONFIGDATA_FILE];
+ String content = entry.length > NetworkModel.CONFIGDATA_CONTENT
+ ? entry[NetworkModel.CONFIGDATA_CONTENT] : null;
+ if (content == null) content = "";
+
+ byte[] contentBytes;
+ if (NetworkModel.USERDATA_DIR.equals(dir) && NetworkModel.USERDATA_FILE.equals(file)) {
+ // user-data is stored as base64 in CloudStack DB; decode it for the wrapper
+ try {
+ contentBytes = Base64.getDecoder().decode(content);
+ } catch (Exception e) {
+ contentBytes = content.getBytes(StandardCharsets.UTF_8);
+ }
+ } else {
+ contentBytes = content.getBytes(StandardCharsets.UTF_8);
+ }
+
+ if (!first) json.append(",");
+ first = false;
+ json.append("{\"dir\":\"").append(jsonEscape(dir))
+ .append("\",\"file\":\"").append(jsonEscape(file))
+ .append("\",\"content\":\"")
+ .append(Base64.getEncoder().encodeToString(contentBytes))
+ .append("\"}");
+ }
+ json.append("]");
+
+ // Wrap the entire JSON as base64 to avoid any shell quoting / escaping issues
+ String vmDataArg = Base64.getEncoder().encodeToString(
+ json.toString().getBytes(StandardCharsets.UTF_8));
+
+ List args = new ArrayList<>();
+ args.add("--network-id"); args.add(String.valueOf(network.getId()));
+ args.add("--ip"); args.add(safeStr(nicIpAddress));
+ args.add("--gateway"); args.add(safeStr(nic.getIPv4Gateway()));
+ args.add("--extension-ip"); args.add(safeStr(ensureExtensionIp(network)));
+ args.addAll(getVpcIdArgs(network));
+ return executeScriptWithFilePayload(network, "save-vm-data", "--vm-data-file",
+ vmDataArg, args.toArray(new String[0]));
+ }
+
+ @Override
+ public boolean savePassword(Network network, NicProfile nic, VirtualMachineProfile vm)
+ throws ResourceUnavailableException {
+ if (!canHandle(network, Service.UserData)) {
+ return false;
+ }
+ String password = (String) vm.getParameter(VirtualMachineProfile.Param.VmPassword);
+ if (password == null || password.isEmpty()) {
+ return true;
+ }
+ logger.debug("savePassword: network={} ip={}", network.getId(), nic.getIPv4Address());
+ String extensionIp = ensureExtensionIp(network);
+ List args = new ArrayList<>();
+ args.add("--network-id"); args.add(String.valueOf(network.getId()));
+ args.add("--ip"); args.add(safeStr(nic.getIPv4Address()));
+ args.add("--gateway"); args.add(safeStr(nic.getIPv4Gateway()));
+ args.add("--password"); args.add(password);
+ args.add("--extension-ip"); args.add(safeStr(extensionIp));
+ args.addAll(getVpcIdArgs(network));
+ return executeScript(network, "save-password", args.toArray(new String[0]));
+ }
+
+ @Override
+ public boolean saveUserData(Network network, NicProfile nic, VirtualMachineProfile vm)
+ throws ResourceUnavailableException {
+ if (!canHandle(network, Service.UserData)) {
+ return false;
+ }
+ String userData = null;
+ if (vm.getVirtualMachine() instanceof UserVm) {
+ userData = ((UserVm) vm.getVirtualMachine()).getUserData();
+ }
+ if (userData == null || userData.isEmpty()) {
+ return true;
+ }
+ logger.debug("saveUserData: network={} ip={}", network.getId(), nic.getIPv4Address());
+ // userData is stored as base64; pass it directly so the script can decode it
+ String extensionIp = ensureExtensionIp(network);
+ List args = new ArrayList<>();
+ args.add("--network-id"); args.add(String.valueOf(network.getId()));
+ args.add("--ip"); args.add(safeStr(nic.getIPv4Address()));
+ args.add("--gateway"); args.add(safeStr(nic.getIPv4Gateway()));
+ args.add("--userdata"); args.add(userData);
+ args.add("--extension-ip"); args.add(safeStr(extensionIp));
+ args.addAll(getVpcIdArgs(network));
+ return executeScript(network, "save-userdata", args.toArray(new String[0]));
+ }
+
+ @Override
+ public boolean saveSSHKey(Network network, NicProfile nic, VirtualMachineProfile vm,
+ String sshPublicKey) throws ResourceUnavailableException {
+ if (!canHandle(network, Service.UserData)) {
+ return false;
+ }
+ if (sshPublicKey == null || sshPublicKey.isEmpty()) {
+ return true;
+ }
+ logger.debug("saveSSHKey: network={} ip={}", network.getId(), nic.getIPv4Address());
+ // Encode SSH key as base64 to safely pass via CLI
+ String sshKeyBase64 = Base64.getEncoder().encodeToString(sshPublicKey.getBytes(java.nio.charset.StandardCharsets.UTF_8));
+ String extensionIp = ensureExtensionIp(network);
+ List args = new ArrayList<>();
+ args.add("--network-id"); args.add(String.valueOf(network.getId()));
+ args.add("--ip"); args.add(safeStr(nic.getIPv4Address()));
+ args.add("--gateway"); args.add(safeStr(nic.getIPv4Gateway()));
+ args.add("--sshkey"); args.add(sshKeyBase64);
+ args.add("--extension-ip"); args.add(safeStr(extensionIp));
+ args.addAll(getVpcIdArgs(network));
+ return executeScript(network, "save-sshkey", args.toArray(new String[0]));
+ }
+
+ @Override
+ public boolean saveHypervisorHostname(NicProfile nic, Network network, VirtualMachineProfile vm,
+ DeployDestination dest) throws ResourceUnavailableException {
+ if (!canHandle(network, Service.UserData)) {
+ return false;
+ }
+ String hostname = dest != null && dest.getHost() != null ? dest.getHost().getName() : null;
+ if (hostname == null || hostname.isEmpty()) {
+ return true;
+ }
+ logger.debug("saveHypervisorHostname: network={} ip={} host={}", network.getId(),
+ nic.getIPv4Address(), hostname);
+ String extensionIp = ensureExtensionIp(network);
+ List args = new ArrayList<>();
+ args.add("--network-id"); args.add(String.valueOf(network.getId()));
+ args.add("--ip"); args.add(safeStr(nic.getIPv4Address()));
+ args.add("--gateway"); args.add(safeStr(nic.getIPv4Gateway()));
+ args.add("--hypervisor-hostname"); args.add(hostname);
+ args.add("--extension-ip"); args.add(safeStr(extensionIp));
+ args.addAll(getVpcIdArgs(network));
+ return executeScript(network, "save-hypervisor-hostname", args.toArray(new String[0]));
+ }
+
+ // ---- LoadBalancingServiceProvider ----
+
+ @Override
+ public boolean applyLBRules(Network network, List rules)
+ throws ResourceUnavailableException {
+ if (rules == null || rules.isEmpty()) {
+ return true;
+ }
+ if (!canHandle(network, Service.Lb)) {
+ return false;
+ }
+ logger.info("Applying {} LB rules for network {}", rules.size(), network.getId());
+ String vlanId = getVlanId(network);
+ List vpcArgs = getVpcIdArgs(network);
+
+ // Serialise all rules as a JSON array and pass as a single --lb-rules argument
+ StringBuilder json = new StringBuilder("[");
+ boolean firstRule = true;
+ for (LoadBalancingRule rule : rules) {
+ if (!firstRule) json.append(",");
+ firstRule = false;
+ boolean revoke = rule.getState() == FirewallRule.State.Revoke;
+ json.append("{");
+ json.append("\"id\":").append(rule.getId()).append(",");
+ json.append("\"name\":\"").append(jsonEscape(rule.getName())).append("\",");
+ json.append("\"publicIp\":\"").append(jsonEscape(rule.getSourceIp() != null ? rule.getSourceIp().addr() : "")).append("\",");
+ json.append("\"publicPort\":").append(rule.getSourcePortStart()).append(",");
+ json.append("\"privatePort\":").append(rule.getDefaultPortStart()).append(",");
+ json.append("\"protocol\":\"").append(jsonEscape(safeStr(rule.getProtocol()))).append("\",");
+ json.append("\"algorithm\":\"").append(jsonEscape(safeStr(rule.getAlgorithm()))).append("\",");
+ json.append("\"revoke\":").append(revoke).append(",");
+ json.append("\"backends\":[");
+ if (rule.getDestinations() != null) {
+ boolean firstDest = true;
+ for (LoadBalancingRule.LbDestination dest : rule.getDestinations()) {
+ if (!firstDest) json.append(",");
+ firstDest = false;
+ json.append("{");
+ json.append("\"ip\":\"").append(jsonEscape(dest.getIpAddress())).append("\",");
+ json.append("\"port\":").append(dest.getDestinationPortStart()).append(",");
+ json.append("\"revoked\":").append(dest.isRevoked());
+ json.append("}");
+ }
+ }
+ json.append("]");
+ json.append("}");
+ }
+ json.append("]");
+
+ List args = new ArrayList<>();
+ args.add("--network-id"); args.add(String.valueOf(network.getId()));
+ args.add("--vlan"); args.add(safeStr(vlanId));
+ args.add("--lb-rules"); args.add(json.toString());
+ args.addAll(vpcArgs);
+ boolean result = executeScript(network, "apply-lb-rules", args.toArray(new String[0]));
+ if (!result) {
+ throw new ResourceUnavailableException("Failed to apply LB rules for network " + network.getId(),
+ Network.class, network.getId());
+ }
+ return true;
+ }
+
+ @Override
+ public boolean validateLBRule(Network network, LoadBalancingRule rule) {
+ // Delegate validation to the external script; accept by default
+ return true;
+ }
+
+ @Override
+ public List updateHealthChecks(Network network,
+ List lbrules) {
+ // Health-check state updates are not implemented via this path
+ return new ArrayList<>();
+ }
+
+ @Override
+ public boolean handlesOnlyRulesInTransitionState() {
+ return false;
+ }
+
+ /** Escapes a string for embedding in a JSON string literal. */
+ private static String jsonEscape(String s) {
+ if (s == null) return "";
+ return s.replace("\\", "\\\\").replace("\"", "\\\"").replace("\n", "\\n").replace("\r", "\\r");
+ }
+
+ @Override
+ public IpDeployer getIpDeployer(Network network) {
+ // This element itself implements IpDeployer; return this instance.
+ return this;
+ }
+
+ @Override
+ public boolean rollingRestartSupported() {
+ return false;
+ }
+
+ /**
+ * Applies all active firewall rules for a network to the external network device.
+ *
+ * Three categories of rules are handled:
+ *
+ * - Egress rules ({@link FirewallRule.TrafficType#Egress}) — control outbound
+ * traffic from guest VMs. The network offering's {@code egressDefaultPolicy} flag
+ * is consulted:
+ *
+ * - {@code true} (ALLOW by default) — each egress rule becomes a DROP rule;
+ * a catch-all ACCEPT is appended at the end.
+ * - {@code false} (DENY by default) — each egress rule becomes an ACCEPT rule;
+ * a catch-all DROP is appended at the end.
+ *
+ *
+ * - Ingress rules ({@link FirewallRule.TrafficType#Ingress}) on public IPs
+ * (static NAT, port-forwarding, LB, …) — control inbound access to a specific
+ * public IP. The wrapper script uses {@code conntrack --ctorigdst} to match the
+ * original pre-DNAT destination, so no private-IP lookup is required and all
+ * DNAT-based services (static-NAT, port-forwarding, LB) are handled uniformly.
+ * - Default egress policy — always conveyed via the JSON payload so the
+ * script can enforce it even when the explicit rule list is empty.
+ *
+ *
+ * Full-state rebuild semantics:
+ * {@code applyFWRules} is called with a narrow scope — the firewall manager
+ * passes only the rules for one public IP ({@code applyIngressFirewallRules}) or only
+ * the egress rules ({@code applyEgressFirewallRules}) per call. The script, however,
+ * rebuilds the entire firewall chain from scratch each time it runs. To avoid wiping
+ * the rules for other IPs on every call, this method ignores the {@code rules} parameter
+ * and instead queries the database for all active (non-revoked, non-System)
+ * {@link FirewallRule.Purpose#Firewall} rules for the network.
+ *
+ * Script command: {@code apply-fw-rules}
+ */
+ @Override
+ public boolean applyFWRules(Network network, List rules)
+ throws ResourceUnavailableException {
+ if (!canHandle(network, Service.Firewall)) {
+ return false;
+ }
+
+ // Determine default egress policy from the network offering.
+ // true = ALLOW (default permissive; explicit rules are deny-rules)
+ // false = DENY (default restrictive; explicit rules are allow-rules)
+ NetworkOfferingVO offering = networkOfferingDao.findById(network.getNetworkOfferingId());
+ boolean defaultEgressAllow = offering == null || offering.isEgressDefaultPolicy();
+
+ // Load ALL active (non-revoked) firewall rules for this network from the DB.
+ // applyFWRules is called in a narrow scope (only one public IP's ingress rules, or
+ // only egress rules per call), but the script does a full rebuild of the firewall
+ // chain. Querying the DB ensures every call produces a complete, correct chain.
+ List allRules = firewallRulesDao.listByNetworkAndPurposeAndNotRevoked(
+ network.getId(), FirewallRule.Purpose.Firewall);
+ // Skip System-type rules — the default egress policy is already conveyed by
+ // "default_egress_allow". System rules are transient (not stored in DB), but
+ // guard here anyway in case of future changes.
+ allRules = allRules.stream()
+ .filter(r -> !FirewallRule.FirewallRuleType.System.equals(r.getType()))
+ .collect(Collectors.toList());
+
+ for (FirewallRuleVO r : allRules) {
+ firewallRulesDao.loadSourceCidrs(r);
+ firewallRulesDao.loadDestinationCidrs(r);
+ }
+
+ logger.info("applyFWRules: network={} activeRules={} defaultEgressAllow={}",
+ network.getId(), allRules.size(), defaultEgressAllow);
+
+ // Build JSON payload: { "default_egress_allow": , "cidr": "...", "rules": [...] }
+ StringBuilder json = new StringBuilder();
+ json.append("{\"default_egress_allow\":").append(defaultEgressAllow).append(",");
+ json.append("\"cidr\":\"").append(jsonEscape(safeStr(network.getCidr()))).append("\",");
+ json.append("\"rules\":[");
+
+ boolean first = true;
+ for (FirewallRuleVO rule : allRules) {
+ if (!first) json.append(",");
+ first = false;
+
+ boolean isEgress = FirewallRule.TrafficType.Egress.equals(rule.getTrafficType());
+
+ json.append("{");
+ json.append("\"id\":").append(rule.getId()).append(",");
+ json.append("\"type\":\"").append(isEgress ? "egress" : "ingress").append("\",");
+ json.append("\"protocol\":\"").append(jsonEscape(safeStr(rule.getProtocol()))).append("\",");
+ if (rule.getSourcePortStart() != null) {
+ json.append("\"portStart\":").append(rule.getSourcePortStart()).append(",");
+ }
+ if (rule.getSourcePortEnd() != null) {
+ json.append("\"portEnd\":").append(rule.getSourcePortEnd()).append(",");
+ }
+ if (rule.getIcmpType() != null) {
+ json.append("\"icmpType\":").append(rule.getIcmpType()).append(",");
+ }
+ if (rule.getIcmpCode() != null) {
+ json.append("\"icmpCode\":").append(rule.getIcmpCode()).append(",");
+ }
+ // For ingress rules include the public IP the rule is associated with.
+ if (!isEgress) {
+ json.append("\"publicIp\":\"")
+ .append(jsonEscape(getIpAddress(rule.getSourceIpAddressId())))
+ .append("\",");
+ }
+ // sourceCidrs: for ingress = allowed external source IPs;
+ // for egress = allowed VM source IP ranges
+ json.append("\"sourceCidrs\":[");
+ List sourceCidrs = rule.getSourceCidrList();
+ if (sourceCidrs != null && !sourceCidrs.isEmpty()) {
+ boolean firstCidr = true;
+ for (String cidr : sourceCidrs) {
+ if (!firstCidr) json.append(",");
+ firstCidr = false;
+ json.append("\"").append(jsonEscape(cidr)).append("\"");
+ }
+ }
+ json.append("]");
+ // destCidrs: optional destination CIDR filter (meaningful for egress rules)
+ List destCidrs = rule.getDestinationCidrList();
+ json.append(",\"destCidrs\":[");
+ if (destCidrs != null && !destCidrs.isEmpty()) {
+ boolean firstCidr = true;
+ for (String cidr : destCidrs) {
+ if (!firstCidr) json.append(",");
+ firstCidr = false;
+ json.append("\"").append(jsonEscape(cidr)).append("\"");
+ }
+ }
+ json.append("]");
+ json.append("}");
+ }
+ json.append("]}");
+
+ String rulesBase64 = Base64.getEncoder().encodeToString(
+ json.toString().getBytes(StandardCharsets.UTF_8));
+
+ List args = new ArrayList<>();
+ args.add("--network-id"); args.add(String.valueOf(network.getId()));
+ args.add("--vlan"); args.add(safeStr(getVlanId(network)));
+ args.add("--gateway"); args.add(safeStr(network.getGateway()));
+ args.add("--cidr"); args.add(safeStr(network.getCidr()));
+ args.addAll(getVpcIdArgs(network));
+
+ boolean result = executeScriptWithFilePayload(network, "apply-fw-rules", "--fw-rules-file",
+ rulesBase64, args.toArray(new String[0]));
+ if (!result) {
+ throw new ResourceUnavailableException(
+ "Failed to apply firewall rules for network " + network.getId(),
+ Network.class, network.getId());
+ }
+ return true;
+ }
+
+ // ---- AggregatedCommandExecutor ----
+
+ /**
+ * Called at the start of a network-restart cycle (before rules are re-programmed).
+ * We have nothing to "start" here — the batch restore is driven by
+ * {@link #completeAggregatedExecution}.
+ */
+ @Override
+ public boolean prepareAggregatedExecution(Network network, DeployDestination dest)
+ throws ResourceUnavailableException {
+ if (!canHandle(network, null)) {
+ return true;
+ }
+ logger.debug("prepareAggregatedExecution: network={}", network.getId());
+ return true;
+ }
+
+ /**
+ * Called after all firewall/NAT/LB rules have been re-applied during a network restart.
+ *
+ * Queries all active User-VM NICs on this network from the database, builds a single
+ * batch JSON payload containing DHCP/DNS/metadata entries for every VM, and sends it to
+ * the wrapper script as a single {@code restore-network} call. This avoids N script
+ * invocations (one per VM) and instead performs the full restore in one shot.
+ */
+ @Override
+ public boolean completeAggregatedExecution(Network network, DeployDestination dest)
+ throws ResourceUnavailableException {
+ if (!canHandle(network, null)) {
+ return true;
+ }
+
+ logger.info("completeAggregatedExecution: restoring all VM network data for network={}", network.getId());
+
+ boolean dhcpEnabled = networkModel.areServicesSupportedInNetwork(network.getId(), Service.Dhcp)
+ && networkModel.isProviderSupportServiceInNetwork(network.getId(), Service.Dhcp, getProvider());
+ boolean dnsEnabled = networkModel.areServicesSupportedInNetwork(network.getId(), Service.Dns)
+ && networkModel.isProviderSupportServiceInNetwork(network.getId(), Service.Dns, getProvider());
+ boolean userdataEnabled = networkModel.areServicesSupportedInNetwork(network.getId(), Service.UserData)
+ && networkModel.isProviderSupportServiceInNetwork(network.getId(), Service.UserData, getProvider());
+
+ if (!dhcpEnabled && !dnsEnabled && !userdataEnabled) {
+ logger.debug("completeAggregatedExecution: no DHCP/DNS/UserData service for network={}, skipping", network.getId());
+ return true;
+ }
+
+ // Query all active User-VM NICs on this network
+ List nics = nicDao.listByNetworkIdAndType(network.getId(), VirtualMachine.Type.User);
+ if (nics == null || nics.isEmpty()) {
+ logger.debug("completeAggregatedExecution: no user VM NICs on network={}, skipping", network.getId());
+ return true;
+ }
+
+ logger.info("completeAggregatedExecution: building batch restore for {} VMs on network={}",
+ nics.size(), network.getId());
+
+ String restoreDataBase64 = buildRestoreNetworkData(network, nics, dhcpEnabled, dnsEnabled, userdataEnabled);
+
+ String extensionIp = ensureExtensionIp(network);
+ List args = new ArrayList<>();
+ args.add("--network-id"); args.add(String.valueOf(network.getId()));
+ args.add("--gateway"); args.add(safeStr(network.getGateway()));
+ args.add("--cidr"); args.add(safeStr(network.getCidr()));
+ args.add("--vlan"); args.add(safeStr(getVlanId(network)));
+ args.add("--extension-ip"); args.add(safeStr(extensionIp));
+ args.add("--dns"); args.add(safeStr(getNetworkDns(network)));
+ args.add("--domain"); args.add(safeStr(network.getNetworkDomain()));
+ args.addAll(getVpcIdArgs(network));
+
+ return executeScriptWithFilePayload(network, "restore-network", "--restore-data-file",
+ restoreDataBase64, args.toArray(new String[0]));
+ }
+
+ /**
+ * Called in the {@code finally} block of the network-restart cycle to clean up any
+ * temporary state created by {@link #prepareAggregatedExecution}.
+ * Nothing to clean up here.
+ */
+ @Override
+ public boolean cleanupAggregatedExecution(Network network, DeployDestination dest)
+ throws ResourceUnavailableException {
+ return true;
+ }
+
+ /**
+ * Builds the base64-encoded JSON payload for {@code restore-network}.
+ *
+ * The JSON structure is:
+ *
+ * {
+ * "dhcp_enabled": true,
+ * "dns_enabled": true,
+ * "userdata_enabled": true,
+ * "vms": [
+ * {
+ * "ip": "10.0.0.10",
+ * "mac": "02:00:00:00:00:01",
+ * "hostname": "vm-1",
+ * "default_nic": true,
+ * "vm_data": [
+ * { "dir": "userdata", "file": "user-data", "content": "" },
+ * { "dir": "meta-data", "file": "instance-id", "content": "" },
+ * ...
+ * ]
+ * },
+ * ...
+ * ]
+ * }
+ *
+ *
+ * Each {@code vm_data} entry has its {@code content} base64-encoded (the same
+ * encoding used by the per-VM {@code save-vm-data} command), so the wrapper script
+ * can handle both paths with the same decoder.
+ */
+ private String buildRestoreNetworkData(Network network, List nics,
+ boolean dhcpEnabled, boolean dnsEnabled, boolean userdataEnabled) {
+
+ // Precompute service-offering display text keyed by offering ID to avoid repeated DB hits
+ Map offeringNameCache = new HashMap<>();
+
+ StringBuilder json = new StringBuilder("{");
+ json.append("\"dhcp_enabled\":").append(dhcpEnabled).append(",");
+ json.append("\"dns_enabled\":").append(dnsEnabled).append(",");
+ json.append("\"userdata_enabled\":").append(userdataEnabled).append(",");
+ json.append("\"vms\":[");
+
+ boolean firstVm = true;
+ for (NicVO nic : nics) {
+ if (nic.getState() != Nic.State.Reserved && nic.getState() != Nic.State.Allocated) {
+ continue;
+ }
+ if (nic.getIPv4Address() == null || nic.getMacAddress() == null) {
+ continue;
+ }
+
+ Long instanceId = nic.getInstanceId();
+ if (instanceId == null) {
+ continue;
+ }
+
+ UserVmVO userVm = userVmDao.findById(instanceId);
+ if (userVm == null) {
+ continue;
+ }
+
+ // Per-VM data array (only if UserData service is enabled)
+ List vmData = null;
+ if (userdataEnabled) {
+ try {
+ // Service offering display text
+ String offeringName = offeringNameCache.computeIfAbsent(userVm.getServiceOfferingId(), id -> {
+ try {
+ ServiceOfferingVO so = serviceOfferingDao.findById(id);
+ return so != null ? so.getDisplayText() : "";
+ } catch (Exception e) {
+ return "";
+ }
+ });
+
+ // SSH public key
+ String sshPublicKey = null;
+ try {
+ VMInstanceDetailVO sshKeyDetail = vmInstanceDetailsDao.findDetail(instanceId, VmDetailConstants.SSH_PUBLIC_KEY);
+ if (sshKeyDetail != null) {
+ sshPublicKey = sshKeyDetail.getValue();
+ }
+ } catch (Exception e) {
+ logger.debug("Could not fetch SSH key for VM {}: {}", instanceId, e.getMessage());
+ }
+
+ // Is Windows?
+ boolean isWindows = false;
+ try {
+ isWindows = guestOSCategoryDao
+ .findById(guestOSDao.findById(userVm.getGuestOSId()).getCategoryId())
+ .getName().equalsIgnoreCase("Windows");
+ } catch (Exception ignored) { }
+
+ // Hypervisor hostname from current host
+ String destHostname = null;
+ try {
+ if (userVm.getHostId() != null) {
+ destHostname = VirtualMachineManager.getHypervisorHostname(
+ hostDao.findById(userVm.getHostId()).getName());
+ }
+ } catch (Exception ignored) { }
+
+ vmData = networkModel.generateVmData(
+ userVm.getUserData(),
+ userVm.getUserDataDetails(),
+ offeringName,
+ userVm.getDataCenterId(),
+ userVm.getInstanceName(),
+ userVm.getHostName(),
+ userVm.getId(),
+ userVm.getUuid(),
+ nic.getIPv4Address(),
+ sshPublicKey,
+ null, // password — not re-issued on restore
+ isWindows,
+ destHostname);
+ } catch (Exception e) {
+ logger.warn("Could not generate vmData for VM {} on network {}: {}", instanceId, network.getId(), e.getMessage());
+ }
+ }
+
+ // Build VM JSON entry
+ if (!firstVm) json.append(",");
+ firstVm = false;
+
+ json.append("{");
+ json.append("\"ip\":\"").append(jsonEscape(nic.getIPv4Address())).append("\",");
+ json.append("\"mac\":\"").append(jsonEscape(nic.getMacAddress())).append("\",");
+ json.append("\"hostname\":\"").append(jsonEscape(safeStr(userVm.getHostName()))).append("\",");
+ json.append("\"default_nic\":").append(nic.isDefaultNic()).append(",");
+ json.append("\"vm_data\":[");
+
+ if (vmData != null && !vmData.isEmpty()) {
+ boolean firstEntry = true;
+ for (String[] entry : vmData) {
+ String dir = entry[NetworkModel.CONFIGDATA_DIR];
+ String file = entry[NetworkModel.CONFIGDATA_FILE];
+ String content = entry.length > NetworkModel.CONFIGDATA_CONTENT
+ ? entry[NetworkModel.CONFIGDATA_CONTENT] : null;
+ if (content == null) content = "";
+
+ byte[] contentBytes;
+ if (NetworkModel.USERDATA_DIR.equals(dir) && NetworkModel.USERDATA_FILE.equals(file)) {
+ try {
+ contentBytes = Base64.getDecoder().decode(content);
+ } catch (Exception e) {
+ contentBytes = content.getBytes(StandardCharsets.UTF_8);
+ }
+ } else {
+ contentBytes = content.getBytes(StandardCharsets.UTF_8);
+ }
+
+ if (!firstEntry) json.append(",");
+ firstEntry = false;
+ json.append("{\"dir\":\"").append(jsonEscape(dir))
+ .append("\",\"file\":\"").append(jsonEscape(file))
+ .append("\",\"content\":\"")
+ .append(Base64.getEncoder().encodeToString(contentBytes))
+ .append("\"}");
+ }
+ }
+
+ json.append("]"); // vm_data
+ json.append("}"); // vm object
+ }
+
+ json.append("]"); // vms
+ json.append("}"); // root
+
+ return Base64.getEncoder().encodeToString(json.toString().getBytes(StandardCharsets.UTF_8));
+ }
+
+ // ---- VpcProvider ----
+
+ /**
+ * Finds the extension + physical-network pair for the given VPC by scanning the
+ * physical networks in the VPC's zone for a registered NetworkOrchestrator extension.
+ * Returns {@code null} when no suitable extension is found.
+ */
+ protected Pair resolveExtensionForVpc(Vpc vpc) {
+ List physNetworks = physicalNetworkDao.listByZone(vpc.getZoneId());
+ if (physNetworks == null || physNetworks.isEmpty()) {
+ return null;
+ }
+ for (PhysicalNetworkVO pn : physNetworks) {
+ Extension ext;
+ if (providerName != null && !providerName.isBlank()) {
+ ext = extensionHelper.getExtensionForPhysicalNetworkAndProvider(pn.getId(), providerName);
+ } else {
+ ext = extensionHelper.getExtensionForPhysicalNetwork(pn.getId());
+ }
+ if (ext != null) {
+ return new Pair<>(pn.getId(), ext);
+ }
+ }
+ return null;
+ }
+
+ /**
+ * Resolves the script file for a VPC-level operation (no network object required).
+ */
+ protected File resolveScriptFileForVpc(Long physicalNetworkId, Extension extension) {
+ if (physicalNetworkId == null) {
+ throw new CloudRuntimeException("No physical network ID for VPC extension");
+ }
+ if (extension == null) {
+ throw new CloudRuntimeException("No extension found for physical network " + physicalNetworkId);
+ }
+ if (!Extension.Type.NetworkOrchestrator.equals(extension.getType())) {
+ throw new CloudRuntimeException("Extension " + extension.getName() + " is not of type NetworkOrchestrator");
+ }
+ if (!Extension.State.Enabled.equals(extension.getState())) {
+ throw new CloudRuntimeException("Extension " + extension.getName() + " is not enabled");
+ }
+ if (!extension.isPathReady()) {
+ throw new CloudRuntimeException("Extension " + extension.getName() + " path is not ready");
+ }
+ String extensionPath = extensionHelper.getExtensionScriptPath(extension);
+ if (extensionPath == null) {
+ throw new CloudRuntimeException("Could not resolve path for extension " + extension.getName());
+ }
+ File extensionDir = new File(extensionPath);
+ File namedScript = new File(extensionDir, extension.getName() + ".sh");
+ if (namedScript.exists() && namedScript.canExecute()) {
+ return namedScript;
+ }
+ if (extensionDir.isFile() && extensionDir.canExecute()) {
+ return extensionDir;
+ }
+ throw new CloudRuntimeException(
+ "No executable script found in extension path " + extensionPath
+ + ". Expected '" + extension.getName() + ".sh'.");
+ }
+
+ /**
+ * Calls {@code ensure-network-device} with VPC-level args (no {@code --network-id}).
+ * The returned JSON is persisted in {@code vpc_details} under key
+ * {@value #NETWORK_DETAIL_EXTENSION_DETAILS}. VPC tier networks then inherit
+ * these details via {@link #ensureExtensionDetails(Network)}.
+ */
+ protected void ensureExtensionDetails(Vpc vpc) {
+ Map stored = vpcDetailsDao.listDetailsKeyPairs(vpc.getId());
+ String currentDetails = stored != null
+ ? stored.getOrDefault(NETWORK_DETAIL_EXTENSION_DETAILS, "{}") : "{}";
+
+ logger.info("Ensuring extension device for VPC {} (current={})", vpc.getId(), currentDetails);
+
+ Pair physNetAndExt = resolveExtensionForVpc(vpc);
+ if (physNetAndExt == null) {
+ logger.warn("ensureExtensionDetails(vpc): no extension found for VPC {} zone {}",
+ vpc.getId(), vpc.getZoneId());
+ return;
+ }
+ Long physicalNetworkId = physNetAndExt.first();
+ Extension extension = physNetAndExt.second();
+ File scriptFile = resolveScriptFileForVpc(physicalNetworkId, extension);
+ String physicalNetworkDetailsJson = buildPhysicalNetworkDetailsJson(physicalNetworkId, extension);
+
+ List cmdLine = new ArrayList<>();
+ cmdLine.add(scriptFile.getAbsolutePath());
+ cmdLine.add("ensure-network-device");
+ cmdLine.add("--vpc-id");
+ cmdLine.add(String.valueOf(vpc.getId()));
+ cmdLine.add("--zone-id");
+ cmdLine.add(String.valueOf(vpc.getZoneId()));
+ cmdLine.add("--current-details");
+ cmdLine.add(currentDetails);
+ cmdLine.add(ARG_PHYSICAL_NETWORK_EXTENSION_DETAILS);
+ cmdLine.add(physicalNetworkDetailsJson);
+ cmdLine.add(ARG_NETWORK_EXTENSION_DETAILS);
+ cmdLine.add(currentDetails);
+
+ try {
+ ProcessBuilder pb = new ProcessBuilder(cmdLine);
+ pb.redirectErrorStream(true);
+ Process process = pb.start();
+ String output = new String(process.getInputStream().readAllBytes()).trim();
+ int exitCode = process.waitFor();
+
+ logger.debug("Ensuring VPC network device script: {}", String.join(" ", cmdLine));
+
+ if (exitCode != 0) {
+ logger.warn("ensure-network-device exited {} for VPC {} — keeping current details",
+ exitCode, vpc.getId());
+ if ("{}".equals(currentDetails)) {
+ vpcDetailsDao.addDetail(vpc.getId(), NETWORK_DETAIL_EXTENSION_DETAILS, "{}", false);
+ }
+ return;
+ }
+ if (output.isEmpty()) {
+ output = "{}".equals(currentDetails) ? "{}" : currentDetails;
+ }
+ if (!output.equals(currentDetails)) {
+ logger.info("VPC extension device updated for VPC {}: {}", vpc.getId(), output);
+ vpcDetailsDao.addDetail(vpc.getId(), NETWORK_DETAIL_EXTENSION_DETAILS, output, false);
+ } else {
+ logger.debug("VPC extension device unchanged for VPC {}: {}", vpc.getId(), output);
+ }
+ } catch (Exception e) {
+ logger.warn("Failed ensure-network-device for VPC {}: {}", vpc.getId(), e.getMessage());
+ if ("{}".equals(currentDetails)) {
+ vpcDetailsDao.addDetail(vpc.getId(), NETWORK_DETAIL_EXTENSION_DETAILS, "{}", false);
+ }
+ }
+ }
+
+ /**
+ * Returns the per-VPC extension-details JSON from {@code vpc_details}
+ * (returns {@code {}} if not yet set).
+ */
+ private String getVpcExtensionDetailsJson(long vpcId) {
+ Map vpcDetails = vpcDetailsDao.listDetailsKeyPairs(vpcId);
+ return vpcDetails != null
+ ? vpcDetails.getOrDefault(NETWORK_DETAIL_EXTENSION_DETAILS, "{}") : "{}";
+ }
+
+ /**
+ * Executes the extension script for a VPC-level command (no tier network required).
+ * Uses VPC-level details from {@code vpc_details}.
+ */
+ protected boolean executeVpcScript(Vpc vpc, String command, String... args) {
+ Pair physNetAndExt = resolveExtensionForVpc(vpc);
+ if (physNetAndExt == null) {
+ logger.warn("executeVpcScript: no extension found for VPC {} zone {}", vpc.getId(), vpc.getZoneId());
+ return false;
+ }
+ Long physicalNetworkId = physNetAndExt.first();
+ Extension extension = physNetAndExt.second();
+ File scriptFile = resolveScriptFileForVpc(physicalNetworkId, extension);
+
+ String physicalNetworkDetailsJson = buildPhysicalNetworkDetailsJson(physicalNetworkId, extension);
+ String vpcExtDetailsJson = getVpcExtensionDetailsJson(vpc.getId());
+
+ logger.debug("Physical network details JSON: {}", physicalNetworkDetailsJson);
+ logger.debug("VPC extension details JSON: {}", vpcExtDetailsJson);
+
+ List cmdLine = new ArrayList<>();
+ cmdLine.add(scriptFile.getAbsolutePath());
+ cmdLine.add(command);
+ cmdLine.addAll(Arrays.asList(args));
+ cmdLine.add(ARG_PHYSICAL_NETWORK_EXTENSION_DETAILS);
+ cmdLine.add(physicalNetworkDetailsJson);
+ cmdLine.add(ARG_NETWORK_EXTENSION_DETAILS);
+ cmdLine.add(vpcExtDetailsJson);
+
+ logger.debug("Executing VPC extension script: {}", String.join(" ", cmdLine));
+
+ try {
+ ProcessBuilder pb = new ProcessBuilder(cmdLine);
+ pb.redirectErrorStream(true);
+ Process process = pb.start();
+ byte[] output = process.getInputStream().readAllBytes();
+ int exitCode = process.waitFor();
+
+ String outputStr = new String(output).trim();
+ if (!outputStr.isEmpty()) {
+ logger.debug("Script output: {}", outputStr);
+ }
+ if (exitCode != 0) {
+ logger.error("VPC extension script {} failed with exit code {}: {}", command, exitCode, outputStr);
+ return false;
+ }
+ return true;
+ } catch (Exception e) {
+ logger.error("Failed to execute VPC extension script {}: {}", command, e.getMessage(), e);
+ throw new CloudRuntimeException("Failed to execute VPC extension script: " + command, e);
+ }
+ }
+
+ protected PublicIpAddress getVpcSourceNatIp(long vpcId) {
+ final List ips = ipAddressDao.listByAssociatedVpc(vpcId, true);
+ if (ips == null || ips.isEmpty()) {
+ return null;
+ }
+ IPAddressVO selected = null;
+ for (final IPAddressVO ip : ips) {
+ if (ip.getState() != IpAddress.State.Releasing) {
+ selected = ip;
+ break;
+ }
+ }
+ if (selected == null) {
+ selected = ips.get(0);
+ }
+
+ final VlanVO vlan = vlanDao.findById(selected.getVlanId());
+ if (vlan == null) {
+ logger.warn("No VLAN found for VPC source NAT IP {} (vpc={})", selected.getAddress(), vpcId);
+ return null;
+ }
+ return PublicIp.createFromAddrAndVlan(selected, vlan);
+ }
+
+ /**
+ * Implements the VPC by:
+ *
+ * - Calling {@link #ensureExtensionDetails(Vpc)} to select a host and
+ * save the VPC-level details (does not use any anchor tier network).
+ * - Calling the script's {@code implement-vpc} command to create the VPC
+ * namespace and VPC-level networking state.
+ * - Applying VPC source NAT if a source-NAT IP already exists (the script's
+ * {@code assign-ip} sets up the public veth + SNAT rule for the VPC CIDR).
+ *
+ */
+ @Override
+ public boolean implementVpc(Vpc vpc, DeployDestination dest, ReservationContext context)
+ throws ConcurrentOperationException, ResourceUnavailableException, InsufficientCapacityException {
+
+ // Step 1: Ensure a VPC extension device is selected and details saved at VPC level.
+ ensureExtensionDetails(vpc);
+
+ // Step 2: Create the VPC namespace (no anchor tier network needed).
+ List implArgs = new ArrayList<>();
+ implArgs.add("--vpc-id"); implArgs.add(String.valueOf(vpc.getId()));
+ implArgs.add("--cidr"); implArgs.add(safeStr(vpc.getCidr()));
+
+ // Include source NAT IP if already allocated, so the script can set up the
+ // VPC-level SNAT rule for the entire VPC CIDR.
+ final PublicIpAddress sourceNatIp = getVpcSourceNatIp(vpc.getId());
+ if (sourceNatIp != null) {
+ implArgs.add("--public-ip"); implArgs.add(safeStr(sourceNatIp.getAddress().addr()));
+ implArgs.add("--public-vlan"); implArgs.add(safeStr(getPublicVlanTag(sourceNatIp.getId())));
+ implArgs.add("--public-gateway"); implArgs.add(safeStr(sourceNatIp.getGateway()));
+ implArgs.add("--public-cidr"); implArgs.add(safeStr(getPublicCidr(sourceNatIp.getId())));
+ implArgs.add("--source-nat"); implArgs.add("true");
+ }
+
+ if (!executeVpcScript(vpc, "implement-vpc", implArgs.toArray(new String[0]))) {
+ return false;
+ }
+
+ return true;
+ }
+
+ /**
+ * Shuts down the VPC by:
+ *
+ * - Calling {@code destroy-network} for each extension-backed VPC tier (removes
+ * tier resources but preserves the shared VPC namespace).
+ * - Calling {@code shutdown-vpc} to remove the VPC namespace and state after
+ * all tiers have been cleaned up.
+ *
+ */
+ @Override
+ public boolean shutdownVpc(Vpc vpc, ReservationContext context)
+ throws ConcurrentOperationException, ResourceUnavailableException {
+ final List networks = networkModel.listNetworksByVpc(vpc.getId());
+
+ boolean result = true;
+ if (networks != null) {
+ for (final Network network : networks) {
+ if (!canHandle(network, null)) {
+ continue;
+ }
+
+ final List args = new ArrayList<>();
+ args.add("--network-id"); args.add(String.valueOf(network.getId()));
+ args.add("--vlan"); args.add(safeStr(getVlanId(network)));
+ args.addAll(getVpcIdArgs(network));
+
+ final boolean tierResult = executeScript(network, "destroy-network", args.toArray(new String[0]));
+ result = result && tierResult;
+ }
+ }
+
+ // Remove the VPC namespace and VPC-level details regardless of tier result.
+ List vpcArgs = new ArrayList<>();
+ vpcArgs.add("--vpc-id"); vpcArgs.add(String.valueOf(vpc.getId()));
+ boolean vpcResult = executeVpcScript(vpc, "shutdown-vpc", vpcArgs.toArray(new String[0]));
+ if (vpcResult) {
+ try {
+ vpcDetailsDao.removeDetail(vpc.getId(), NETWORK_DETAIL_EXTENSION_DETAILS);
+ } catch (Exception e) {
+ logger.warn("Failed to remove VPC extension details for VPC {}: {}", vpc.getId(), e.getMessage());
+ }
+ }
+ result = result && vpcResult;
+
+ return result;
+ }
+
+ @Override
+ public boolean createPrivateGateway(PrivateGateway gateway)
+ throws ConcurrentOperationException, ResourceUnavailableException {
+ throw new UnsupportedOperationException("Private gateways are not supported by the network extension element.");
+ }
+
+ /** Private gateways are not supported by the network extension element. */
+ @Override
+ public boolean deletePrivateGateway(PrivateGateway gateway)
+ throws ConcurrentOperationException, ResourceUnavailableException {
+ throw new UnsupportedOperationException("Private gateways are not supported by the network extension element.");
+ }
+
+ /** Static routes are not supported by the network extension element. */
+ @Override
+ public boolean applyStaticRoutes(Vpc vpc, List routes)
+ throws ResourceUnavailableException {
+ throw new UnsupportedOperationException("Static routes are not supported by the network extension element.");
+ }
+
+ /** ACL items on private gateways are not supported by the network extension element. */
+ @Override
+ public boolean applyACLItemsToPrivateGw(PrivateGateway gateway, List rules)
+ throws ResourceUnavailableException {
+ throw new UnsupportedOperationException("ACL items on private gateways are not supported by the network extension element.");
+ }
+
+ @Override
+ public boolean updateVpcSourceNatIp(Vpc vpc, IpAddress address) {
+ if (vpc == null || address == null || address.getAddress() == null) {
+ logger.warn("updateVpcSourceNatIp: invalid input (vpc={}, address={})", vpc, address);
+ return false;
+ }
+
+ final List args = new ArrayList<>();
+ final VlanVO vlan = vlanDao.findById(address.getVlanId());
+ args.add("--vpc-id"); args.add(String.valueOf(vpc.getId()));
+ args.add("--cidr"); args.add(safeStr(vpc.getCidr()));
+ args.add("--public-ip"); args.add(safeStr(address.getAddress().addr()));
+ args.add("--public-vlan"); args.add(safeStr(getPublicVlanTag(address.getId())));
+ args.add("--public-gateway"); args.add(vlan != null ? safeStr(vlan.getVlanGateway()) : "");
+ args.add("--public-cidr"); args.add(safeStr(getPublicCidr(address.getId())));
+ args.add("--source-nat"); args.add("true");
+
+ final boolean result = executeVpcScript(vpc, "update-vpc-source-nat-ip", args.toArray(new String[0]));
+ if (!result) {
+ logger.warn("updateVpcSourceNatIp: failed to update source NAT IP for VPC {} to {}",
+ vpc.getId(), address.getAddress().addr());
+ }
+ return result;
+ }
+
+ /**
+ * Applies VPC network ACL rules for a VPC tier network via the script's
+ * {@code apply-network-acl} command. Rules are serialised as a Base64-encoded
+ * JSON array and passed via a temporary payload file.
+ *
+ * Script command: {@code apply-network-acl}
+ */
+ @Override
+ public boolean applyNetworkACLs(Network config, List rules)
+ throws ResourceUnavailableException {
+ if (!canHandle(config, Service.NetworkACL)) {
+ return true;
+ }
+
+ // Rebuild the ACL chain from all non-revoked rules.
+ List activeRules = rules == null ? List.of() :
+ rules.stream()
+ .filter(r -> r.getState() != NetworkACLItem.State.Revoke)
+ .collect(Collectors.toList());
+
+ logger.info("applyNetworkACLs: network={} activeRules={}", config.getId(), activeRules.size());
+
+ String aclRulesBase64 = buildAclRulesBase64(activeRules);
+
+ List args = new ArrayList<>();
+ args.add("--network-id"); args.add(String.valueOf(config.getId()));
+ args.add("--vlan"); args.add(safeStr(getVlanId(config)));
+ args.add("--gateway"); args.add(safeStr(config.getGateway()));
+ args.add("--cidr"); args.add(safeStr(config.getCidr()));
+ args.addAll(getVpcIdArgs(config));
+
+ boolean result = executeScriptWithFilePayload(config, "apply-network-acl",
+ "--acl-rules-file", aclRulesBase64, args.toArray(new String[0]));
+ if (!result) {
+ throw new ResourceUnavailableException(
+ "Failed to apply network ACL rules for network " + config.getId(),
+ Network.class, config.getId());
+ }
+ return true;
+ }
+
+ /**
+ * Re-applies ACL rules for all extension-backed networks in a VPC after a rule reorder.
+ * Calls {@code apply-network-acl} for each affected network with the full ACL item list.
+ */
+ @Override
+ public boolean reorderAclRules(Vpc vpc, List networks,
+ List networkACLItems) {
+ if (networks == null || networks.isEmpty()) {
+ return true;
+ }
+
+ List activeRules = networkACLItems == null ? List.of() :
+ networkACLItems.stream()
+ .filter(r -> r.getState() != NetworkACLItem.State.Revoke)
+ .collect(Collectors.toList());
+
+ boolean result = true;
+ for (Network network : networks) {
+ if (!canHandle(network, Service.NetworkACL)) {
+ continue;
+ }
+ try {
+ String aclRulesBase64 = buildAclRulesBase64(activeRules);
+
+ List args = new ArrayList<>();
+ args.add("--network-id"); args.add(String.valueOf(network.getId()));
+ args.add("--vlan"); args.add(safeStr(getVlanId(network)));
+ args.add("--gateway"); args.add(safeStr(network.getGateway()));
+ args.add("--cidr"); args.add(safeStr(network.getCidr()));
+ args.addAll(getVpcIdArgs(network));
+
+ boolean r = executeScriptWithFilePayload(network, "apply-network-acl",
+ "--acl-rules-file", aclRulesBase64, args.toArray(new String[0]));
+ result = result && r;
+ } catch (Exception e) {
+ logger.warn("reorderAclRules: failed for network {}: {}", network.getId(), e.getMessage());
+ result = false;
+ }
+ }
+ return result;
+ }
+
+ /**
+ * Serialises a list of {@link NetworkACLItem}s to a Base64-encoded JSON array
+ * suitable for passing to the {@code apply-network-acl} script command.
+ * Rules are sorted by their number (priority order).
+ */
+ private String buildAclRulesBase64(List rules) {
+ StringBuilder json = new StringBuilder("[");
+ boolean first = true;
+ List sorted = rules.stream()
+ .sorted(java.util.Comparator.comparingInt(NetworkACLItem::getNumber))
+ .collect(Collectors.toList());
+ for (NetworkACLItem rule : sorted) {
+ if (!first) json.append(",");
+ first = false;
+ json.append("{");
+ json.append("\"number\":").append(rule.getNumber()).append(",");
+ json.append("\"action\":\"").append(rule.getAction().name().toLowerCase()).append("\",");
+ json.append("\"trafficType\":\"").append(rule.getTrafficType().name().toLowerCase()).append("\",");
+ json.append("\"protocol\":\"").append(jsonEscape(safeStr(rule.getProtocol()))).append("\"");
+ if (rule.getSourcePortStart() != null) {
+ json.append(",\"portStart\":").append(rule.getSourcePortStart());
+ }
+ if (rule.getSourcePortEnd() != null) {
+ json.append(",\"portEnd\":").append(rule.getSourcePortEnd());
+ }
+ if (rule.getIcmpType() != null) {
+ json.append(",\"icmpType\":").append(rule.getIcmpType());
+ }
+ if (rule.getIcmpCode() != null) {
+ json.append(",\"icmpCode\":").append(rule.getIcmpCode());
+ }
+ json.append(",\"sourceCidrs\":[");
+ List sourceCidrs = rule.getSourceCidrList();
+ if (sourceCidrs != null && !sourceCidrs.isEmpty()) {
+ boolean firstCidr = true;
+ for (String cidr : sourceCidrs) {
+ if (!firstCidr) json.append(",");
+ firstCidr = false;
+ json.append("\"").append(jsonEscape(cidr)).append("\"");
+ }
+ }
+ json.append("]");
+ json.append("}");
+ }
+ json.append("]");
+ return Base64.getEncoder().encodeToString(
+ json.toString().getBytes(StandardCharsets.UTF_8));
+ }
+}
diff --git a/framework/extensions/src/main/java/org/apache/cloudstack/framework/extensions/network/README.md b/framework/extensions/src/main/java/org/apache/cloudstack/framework/extensions/network/README.md
new file mode 100644
index 000000000000..964677f261b6
--- /dev/null
+++ b/framework/extensions/src/main/java/org/apache/cloudstack/framework/extensions/network/README.md
@@ -0,0 +1,1092 @@
+
+
+# Network Extension Script Protocol
+
+This document describes the complete interface between Apache CloudStack's
+`NetworkExtensionElement` and the external script (Bash, Python, or any
+executable) that implements network services for a custom device.
+
+Any executable that handles the commands listed below can be registered as a
+**NetworkOrchestrator extension** and used as the provider for one or more
+CloudStack network services (DHCP, DNS, UserData, SourceNat, StaticNat,
+PortForwarding, Firewall, Lb, NetworkACL, Gateway).
+
+The reference implementation is the `network-namespace` extension at
+`extensions/network-namespace/`, which uses Linux network namespaces on KVM
+hosts. Use it as a working example.
+
+---
+
+## Table of Contents
+
+1. [Architecture Overview](#architecture-overview)
+2. [Script Placement Convention](#script-placement-convention)
+3. [CloudStack Setup Steps](#cloudstack-setup-steps)
+4. [Always-present CLI Arguments](#always-present-cli-arguments)
+5. [Shared Arguments Reference](#shared-arguments-reference)
+6. [Command Reference](#command-reference)
+ - [ensure-network-device](#ensure-network-device)
+ - [implement](#implement)
+ - [shutdown](#shutdown)
+ - [destroy](#destroy)
+ - [assign-ip / release-ip](#assign-ip--release-ip)
+ - [add-static-nat / delete-static-nat](#add-static-nat--delete-static-nat)
+ - [add-port-forward / delete-port-forward](#add-port-forward--delete-port-forward)
+ - [apply-fw-rules](#apply-fw-rules)
+ - [add-dhcp-entry / remove-dhcp-entry](#add-dhcp-entry--remove-dhcp-entry)
+ - [config-dhcp-subnet / remove-dhcp-subnet](#config-dhcp-subnet--remove-dhcp-subnet)
+ - [set-dhcp-options](#set-dhcp-options)
+ - [add-dns-entry](#add-dns-entry)
+ - [config-dns-subnet / remove-dns-subnet](#config-dns-subnet--remove-dns-subnet)
+ - [save-vm-data](#save-vm-data)
+ - [save-password](#save-password)
+ - [save-userdata](#save-userdata)
+ - [save-sshkey](#save-sshkey)
+ - [save-hypervisor-hostname](#save-hypervisor-hostname)
+ - [apply-lb-rules](#apply-lb-rules)
+ - [restore-network](#restore-network)
+ - [custom-action](#custom-action)
+7. [Service-to-Command Mapping](#service-to-command-mapping)
+8. [Capabilities Configuration](#capabilities-configuration)
+9. [VPC Networks](#vpc-networks)
+10. [Extension IP](#extension-ip)
+11. [Exit Codes](#exit-codes)
+12. [Minimal Script Skeleton](#minimal-script-skeleton)
+
+---
+
+## Architecture Overview
+
+```
+CloudStack Management Server
+ │
+ │ exec /.sh [args...]
+ │ --physical-network-extension-details '{...}'
+ │ --network-extension-details '{...}'
+ ▼
+ Your Script (Bash / Python / Go / …)
+ │
+ │ configures / queries your device:
+ │ • KVM host over SSH
+ │ • SDN controller REST API
+ │ • Hardware appliance CLI
+ │ • Cloud provider API
+ ▼
+ External Network Device
+```
+
+CloudStack calls the script synchronously (blocking process execution) on the
+**management server** for every network event. The script is responsible for
+translating those events into configuration changes on the actual device.
+
+The script must:
+
+- **Exit 0** on success.
+- **Exit non-zero** on failure (CloudStack will log the error and may retry).
+- For `ensure-network-device` only, **print a single-line JSON object** to
+ stdout (see [ensure-network-device](#ensure-network-device)).
+
+All other commands must produce no output on stdout (any output is logged at
+DEBUG level and ignored).
+
+---
+
+## Script Placement Convention
+
+CloudStack resolves the executable in this order (first match wins):
+
+1. **`/.sh`** — preferred convention.
+ Example: extension named `my-sdn` → script at
+ `.../my-sdn/my-sdn.sh`.
+2. **`` itself**, if it is a regular file and is executable.
+
+The `