CloudStack 4.20.3.0 - Inter-VPC Routing via Private Gateway Does Not Support Multiple Next-Hops in Hub-and-Spoke Architecture

[muthukrishnang1100]

"In CloudStack 4.20.3.0 VPC Static Routes, how can a Hub VPC route to multiple spoke VPCs connected via Private Gateways on a shared transit VLAN when static routes only allow destination CIDR and no per-route next-hop?"

Environment
CloudStack Version: 4.20.3.0
VPC Offering: Redundant VPC
Transit VLAN: 1931
Transit Network: 10.31.252.0/22
Private Gateways used for Inter-VPC communication
Architecture
Common-Hub VPC
CIDR: 10.30.64.0/23
Private Gateway IP: 10.31.252.5
Gateway: 10.31.252.1
Development VPC
CIDR: 10.30.70.0/24
Private Gateway IP: 10.31.252.6
Gateway: 10.31.252.5
Cloud-Test VPC
CIDR: 10.30.74.0/24
Private Gateway IP: 10.31.252.7
Gateway: 10.31.252.5

The design is Hub-and-Spoke.

Common-Hub VPC provides shared services such as:

DNS
LDAP
Monitoring
Backup
Shared Infrastructure Services

Development and Cloud-Test VPCs should communicate with Common-Hub through the transit network.

Expected Behaviour

Common-Hub VPC should be able to reach:

Development VPC (10.30.70.0/24) via next-hop 10.31.252.6
Cloud-Test VPC (10.30.74.0/24) via next-hop 10.31.252.7

Expected routing on Common-Hub VR:

10.30.70.0/24 via 10.31.252.6

10.30.74.0/24 via 10.31.252.7

This would allow bidirectional communication between Hub and Spoke VPCs.

Actual Behaviour

When static routes are created on the Common-Hub Private Gateway through the CloudStack UI, only the Destination CIDR field is available.

There is no option to specify a route-specific next-hop IP.

Static Routes configured:

10.30.70.0/24

10.30.74.0/24

CloudStack automatically generates the following routes on the VPC Router:

10.30.70.0/24 via 10.31.252.1 dev eth4

10.30.74.0/24 via 10.31.252.1 dev eth4

The next-hop is automatically taken from the Private Gateway "Gateway" field (10.31.252.1).

However, the correct next-hops should be:

10.30.70.0/24 via 10.31.252.6

10.30.74.0/24 via 10.31.252.7

As a result:

Development VPC -> Common-Hub VPC = SUCCESS

Cloud-Test VPC -> Common-Hub VPC = SUCCESS

Common-Hub VPC -> Development VPC = FAILED

Common-Hub VPC -> Cloud-Test VPC = FAILED

Proof 1 - Generated Route on Common-Hub VR

Output from active Common-Hub VPC Router:

ip route

Output:

10.30.70.0/24 via 10.31.252.1 dev eth4

10.30.74.0/24 via 10.31.252.1 dev eth4

These routes are generated automatically by CloudStack.

Proof 2 - Transit Network Connectivity Exists

From Common-Hub VR:

ping 10.31.252.6

Result:

64 bytes from 10.31.252.6
0% packet loss

From Common-Hub VR:

ping 10.31.252.7

Result:

64 bytes from 10.31.252.7
0% packet loss

This proves:

Transit VLAN is working
Layer 2 connectivity exists
Layer 3 connectivity exists
Private Gateways can reach each other
Proof 3 - Manual Route Fix Works

When manually adding routes on the active Common-Hub VPC Router:

ip route add 10.30.70.0/24 via 10.31.252.6

ip route add 10.30.74.0/24 via 10.31.252.7

Communication immediately starts working.

Common-Hub -> Development = SUCCESS

Common-Hub -> Cloud-Test = SUCCESS

Additional Problem

Manual routes are not persistent.

The manually added routes disappear after:

VPC Router restart
VPC Router failover
Management Server restart
Router reconfigure
Router stop/start

CloudStack regenerates the routes and restores:

10.30.70.0/24 via 10.31.252.1

10.30.74.0/24 via 10.31.252.1

which breaks communication again.

Database Observation

Table:

static_routes

Columns:

id

uuid

vpc_gateway_id

cidr

state

vpc_id

account_id

domain_id

created

There does not appear to be any column that stores a route-specific next-hop IP.

This suggests CloudStack derives the next-hop from the Private Gateway gateway field rather than storing a next-hop per route.

Question

Is this expected behaviour in CloudStack 4.20.3.0?

If yes, how should a Hub-and-Spoke Inter-VPC architecture be implemented when a Hub VPC must route to multiple spoke VPCs using different Private Gateway IP addresses?

If not, is this a bug?

Feature Request

Please consider supporting route-specific next-hop selection for VPC Private Gateway Static Routes through:

CloudStack UI
API
Database model

This would allow implementation of standard Hub-and-Spoke Inter-VPC routing designs using Private Gateways without requiring manual route injection on VPC Routers.