Repository navigation
Kubernetes Clusters cannot be deployed in VPCs #6723
Description
ISSUE TYPE
- Bug Report
- Improvement Request
- Documentation Report
COMPONENT NAME
Kubernetes Provider and VPC
CLOUDSTACK VERSION
4.17.0.1
CONFIGURATION
Advanced Networking
OS / ENVIRONMENT
Ubuntu based hosts using KVM Hypervisor
SUMMARY
According to the documentation Cloudstack Kubernetes Service a Kubernetes Cluster can be deployed in VPC Networks.
Unfortunately, creating a Kubernetes Cluster in a VPC (VPC enabled networks) is failing, because of the missing "Firewall Service" in all networks being VPC enabled.
Creating a NetworkOffering with VPC support does not allow to enable the "Firewall Service" via the provider VpcVirtualRouter, which makes sense, because VPC enabled network offerings use the "Network ACL Service".
Therefore i think either the documentation is not specific enough to exclude VPC, or the Kubernetes Service might require additional functionality to support VPC enabled networks and their "Network ACL Service" instead of the "Firewall Service".
Or i miss something obvious / misunderstand the documentation.
STEPS TO REPRODUCE
- Create a VPC if not existing
- Create a Tier within the VPC if not existing, for example using the DefaultIsolatedNetworkOfferingForVpcNetworks Network Offering
- Try to create a Kubernetes Cluster selecting either the previously created VPC Tier network or any other VPC Tier network
Cloudstack will complain about the missing "Firewall Service" of the chosen network.
EXPECTED RESULTS
Either the documentation should not mention the possibility to deploy Kubernetes Clusters in VPCs, or the Kubernetes Cluster Deployment should be able to handle "Network ACL Service" in addition the the currently supported "Firewall Service"
ACTUAL RESULTS
Cloudstack will complain about the missing "Firewall Service" of the chosen network when trying to deploy a Kubernetes Cluster in any kind of VPC enabled network.
It would be great to see Kubernetes Clusters working in VPCs, because of the richer feature set the virtual router provides on VPCs, for example like easy possibility for Tiering / subnet separation and the possibility of Private Gateways with static routes.
Activity
StepBee commentedon Apr 23, 2023
I kind of worked around it for now by creating a Network Service Offering based on the Default Isolated network offering but with the option to specify the vlan.
The VLAN is known to all switches and routers outside of Cloudstack and i can do the specific routing outside of cloudstack, while still keeping VR functionality in direction to public and make use of the Kubernetes provider integration with the Virtual Router.
To implement the static route(s) in direction to the other involved networks on the Kubernetes nodes i use a DaemonSet with privileged rights and host network. The pod deployed by the DaemonSet with host network is implementing the required routes on each Kubernetes node.
Still i would really like to see the possibility to Kubernetes Clusters in VPC networks.
weizhouapache commentedon Apr 23, 2023
added to 4.18.1.0 milestone. cc @shwstppr
yadvr commentedon Jun 1, 2023
Fixed in #7479